2024-09-15T03:36:12.883 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T03:36:12.901 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T03:36:12.975 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905308 branch: wip-bharath9-testing-2024-09-13-1444-quincy description: rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} email: skanta@redhat.com first_in_suite: false job_id: '7905308' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath9-testing-2024-09-13-1444-quincy ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 2500 ms tcp read timeout: 5 osd_async_recovery_min_cost: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd max backfills: 6 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: filestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd sloppy crc: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - slow request - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - mons down - mon down - out of quorum sha1: d75f23820806da622f561ee5ee7551015285993f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: osd objectstore: filestore osd sloppy crc: true filestore: true fs: xfs install: ceph: enable_coprs: - ceph/el9 flavor: default sha1: d75f23820806da622f561ee5ee7551015285993f workunit: branch: wip-bharath9-testing-2024-09-13-1444-quincy sha1: d75f23820806da622f561ee5ee7551015285993f owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - mon.a - - osd.4 - osd.5 - osd.6 - osd.7 - mon.b - mgr.x - - osd.8 - osd.9 - osd.10 - osd.11 - mon.c seed: 2126 sha1: d75f23820806da622f561ee5ee7551015285993f sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath9-testing-2024-09-13-1444-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d75f23820806da622f561ee5ee7551015285993f targets: smithi001.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA3rs2DhrBM6++QsY9hBrsnj2CCrIknFhDwNIAqmuhKmjPHgxiztD+Dz4jK+ZoK6FDABuTtWCHD1q6uc80jpEp0= smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN8WEoelaeFizgFkIx1baxTd76J3XY71ZvQNlmPsKUT5z2biyJkZL/BKq9asql2W4jeU3y3/zCxL7zXxwxaJ6IE= smithi190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD3ndSH9u14htsEoJRyYt+lnfEDhk3s2fNJubG0o7+k5SrAhVk1wVqw7iFgVz5tUGkQ3QzULLej4O8T9lgemhj8= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: aggressive_pg_num_changes: false chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: crush-failure-domain: osd k: 4 l: 3 m: 2 name: lrcprofile plugin: lrc objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_00:54:39 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T03:36:12.975 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa; will attempt to use it 2024-09-15T03:36:12.976 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa/tasks 2024-09-15T03:36:12.976 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T03:36:12.978 INFO:teuthology.task.internal:Checking packages... 2024-09-15T03:36:12.999 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd75f23820806da622f561ee5ee7551015285993f' 2024-09-15T03:36:12.999 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T03:36:12.999 INFO:teuthology.packaging:ref: None 2024-09-15T03:36:12.999 INFO:teuthology.packaging:tag: None 2024-09-15T03:36:13.000 INFO:teuthology.packaging:branch: wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T03:36:13.000 INFO:teuthology.packaging:sha1: d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:36:13.000 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T03:36:13.145 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1643.gd75f2382 2024-09-15T03:36:13.147 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T03:36:13.154 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T03:36:13.154 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T03:36:13.168 INFO:teuthology.task.internal:Saving configuration 2024-09-15T03:36:13.184 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T03:36:13.190 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T03:36:13.214 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905308', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 03:31:09.636981', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA3rs2DhrBM6++QsY9hBrsnj2CCrIknFhDwNIAqmuhKmjPHgxiztD+Dz4jK+ZoK6FDABuTtWCHD1q6uc80jpEp0='} 2024-09-15T03:36:13.238 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905308', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 03:31:09.637998', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN8WEoelaeFizgFkIx1baxTd76J3XY71ZvQNlmPsKUT5z2biyJkZL/BKq9asql2W4jeU3y3/zCxL7zXxwxaJ6IE='} 2024-09-15T03:36:13.263 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905308', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 03:31:09.635744', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD3ndSH9u14htsEoJRyYt+lnfEDhk3s2fNJubG0o7+k5SrAhVk1wVqw7iFgVz5tUGkQ3QzULLej4O8T9lgemhj8='} 2024-09-15T03:36:13.263 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T03:36:13.270 INFO:teuthology.task.internal:roles: ubuntu@smithi001.front.sepia.ceph.com - ['osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.a'] 2024-09-15T03:36:13.270 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.b', 'mgr.x'] 2024-09-15T03:36:13.270 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['osd.8', 'osd.9', 'osd.10', 'osd.11', 'mon.c'] 2024-09-15T03:36:13.270 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T03:36:13.345 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa341463ac0>, signals=[15]) 2024-09-15T03:36:13.346 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T03:36:13.352 INFO:teuthology.task.internal:Opening connections... 2024-09-15T03:36:13.352 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi001.front.sepia.ceph.com 2024-09-15T03:36:13.353 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:36:13.430 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-09-15T03:36:13.431 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:36:13.510 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2024-09-15T03:36:13.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:36:13.590 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T03:36:13.596 DEBUG:teuthology.orchestra.run.smithi001:> uname -m 2024-09-15T03:36:13.615 INFO:teuthology.orchestra.run.smithi001.stdout:x86_64 2024-09-15T03:36:13.616 DEBUG:teuthology.orchestra.run.smithi001:> cat /etc/os-release 2024-09-15T03:36:13.675 INFO:teuthology.orchestra.run.smithi001.stdout:NAME="CentOS Stream" 2024-09-15T03:36:13.675 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION="9" 2024-09-15T03:36:13.675 INFO:teuthology.orchestra.run.smithi001.stdout:ID="centos" 2024-09-15T03:36:13.675 INFO:teuthology.orchestra.run.smithi001.stdout:ID_LIKE="rhel fedora" 2024-09-15T03:36:13.675 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION_ID="9" 2024-09-15T03:36:13.675 INFO:teuthology.orchestra.run.smithi001.stdout:PLATFORM_ID="platform:el9" 2024-09-15T03:36:13.675 INFO:teuthology.orchestra.run.smithi001.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T03:36:13.675 INFO:teuthology.orchestra.run.smithi001.stdout:ANSI_COLOR="0;31" 2024-09-15T03:36:13.676 INFO:teuthology.orchestra.run.smithi001.stdout:LOGO="fedora-logo-icon" 2024-09-15T03:36:13.676 INFO:teuthology.orchestra.run.smithi001.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T03:36:13.676 INFO:teuthology.orchestra.run.smithi001.stdout:HOME_URL="https://centos.org/" 2024-09-15T03:36:13.676 INFO:teuthology.orchestra.run.smithi001.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T03:36:13.676 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T03:36:13.676 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T03:36:13.677 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server 2024-09-15T03:36:13.705 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-09-15T03:36:13.724 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-09-15T03:36:13.724 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-09-15T03:36:13.782 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Stream" 2024-09-15T03:36:13.782 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="9" 2024-09-15T03:36:13.782 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2024-09-15T03:36:13.782 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2024-09-15T03:36:13.782 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="9" 2024-09-15T03:36:13.782 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el9" 2024-09-15T03:36:13.782 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T03:36:13.782 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2024-09-15T03:36:13.782 INFO:teuthology.orchestra.run.smithi067.stdout:LOGO="fedora-logo-icon" 2024-09-15T03:36:13.782 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T03:36:13.783 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://centos.org/" 2024-09-15T03:36:13.783 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T03:36:13.783 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T03:36:13.783 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T03:36:13.783 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-09-15T03:36:13.801 DEBUG:teuthology.orchestra.run.smithi190:> uname -m 2024-09-15T03:36:13.820 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2024-09-15T03:36:13.820 DEBUG:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2024-09-15T03:36:13.878 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="CentOS Stream" 2024-09-15T03:36:13.878 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="9" 2024-09-15T03:36:13.878 INFO:teuthology.orchestra.run.smithi190.stdout:ID="centos" 2024-09-15T03:36:13.878 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE="rhel fedora" 2024-09-15T03:36:13.878 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="9" 2024-09-15T03:36:13.878 INFO:teuthology.orchestra.run.smithi190.stdout:PLATFORM_ID="platform:el9" 2024-09-15T03:36:13.878 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T03:36:13.878 INFO:teuthology.orchestra.run.smithi190.stdout:ANSI_COLOR="0;31" 2024-09-15T03:36:13.878 INFO:teuthology.orchestra.run.smithi190.stdout:LOGO="fedora-logo-icon" 2024-09-15T03:36:13.879 INFO:teuthology.orchestra.run.smithi190.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T03:36:13.879 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://centos.org/" 2024-09-15T03:36:13.879 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T03:36:13.879 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T03:36:13.879 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T03:36:13.879 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2024-09-15T03:36:13.901 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T03:36:13.916 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T03:36:13.922 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T03:36:13.922 DEBUG:teuthology.orchestra.run.smithi001:> test '!' -e /home/ubuntu/cephtest 2024-09-15T03:36:13.925 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-09-15T03:36:13.927 DEBUG:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2024-09-15T03:36:13.945 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T03:36:13.951 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T03:36:13.951 DEBUG:teuthology.orchestra.run.smithi001:> test -z $(ls -A /var/lib/ceph) 2024-09-15T03:36:13.982 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-09-15T03:36:13.984 DEBUG:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2024-09-15T03:36:14.082 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T03:36:14.158 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T03:36:14.171 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-15T03:36:14.171 INFO:teuthology.task.kernel:config {'osd.0': {'kdb': 1, 'sha1': 'distro'}, 'osd.4': {'kdb': 1, 'sha1': 'distro'}, 'osd.8': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-15T03:36:14.172 DEBUG:teuthology.orchestra.run.smithi001:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T03:36:14.172 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T03:36:14.173 DEBUG:teuthology.orchestra.run.smithi190:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T03:36:14.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:36:14.190 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-09-15T03:36:14.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:36:14.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:36:14.192 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-09-15T03:36:14.192 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-09-15T03:36:14.248 INFO:teuthology.orchestra.run.smithi001.stdout:5.14.0-503.el9.x86_64 2024-09-15T03:36:14.248 INFO:teuthology.task.kernel:Running kernel on smithi001: 5.14.0-503.el9.x86_64 2024-09-15T03:36:14.248 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum install -y kernel 2024-09-15T03:36:14.249 INFO:teuthology.orchestra.run.smithi190.stdout:5.14.0-503.el9.x86_64 2024-09-15T03:36:14.250 INFO:teuthology.task.kernel:Running kernel on smithi190: 5.14.0-503.el9.x86_64 2024-09-15T03:36:14.250 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum install -y kernel 2024-09-15T03:36:14.251 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-503.el9.x86_64 2024-09-15T03:36:14.251 INFO:teuthology.task.kernel:Running kernel on smithi067: 5.14.0-503.el9.x86_64 2024-09-15T03:36:14.251 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum install -y kernel 2024-09-15T03:36:16.752 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-09-15T03:36:16.785 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.2 MB 00:00 2024-09-15T03:36:18.594 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 29 MB/s | 20 MB 00:00 2024-09-15T03:36:20.760 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 1.7 MB/s | 8.2 MB 00:04 2024-09-15T03:36:22.477 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 36 MB/s | 20 MB 00:00 2024-09-15T03:36:22.853 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-15T03:36:24.520 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 19 kB 00:00 2024-09-15T03:36:25.010 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 2.9 MB/s | 20 MB 00:07 2024-09-15T03:36:25.508 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 26 MB/s | 23 MB 00:00 2024-09-15T03:36:27.903 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.5 MB 00:01 2024-09-15T03:36:29.621 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 19 kB 00:00 2024-09-15T03:36:30.416 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-15T03:36:30.672 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-15T03:36:31.163 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-15T03:36:32.186 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 79 kB/s | 19 kB 00:00 2024-09-15T03:36:33.157 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 27 MB/s | 23 MB 00:00 2024-09-15T03:36:33.170 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T03:36:33.170 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T03:36:33.266 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:36:33.278 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:36:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-09-15T03:36:33.278 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:36:33.278 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-15T03:36:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T03:36:33.278 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-15T03:36:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T03:36:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T03:36:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T03:36:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:36:33.279 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-15T03:36:33.279 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:36:33.279 INFO:teuthology.orchestra.run.smithi190.stdout:Install 4 Packages 2024-09-15T03:36:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:36:33.279 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 79 M 2024-09-15T03:36:33.279 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 126 M 2024-09-15T03:36:33.279 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-15T03:36:33.590 INFO:teuthology.orchestra.run.smithi190.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 79 kB/s | 23 kB 00:00 2024-09-15T03:36:36.047 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-15T03:36:37.823 INFO:teuthology.orchestra.run.smithi190.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 3.5 MB/s | 16 MB 00:04 2024-09-15T03:36:38.162 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T03:36:38.162 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T03:36:38.248 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:36:38.260 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:36:38.261 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-09-15T03:36:38.261 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:36:38.261 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-15T03:36:38.261 INFO:teuthology.orchestra.run.smithi001.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T03:36:38.261 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-15T03:36:38.261 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T03:36:38.261 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T03:36:38.261 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T03:36:38.261 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:36:38.261 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-15T03:36:38.262 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:36:38.262 INFO:teuthology.orchestra.run.smithi001.stdout:Install 4 Packages 2024-09-15T03:36:38.262 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:36:38.262 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 79 M 2024-09-15T03:36:38.263 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 126 M 2024-09-15T03:36:38.263 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-15T03:36:38.882 INFO:teuthology.orchestra.run.smithi001.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 38 kB/s | 23 kB 00:00 2024-09-15T03:36:39.013 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-15T03:36:39.807 INFO:teuthology.orchestra.run.smithi001.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-09-15T03:36:40.574 INFO:teuthology.orchestra.run.smithi001.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 15 MB/s | 35 MB 00:02 2024-09-15T03:36:41.086 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T03:36:41.087 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T03:36:41.175 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:36:41.180 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:36:41.181 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-09-15T03:36:41.181 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:36:41.181 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T03:36:41.181 INFO:teuthology.orchestra.run.smithi067.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T03:36:41.181 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-09-15T03:36:41.181 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T03:36:41.181 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T03:36:41.181 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T03:36:41.181 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:36:41.181 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T03:36:41.181 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:36:41.181 INFO:teuthology.orchestra.run.smithi067.stdout:Install 4 Packages 2024-09-15T03:36:41.182 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:36:41.182 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 79 M 2024-09-15T03:36:41.182 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 126 M 2024-09-15T03:36:41.182 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T03:36:41.515 INFO:teuthology.orchestra.run.smithi067.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 291 kB/s | 23 kB 00:00 2024-09-15T03:36:41.706 INFO:teuthology.orchestra.run.smithi190.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 4.1 MB/s | 35 MB 00:08 2024-09-15T03:36:41.907 INFO:teuthology.orchestra.run.smithi001.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 9.4 MB/s | 29 MB 00:03 2024-09-15T03:36:41.908 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:36:41.908 INFO:teuthology.orchestra.run.smithi001.stdout:Total 22 MB/s | 79 MB 00:03 2024-09-15T03:36:42.407 INFO:teuthology.orchestra.run.smithi067.stdout:(2/4): kernel-modules-core-5.14.0-505.el9.x86_6 32 MB/s | 29 MB 00:00 2024-09-15T03:36:42.557 INFO:teuthology.orchestra.run.smithi067.stdout:(3/4): kernel-core-5.14.0-505.el9.x86_64.rpm 14 MB/s | 16 MB 00:01 2024-09-15T03:36:42.617 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-15T03:36:42.978 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-15T03:36:42.978 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-15T03:36:43.015 INFO:teuthology.orchestra.run.smithi067.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 22 MB/s | 35 MB 00:01 2024-09-15T03:36:43.016 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:36:43.016 INFO:teuthology.orchestra.run.smithi067.stdout:Total 43 MB/s | 79 MB 00:01 2024-09-15T03:36:43.723 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T03:36:43.944 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-15T03:36:43.944 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-15T03:36:44.095 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T03:36:44.095 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T03:36:44.106 INFO:teuthology.orchestra.run.smithi190.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 2.7 MB/s | 29 MB 00:10 2024-09-15T03:36:44.106 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:36:44.106 INFO:teuthology.orchestra.run.smithi190.stdout:Total 7.3 MB/s | 79 MB 00:10 2024-09-15T03:36:44.792 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-15T03:36:45.086 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T03:36:45.086 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T03:36:45.143 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-15T03:36:45.143 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-15T03:36:46.105 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-15T03:36:46.105 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-15T03:36:46.555 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-15T03:36:47.241 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:36:47.341 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:36:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T03:36:48.316 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:36:48.399 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:36:48.556 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-15T03:36:49.263 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:36:49.347 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:36:50.114 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:36:50.253 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:36:51.236 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:36:51.361 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:36:52.132 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:36:52.268 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:36:55.378 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:36:55.420 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:36:56.642 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:36:56.687 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:36:57.089 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:36:57.192 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:37:00.568 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:37:01.900 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:37:02.060 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:37:39.654 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:37:39.695 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:37:40.082 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:37:40.082 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:37:40.083 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:37:40.083 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:37:40.429 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:37:40.467 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:37:40.754 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:37:40.760 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:37:40.760 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:37:40.761 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:37:40.761 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:37:40.797 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:37:40.936 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:37:40.936 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:37:40.936 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-15T03:37:40.936 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T03:37:40.936 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T03:37:40.936 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T03:37:40.936 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T03:37:40.936 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:37:40.936 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:37:41.062 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:37:41.062 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:37:41.063 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:37:41.063 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:37:41.293 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:37:41.381 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:37:41.381 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:37:41.382 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T03:37:41.382 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T03:37:41.382 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T03:37:41.382 INFO:teuthology.task.kernel:Installing distro kernel on osd.0... 2024-09-15T03:37:41.382 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi001.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T03:37:41.382 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:37:41.441 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:37:41.441 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:37:41.441 DEBUG:teuthology.orchestra.run.smithi001:> sudo rpm -qi grub2-tools 2024-09-15T03:37:41.478 INFO:teuthology.orchestra.run.smithi001.stdout:Name : grub2-tools 2024-09-15T03:37:41.478 INFO:teuthology.orchestra.run.smithi001.stdout:Epoch : 1 2024-09-15T03:37:41.478 INFO:teuthology.orchestra.run.smithi001.stdout:Version : 2.06 2024-09-15T03:37:41.479 INFO:teuthology.orchestra.run.smithi001.stdout:Release : 82.el9 2024-09-15T03:37:41.479 INFO:teuthology.orchestra.run.smithi001.stdout:Architecture: x86_64 2024-09-15T03:37:41.479 INFO:teuthology.orchestra.run.smithi001.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T03:37:41.479 INFO:teuthology.orchestra.run.smithi001.stdout:Group : Unspecified 2024-09-15T03:37:41.479 INFO:teuthology.orchestra.run.smithi001.stdout:Size : 8274670 2024-09-15T03:37:41.479 INFO:teuthology.orchestra.run.smithi001.stdout:License : GPLv3+ 2024-09-15T03:37:41.479 INFO:teuthology.orchestra.run.smithi001.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T03:37:41.479 INFO:teuthology.orchestra.run.smithi001.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T03:37:41.479 INFO:teuthology.orchestra.run.smithi001.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T03:37:41.479 INFO:teuthology.orchestra.run.smithi001.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T03:37:41.479 INFO:teuthology.orchestra.run.smithi001.stdout:Packager : builder@centos.org 2024-09-15T03:37:41.479 INFO:teuthology.orchestra.run.smithi001.stdout:Vendor : CentOS 2024-09-15T03:37:41.479 INFO:teuthology.orchestra.run.smithi001.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T03:37:41.480 INFO:teuthology.orchestra.run.smithi001.stdout:Summary : Support tools for GRUB. 2024-09-15T03:37:41.480 INFO:teuthology.orchestra.run.smithi001.stdout:Description : 2024-09-15T03:37:41.480 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:37:41.480 INFO:teuthology.orchestra.run.smithi001.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T03:37:41.480 INFO:teuthology.orchestra.run.smithi001.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T03:37:41.480 INFO:teuthology.orchestra.run.smithi001.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T03:37:41.480 INFO:teuthology.orchestra.run.smithi001.stdout:hardware devices. 2024-09-15T03:37:41.480 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:37:41.480 INFO:teuthology.orchestra.run.smithi001.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T03:37:41.481 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T03:37:41.482 INFO:teuthology.task.kernel:Updating grub on smithi001 to boot 5.14.0-505.el9.x86_64 2024-09-15T03:37:41.482 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T03:37:41.582 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:37:41.582 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:37:41.582 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-15T03:37:41.582 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T03:37:41.583 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T03:37:41.583 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T03:37:41.583 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T03:37:41.583 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:37:41.583 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:37:41.888 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:37:41.889 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:37:41.889 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T03:37:41.889 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T03:37:41.889 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T03:37:41.889 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T03:37:41.889 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T03:37:41.889 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:37:41.889 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:37:41.922 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:37:42.010 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:37:42.011 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:37:42.011 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T03:37:42.011 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T03:37:42.012 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T03:37:42.012 INFO:teuthology.task.kernel:Installing distro kernel on osd.8... 2024-09-15T03:37:42.012 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi190.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T03:37:42.012 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:37:42.066 INFO:teuthology.orchestra.run.smithi001.stderr:Generating grub configuration file ... 2024-09-15T03:37:42.069 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:37:42.070 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:37:42.070 DEBUG:teuthology.orchestra.run.smithi190:> sudo rpm -qi grub2-tools 2024-09-15T03:37:42.102 INFO:teuthology.orchestra.run.smithi190.stdout:Name : grub2-tools 2024-09-15T03:37:42.102 INFO:teuthology.orchestra.run.smithi190.stdout:Epoch : 1 2024-09-15T03:37:42.102 INFO:teuthology.orchestra.run.smithi190.stdout:Version : 2.06 2024-09-15T03:37:42.102 INFO:teuthology.orchestra.run.smithi190.stdout:Release : 82.el9 2024-09-15T03:37:42.102 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: x86_64 2024-09-15T03:37:42.102 INFO:teuthology.orchestra.run.smithi190.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T03:37:42.103 INFO:teuthology.orchestra.run.smithi190.stdout:Group : Unspecified 2024-09-15T03:37:42.103 INFO:teuthology.orchestra.run.smithi190.stdout:Size : 8274670 2024-09-15T03:37:42.103 INFO:teuthology.orchestra.run.smithi190.stdout:License : GPLv3+ 2024-09-15T03:37:42.103 INFO:teuthology.orchestra.run.smithi190.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T03:37:42.103 INFO:teuthology.orchestra.run.smithi190.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T03:37:42.103 INFO:teuthology.orchestra.run.smithi190.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T03:37:42.103 INFO:teuthology.orchestra.run.smithi190.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T03:37:42.103 INFO:teuthology.orchestra.run.smithi190.stdout:Packager : builder@centos.org 2024-09-15T03:37:42.103 INFO:teuthology.orchestra.run.smithi190.stdout:Vendor : CentOS 2024-09-15T03:37:42.103 INFO:teuthology.orchestra.run.smithi190.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T03:37:42.103 INFO:teuthology.orchestra.run.smithi190.stdout:Summary : Support tools for GRUB. 2024-09-15T03:37:42.103 INFO:teuthology.orchestra.run.smithi190.stdout:Description : 2024-09-15T03:37:42.103 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:37:42.103 INFO:teuthology.orchestra.run.smithi190.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T03:37:42.104 INFO:teuthology.orchestra.run.smithi190.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T03:37:42.104 INFO:teuthology.orchestra.run.smithi190.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T03:37:42.104 INFO:teuthology.orchestra.run.smithi190.stdout:hardware devices. 2024-09-15T03:37:42.104 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:37:42.104 INFO:teuthology.orchestra.run.smithi190.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T03:37:42.105 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T03:37:42.105 INFO:teuthology.task.kernel:Updating grub on smithi190 to boot 5.14.0-505.el9.x86_64 2024-09-15T03:37:42.105 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T03:37:42.248 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:37:42.335 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:37:42.336 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:37:42.336 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T03:37:42.336 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T03:37:42.336 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T03:37:42.336 INFO:teuthology.task.kernel:Installing distro kernel on osd.4... 2024-09-15T03:37:42.336 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi067.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T03:37:42.336 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:37:42.396 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:37:42.397 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:37:42.397 DEBUG:teuthology.orchestra.run.smithi067:> sudo rpm -qi grub2-tools 2024-09-15T03:37:42.437 INFO:teuthology.orchestra.run.smithi067.stdout:Name : grub2-tools 2024-09-15T03:37:42.437 INFO:teuthology.orchestra.run.smithi067.stdout:Epoch : 1 2024-09-15T03:37:42.437 INFO:teuthology.orchestra.run.smithi067.stdout:Version : 2.06 2024-09-15T03:37:42.437 INFO:teuthology.orchestra.run.smithi067.stdout:Release : 82.el9 2024-09-15T03:37:42.437 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: x86_64 2024-09-15T03:37:42.437 INFO:teuthology.orchestra.run.smithi067.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T03:37:42.437 INFO:teuthology.orchestra.run.smithi067.stdout:Group : Unspecified 2024-09-15T03:37:42.437 INFO:teuthology.orchestra.run.smithi067.stdout:Size : 8274670 2024-09-15T03:37:42.437 INFO:teuthology.orchestra.run.smithi067.stdout:License : GPLv3+ 2024-09-15T03:37:42.438 INFO:teuthology.orchestra.run.smithi067.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T03:37:42.438 INFO:teuthology.orchestra.run.smithi067.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T03:37:42.438 INFO:teuthology.orchestra.run.smithi067.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T03:37:42.438 INFO:teuthology.orchestra.run.smithi067.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T03:37:42.438 INFO:teuthology.orchestra.run.smithi067.stdout:Packager : builder@centos.org 2024-09-15T03:37:42.438 INFO:teuthology.orchestra.run.smithi067.stdout:Vendor : CentOS 2024-09-15T03:37:42.438 INFO:teuthology.orchestra.run.smithi067.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T03:37:42.438 INFO:teuthology.orchestra.run.smithi067.stdout:Summary : Support tools for GRUB. 2024-09-15T03:37:42.438 INFO:teuthology.orchestra.run.smithi067.stdout:Description : 2024-09-15T03:37:42.438 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:37:42.438 INFO:teuthology.orchestra.run.smithi067.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T03:37:42.438 INFO:teuthology.orchestra.run.smithi067.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T03:37:42.438 INFO:teuthology.orchestra.run.smithi067.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T03:37:42.438 INFO:teuthology.orchestra.run.smithi067.stdout:hardware devices. 2024-09-15T03:37:42.438 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:37:42.439 INFO:teuthology.orchestra.run.smithi067.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T03:37:42.439 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T03:37:42.439 INFO:teuthology.task.kernel:Updating grub on smithi067 to boot 5.14.0-505.el9.x86_64 2024-09-15T03:37:42.440 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T03:37:42.700 INFO:teuthology.orchestra.run.smithi190.stderr:Generating grub configuration file ... 2024-09-15T03:37:43.062 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2024-09-15T03:37:43.621 INFO:teuthology.orchestra.run.smithi001.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T03:37:43.660 INFO:teuthology.orchestra.run.smithi001.stderr:done 2024-09-15T03:37:43.661 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-09-15T03:37:43.676 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.sSFm4o3RBx 2024-09-15T03:37:43.676 DEBUG:teuthology.orchestra.run.smithi001:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.sSFm4o3RBx 2024-09-15T03:37:43.743 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.sSFm4o3RBx 2024-09-15T03:37:43.890 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.sSFm4o3RBx is 6KB 2024-09-15T03:37:43.940 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.sSFm4o3RBx 2024-09-15T03:37:43.955 DEBUG:teuthology.orchestra.run.smithi001:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T03:37:44.017 INFO:teuthology.orchestra.run.smithi001.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T03:37:44.017 INFO:teuthology.orchestra.run.smithi001.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T03:37:44.017 INFO:teuthology.orchestra.run.smithi001.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T03:37:44.017 INFO:teuthology.orchestra.run.smithi001.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T03:37:44.018 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T03:37:44.059 INFO:teuthology.orchestra.run.smithi190.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T03:37:44.097 INFO:teuthology.orchestra.run.smithi190.stderr:done 2024-09-15T03:37:44.100 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-09-15T03:37:44.115 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.fTb3fGWKZp 2024-09-15T03:37:44.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.fTb3fGWKZp 2024-09-15T03:37:44.180 DEBUG:teuthology.orchestra.run.smithi001:> sudo shutdown -r now 2024-09-15T03:37:44.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.fTb3fGWKZp 2024-09-15T03:37:44.334 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.fTb3fGWKZp is 6KB 2024-09-15T03:37:44.383 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.fTb3fGWKZp 2024-09-15T03:37:44.388 INFO:teuthology.orchestra.run.smithi067.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T03:37:44.398 DEBUG:teuthology.orchestra.run.smithi190:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T03:37:44.414 INFO:teuthology.orchestra.run.smithi067.stderr:done 2024-09-15T03:37:44.416 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-09-15T03:37:44.432 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.mCwlt1gr5A 2024-09-15T03:37:44.432 DEBUG:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mCwlt1gr5A 2024-09-15T03:37:44.467 INFO:teuthology.orchestra.run.smithi190.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T03:37:44.467 INFO:teuthology.orchestra.run.smithi190.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T03:37:44.467 INFO:teuthology.orchestra.run.smithi190.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T03:37:44.468 INFO:teuthology.orchestra.run.smithi190.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T03:37:44.469 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T03:37:44.499 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.mCwlt1gr5A 2024-09-15T03:37:44.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo shutdown -r now 2024-09-15T03:37:44.642 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.mCwlt1gr5A is 6KB 2024-09-15T03:37:44.691 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.mCwlt1gr5A 2024-09-15T03:37:44.706 DEBUG:teuthology.orchestra.run.smithi067:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T03:37:44.768 INFO:teuthology.orchestra.run.smithi067.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T03:37:44.768 INFO:teuthology.orchestra.run.smithi067.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T03:37:44.768 INFO:teuthology.orchestra.run.smithi067.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T03:37:44.768 INFO:teuthology.orchestra.run.smithi067.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T03:37:44.769 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T03:37:44.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo shutdown -r now 2024-09-15T03:38:14.210 INFO:teuthology.task.kernel:Checking client osd.0 for new kernel version... 2024-09-15T03:38:14.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-15T03:38:14.212 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:38:14.628 INFO:teuthology.task.kernel:Checking client osd.8 for new kernel version... 2024-09-15T03:38:14.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-15T03:38:14.629 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:38:14.925 INFO:teuthology.task.kernel:Checking client osd.4 for new kernel version... 2024-09-15T03:38:14.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-15T03:38:14.926 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:38:32.637 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.1 2024-09-15T03:38:33.152 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-09-15T03:38:33.405 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-09-15T03:38:41.645 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-15T03:38:41.646 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:38:42.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-15T03:38:42.154 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:38:42.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-15T03:38:42.407 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:38:44.705 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.1 2024-09-15T03:38:45.213 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-09-15T03:38:45.473 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-09-15T03:38:56.717 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-15T03:38:56.718 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:38:57.214 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-15T03:38:57.215 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:38:57.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-15T03:38:57.475 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:39:15.905 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-09-15T03:39:28.517 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-09-15T03:39:29.075 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-09-15T03:39:29.083 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-15T03:39:29.083 INFO:teuthology.task.kernel:Checking kernel version of osd.0, want "5.14.0-505.el9.x86_64"... 2024-09-15T03:39:29.083 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-09-15T03:39:29.098 INFO:teuthology.orchestra.run.smithi001.stdout:5.14.0-505.el9.x86_64 2024-09-15T03:39:29.098 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T03:39:29.098 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T03:39:29.098 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T03:39:29.414 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-15T03:39:29.414 INFO:teuthology.task.kernel:Checking kernel version of osd.8, want "5.14.0-505.el9.x86_64"... 2024-09-15T03:39:29.414 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-09-15T03:39:29.429 INFO:teuthology.orchestra.run.smithi190.stdout:5.14.0-505.el9.x86_64 2024-09-15T03:39:29.430 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T03:39:29.430 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T03:39:29.430 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T03:39:30.099 INFO:teuthology.task.kernel:Enabling kdb on osd.0... 2024-09-15T03:39:30.099 DEBUG:teuthology.orchestra.run.smithi001:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T03:39:30.202 INFO:teuthology.orchestra.run.smithi001.stdout:ttyS1 2024-09-15T03:39:30.229 DEBUG:teuthology.parallel:result is None 2024-09-15T03:39:30.430 INFO:teuthology.task.kernel:Enabling kdb on osd.8... 2024-09-15T03:39:30.430 DEBUG:teuthology.orchestra.run.smithi190:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T03:39:30.502 INFO:teuthology.orchestra.run.smithi190.stdout:ttyS1 2024-09-15T03:39:30.524 DEBUG:teuthology.parallel:result is None 2024-09-15T03:39:30.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-15T03:39:30.907 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:39:31.160 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-09-15T03:39:31.696 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-15T03:39:31.696 INFO:teuthology.task.kernel:Checking kernel version of osd.4, want "5.14.0-505.el9.x86_64"... 2024-09-15T03:39:31.697 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-09-15T03:39:31.722 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-505.el9.x86_64 2024-09-15T03:39:31.722 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T03:39:31.722 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T03:39:31.722 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T03:39:32.723 INFO:teuthology.task.kernel:Enabling kdb on osd.4... 2024-09-15T03:39:32.724 DEBUG:teuthology.orchestra.run.smithi067:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T03:39:32.796 INFO:teuthology.orchestra.run.smithi067.stdout:ttyS1 2024-09-15T03:39:32.826 DEBUG:teuthology.parallel:result is None 2024-09-15T03:39:32.826 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-15T03:39:32.833 INFO:teuthology.task.internal:Creating test directory... 2024-09-15T03:39:32.833 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T03:39:32.835 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T03:39:32.837 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T03:39:32.856 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-15T03:39:32.862 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-15T03:39:32.873 INFO:teuthology.task.internal:Creating archive directory... 2024-09-15T03:39:32.873 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T03:39:32.891 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T03:39:32.899 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T03:39:33.020 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-15T03:39:33.027 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-15T03:39:33.027 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T03:39:33.029 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T03:39:33.031 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T03:39:33.053 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:39:33.063 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:39:33.065 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:39:33.077 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:39:33.170 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:39:33.181 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:39:33.183 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-15T03:39:33.190 INFO:teuthology.task.internal:Configuring sudo... 2024-09-15T03:39:33.191 DEBUG:teuthology.orchestra.run.smithi001:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T03:39:33.193 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T03:39:33.195 DEBUG:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T03:39:33.223 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-15T03:39:33.233 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-15T03:39:33.234 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T03:39:33.260 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T03:39:33.265 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T03:39:33.281 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:39:33.356 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:39:33.404 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:39:33.485 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:39:33.527 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-15T03:39:33.527 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T03:39:33.593 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:39:33.636 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:39:33.689 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:39:33.776 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:39:33.820 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T03:39:33.820 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T03:39:33.888 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:39:33.934 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:39:33.994 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:39:34.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:39:34.081 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-15T03:39:34.081 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T03:39:34.148 DEBUG:teuthology.orchestra.run.smithi001:> sudo service rsyslog restart 2024-09-15T03:39:34.150 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-09-15T03:39:34.152 DEBUG:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2024-09-15T03:39:34.191 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:39:34.205 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:39:34.233 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:39:34.675 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-15T03:39:34.683 INFO:teuthology.task.internal:Starting timer... 2024-09-15T03:39:34.683 INFO:teuthology.run_tasks:Running task pcp... 2024-09-15T03:39:34.694 INFO:teuthology.run_tasks:Running task selinux... 2024-09-15T03:39:34.704 DEBUG:teuthology.orchestra.run.smithi001:> sudo service auditd rotate 2024-09-15T03:39:34.762 INFO:teuthology.orchestra.run.smithi001.stdout:Rotating logs: 2024-09-15T03:39:34.764 DEBUG:teuthology.orchestra.run.smithi067:> sudo service auditd rotate 2024-09-15T03:39:34.816 INFO:teuthology.orchestra.run.smithi067.stdout:Rotating logs: 2024-09-15T03:39:34.818 DEBUG:teuthology.orchestra.run.smithi190:> sudo service auditd rotate 2024-09-15T03:39:34.861 INFO:teuthology.orchestra.run.smithi190.stdout:Rotating logs: 2024-09-15T03:39:34.863 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-15T03:39:34.863 DEBUG:teuthology.orchestra.run.smithi001:> /usr/sbin/getenforce 2024-09-15T03:39:34.912 INFO:teuthology.orchestra.run.smithi001.stdout:Permissive 2024-09-15T03:39:34.913 DEBUG:teuthology.orchestra.run.smithi067:> /usr/sbin/getenforce 2024-09-15T03:39:34.956 INFO:teuthology.orchestra.run.smithi067.stdout:Permissive 2024-09-15T03:39:34.957 DEBUG:teuthology.orchestra.run.smithi190:> /usr/sbin/getenforce 2024-09-15T03:39:34.978 INFO:teuthology.orchestra.run.smithi190.stdout:Permissive 2024-09-15T03:39:34.978 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi001.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi067.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi190.front.sepia.ceph.com': 'permissive'} 2024-09-15T03:39:34.978 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:39:35.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:39:35.007 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:39:35.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:39:35.034 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:39:35.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:39:35.061 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-15T03:39:35.061 DEBUG:teuthology.orchestra.run.smithi001:> sudo /usr/sbin/setenforce permissive 2024-09-15T03:39:35.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo /usr/sbin/setenforce permissive 2024-09-15T03:39:35.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo /usr/sbin/setenforce permissive 2024-09-15T03:39:35.169 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-15T03:39:35.179 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-15T03:39:35.181 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-15T03:39:35.297 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-15T03:39:35.310 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-15T03:39:35.311 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi001.front.sepia.ceph.com,smithi067.front.sepia.ceph.com,smithi190.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-15T03:43:24.700 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi001.front.sepia.ceph.com'), Remote(name='ubuntu@smithi067.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com')] 2024-09-15T03:43:24.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-15T03:43:24.702 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:43:24.786 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-09-15T03:43:24.866 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-15T03:43:24.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-15T03:43:24.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:43:24.953 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-09-15T03:43:25.034 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-15T03:43:25.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-15T03:43:25.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:43:25.113 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-09-15T03:43:25.193 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-15T03:43:25.194 INFO:teuthology.run_tasks:Running task clock... 2024-09-15T03:43:25.205 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-15T03:43:25.205 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T03:43:25.206 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:43:25.209 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T03:43:25.209 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:43:25.211 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T03:43:25.211 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:43:25.242 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T03:43:25.243 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T03:43:25.258 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T03:43:25.259 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T03:43:25.265 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T03:43:25.280 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T03:43:25.285 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: ntpd: command not found 2024-09-15T03:43:25.286 INFO:teuthology.orchestra.run.smithi001.stderr:sudo: ntpd: command not found 2024-09-15T03:43:25.297 INFO:teuthology.orchestra.run.smithi067.stdout:506 Cannot talk to daemon 2024-09-15T03:43:25.298 INFO:teuthology.orchestra.run.smithi001.stdout:506 Cannot talk to daemon 2024-09-15T03:43:25.306 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: ntpd: command not found 2024-09-15T03:43:25.313 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T03:43:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T03:43:25.317 INFO:teuthology.orchestra.run.smithi190.stdout:506 Cannot talk to daemon 2024-09-15T03:43:25.328 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T03:43:25.330 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T03:43:25.331 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T03:43:25.346 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T03:43:25.385 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:43:25.386 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:43:25.389 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:43:25.389 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-09-15T03:43:25.389 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:43:25.389 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:43:25.389 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:43:25.389 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:43:25.391 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:43:25.391 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-09-15T03:43:25.391 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:43:25.391 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:43:25.391 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:43:25.391 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:43:25.402 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:43:25.406 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:43:25.406 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-09-15T03:43:25.406 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:43:25.406 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:43:25.406 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:43:25.406 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:43:25.407 INFO:teuthology.run_tasks:Running task install... 2024-09-15T03:43:25.416 DEBUG:teuthology.task.install:project ceph 2024-09-15T03:43:25.416 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-15T03:43:25.416 DEBUG:teuthology.task.install:config {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'} 2024-09-15T03:43:25.416 INFO:teuthology.task.install:Using flavor: default 2024-09-15T03:43:25.422 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-15T03:43:25.423 INFO:teuthology.task.install:extra packages: [] 2024-09-15T03:43:25.423 DEBUG:teuthology.orchestra.run.smithi001:> sudo dnf -y install 'dnf-command(copr)' 2024-09-15T03:43:25.423 DEBUG:teuthology.orchestra.run.smithi067:> sudo dnf -y install 'dnf-command(copr)' 2024-09-15T03:43:25.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo dnf -y install 'dnf-command(copr)' 2024-09-15T03:43:26.004 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:49 ago on Sun 15 Sep 2024 03:42:36 AM UTC. 2024-09-15T03:43:26.012 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:45 ago on Sun 15 Sep 2024 03:42:40 AM UTC. 2024-09-15T03:43:26.012 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:45 ago on Sun 15 Sep 2024 03:42:40 AM UTC. 2024-09-15T03:43:26.092 INFO:teuthology.orchestra.run.smithi190.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-15T03:43:26.102 INFO:teuthology.orchestra.run.smithi067.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-15T03:43:26.102 INFO:teuthology.orchestra.run.smithi001.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-15T03:43:26.134 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:43:26.135 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-15T03:43:26.135 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:43:26.146 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:43:26.147 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:43:26.148 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-09-15T03:43:26.148 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:43:26.148 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-15T03:43:26.148 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:43:26.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo dnf -y copr enable ceph/el9 2024-09-15T03:43:26.203 DEBUG:teuthology.orchestra.run.smithi067:> sudo dnf -y copr enable ceph/el9 2024-09-15T03:43:26.210 DEBUG:teuthology.orchestra.run.smithi001:> sudo dnf -y copr enable ceph/el9 2024-09-15T03:43:26.365 INFO:teuthology.orchestra.run.smithi190.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-15T03:43:26.365 INFO:teuthology.orchestra.run.smithi190.stderr:of the main distribution, and quality may vary. 2024-09-15T03:43:26.366 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-15T03:43:26.366 INFO:teuthology.orchestra.run.smithi190.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-15T03:43:26.366 INFO:teuthology.orchestra.run.smithi190.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-15T03:43:26.366 INFO:teuthology.orchestra.run.smithi190.stderr:, 2024-09-15T03:43:26.366 INFO:teuthology.orchestra.run.smithi190.stderr:and packages are not held to any quality or security level. 2024-09-15T03:43:26.366 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-15T03:43:26.366 INFO:teuthology.orchestra.run.smithi190.stderr:Please do not file bug reports about these packages in Fedora 2024-09-15T03:43:26.367 INFO:teuthology.orchestra.run.smithi190.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-15T03:43:26.382 INFO:teuthology.orchestra.run.smithi067.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-15T03:43:26.382 INFO:teuthology.orchestra.run.smithi067.stderr:of the main distribution, and quality may vary. 2024-09-15T03:43:26.382 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T03:43:26.382 INFO:teuthology.orchestra.run.smithi067.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-15T03:43:26.383 INFO:teuthology.orchestra.run.smithi067.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-15T03:43:26.383 INFO:teuthology.orchestra.run.smithi067.stderr:, 2024-09-15T03:43:26.383 INFO:teuthology.orchestra.run.smithi067.stderr:and packages are not held to any quality or security level. 2024-09-15T03:43:26.383 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T03:43:26.383 INFO:teuthology.orchestra.run.smithi067.stderr:Please do not file bug reports about these packages in Fedora 2024-09-15T03:43:26.383 INFO:teuthology.orchestra.run.smithi067.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-15T03:43:26.389 INFO:teuthology.orchestra.run.smithi001.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-15T03:43:26.390 INFO:teuthology.orchestra.run.smithi001.stderr:of the main distribution, and quality may vary. 2024-09-15T03:43:26.390 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-09-15T03:43:26.390 INFO:teuthology.orchestra.run.smithi001.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-15T03:43:26.390 INFO:teuthology.orchestra.run.smithi001.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-15T03:43:26.390 INFO:teuthology.orchestra.run.smithi001.stderr:, 2024-09-15T03:43:26.390 INFO:teuthology.orchestra.run.smithi001.stderr:and packages are not held to any quality or security level. 2024-09-15T03:43:26.390 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-09-15T03:43:26.390 INFO:teuthology.orchestra.run.smithi001.stderr:Please do not file bug reports about these packages in Fedora 2024-09-15T03:43:26.390 INFO:teuthology.orchestra.run.smithi001.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-15T03:43:26.523 INFO:teuthology.orchestra.run.smithi067.stdout:Repository successfully enabled. 2024-09-15T03:43:26.525 INFO:teuthology.orchestra.run.smithi190.stdout:Repository successfully enabled. 2024-09-15T03:43:26.526 INFO:teuthology.orchestra.run.smithi001.stdout:Repository successfully enabled. 2024-09-15T03:43:26.559 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-15T03:43:26.560 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:43:26.566 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-15T03:43:26.566 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:43:26.567 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-15T03:43:26.567 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:43:26.719 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-15T03:43:26.719 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-15T03:43:26.723 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-15T03:43:26.723 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-15T03:43:26.735 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-15T03:43:26.735 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-15T03:43:26.828 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T03:43:26.829 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-15T03:43:26.829 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T03:43:26.835 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T03:43:26.835 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T03:43:26.835 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T03:43:26.843 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T03:43:26.843 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-15T03:43:26.843 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T03:43:26.854 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-15T03:43:26.854 DEBUG:teuthology.orchestra.run.smithi190:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T03:43:26.862 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-15T03:43:26.862 DEBUG:teuthology.orchestra.run.smithi067:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T03:43:26.873 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-15T03:43:26.873 DEBUG:teuthology.orchestra.run.smithi001:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T03:43:26.921 DEBUG:teuthology.orchestra.run.smithi190:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T03:43:26.929 DEBUG:teuthology.orchestra.run.smithi067:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T03:43:26.941 DEBUG:teuthology.orchestra.run.smithi001:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T03:43:26.996 DEBUG:teuthology.orchestra.run.smithi190:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T03:43:27.006 DEBUG:teuthology.orchestra.run.smithi067:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T03:43:27.018 DEBUG:teuthology.orchestra.run.smithi001:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T03:43:27.061 INFO:teuthology.orchestra.run.smithi190.stdout:check_obsoletes = 1 2024-09-15T03:43:27.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum clean all 2024-09-15T03:43:27.071 INFO:teuthology.orchestra.run.smithi067.stdout:check_obsoletes = 1 2024-09-15T03:43:27.072 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum clean all 2024-09-15T03:43:27.083 INFO:teuthology.orchestra.run.smithi001.stdout:check_obsoletes = 1 2024-09-15T03:43:27.085 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum clean all 2024-09-15T03:43:27.297 INFO:teuthology.orchestra.run.smithi190.stdout:45 files removed 2024-09-15T03:43:27.316 INFO:teuthology.orchestra.run.smithi067.stdout:45 files removed 2024-09-15T03:43:27.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-radosgw 2024-09-15T03:43:27.326 INFO:teuthology.orchestra.run.smithi001.stdout:45 files removed 2024-09-15T03:43:27.344 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-radosgw 2024-09-15T03:43:27.353 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-radosgw 2024-09-15T03:43:27.877 INFO:teuthology.orchestra.run.smithi190.stdout:ceph packages for x86_64 291 kB/s | 77 kB 00:00 2024-09-15T03:43:27.911 INFO:teuthology.orchestra.run.smithi001.stdout:ceph packages for x86_64 298 kB/s | 77 kB 00:00 2024-09-15T03:43:27.917 INFO:teuthology.orchestra.run.smithi067.stdout:ceph packages for x86_64 282 kB/s | 77 kB 00:00 2024-09-15T03:43:28.161 INFO:teuthology.orchestra.run.smithi190.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-15T03:43:28.195 INFO:teuthology.orchestra.run.smithi001.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-15T03:43:28.209 INFO:teuthology.orchestra.run.smithi067.stdout:ceph noarch packages 44 kB/s | 11 kB 00:00 2024-09-15T03:43:28.437 INFO:teuthology.orchestra.run.smithi190.stdout:ceph source packages 7.5 kB/s | 1.8 kB 00:00 2024-09-15T03:43:28.479 INFO:teuthology.orchestra.run.smithi001.stdout:ceph source packages 7.3 kB/s | 1.8 kB 00:00 2024-09-15T03:43:28.520 INFO:teuthology.orchestra.run.smithi067.stdout:ceph source packages 6.6 kB/s | 1.8 kB 00:00 2024-09-15T03:43:28.623 INFO:teuthology.orchestra.run.smithi190.stdout:Copr repo for el9 owned by ceph 51 kB/s | 7.6 kB 00:00 2024-09-15T03:43:28.629 INFO:teuthology.orchestra.run.smithi001.stdout:Copr repo for el9 owned by ceph 67 kB/s | 7.6 kB 00:00 2024-09-15T03:43:28.685 INFO:teuthology.orchestra.run.smithi067.stdout:Copr repo for el9 owned by ceph 59 kB/s | 7.6 kB 00:00 2024-09-15T03:43:29.202 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.2 MB 00:00 2024-09-15T03:43:29.212 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.2 MB 00:00 2024-09-15T03:43:30.265 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.2 MB 00:01 2024-09-15T03:43:31.178 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 25 MB/s | 20 MB 00:00 2024-09-15T03:43:32.458 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 9.7 MB/s | 20 MB 00:02 2024-09-15T03:43:35.923 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 6.5 MB 00:00 2024-09-15T03:43:36.892 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-15T03:43:37.640 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 19 kB 00:00 2024-09-15T03:43:38.140 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 3.0 MB/s | 20 MB 00:06 2024-09-15T03:43:38.434 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 19 kB 00:00 2024-09-15T03:43:38.486 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 32 MB/s | 23 MB 00:00 2024-09-15T03:43:39.547 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-15T03:43:42.832 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 7.1 MB/s | 6.5 MB 00:00 2024-09-15T03:43:44.175 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-15T03:43:44.508 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 19 kB 00:00 2024-09-15T03:43:45.152 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-09-15T03:43:45.379 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-15T03:43:46.737 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:43:46.755 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:43:46.755 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-09-15T03:43:46.755 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:43:46.755 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T03:43:46.755 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-15T03:43:46.756 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrading: 2024-09-15T03:43:46.756 INFO:teuthology.orchestra.run.smithi067.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-15T03:43:46.756 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:43:46.756 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-09-15T03:43:46.756 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-15T03:43:46.756 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-15T03:43:46.756 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:43:46.756 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T03:43:46.756 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-15T03:43:46.756 INFO:teuthology.orchestra.run.smithi067.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T03:43:46.757 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T03:43:46.757 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T03:43:46.757 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-15T03:43:46.757 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T03:43:46.757 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-15T03:43:46.757 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T03:43:46.757 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T03:43:46.757 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-15T03:43:46.757 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-15T03:43:46.757 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-15T03:43:46.758 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T03:43:46.758 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-15T03:43:46.758 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-15T03:43:46.758 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-15T03:43:46.758 INFO:teuthology.orchestra.run.smithi067.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T03:43:46.758 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:43:46.758 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T03:43:46.758 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:43:46.758 INFO:teuthology.orchestra.run.smithi067.stdout:Install 22 Packages 2024-09-15T03:43:46.758 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade 2 Packages 2024-09-15T03:43:46.758 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:43:46.759 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 58 M 2024-09-15T03:43:46.759 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T03:43:47.722 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:43:47.731 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:43:47.731 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-15T03:43:47.731 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:43:47.731 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-15T03:43:47.731 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-15T03:43:47.731 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrading: 2024-09-15T03:43:47.732 INFO:teuthology.orchestra.run.smithi001.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-15T03:43:47.732 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:43:47.732 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-15T03:43:47.732 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-15T03:43:47.732 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-15T03:43:47.732 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:43:47.732 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T03:43:47.732 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-15T03:43:47.732 INFO:teuthology.orchestra.run.smithi001.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T03:43:47.732 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T03:43:47.732 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T03:43:47.732 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-15T03:43:47.732 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T03:43:47.733 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-15T03:43:47.733 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T03:43:47.733 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T03:43:47.733 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-15T03:43:47.733 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-15T03:43:47.733 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-15T03:43:47.733 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T03:43:47.733 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-15T03:43:47.733 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-15T03:43:47.733 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-15T03:43:47.733 INFO:teuthology.orchestra.run.smithi001.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T03:43:47.733 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:43:47.734 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-15T03:43:47.734 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:43:47.734 INFO:teuthology.orchestra.run.smithi001.stdout:Install 22 Packages 2024-09-15T03:43:47.734 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrade 2 Packages 2024-09-15T03:43:47.734 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:43:47.734 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 58 M 2024-09-15T03:43:47.735 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-15T03:43:47.785 INFO:teuthology.orchestra.run.smithi067.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 8.1 MB/s | 5.0 MB 00:00 2024-09-15T03:43:47.852 INFO:teuthology.orchestra.run.smithi067.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 377 kB/s | 25 kB 00:00 2024-09-15T03:43:47.944 INFO:teuthology.orchestra.run.smithi067.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 6.8 MB/s | 637 kB 00:00 2024-09-15T03:43:48.027 INFO:teuthology.orchestra.run.smithi067.stdout:(4/24): libradosstriper1-17.2.7-1643.gd75f2382. 5.4 MB/s | 458 kB 00:00 2024-09-15T03:43:48.169 INFO:teuthology.orchestra.run.smithi067.stdout:(5/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 15 MB/s | 14 MB 00:00 2024-09-15T03:43:48.219 INFO:teuthology.orchestra.run.smithi067.stdout:(6/24): python3-ceph-argparse-17.2.7-1643.gd75f 911 kB/s | 45 kB 00:00 2024-09-15T03:43:48.436 INFO:teuthology.orchestra.run.smithi067.stdout:(7/24): ceph-common-17.2.7-1643.gd75f2382.el9.x 18 MB/s | 22 MB 00:01 2024-09-15T03:43:48.461 INFO:teuthology.orchestra.run.smithi067.stdout:(8/24): python3-ceph-common-17.2.7-1643.gd75f23 497 kB/s | 120 kB 00:00 2024-09-15T03:43:48.470 INFO:teuthology.orchestra.run.smithi001.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 8.8 MB/s | 5.0 MB 00:00 2024-09-15T03:43:48.487 INFO:teuthology.orchestra.run.smithi067.stdout:(9/24): python3-cephfs-17.2.7-1643.gd75f2382.el 3.1 MB/s | 156 kB 00:00 2024-09-15T03:43:48.512 INFO:teuthology.orchestra.run.smithi067.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 6.2 MB/s | 320 kB 00:00 2024-09-15T03:43:48.529 INFO:teuthology.orchestra.run.smithi001.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 431 kB/s | 25 kB 00:00 2024-09-15T03:43:48.537 INFO:teuthology.orchestra.run.smithi067.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 5.7 MB/s | 295 kB 00:00 2024-09-15T03:43:48.562 INFO:teuthology.orchestra.run.smithi067.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.9 MB/s | 100 kB 00:00 2024-09-15T03:43:48.596 INFO:teuthology.orchestra.run.smithi001.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 9.4 MB/s | 637 kB 00:00 2024-09-15T03:43:48.630 INFO:teuthology.orchestra.run.smithi067.stdout:(13/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_ 8.2 MB/s | 4.9 MB 00:00 2024-09-15T03:43:48.671 INFO:teuthology.orchestra.run.smithi001.stdout:(4/24): libradosstriper1-17.2.7-1643.gd75f2382. 6.0 MB/s | 458 kB 00:00 2024-09-15T03:43:48.697 INFO:teuthology.orchestra.run.smithi067.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-15T03:43:48.722 INFO:teuthology.orchestra.run.smithi067.stdout:(15/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 284 kB/s | 45 kB 00:00 2024-09-15T03:43:48.764 INFO:teuthology.orchestra.run.smithi067.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-15T03:43:48.789 INFO:teuthology.orchestra.run.smithi067.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-09-15T03:43:48.847 INFO:teuthology.orchestra.run.smithi067.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.0 MB/s | 662 kB 00:00 2024-09-15T03:43:48.881 INFO:teuthology.orchestra.run.smithi067.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 529 kB/s | 49 kB 00:00 2024-09-15T03:43:48.907 INFO:teuthology.orchestra.run.smithi067.stdout:(20/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.1 MB/s | 308 kB 00:00 2024-09-15T03:43:48.932 INFO:teuthology.orchestra.run.smithi067.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 824 kB/s | 67 kB 00:00 2024-09-15T03:43:48.980 INFO:teuthology.orchestra.run.smithi001.stdout:(5/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 16 MB/s | 4.9 MB 00:00 2024-09-15T03:43:49.038 INFO:teuthology.orchestra.run.smithi001.stdout:(6/24): python3-ceph-argparse-17.2.7-1643.gd75f 776 kB/s | 45 kB 00:00 2024-09-15T03:43:49.040 INFO:teuthology.orchestra.run.smithi067.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-09-15T03:43:49.224 INFO:teuthology.orchestra.run.smithi067.stdout:(23/24): librados2-17.2.7-1643.gd75f2382.el9.x8 11 MB/s | 3.1 MB 00:00 2024-09-15T03:43:49.282 INFO:teuthology.orchestra.run.smithi067.stdout:(24/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 8.5 MB/s | 2.9 MB 00:00 2024-09-15T03:43:49.287 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:43:49.287 INFO:teuthology.orchestra.run.smithi067.stdout:Total 23 MB/s | 58 MB 00:02 2024-09-15T03:43:49.305 INFO:teuthology.orchestra.run.smithi001.stdout:(7/24): ceph-common-17.2.7-1643.gd75f2382.el9.x 16 MB/s | 22 MB 00:01 2024-09-15T03:43:49.330 INFO:teuthology.orchestra.run.smithi001.stdout:(8/24): python3-ceph-common-17.2.7-1643.gd75f23 412 kB/s | 120 kB 00:00 2024-09-15T03:43:49.351 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T03:43:49.356 INFO:teuthology.orchestra.run.smithi001.stdout:(9/24): python3-cephfs-17.2.7-1643.gd75f2382.el 3.1 MB/s | 156 kB 00:00 2024-09-15T03:43:49.387 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T03:43:49.388 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T03:43:49.389 INFO:teuthology.orchestra.run.smithi001.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 5.3 MB/s | 320 kB 00:00 2024-09-15T03:43:49.423 INFO:teuthology.orchestra.run.smithi001.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 4.3 MB/s | 295 kB 00:00 2024-09-15T03:43:49.440 INFO:teuthology.orchestra.run.smithi001.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.9 MB/s | 100 kB 00:00 2024-09-15T03:43:49.607 INFO:teuthology.orchestra.run.smithi001.stdout:(13/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9 8.5 MB/s | 14 MB 00:01 2024-09-15T03:43:49.624 INFO:teuthology.orchestra.run.smithi001.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 246 kB/s | 45 kB 00:00 2024-09-15T03:43:49.666 INFO:teuthology.orchestra.run.smithi001.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 661 kB/s | 160 kB 00:00 2024-09-15T03:43:49.699 INFO:teuthology.orchestra.run.smithi001.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 42 kB 00:00 2024-09-15T03:43:49.725 INFO:teuthology.orchestra.run.smithi001.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-15T03:43:49.775 INFO:teuthology.orchestra.run.smithi001.stdout:(18/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 4.0 MB/s | 308 kB 00:00 2024-09-15T03:43:49.800 INFO:teuthology.orchestra.run.smithi001.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2024-09-15T03:43:49.826 INFO:teuthology.orchestra.run.smithi001.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-15T03:43:49.851 INFO:teuthology.orchestra.run.smithi001.stdout:(21/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.7 MB/s | 662 kB 00:00 2024-09-15T03:43:49.901 INFO:teuthology.orchestra.run.smithi001.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-15T03:43:50.001 INFO:teuthology.orchestra.run.smithi001.stdout:(23/24): librados2-17.2.7-1643.gd75f2382.el9.x8 18 MB/s | 3.1 MB 00:00 2024-09-15T03:43:50.068 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T03:43:50.068 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T03:43:50.085 INFO:teuthology.orchestra.run.smithi001.stdout:(24/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 13 MB/s | 2.9 MB 00:00 2024-09-15T03:43:50.089 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:43:50.089 INFO:teuthology.orchestra.run.smithi001.stdout:Total 25 MB/s | 58 MB 00:02 2024-09-15T03:43:50.152 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-15T03:43:50.189 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-15T03:43:50.189 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-15T03:43:50.785 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-09-15T03:43:50.879 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-15T03:43:50.879 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-15T03:43:51.369 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T03:43:51.777 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-15T03:43:51.818 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:43:51.911 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:43:51.991 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:43:52.019 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:43:52.093 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:43:52.137 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-15T03:43:52.138 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-15T03:43:52.166 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-15T03:43:52.527 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-15T03:43:52.587 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:43:52.629 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-15T03:43:52.656 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:43:52.680 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:43:52.718 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:43:52.746 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-15T03:43:52.760 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:43:52.782 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-15T03:43:52.788 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:43:52.819 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:43:52.879 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:43:52.882 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:43:52.907 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:43:52.923 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-15T03:43:52.960 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-15T03:43:52.973 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:43:53.029 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-15T03:43:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-15T03:43:53.116 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-15T03:43:53.302 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:43:53.317 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:43:53.317 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-15T03:43:53.318 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:43:53.318 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-15T03:43:53.318 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-15T03:43:53.318 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrading: 2024-09-15T03:43:53.318 INFO:teuthology.orchestra.run.smithi190.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-15T03:43:53.318 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:43:53.318 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-15T03:43:53.318 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-15T03:43:53.318 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-15T03:43:53.318 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:43:53.318 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T03:43:53.318 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-15T03:43:53.318 INFO:teuthology.orchestra.run.smithi190.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T03:43:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T03:43:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T03:43:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-15T03:43:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T03:43:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-15T03:43:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T03:43:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T03:43:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-15T03:43:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-15T03:43:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-15T03:43:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T03:43:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-15T03:43:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-15T03:43:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-15T03:43:53.320 INFO:teuthology.orchestra.run.smithi190.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T03:43:53.320 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:43:53.320 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-15T03:43:53.320 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:43:53.320 INFO:teuthology.orchestra.run.smithi190.stdout:Install 22 Packages 2024-09-15T03:43:53.320 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrade 2 Packages 2024-09-15T03:43:53.320 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:43:53.321 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 58 M 2024-09-15T03:43:53.321 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-15T03:43:53.422 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-15T03:43:53.425 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-15T03:43:53.449 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:43:53.461 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:43:53.506 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:43:53.536 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:43:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-15T03:43:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-15T03:43:53.611 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-15T03:43:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:43:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-15T03:43:53.694 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:43:53.719 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:43:53.785 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:43:53.824 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-15T03:43:53.861 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-15T03:43:53.894 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-15T03:43:54.204 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-15T03:43:54.248 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:43:54.278 INFO:teuthology.orchestra.run.smithi190.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 10 MB/s | 5.0 MB 00:00 2024-09-15T03:43:54.339 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:43:54.345 INFO:teuthology.orchestra.run.smithi190.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 377 kB/s | 25 kB 00:00 2024-09-15T03:43:54.414 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-15T03:43:54.450 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-15T03:43:54.453 INFO:teuthology.orchestra.run.smithi190.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 5.8 MB/s | 637 kB 00:00 2024-09-15T03:43:54.563 INFO:teuthology.orchestra.run.smithi190.stdout:(4/24): libradosstriper1-17.2.7-1643.gd75f2382. 4.1 MB/s | 458 kB 00:00 2024-09-15T03:43:54.738 INFO:teuthology.orchestra.run.smithi190.stdout:(5/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 15 MB/s | 14 MB 00:00 2024-09-15T03:43:54.779 INFO:teuthology.orchestra.run.smithi190.stdout:(6/24): python3-ceph-argparse-17.2.7-1643.gd75f 1.1 MB/s | 45 kB 00:00 2024-09-15T03:43:54.863 INFO:teuthology.orchestra.run.smithi190.stdout:(7/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 16 MB/s | 4.9 MB 00:00 2024-09-15T03:43:54.897 INFO:teuthology.orchestra.run.smithi190.stdout:(8/24): python3-ceph-common-17.2.7-1643.gd75f23 1.0 MB/s | 120 kB 00:00 2024-09-15T03:43:54.922 INFO:teuthology.orchestra.run.smithi190.stdout:(9/24): python3-cephfs-17.2.7-1643.gd75f2382.el 2.6 MB/s | 156 kB 00:00 2024-09-15T03:43:54.955 INFO:teuthology.orchestra.run.smithi190.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 5.3 MB/s | 320 kB 00:00 2024-09-15T03:43:54.981 INFO:teuthology.orchestra.run.smithi190.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 4.9 MB/s | 295 kB 00:00 2024-09-15T03:43:55.006 INFO:teuthology.orchestra.run.smithi190.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.9 MB/s | 100 kB 00:00 2024-09-15T03:43:55.223 INFO:teuthology.orchestra.run.smithi190.stdout:(13/24): ceph-common-17.2.7-1643.gd75f2382.el9. 16 MB/s | 22 MB 00:01 2024-09-15T03:43:55.332 INFO:teuthology.orchestra.run.smithi190.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 139 kB/s | 45 kB 00:00 2024-09-15T03:43:55.365 INFO:teuthology.orchestra.run.smithi190.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 417 kB/s | 160 kB 00:00 2024-09-15T03:43:55.416 INFO:teuthology.orchestra.run.smithi190.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 845 kB/s | 42 kB 00:00 2024-09-15T03:43:55.449 INFO:teuthology.orchestra.run.smithi190.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-09-15T03:43:55.491 INFO:teuthology.orchestra.run.smithi190.stdout:(18/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 4.0 MB/s | 308 kB 00:00 2024-09-15T03:43:55.525 INFO:teuthology.orchestra.run.smithi190.stdout:(19/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.1 MB/s | 662 kB 00:00 2024-09-15T03:43:55.550 INFO:teuthology.orchestra.run.smithi190.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 492 kB/s | 49 kB 00:00 2024-09-15T03:43:55.584 INFO:teuthology.orchestra.run.smithi190.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 746 kB/s | 67 kB 00:00 2024-09-15T03:43:55.643 INFO:teuthology.orchestra.run.smithi190.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-15T03:43:55.728 INFO:teuthology.orchestra.run.smithi190.stdout:(23/24): librados2-17.2.7-1643.gd75f2382.el9.x8 18 MB/s | 3.1 MB 00:00 2024-09-15T03:43:55.801 INFO:teuthology.orchestra.run.smithi190.stdout:(24/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 14 MB/s | 2.9 MB 00:00 2024-09-15T03:43:55.807 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:43:55.807 INFO:teuthology.orchestra.run.smithi190.stdout:Total 23 MB/s | 58 MB 00:02 2024-09-15T03:43:55.869 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-15T03:43:55.903 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-15T03:43:55.903 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-15T03:43:56.172 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:43:56.209 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:43:56.590 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-15T03:43:56.591 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-15T03:43:56.760 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:43:56.804 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:43:56.883 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:43:56.883 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T03:43:56.883 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T03:43:56.884 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:43:56.914 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:43:56.998 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:43:57.038 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:43:57.587 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:43:57.633 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:43:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:43:57.721 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T03:43:57.721 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T03:43:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:43:57.759 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:43:57.820 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-15T03:43:58.210 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-15T03:43:58.270 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:43:58.345 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:43:58.432 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:43:58.463 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:43:58.552 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:43:58.597 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-15T03:43:58.635 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-15T03:43:59.074 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-15T03:43:59.100 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:43:59.146 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:43:59.183 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-15T03:43:59.226 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-15T03:43:59.263 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:43:59.325 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:43:59.361 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:43:59.427 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:43:59.491 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-15T03:43:59.528 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-15T03:43:59.561 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-15T03:43:59.854 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-15T03:43:59.890 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:43:59.980 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:44:00.053 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-15T03:44:00.084 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-15T03:44:02.548 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:44:02.581 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:44:03.110 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:44:03.151 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:44:03.247 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:44:03.247 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T03:44:03.247 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T03:44:03.248 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:03.277 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:44:07.438 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:44:07.438 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /sys 2024-09-15T03:44:07.438 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /proc 2024-09-15T03:44:07.438 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /mnt 2024-09-15T03:44:07.438 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /var/tmp 2024-09-15T03:44:07.438 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /home 2024-09-15T03:44:07.438 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /root 2024-09-15T03:44:07.438 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /tmp 2024-09-15T03:44:07.438 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:08.108 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:44:08.108 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /sys 2024-09-15T03:44:08.108 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /proc 2024-09-15T03:44:08.108 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /mnt 2024-09-15T03:44:08.108 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /var/tmp 2024-09-15T03:44:08.108 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /home 2024-09-15T03:44:08.108 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /root 2024-09-15T03:44:08.108 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /tmp 2024-09-15T03:44:08.108 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:08.812 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:44:08.892 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:44:08.892 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:44:08.892 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T03:44:08.892 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:44:08.892 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:44:08.893 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:08.894 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:44:08.982 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:44:08.982 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:44:09.468 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:44:09.547 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:44:09.548 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:44:09.548 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T03:44:09.548 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:44:09.548 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:44:09.548 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:09.549 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:44:09.637 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:44:09.637 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:44:10.263 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:44:10.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-15T03:44:10.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:44:10.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:44:10.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:44:10.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-15T03:44:10.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-15T03:44:10.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-15T03:44:10.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-15T03:44:10.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-15T03:44:10.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-15T03:44:10.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:44:10.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:44:10.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-15T03:44:10.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-15T03:44:10.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-15T03:44:10.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-15T03:44:10.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-15T03:44:10.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-15T03:44:10.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-15T03:44:10.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-15T03:44:10.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-15T03:44:10.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-15T03:44:10.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:44:10.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-15T03:44:10.266 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-15T03:44:10.920 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:44:10.920 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-15T03:44:10.920 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:44:10.921 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:44:10.921 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:44:10.921 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-15T03:44:10.921 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-15T03:44:10.921 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-15T03:44:10.921 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-15T03:44:10.921 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-15T03:44:10.921 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-15T03:44:10.921 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:44:10.921 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:44:10.921 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-15T03:44:10.922 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-15T03:44:10.922 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-15T03:44:10.922 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-15T03:44:10.922 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-15T03:44:10.922 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-15T03:44:10.922 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-15T03:44:10.922 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-15T03:44:10.922 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-15T03:44:10.922 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-15T03:44:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:44:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-15T03:44:10.923 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-15T03:44:11.260 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:44:11.261 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:11.261 INFO:teuthology.orchestra.run.smithi067.stdout:Upgraded: 2024-09-15T03:44:11.261 INFO:teuthology.orchestra.run.smithi067.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.261 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.261 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T03:44:11.261 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.261 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.261 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.261 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.261 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T03:44:11.261 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.262 INFO:teuthology.orchestra.run.smithi067.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T03:44:11.262 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T03:44:11.262 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T03:44:11.262 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.262 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T03:44:11.262 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.262 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T03:44:11.262 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T03:44:11.262 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.262 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.262 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.263 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T03:44:11.263 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.263 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.263 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.263 INFO:teuthology.orchestra.run.smithi067.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T03:44:11.263 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:11.263 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:44:11.637 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-test 2024-09-15T03:44:11.769 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:44:11.769 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:11.769 INFO:teuthology.orchestra.run.smithi001.stdout:Upgraded: 2024-09-15T03:44:11.769 INFO:teuthology.orchestra.run.smithi001.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.769 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.769 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-15T03:44:11.769 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.769 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.769 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.769 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.770 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T03:44:11.770 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.770 INFO:teuthology.orchestra.run.smithi001.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T03:44:11.770 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T03:44:11.770 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T03:44:11.770 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.770 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T03:44:11.770 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.770 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T03:44:11.770 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T03:44:11.770 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.770 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.770 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.771 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T03:44:11.771 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.771 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.771 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:11.771 INFO:teuthology.orchestra.run.smithi001.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T03:44:11.771 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:11.771 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:44:12.098 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-test 2024-09-15T03:44:12.223 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:28 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:44:12.369 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:44:12.370 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:44:12.370 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-09-15T03:44:12.370 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:44:12.370 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T03:44:12.370 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-15T03:44:12.370 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-09-15T03:44:12.370 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-15T03:44:12.370 INFO:teuthology.orchestra.run.smithi067.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T03:44:12.371 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T03:44:12.371 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:12.371 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T03:44:12.371 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:44:12.371 INFO:teuthology.orchestra.run.smithi067.stdout:Install 4 Packages 2024-09-15T03:44:12.371 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:12.371 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 48 M 2024-09-15T03:44:12.372 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 193 M 2024-09-15T03:44:12.372 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T03:44:12.619 INFO:teuthology.orchestra.run.smithi067.stdout:(1/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 816 kB/s | 163 kB 00:00 2024-09-15T03:44:12.644 INFO:teuthology.orchestra.run.smithi067.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.3 MB/s | 304 kB 00:00 2024-09-15T03:44:12.661 INFO:teuthology.orchestra.run.smithi067.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-15T03:44:12.679 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:27 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:44:12.823 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:44:12.824 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:44:12.824 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-15T03:44:12.824 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:44:12.824 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-15T03:44:12.824 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-15T03:44:12.824 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-15T03:44:12.824 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-15T03:44:12.824 INFO:teuthology.orchestra.run.smithi001.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T03:44:12.825 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T03:44:12.825 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:12.825 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-15T03:44:12.825 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:44:12.825 INFO:teuthology.orchestra.run.smithi001.stdout:Install 4 Packages 2024-09-15T03:44:12.825 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:12.826 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 48 M 2024-09-15T03:44:12.826 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 193 M 2024-09-15T03:44:12.826 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-15T03:44:13.045 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:44:13.045 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /sys 2024-09-15T03:44:13.045 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /proc 2024-09-15T03:44:13.045 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /mnt 2024-09-15T03:44:13.045 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /var/tmp 2024-09-15T03:44:13.045 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /home 2024-09-15T03:44:13.046 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /root 2024-09-15T03:44:13.046 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /tmp 2024-09-15T03:44:13.046 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:13.050 INFO:teuthology.orchestra.run.smithi001.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-15T03:44:13.101 INFO:teuthology.orchestra.run.smithi001.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-15T03:44:13.242 INFO:teuthology.orchestra.run.smithi001.stdout:(3/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 413 kB/s | 163 kB 00:00 2024-09-15T03:44:13.962 INFO:teuthology.orchestra.run.smithi067.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 31 MB/s | 48 MB 00:01 2024-09-15T03:44:13.964 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:44:13.964 INFO:teuthology.orchestra.run.smithi067.stdout:Total 30 MB/s | 48 MB 00:01 2024-09-15T03:44:13.993 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T03:44:14.015 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T03:44:14.015 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T03:44:14.398 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:44:14.403 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T03:44:14.403 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T03:44:14.477 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:44:14.477 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:44:14.477 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T03:44:14.477 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:44:14.477 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:44:14.477 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:14.479 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:44:14.559 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:44:14.559 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:44:14.953 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T03:44:15.031 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T03:44:15.070 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T03:44:15.097 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:44:15.368 INFO:teuthology.orchestra.run.smithi001.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 19 MB/s | 48 MB 00:02 2024-09-15T03:44:15.369 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:44:15.370 INFO:teuthology.orchestra.run.smithi001.stdout:Total 19 MB/s | 48 MB 00:02 2024-09-15T03:44:15.398 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-15T03:44:15.420 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-15T03:44:15.420 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-15T03:44:15.805 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-15T03:44:15.805 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-15T03:44:15.878 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:44:15.878 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-15T03:44:15.878 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:44:15.879 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:44:15.879 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:44:15.879 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-15T03:44:15.879 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-15T03:44:15.879 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-15T03:44:15.879 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-15T03:44:15.879 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-15T03:44:15.879 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-15T03:44:15.879 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:44:15.879 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:44:15.879 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-15T03:44:15.879 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-15T03:44:15.880 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-15T03:44:15.880 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-15T03:44:15.880 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-15T03:44:15.880 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-15T03:44:15.880 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-15T03:44:15.880 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-15T03:44:15.880 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-15T03:44:15.880 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-15T03:44:15.880 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:44:15.880 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-15T03:44:15.880 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-15T03:44:16.465 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-15T03:44:16.545 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T03:44:16.584 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T03:44:16.612 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:44:16.942 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:44:16.942 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:16.942 INFO:teuthology.orchestra.run.smithi190.stdout:Upgraded: 2024-09-15T03:44:16.942 INFO:teuthology.orchestra.run.smithi190.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:16.943 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:16.943 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-15T03:44:16.943 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:16.943 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:16.943 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:16.943 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:16.943 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T03:44:16.943 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:16.943 INFO:teuthology.orchestra.run.smithi190.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T03:44:16.943 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T03:44:16.943 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T03:44:16.944 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:16.944 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T03:44:16.944 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:16.944 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T03:44:16.944 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T03:44:16.944 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:16.944 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:16.944 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:16.944 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T03:44:16.944 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:16.945 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:16.945 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:16.945 INFO:teuthology.orchestra.run.smithi190.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T03:44:16.945 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:16.945 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:44:17.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-test 2024-09-15T03:44:17.920 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:27 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:44:18.052 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:44:18.052 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:44:18.052 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-15T03:44:18.052 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:44:18.053 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-15T03:44:18.053 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-15T03:44:18.053 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-15T03:44:18.053 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-15T03:44:18.053 INFO:teuthology.orchestra.run.smithi190.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T03:44:18.053 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T03:44:18.053 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:18.053 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-15T03:44:18.053 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:44:18.053 INFO:teuthology.orchestra.run.smithi190.stdout:Install 4 Packages 2024-09-15T03:44:18.053 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:18.054 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 48 M 2024-09-15T03:44:18.054 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 193 M 2024-09-15T03:44:18.054 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-15T03:44:18.373 INFO:teuthology.orchestra.run.smithi190.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-15T03:44:18.398 INFO:teuthology.orchestra.run.smithi190.stdout:(2/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 724 kB/s | 163 kB 00:00 2024-09-15T03:44:18.423 INFO:teuthology.orchestra.run.smithi190.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-15T03:44:19.682 INFO:teuthology.orchestra.run.smithi190.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 32 MB/s | 48 MB 00:01 2024-09-15T03:44:19.684 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:44:19.684 INFO:teuthology.orchestra.run.smithi190.stdout:Total 30 MB/s | 48 MB 00:01 2024-09-15T03:44:19.711 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-15T03:44:19.731 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-15T03:44:19.731 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-15T03:44:19.785 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:44:19.864 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:44:20.094 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-15T03:44:20.094 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-15T03:44:20.646 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-15T03:44:20.708 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T03:44:20.748 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T03:44:20.776 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:44:21.310 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:44:21.387 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:44:22.005 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:44:22.005 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-15T03:44:22.005 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-15T03:44:22.005 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T03:44:22.308 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T03:44:22.308 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:22.308 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T03:44:22.308 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:22.308 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:22.309 INFO:teuthology.orchestra.run.smithi067.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T03:44:22.309 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T03:44:22.309 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:22.309 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:44:22.588 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph 2024-09-15T03:44:23.175 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:39 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:44:23.321 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:44:23.324 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:44:23.324 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-09-15T03:44:23.324 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:44:23.324 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T03:44:23.324 INFO:teuthology.orchestra.run.smithi067.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-15T03:44:23.324 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-09-15T03:44:23.324 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-15T03:44:23.324 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-15T03:44:23.324 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-15T03:44:23.324 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-15T03:44:23.324 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-15T03:44:23.325 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T03:44:23.325 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T03:44:23.325 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T03:44:23.325 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T03:44:23.325 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T03:44:23.325 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T03:44:23.325 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T03:44:23.325 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections 2024-09-15T03:44:23.325 INFO:teuthology.orchestra.run.smithi067.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-15T03:44:23.325 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T03:44:23.325 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T03:44:23.325 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T03:44:23.325 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T03:44:23.326 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T03:44:23.326 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T03:44:23.326 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T03:44:23.326 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T03:44:23.326 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T03:44:23.326 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T03:44:23.326 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T03:44:23.326 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T03:44:23.326 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T03:44:23.326 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T03:44:23.326 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T03:44:23.326 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T03:44:23.327 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T03:44:23.327 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T03:44:23.327 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T03:44:23.327 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T03:44:23.327 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:23.327 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T03:44:23.327 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:44:23.327 INFO:teuthology.orchestra.run.smithi067.stdout:Install 34 Packages 2024-09-15T03:44:23.327 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:23.328 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 27 M 2024-09-15T03:44:23.328 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 96 M 2024-09-15T03:44:23.328 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T03:44:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:44:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-15T03:44:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-15T03:44:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T03:44:23.923 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T03:44:23.924 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:23.924 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-15T03:44:23.924 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:23.924 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:23.924 INFO:teuthology.orchestra.run.smithi001.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T03:44:23.924 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T03:44:23.924 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:23.924 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:44:24.119 INFO:teuthology.orchestra.run.smithi067.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 23 kB/s | 6.5 kB 00:00 2024-09-15T03:44:24.236 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph 2024-09-15T03:44:24.728 INFO:teuthology.orchestra.run.smithi067.stdout:(2/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 1.6 MB/s | 1.4 MB 00:00 2024-09-15T03:44:24.833 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:39 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:44:24.861 INFO:teuthology.orchestra.run.smithi067.stdout:(3/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 2.0 MB/s | 2.1 MB 00:01 2024-09-15T03:44:24.961 INFO:teuthology.orchestra.run.smithi067.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 2.4 MB/s | 240 kB 00:00 2024-09-15T03:44:24.974 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:44:24.977 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:44:24.977 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-15T03:44:24.977 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:44:24.977 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-15T03:44:24.977 INFO:teuthology.orchestra.run.smithi001.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-15T03:44:24.977 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-15T03:44:24.977 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-15T03:44:24.977 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-15T03:44:24.977 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-15T03:44:24.978 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-15T03:44:24.978 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-15T03:44:24.978 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T03:44:24.978 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T03:44:24.978 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T03:44:24.978 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T03:44:24.978 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T03:44:24.978 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T03:44:24.978 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T03:44:24.978 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections 2024-09-15T03:44:24.978 INFO:teuthology.orchestra.run.smithi001.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-15T03:44:24.978 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T03:44:24.978 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T03:44:24.979 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T03:44:24.979 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T03:44:24.979 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T03:44:24.979 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T03:44:24.979 INFO:teuthology.orchestra.run.smithi001.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T03:44:24.979 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T03:44:24.979 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T03:44:24.979 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T03:44:24.979 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T03:44:24.979 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T03:44:24.979 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T03:44:24.979 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T03:44:24.979 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T03:44:24.980 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T03:44:24.980 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T03:44:24.980 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T03:44:24.980 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T03:44:24.980 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T03:44:24.980 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:24.980 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-15T03:44:24.980 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:44:24.980 INFO:teuthology.orchestra.run.smithi001.stdout:Install 34 Packages 2024-09-15T03:44:24.980 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:24.981 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 27 M 2024-09-15T03:44:24.981 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 96 M 2024-09-15T03:44:24.981 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-15T03:44:25.078 INFO:teuthology.orchestra.run.smithi067.stdout:(5/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-09-15T03:44:25.162 INFO:teuthology.orchestra.run.smithi067.stdout:(6/34): python3-cryptography-36.0.1-4.el9.x86_6 15 MB/s | 1.2 MB 00:00 2024-09-15T03:44:25.190 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:44:25.204 INFO:teuthology.orchestra.run.smithi067.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-15T03:44:25.246 INFO:teuthology.orchestra.run.smithi067.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-15T03:44:25.264 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:44:25.288 INFO:teuthology.orchestra.run.smithi067.stdout:(9/34): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-09-15T03:44:25.330 INFO:teuthology.orchestra.run.smithi067.stdout:(10/34): python3-urllib3-1.26.5-6.el9.noarch.rp 5.1 MB/s | 215 kB 00:00 2024-09-15T03:44:25.539 INFO:teuthology.orchestra.run.smithi067.stdout:(11/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86 2.8 MB/s | 3.9 MB 00:01 2024-09-15T03:44:25.559 INFO:teuthology.orchestra.run.smithi001.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 24 kB/s | 6.5 kB 00:00 2024-09-15T03:44:25.808 INFO:teuthology.orchestra.run.smithi067.stdout:(12/34): python3-markupsafe-1.1.1-12.el9.x86_64 130 kB/s | 35 kB 00:00 2024-09-15T03:44:25.822 INFO:teuthology.orchestra.run.smithi067.stdout:(13/34): python3-mako-1.1.4-6.el9.noarch.rpm 350 kB/s | 172 kB 00:00 2024-09-15T03:44:26.148 INFO:teuthology.orchestra.run.smithi067.stdout:(14/34): python3-packaging-20.9-5.el9.noarch.rp 226 kB/s | 77 kB 00:00 2024-09-15T03:44:26.173 INFO:teuthology.orchestra.run.smithi067.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 145 kB/s | 51 kB 00:00 2024-09-15T03:44:26.215 INFO:teuthology.orchestra.run.smithi067.stdout:(16/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-09-15T03:44:26.240 INFO:teuthology.orchestra.run.smithi067.stdout:(17/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 6.8 MB/s | 172 kB 00:00 2024-09-15T03:44:26.274 INFO:teuthology.orchestra.run.smithi067.stdout:(18/34): python3-cherrypy-18.6.1-2.el9.noarch.r 10 MB/s | 358 kB 00:00 2024-09-15T03:44:26.299 INFO:teuthology.orchestra.run.smithi067.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 428 kB/s | 11 kB 00:00 2024-09-15T03:44:26.324 INFO:teuthology.orchestra.run.smithi067.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 711 kB/s | 18 kB 00:00 2024-09-15T03:44:26.350 INFO:teuthology.orchestra.run.smithi067.stdout:(21/34): python3-jaraco-collections-3.0.0-8.el9 927 kB/s | 23 kB 00:00 2024-09-15T03:44:26.375 INFO:teuthology.orchestra.run.smithi067.stdout:(22/34): python3-jaraco-functools-3.5.0-2.el9.n 783 kB/s | 19 kB 00:00 2024-09-15T03:44:26.400 INFO:teuthology.orchestra.run.smithi067.stdout:(23/34): python3-jaraco-text-3.2.0-6.el9.noarch 785 kB/s | 20 kB 00:00 2024-09-15T03:44:26.580 INFO:teuthology.orchestra.run.smithi067.stdout:(24/34): python3-logutils-0.3.5-21.el9.noarch.r 257 kB/s | 46 kB 00:00 2024-09-15T03:44:26.618 INFO:teuthology.orchestra.run.smithi067.stdout:(25/34): python3-toml-0.10.2-6.el9.noarch.rpm 89 kB/s | 42 kB 00:00 2024-09-15T03:44:26.643 INFO:teuthology.orchestra.run.smithi067.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2024-09-15T03:44:26.668 INFO:teuthology.orchestra.run.smithi001.stdout:(2/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 1.0 MB/s | 1.4 MB 00:01 2024-09-15T03:44:26.668 INFO:teuthology.orchestra.run.smithi067.stdout:(27/34): python3-portend-3.1.0-2.el9.noarch.rpm 960 kB/s | 16 kB 00:00 2024-09-15T03:44:26.693 INFO:teuthology.orchestra.run.smithi067.stdout:(28/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.8 MB/s | 90 kB 00:00 2024-09-15T03:44:26.710 INFO:teuthology.orchestra.run.smithi067.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 2.1 MB/s | 36 kB 00:00 2024-09-15T03:44:26.735 INFO:teuthology.orchestra.run.smithi067.stdout:(30/34): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-09-15T03:44:26.761 INFO:teuthology.orchestra.run.smithi067.stdout:(31/34): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-15T03:44:26.786 INFO:teuthology.orchestra.run.smithi067.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 801 kB/s | 20 kB 00:00 2024-09-15T03:44:26.820 INFO:teuthology.orchestra.run.smithi067.stdout:(33/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-09-15T03:44:26.909 INFO:teuthology.orchestra.run.smithi001.stdout:(3/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 1.3 MB/s | 2.1 MB 00:01 2024-09-15T03:44:27.043 INFO:teuthology.orchestra.run.smithi001.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 1.8 MB/s | 240 kB 00:00 2024-09-15T03:44:27.177 INFO:teuthology.orchestra.run.smithi001.stdout:(5/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 253 kB 00:00 2024-09-15T03:44:27.353 INFO:teuthology.orchestra.run.smithi067.stdout:(34/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86 5.8 MB/s | 15 MB 00:02 2024-09-15T03:44:27.357 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:44:27.358 INFO:teuthology.orchestra.run.smithi067.stdout:Total 6.8 MB/s | 27 MB 00:04 2024-09-15T03:44:27.419 INFO:teuthology.orchestra.run.smithi001.stdout:(6/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 2.1 MB/s | 3.9 MB 00:01 2024-09-15T03:44:27.452 INFO:teuthology.orchestra.run.smithi001.stdout:(7/34): python3-cryptography-36.0.1-4.el9.x86_6 4.5 MB/s | 1.2 MB 00:00 2024-09-15T03:44:27.483 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:44:27.484 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-15T03:44:27.484 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-15T03:44:27.484 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T03:44:27.490 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T03:44:27.494 INFO:teuthology.orchestra.run.smithi001.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-15T03:44:27.530 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T03:44:27.530 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T03:44:27.536 INFO:teuthology.orchestra.run.smithi001.stdout:(9/34): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-09-15T03:44:27.561 INFO:teuthology.orchestra.run.smithi001.stdout:(10/34): python3-ply-3.11-14.el9.noarch.rpm 749 kB/s | 106 kB 00:00 2024-09-15T03:44:27.603 INFO:teuthology.orchestra.run.smithi001.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-09-15T03:44:27.629 INFO:teuthology.orchestra.run.smithi001.stdout:(12/34): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-09-15T03:44:27.654 INFO:teuthology.orchestra.run.smithi001.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 693 kB/s | 35 kB 00:00 2024-09-15T03:44:27.680 INFO:teuthology.orchestra.run.smithi001.stdout:(14/34): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-15T03:44:27.705 INFO:teuthology.orchestra.run.smithi001.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-15T03:44:27.730 INFO:teuthology.orchestra.run.smithi001.stdout:(16/34): python3-toml-0.10.2-6.el9.noarch.rpm 833 kB/s | 42 kB 00:00 2024-09-15T03:44:27.780 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T03:44:27.780 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:27.780 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-15T03:44:27.780 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:27.780 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:27.780 INFO:teuthology.orchestra.run.smithi190.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T03:44:27.780 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T03:44:27.780 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:27.780 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:44:27.781 INFO:teuthology.orchestra.run.smithi001.stdout:(17/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 579 kB/s | 43 kB 00:00 2024-09-15T03:44:27.805 INFO:teuthology.orchestra.run.smithi001.stdout:(18/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-09-15T03:44:27.831 INFO:teuthology.orchestra.run.smithi001.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 427 kB/s | 11 kB 00:00 2024-09-15T03:44:27.857 INFO:teuthology.orchestra.run.smithi001.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 712 kB/s | 18 kB 00:00 2024-09-15T03:44:27.881 INFO:teuthology.orchestra.run.smithi001.stdout:(21/34): python3-jaraco-collections-3.0.0-8.el9 931 kB/s | 23 kB 00:00 2024-09-15T03:44:27.907 INFO:teuthology.orchestra.run.smithi001.stdout:(22/34): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-09-15T03:44:27.929 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T03:44:27.929 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T03:44:27.932 INFO:teuthology.orchestra.run.smithi001.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-15T03:44:27.957 INFO:teuthology.orchestra.run.smithi001.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-15T03:44:27.983 INFO:teuthology.orchestra.run.smithi001.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 920 kB/s | 46 kB 00:00 2024-09-15T03:44:28.008 INFO:teuthology.orchestra.run.smithi001.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-15T03:44:28.033 INFO:teuthology.orchestra.run.smithi001.stdout:(27/34): python3-portend-3.1.0-2.el9.noarch.rpm 661 kB/s | 16 kB 00:00 2024-09-15T03:44:28.059 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph 2024-09-15T03:44:28.060 INFO:teuthology.orchestra.run.smithi001.stdout:(28/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-09-15T03:44:28.083 INFO:teuthology.orchestra.run.smithi001.stdout:(29/34): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-09-15T03:44:28.109 INFO:teuthology.orchestra.run.smithi001.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-09-15T03:44:28.143 INFO:teuthology.orchestra.run.smithi001.stdout:(31/34): python3-webob-1.8.8-2.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-09-15T03:44:28.168 INFO:teuthology.orchestra.run.smithi001.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 807 kB/s | 20 kB 00:00 2024-09-15T03:44:28.202 INFO:teuthology.orchestra.run.smithi001.stdout:(33/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-15T03:44:28.633 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:38 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:44:28.777 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:44:28.779 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:44:28.780 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-15T03:44:28.780 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:44:28.780 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-15T03:44:28.780 INFO:teuthology.orchestra.run.smithi190.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-15T03:44:28.780 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-15T03:44:28.780 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-15T03:44:28.780 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-15T03:44:28.780 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-15T03:44:28.780 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-15T03:44:28.780 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-15T03:44:28.780 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T03:44:28.781 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T03:44:28.781 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T03:44:28.781 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T03:44:28.781 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T03:44:28.781 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T03:44:28.781 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T03:44:28.781 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections 2024-09-15T03:44:28.781 INFO:teuthology.orchestra.run.smithi190.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-15T03:44:28.781 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T03:44:28.781 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T03:44:28.781 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T03:44:28.781 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T03:44:28.782 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T03:44:28.782 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T03:44:28.782 INFO:teuthology.orchestra.run.smithi190.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T03:44:28.782 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T03:44:28.782 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T03:44:28.782 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T03:44:28.782 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T03:44:28.782 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T03:44:28.782 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T03:44:28.782 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T03:44:28.782 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T03:44:28.782 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T03:44:28.782 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T03:44:28.783 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T03:44:28.783 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T03:44:28.783 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T03:44:28.783 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:28.783 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-15T03:44:28.783 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:44:28.783 INFO:teuthology.orchestra.run.smithi190.stdout:Install 34 Packages 2024-09-15T03:44:28.783 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:28.784 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 27 M 2024-09-15T03:44:28.784 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 96 M 2024-09-15T03:44:28.784 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-15T03:44:28.835 INFO:teuthology.orchestra.run.smithi001.stdout:(34/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86 7.1 MB/s | 15 MB 00:02 2024-09-15T03:44:28.840 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:44:28.840 INFO:teuthology.orchestra.run.smithi001.stdout:Total 7.1 MB/s | 27 MB 00:03 2024-09-15T03:44:28.978 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-15T03:44:29.020 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-15T03:44:29.020 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-15T03:44:29.316 INFO:teuthology.orchestra.run.smithi190.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 54 kB/s | 6.5 kB 00:00 2024-09-15T03:44:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T03:44:29.390 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-15T03:44:29.422 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-15T03:44:29.440 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-15T03:44:29.440 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-15T03:44:29.464 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-15T03:44:29.500 INFO:teuthology.orchestra.run.smithi190.stdout:(2/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 4.6 MB/s | 1.4 MB 00:00 2024-09-15T03:44:29.559 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-15T03:44:29.584 INFO:teuthology.orchestra.run.smithi190.stdout:(3/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 5.3 MB/s | 2.1 MB 00:00 2024-09-15T03:44:29.659 INFO:teuthology.orchestra.run.smithi190.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 3.1 MB/s | 240 kB 00:00 2024-09-15T03:44:29.743 INFO:teuthology.orchestra.run.smithi190.stdout:(5/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 9.2 MB/s | 3.9 MB 00:00 2024-09-15T03:44:29.746 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-15T03:44:29.791 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-15T03:44:29.827 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-15T03:44:29.859 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-15T03:44:29.893 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-15T03:44:29.909 INFO:teuthology.orchestra.run.smithi190.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-09-15T03:44:29.927 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-15T03:44:29.960 INFO:teuthology.orchestra.run.smithi190.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-15T03:44:29.968 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-15T03:44:30.011 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-15T03:44:30.045 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-15T03:44:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-15T03:44:30.127 INFO:teuthology.orchestra.run.smithi190.stdout:(8/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86_ 24 MB/s | 15 MB 00:00 2024-09-15T03:44:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:44:30.152 INFO:teuthology.orchestra.run.smithi190.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 704 kB/s | 135 kB 00:00 2024-09-15T03:44:30.186 INFO:teuthology.orchestra.run.smithi190.stdout:(10/34): python3-cryptography-36.0.1-4.el9.x86_ 2.8 MB/s | 1.2 MB 00:00 2024-09-15T03:44:30.211 INFO:teuthology.orchestra.run.smithi190.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-15T03:44:30.211 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-15T03:44:30.328 INFO:teuthology.orchestra.run.smithi190.stdout:(12/34): python3-requests-2.25.1-8.el9.noarch.r 622 kB/s | 125 kB 00:00 2024-09-15T03:44:30.362 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-15T03:44:30.444 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-15T03:44:30.495 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-15T03:44:30.541 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-15T03:44:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-15T03:44:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-15T03:44:30.769 INFO:teuthology.orchestra.run.smithi190.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 62 kB/s | 35 kB 00:00 2024-09-15T03:44:30.858 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-15T03:44:30.862 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-15T03:44:30.912 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-15T03:44:30.930 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-15T03:44:30.936 INFO:teuthology.orchestra.run.smithi190.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 229 kB/s | 172 kB 00:00 2024-09-15T03:44:30.970 INFO:teuthology.orchestra.run.smithi190.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 120 kB/s | 77 kB 00:00 2024-09-15T03:44:30.971 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-15T03:44:31.010 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-15T03:44:31.020 INFO:teuthology.orchestra.run.smithi190.stdout:(16/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 873 kB/s | 43 kB 00:00 2024-09-15T03:44:31.020 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-15T03:44:31.045 INFO:teuthology.orchestra.run.smithi190.stdout:(17/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 6.7 MB/s | 172 kB 00:00 2024-09-15T03:44:31.079 INFO:teuthology.orchestra.run.smithi190.stdout:(18/34): python3-cherrypy-18.6.1-2.el9.noarch.r 10 MB/s | 358 kB 00:00 2024-09-15T03:44:31.105 INFO:teuthology.orchestra.run.smithi190.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 436 kB/s | 11 kB 00:00 2024-09-15T03:44:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-15T03:44:31.118 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-15T03:44:31.132 INFO:teuthology.orchestra.run.smithi190.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 710 kB/s | 18 kB 00:00 2024-09-15T03:44:31.144 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-15T03:44:31.155 INFO:teuthology.orchestra.run.smithi190.stdout:(21/34): python3-jaraco-collections-3.0.0-8.el9 928 kB/s | 23 kB 00:00 2024-09-15T03:44:31.180 INFO:teuthology.orchestra.run.smithi190.stdout:(22/34): python3-jaraco-functools-3.5.0-2.el9.n 778 kB/s | 19 kB 00:00 2024-09-15T03:44:31.214 INFO:teuthology.orchestra.run.smithi190.stdout:(23/34): python3-jaraco-text-3.2.0-6.el9.noarch 751 kB/s | 20 kB 00:00 2024-09-15T03:44:31.239 INFO:teuthology.orchestra.run.smithi190.stdout:(24/34): python3-logutils-0.3.5-21.el9.noarch.r 1.8 MB/s | 46 kB 00:00 2024-09-15T03:44:31.264 INFO:teuthology.orchestra.run.smithi190.stdout:(25/34): python3-more-itertools-8.12.0-2.el9.no 3.1 MB/s | 79 kB 00:00 2024-09-15T03:44:31.267 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-15T03:44:31.298 INFO:teuthology.orchestra.run.smithi190.stdout:(26/34): python3-pecan-1.4.2-3.el9.noarch.rpm 8.2 MB/s | 272 kB 00:00 2024-09-15T03:44:31.312 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-15T03:44:31.315 INFO:teuthology.orchestra.run.smithi190.stdout:(27/34): python3-portend-3.1.0-2.el9.noarch.rpm 979 kB/s | 16 kB 00:00 2024-09-15T03:44:31.340 INFO:teuthology.orchestra.run.smithi190.stdout:(28/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-09-15T03:44:31.357 INFO:teuthology.orchestra.run.smithi190.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 2.1 MB/s | 36 kB 00:00 2024-09-15T03:44:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-15T03:44:31.391 INFO:teuthology.orchestra.run.smithi190.stdout:(30/34): python3-webob-1.8.8-2.el9.noarch.rpm 6.8 MB/s | 230 kB 00:00 2024-09-15T03:44:31.391 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-15T03:44:31.425 INFO:teuthology.orchestra.run.smithi190.stdout:(31/34): python3-pytz-2021.1-5.el9.noarch.rpm 78 kB/s | 51 kB 00:00 2024-09-15T03:44:31.425 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-15T03:44:31.429 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-15T03:44:31.458 INFO:teuthology.orchestra.run.smithi190.stdout:(32/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 6.3 MB/s | 427 kB 00:00 2024-09-15T03:44:31.459 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-15T03:44:31.462 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:44:31.483 INFO:teuthology.orchestra.run.smithi190.stdout:(33/34): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2024-09-15T03:44:31.490 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:44:31.490 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:44:31.490 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T03:44:31.491 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:44:31.491 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:44:31.491 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:31.493 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-15T03:44:31.534 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-15T03:44:31.542 INFO:teuthology.orchestra.run.smithi190.stdout:(34/34): python3-toml-0.10.2-6.el9.noarch.rpm 69 kB/s | 42 kB 00:00 2024-09-15T03:44:31.546 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:44:31.547 INFO:teuthology.orchestra.run.smithi190.stdout:Total 9.8 MB/s | 27 MB 00:02 2024-09-15T03:44:31.577 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-15T03:44:31.611 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-15T03:44:31.669 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-15T03:44:31.680 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-15T03:44:31.717 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-15T03:44:31.717 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-15T03:44:31.717 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:44:31.794 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-15T03:44:31.956 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-15T03:44:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-15T03:44:32.091 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-15T03:44:32.092 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-15T03:44:32.104 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-15T03:44:32.141 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-15T03:44:32.200 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-15T03:44:32.264 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-15T03:44:32.464 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-15T03:44:32.512 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-15T03:44:32.595 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-15T03:44:32.693 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-15T03:44:32.730 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-15T03:44:32.896 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:44:32.925 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:44:32.925 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:44:32.925 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T03:44:32.925 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:44:32.926 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:44:32.926 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:33.020 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-15T03:44:33.212 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-15T03:44:33.237 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:44:33.264 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:44:33.264 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:44:33.264 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T03:44:33.264 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:44:33.264 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:44:33.264 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:33.321 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:44:33.330 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-15T03:44:33.346 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:44:33.346 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:44:33.346 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T03:44:33.347 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:44:33.347 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:44:33.347 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:33.395 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-15T03:44:33.427 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-15T03:44:33.460 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-15T03:44:33.551 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-15T03:44:33.556 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:44:33.581 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:44:33.581 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:44:33.581 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T03:44:33.581 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:44:33.581 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:44:33.581 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:33.631 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-15T03:44:33.646 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:44:33.672 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-15T03:44:33.706 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-15T03:44:33.736 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-15T03:44:33.771 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-15T03:44:33.807 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-15T03:44:33.848 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-15T03:44:33.891 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-15T03:44:33.926 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-15T03:44:33.973 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-15T03:44:34.006 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:44:34.080 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-15T03:44:34.243 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-15T03:44:34.323 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-15T03:44:34.376 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-15T03:44:34.414 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-15T03:44:34.464 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-15T03:44:34.518 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-15T03:44:34.688 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:44:34.714 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-15T03:44:34.717 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:44:34.717 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:44:34.717 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T03:44:34.717 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:44:34.717 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:44:34.717 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:34.769 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-15T03:44:34.846 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-15T03:44:34.935 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-15T03:44:34.976 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-15T03:44:35.091 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-15T03:44:35.124 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:44:35.148 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:44:35.148 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:44:35.148 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T03:44:35.148 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:44:35.148 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:44:35.148 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:35.247 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-15T03:44:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:44:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-15T03:44:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-15T03:44:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-15T03:44:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-15T03:44:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-15T03:44:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-15T03:44:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-15T03:44:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-15T03:44:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-15T03:44:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-15T03:44:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-15T03:44:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-15T03:44:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-15T03:44:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-15T03:44:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:44:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-15T03:44:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-15T03:44:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-15T03:44:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-15T03:44:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-15T03:44:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-15T03:44:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-15T03:44:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-15T03:44:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-15T03:44:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-15T03:44:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-15T03:44:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-15T03:44:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-15T03:44:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-15T03:44:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-15T03:44:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-15T03:44:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-15T03:44:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-15T03:44:35.278 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:44:35.304 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:44:35.304 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:44:35.304 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T03:44:35.304 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:44:35.305 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:44:35.305 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:35.365 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:44:35.391 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:44:35.392 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:44:35.392 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T03:44:35.392 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:44:35.392 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:44:35.392 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:35.457 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:44:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-15T03:44:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:36.241 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T03:44:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:44:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T03:44:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T03:44:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T03:44:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T03:44:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T03:44:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T03:44:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T03:44:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T03:44:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T03:44:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T03:44:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T03:44:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T03:44:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T03:44:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T03:44:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T03:44:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T03:44:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T03:44:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T03:44:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T03:44:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T03:44:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T03:44:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T03:44:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T03:44:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T03:44:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T03:44:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T03:44:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T03:44:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T03:44:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:36.244 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:44:36.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-base 2024-09-15T03:44:36.696 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:44:36.725 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:44:36.725 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:44:36.725 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T03:44:36.725 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:44:36.726 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:44:36.726 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:37.005 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:44:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-15T03:44:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-15T03:44:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-15T03:44:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-15T03:44:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-15T03:44:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-15T03:44:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-15T03:44:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-15T03:44:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-15T03:44:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-15T03:44:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-15T03:44:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-15T03:44:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-15T03:44:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-15T03:44:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:44:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-15T03:44:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-15T03:44:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-15T03:44:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-15T03:44:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-15T03:44:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-15T03:44:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-15T03:44:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-15T03:44:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-15T03:44:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-15T03:44:37.008 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-15T03:44:37.008 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-15T03:44:37.008 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-15T03:44:37.008 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-15T03:44:37.008 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-15T03:44:37.008 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-15T03:44:37.009 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-15T03:44:37.009 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-15T03:44:37.100 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:44:37.124 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:44:37.125 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:44:37.125 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T03:44:37.125 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:44:37.125 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:44:37.125 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:37.150 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:53 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:44:37.243 INFO:teuthology.orchestra.run.smithi067.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:44:37.288 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:44:37.289 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-09-15T03:44:37.289 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:44:37.325 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:44:37.349 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:44:37.349 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:44:37.349 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T03:44:37.349 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:44:37.350 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:44:37.350 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:37.351 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install cephadm 2024-09-15T03:44:37.413 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:44:37.937 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:53 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:44:38.043 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-15T03:44:38.044 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:38.044 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-15T03:44:38.044 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:38.044 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:38.044 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:38.044 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:44:38.044 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:38.044 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:38.044 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T03:44:38.044 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T03:44:38.044 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T03:44:38.045 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T03:44:38.045 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T03:44:38.045 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T03:44:38.045 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T03:44:38.045 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T03:44:38.045 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T03:44:38.045 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T03:44:38.045 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T03:44:38.045 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T03:44:38.045 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T03:44:38.045 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T03:44:38.045 INFO:teuthology.orchestra.run.smithi001.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T03:44:38.045 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T03:44:38.046 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T03:44:38.046 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T03:44:38.046 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T03:44:38.046 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T03:44:38.046 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T03:44:38.046 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T03:44:38.046 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T03:44:38.046 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T03:44:38.046 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T03:44:38.046 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T03:44:38.046 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T03:44:38.046 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T03:44:38.046 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:38.047 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:44:38.074 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:44:38.075 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:44:38.075 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-09-15T03:44:38.075 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:44:38.075 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T03:44:38.075 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-15T03:44:38.075 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:38.075 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T03:44:38.075 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:44:38.075 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-09-15T03:44:38.075 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:38.075 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 83 k 2024-09-15T03:44:38.076 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 375 k 2024-09-15T03:44:38.076 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T03:44:38.270 INFO:teuthology.orchestra.run.smithi067.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 426 kB/s | 83 kB 00:00 2024-09-15T03:44:38.271 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:44:38.271 INFO:teuthology.orchestra.run.smithi067.stdout:Total 422 kB/s | 83 kB 00:00 2024-09-15T03:44:38.271 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T03:44:38.277 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T03:44:38.277 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T03:44:38.278 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T03:44:38.279 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T03:44:38.358 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T03:44:38.396 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-base 2024-09-15T03:44:38.755 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:44:38.851 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:44:38.991 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:53 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:44:39.084 INFO:teuthology.orchestra.run.smithi001.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:44:39.111 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:44:39.111 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-15T03:44:39.111 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-15T03:44:39.111 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-15T03:44:39.111 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-15T03:44:39.111 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-15T03:44:39.111 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-15T03:44:39.112 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-15T03:44:39.112 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-15T03:44:39.112 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-15T03:44:39.112 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-15T03:44:39.112 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-15T03:44:39.112 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-15T03:44:39.112 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-15T03:44:39.112 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-15T03:44:39.112 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:44:39.112 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-15T03:44:39.112 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-15T03:44:39.112 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-15T03:44:39.112 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-15T03:44:39.113 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-15T03:44:39.113 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-15T03:44:39.113 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-15T03:44:39.113 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-15T03:44:39.113 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-15T03:44:39.113 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-15T03:44:39.113 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-15T03:44:39.113 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-15T03:44:39.117 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-15T03:44:39.117 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-15T03:44:39.117 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-15T03:44:39.117 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-15T03:44:39.117 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-15T03:44:39.117 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-15T03:44:39.128 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:44:39.129 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-15T03:44:39.129 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:44:39.181 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:44:39.193 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install cephadm 2024-09-15T03:44:39.431 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:44:39.432 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:39.432 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T03:44:39.432 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:44:39.432 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:39.432 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:44:39.644 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T03:44:39.779 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:54 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:44:39.915 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:44:39.916 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:44:39.916 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-15T03:44:39.916 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:44:39.916 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-15T03:44:39.916 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-15T03:44:39.917 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:39.917 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-15T03:44:39.917 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:44:39.917 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-15T03:44:39.917 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:39.917 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 83 k 2024-09-15T03:44:39.917 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 375 k 2024-09-15T03:44:39.917 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-15T03:44:40.123 INFO:teuthology.orchestra.run.smithi001.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 401 kB/s | 83 kB 00:00 2024-09-15T03:44:40.124 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:44:40.124 INFO:teuthology.orchestra.run.smithi001.stdout:Total 398 kB/s | 83 kB 00:00 2024-09-15T03:44:40.124 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-15T03:44:40.130 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-15T03:44:40.130 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-15T03:44:40.132 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-15T03:44:40.132 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-15T03:44:40.202 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-15T03:44:40.233 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-15T03:44:40.233 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:40.233 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-15T03:44:40.233 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:40.233 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:40.233 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:40.233 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:44:40.233 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:40.234 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:40.234 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T03:44:40.234 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T03:44:40.234 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T03:44:40.234 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T03:44:40.234 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T03:44:40.234 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T03:44:40.234 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T03:44:40.234 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T03:44:40.234 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T03:44:40.234 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T03:44:40.234 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T03:44:40.234 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T03:44:40.235 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T03:44:40.235 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T03:44:40.235 INFO:teuthology.orchestra.run.smithi190.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T03:44:40.235 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T03:44:40.235 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T03:44:40.235 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T03:44:40.235 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T03:44:40.235 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T03:44:40.235 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T03:44:40.235 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T03:44:40.235 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T03:44:40.235 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T03:44:40.235 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T03:44:40.236 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T03:44:40.236 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T03:44:40.236 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T03:44:40.236 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:40.236 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:44:40.236 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:56 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:44:40.371 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:44:40.372 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:44:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-09-15T03:44:40.372 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:44:40.372 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T03:44:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-15T03:44:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:40.372 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T03:44:40.372 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:44:40.372 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-09-15T03:44:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:40.373 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 137 k 2024-09-15T03:44:40.373 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 379 k 2024-09-15T03:44:40.373 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T03:44:40.584 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 651 kB/s | 137 kB 00:00 2024-09-15T03:44:40.585 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:44:40.585 INFO:teuthology.orchestra.run.smithi067.stdout:Total 646 kB/s | 137 kB 00:00 2024-09-15T03:44:40.585 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T03:44:40.593 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T03:44:40.593 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T03:44:40.615 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-base 2024-09-15T03:44:40.625 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:44:40.653 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T03:44:40.653 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T03:44:40.738 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:44:40.799 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T03:44:40.860 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:44:40.887 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:44:40.887 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:44:40.887 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T03:44:40.887 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:41.071 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:44:41.210 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:51 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:44:41.301 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:44:41.344 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:44:41.345 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-15T03:44:41.345 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:44:41.350 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:44:41.351 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:41.351 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-15T03:44:41.351 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:44:41.351 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:41.351 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:44:41.414 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install cephadm 2024-09-15T03:44:41.536 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T03:44:41.800 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:44:41.800 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:41.801 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T03:44:41.801 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:41.801 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:41.801 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:44:41.980 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:51 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:44:42.045 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr 2024-09-15T03:44:42.112 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:44:42.113 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:44:42.113 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-15T03:44:42.113 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:44:42.113 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-15T03:44:42.113 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-15T03:44:42.113 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:42.113 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-15T03:44:42.114 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:44:42.114 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-15T03:44:42.114 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:42.114 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 83 k 2024-09-15T03:44:42.114 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 375 k 2024-09-15T03:44:42.114 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-15T03:44:42.124 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:57 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:44:42.261 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:44:42.262 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:44:42.262 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-09-15T03:44:42.262 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:44:42.262 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-15T03:44:42.263 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-15T03:44:42.263 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:42.263 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-15T03:44:42.263 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:44:42.263 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-15T03:44:42.264 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:42.264 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 137 k 2024-09-15T03:44:42.264 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 379 k 2024-09-15T03:44:42.264 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-15T03:44:42.308 INFO:teuthology.orchestra.run.smithi190.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 427 kB/s | 83 kB 00:00 2024-09-15T03:44:42.308 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:44:42.308 INFO:teuthology.orchestra.run.smithi190.stdout:Total 424 kB/s | 83 kB 00:00 2024-09-15T03:44:42.308 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-15T03:44:42.314 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-15T03:44:42.314 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-15T03:44:42.315 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-15T03:44:42.316 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-15T03:44:42.394 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-15T03:44:42.471 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 661 kB/s | 137 kB 00:00 2024-09-15T03:44:42.471 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:44:42.472 INFO:teuthology.orchestra.run.smithi001.stdout:Total 657 kB/s | 137 kB 00:00 2024-09-15T03:44:42.472 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-15T03:44:42.480 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-15T03:44:42.481 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-15T03:44:42.541 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-15T03:44:42.541 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-15T03:44:42.635 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:58 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:44:42.687 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-15T03:44:42.729 INFO:teuthology.orchestra.run.smithi067.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:44:42.755 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:44:42.773 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:44:42.774 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-09-15T03:44:42.774 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:44:42.782 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:44:42.782 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:44:42.782 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T03:44:42.783 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:42.843 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T03:44:42.849 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:44:42.936 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:44:43.295 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:44:43.430 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:59 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:44:43.534 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:44:43.534 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:43.534 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-15T03:44:43.534 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:44:43.534 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:43.535 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:44:43.569 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:44:43.570 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:44:43.570 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-09-15T03:44:43.570 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:44:43.571 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T03:44:43.571 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-15T03:44:43.571 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-09-15T03:44:43.571 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-15T03:44:43.571 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-15T03:44:43.571 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-15T03:44:43.571 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T03:44:43.571 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T03:44:43.571 INFO:teuthology.orchestra.run.smithi067.stdout:Installing weak dependencies: 2024-09-15T03:44:43.571 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-15T03:44:43.571 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:43.571 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T03:44:43.571 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:44:43.572 INFO:teuthology.orchestra.run.smithi067.stdout:Install 7 Packages 2024-09-15T03:44:43.572 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:43.572 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 3.7 M 2024-09-15T03:44:43.572 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 80 M 2024-09-15T03:44:43.572 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T03:44:43.690 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:44:43.690 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:43.690 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-15T03:44:43.691 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:43.691 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:43.691 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:44:43.778 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T03:44:43.927 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr 2024-09-15T03:44:44.012 INFO:teuthology.orchestra.run.smithi067.stdout:(1/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 93 kB/s | 15 kB 00:00 2024-09-15T03:44:44.037 INFO:teuthology.orchestra.run.smithi067.stdout:(2/7): ceph-grafana-dashboards-17.2.7-1643.gd75 133 kB/s | 24 kB 00:00 2024-09-15T03:44:44.079 INFO:teuthology.orchestra.run.smithi067.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-15T03:44:44.104 INFO:teuthology.orchestra.run.smithi067.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-15T03:44:44.129 INFO:teuthology.orchestra.run.smithi067.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 444 kB/s | 41 kB 00:00 2024-09-15T03:44:44.154 INFO:teuthology.orchestra.run.smithi067.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-15T03:44:44.230 INFO:teuthology.orchestra.run.smithi067.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 8.9 MB/s | 3.4 MB 00:00 2024-09-15T03:44:44.233 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:44:44.233 INFO:teuthology.orchestra.run.smithi067.stdout:Total 5.5 MB/s | 3.7 MB 00:00 2024-09-15T03:44:44.234 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T03:44:44.243 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T03:44:44.243 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T03:44:44.333 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T03:44:44.333 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T03:44:44.360 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:54 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:44:44.495 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:44:44.496 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:44:44.496 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-09-15T03:44:44.496 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:44:44.497 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-15T03:44:44.497 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-15T03:44:44.497 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:44.497 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-15T03:44:44.497 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:44:44.497 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-15T03:44:44.497 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:44.497 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 137 k 2024-09-15T03:44:44.497 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 379 k 2024-09-15T03:44:44.497 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-15T03:44:44.518 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:59 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:44:44.611 INFO:teuthology.orchestra.run.smithi001.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:44:44.651 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T03:44:44.657 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:44:44.658 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-15T03:44:44.658 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:44:44.699 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 682 kB/s | 137 kB 00:00 2024-09-15T03:44:44.699 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:44:44.700 INFO:teuthology.orchestra.run.smithi190.stdout:Total 677 kB/s | 137 kB 00:00 2024-09-15T03:44:44.700 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-15T03:44:44.707 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-15T03:44:44.707 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-15T03:44:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-15T03:44:44.729 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T03:44:44.751 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-15T03:44:44.766 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-15T03:44:44.767 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-15T03:44:44.799 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-15T03:44:44.829 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-15T03:44:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-15T03:44:44.907 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-15T03:44:44.975 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:44:45.000 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:44:45.000 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:44:45.001 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T03:44:45.001 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:45.316 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:00 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:44:45.452 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:44:45.453 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:44:45.453 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-15T03:44:45.453 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:44:45.453 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-15T03:44:45.454 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-15T03:44:45.454 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-15T03:44:45.454 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-15T03:44:45.454 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-15T03:44:45.454 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-15T03:44:45.454 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T03:44:45.454 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T03:44:45.454 INFO:teuthology.orchestra.run.smithi001.stdout:Installing weak dependencies: 2024-09-15T03:44:45.454 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-15T03:44:45.454 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:45.454 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-15T03:44:45.455 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:44:45.455 INFO:teuthology.orchestra.run.smithi001.stdout:Install 7 Packages 2024-09-15T03:44:45.455 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:45.455 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 3.7 M 2024-09-15T03:44:45.455 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 80 M 2024-09-15T03:44:45.455 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-15T03:44:45.768 INFO:teuthology.orchestra.run.smithi001.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 162 kB/s | 24 kB 00:00 2024-09-15T03:44:45.794 INFO:teuthology.orchestra.run.smithi001.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 83 kB/s | 15 kB 00:00 2024-09-15T03:44:45.836 INFO:teuthology.orchestra.run.smithi001.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-15T03:44:45.861 INFO:teuthology.orchestra.run.smithi001.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-15T03:44:45.886 INFO:teuthology.orchestra.run.smithi001.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 444 kB/s | 41 kB 00:00 2024-09-15T03:44:45.887 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:44:45.887 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:45.887 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-15T03:44:45.888 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:44:45.888 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:45.888 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:44:45.911 INFO:teuthology.orchestra.run.smithi001.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-15T03:44:45.922 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-15T03:44:45.953 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:44:46.036 INFO:teuthology.orchestra.run.smithi001.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 8.0 MB/s | 3.4 MB 00:00 2024-09-15T03:44:46.040 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:44:46.040 INFO:teuthology.orchestra.run.smithi001.stdout:Total 6.2 MB/s | 3.7 MB 00:00 2024-09-15T03:44:46.040 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-15T03:44:46.049 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-15T03:44:46.049 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-15T03:44:46.087 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr 2024-09-15T03:44:46.142 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-15T03:44:46.143 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-15T03:44:46.453 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-15T03:44:46.521 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-15T03:44:46.556 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-15T03:44:46.612 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-15T03:44:46.642 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-15T03:44:46.676 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:56 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:44:46.680 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-15T03:44:46.765 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:44:46.808 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:44:46.809 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-15T03:44:46.809 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:44:46.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T03:44:46.966 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:44:46.967 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-15T03:44:46.967 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-15T03:44:46.967 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-15T03:44:46.967 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-15T03:44:46.967 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-15T03:44:46.967 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-15T03:44:47.258 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-15T03:44:47.259 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:47.259 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T03:44:47.259 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:44:47.259 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:44:47.259 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:44:47.259 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-15T03:44:47.259 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-15T03:44:47.259 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T03:44:47.259 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T03:44:47.259 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:47.259 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:44:47.436 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:57 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:44:47.454 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T03:44:47.570 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:44:47.571 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:44:47.571 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-15T03:44:47.571 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:44:47.571 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-15T03:44:47.571 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-15T03:44:47.571 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-15T03:44:47.571 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-15T03:44:47.571 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-15T03:44:47.572 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-15T03:44:47.572 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T03:44:47.572 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T03:44:47.572 INFO:teuthology.orchestra.run.smithi190.stdout:Installing weak dependencies: 2024-09-15T03:44:47.572 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-15T03:44:47.572 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:47.572 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-15T03:44:47.572 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:44:47.572 INFO:teuthology.orchestra.run.smithi190.stdout:Install 7 Packages 2024-09-15T03:44:47.572 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:47.573 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 3.7 M 2024-09-15T03:44:47.573 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 80 M 2024-09-15T03:44:47.573 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-15T03:44:47.735 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-15T03:44:47.766 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:44:48.038 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:04 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:44:48.087 INFO:teuthology.orchestra.run.smithi190.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 165 kB/s | 24 kB 00:00 2024-09-15T03:44:48.104 INFO:teuthology.orchestra.run.smithi190.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 88 kB/s | 15 kB 00:00 2024-09-15T03:44:48.138 INFO:teuthology.orchestra.run.smithi190.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 181 kB/s | 9.0 kB 00:00 2024-09-15T03:44:48.154 INFO:teuthology.orchestra.run.smithi190.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 2.0 MB/s | 31 kB 00:00 2024-09-15T03:44:48.171 INFO:teuthology.orchestra.run.smithi190.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 608 kB/s | 41 kB 00:00 2024-09-15T03:44:48.181 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:44:48.181 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================= 2024-09-15T03:44:48.182 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-09-15T03:44:48.182 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================= 2024-09-15T03:44:48.182 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T03:44:48.182 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-15T03:44:48.182 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-09-15T03:44:48.182 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T03:44:48.182 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T03:44:48.182 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T03:44:48.182 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T03:44:48.182 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T03:44:48.182 INFO:teuthology.orchestra.run.smithi067.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T03:44:48.182 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T03:44:48.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T03:44:48.183 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T03:44:48.183 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T03:44:48.183 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T03:44:48.183 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:48.183 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T03:44:48.183 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================= 2024-09-15T03:44:48.183 INFO:teuthology.orchestra.run.smithi067.stdout:Install 12 Packages 2024-09-15T03:44:48.183 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:44:48.183 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 38 M 2024-09-15T03:44:48.183 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 212 M 2024-09-15T03:44:48.183 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T03:44:48.205 INFO:teuthology.orchestra.run.smithi190.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-15T03:44:48.305 INFO:teuthology.orchestra.run.smithi190.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 9.2 MB/s | 3.4 MB 00:00 2024-09-15T03:44:48.308 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:44:48.309 INFO:teuthology.orchestra.run.smithi190.stdout:Total 5.0 MB/s | 3.7 MB 00:00 2024-09-15T03:44:48.309 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-15T03:44:48.318 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-15T03:44:48.318 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-15T03:44:48.407 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-15T03:44:48.408 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-15T03:44:48.698 INFO:teuthology.orchestra.run.smithi067.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 796 kB/s | 191 kB 00:00 2024-09-15T03:44:48.739 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-15T03:44:48.746 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:44:48.746 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-15T03:44:48.746 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-15T03:44:48.746 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-15T03:44:48.746 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-15T03:44:48.746 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-15T03:44:48.746 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-15T03:44:48.756 INFO:teuthology.orchestra.run.smithi067.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.6 MB/s | 801 kB 00:00 2024-09-15T03:44:48.798 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-15T03:44:48.806 INFO:teuthology.orchestra.run.smithi067.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 301 kB/s | 33 kB 00:00 2024-09-15T03:44:48.832 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-15T03:44:48.857 INFO:teuthology.orchestra.run.smithi067.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 365 kB/s | 18 kB 00:00 2024-09-15T03:44:48.886 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-15T03:44:48.918 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-15T03:44:48.948 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-15T03:44:48.974 INFO:teuthology.orchestra.run.smithi067.stdout:(5/12): ceph-mgr-diskprediction-local-17.2.7-16 14 MB/s | 7.4 MB 00:00 2024-09-15T03:44:48.999 INFO:teuthology.orchestra.run.smithi067.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 270 kB/s | 38 kB 00:00 2024-09-15T03:44:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-15T03:44:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:49.042 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-15T03:44:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:44:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:44:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:44:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-15T03:44:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-15T03:44:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T03:44:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T03:44:49.043 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:49.043 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:44:49.082 INFO:teuthology.orchestra.run.smithi067.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-09-15T03:44:49.268 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T03:44:49.316 INFO:teuthology.orchestra.run.smithi067.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.3 MB/s | 3.0 MB 00:00 2024-09-15T03:44:49.376 INFO:teuthology.orchestra.run.smithi067.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.2 MB/s | 250 kB 00:00 2024-09-15T03:44:49.608 INFO:teuthology.orchestra.run.smithi067.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.7 MB/s | 5.1 MB 00:00 2024-09-15T03:44:49.859 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:04 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:44:49.937 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-15T03:44:49.985 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:44:50.000 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:44:50.001 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-09-15T03:44:50.001 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-15T03:44:50.001 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-09-15T03:44:50.001 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-15T03:44:50.001 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-15T03:44:50.001 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-15T03:44:50.001 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T03:44:50.002 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T03:44:50.002 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T03:44:50.002 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T03:44:50.002 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T03:44:50.002 INFO:teuthology.orchestra.run.smithi001.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T03:44:50.002 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T03:44:50.002 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T03:44:50.002 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T03:44:50.002 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T03:44:50.002 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T03:44:50.002 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:50.002 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-15T03:44:50.002 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-09-15T03:44:50.002 INFO:teuthology.orchestra.run.smithi001.stdout:Install 12 Packages 2024-09-15T03:44:50.003 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:50.003 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 38 M 2024-09-15T03:44:50.003 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 212 M 2024-09-15T03:44:50.003 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-15T03:44:50.092 INFO:teuthology.orchestra.run.smithi067.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.4 MB/s | 4.9 MB 00:01 2024-09-15T03:44:50.487 INFO:teuthology.orchestra.run.smithi001.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.2 MB/s | 191 kB 00:00 2024-09-15T03:44:50.528 INFO:teuthology.orchestra.run.smithi001.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.1 MB/s | 801 kB 00:00 2024-09-15T03:44:50.587 INFO:teuthology.orchestra.run.smithi001.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 327 kB/s | 33 kB 00:00 2024-09-15T03:44:50.629 INFO:teuthology.orchestra.run.smithi001.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 436 kB/s | 18 kB 00:00 2024-09-15T03:44:50.687 INFO:teuthology.orchestra.run.smithi001.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 662 kB/s | 38 kB 00:00 2024-09-15T03:44:50.754 INFO:teuthology.orchestra.run.smithi001.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-09-15T03:44:50.863 INFO:teuthology.orchestra.run.smithi001.stdout:(7/12): ceph-mgr-diskprediction-local-17.2.7-16 14 MB/s | 7.4 MB 00:00 2024-09-15T03:44:50.888 INFO:teuthology.orchestra.run.smithi001.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.8 MB/s | 245 kB 00:00 2024-09-15T03:44:50.930 INFO:teuthology.orchestra.run.smithi001.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.9 MB/s | 250 kB 00:00 2024-09-15T03:44:51.030 INFO:teuthology.orchestra.run.smithi001.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 14 MB/s | 4.9 MB 00:00 2024-09-15T03:44:51.035 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:44:51.035 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-15T03:44:51.035 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-15T03:44:51.035 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-15T03:44:51.035 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-15T03:44:51.035 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-15T03:44:51.035 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-15T03:44:51.272 INFO:teuthology.orchestra.run.smithi001.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-09-15T03:44:51.302 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-15T03:44:51.302 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:51.302 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-15T03:44:51.302 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:44:51.302 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:44:51.303 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:44:51.303 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-15T03:44:51.303 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-15T03:44:51.303 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T03:44:51.303 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T03:44:51.303 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:51.303 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:44:51.339 INFO:teuthology.orchestra.run.smithi001.stdout:(12/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-09-15T03:44:51.340 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:44:51.340 INFO:teuthology.orchestra.run.smithi001.stdout:Total 29 MB/s | 38 MB 00:01 2024-09-15T03:44:51.526 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T03:44:51.710 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-15T03:44:51.737 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-15T03:44:51.737 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-15T03:44:52.092 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:02 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:44:52.227 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:44:52.228 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================= 2024-09-15T03:44:52.228 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-15T03:44:52.228 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================= 2024-09-15T03:44:52.229 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-15T03:44:52.229 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-15T03:44:52.229 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-15T03:44:52.229 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T03:44:52.229 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T03:44:52.229 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T03:44:52.229 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T03:44:52.229 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T03:44:52.230 INFO:teuthology.orchestra.run.smithi190.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T03:44:52.230 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T03:44:52.230 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T03:44:52.230 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T03:44:52.230 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T03:44:52.230 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T03:44:52.231 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:52.231 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-15T03:44:52.231 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================= 2024-09-15T03:44:52.231 INFO:teuthology.orchestra.run.smithi190.stdout:Install 12 Packages 2024-09-15T03:44:52.231 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:44:52.232 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 38 M 2024-09-15T03:44:52.232 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 212 M 2024-09-15T03:44:52.232 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-15T03:44:52.256 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-15T03:44:52.256 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-15T03:44:52.946 INFO:teuthology.orchestra.run.smithi190.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-15T03:44:52.980 INFO:teuthology.orchestra.run.smithi190.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.5 MB/s | 801 kB 00:00 2024-09-15T03:44:53.049 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-15T03:44:53.080 INFO:teuthology.orchestra.run.smithi190.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 245 kB/s | 33 kB 00:00 2024-09-15T03:44:53.119 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T03:44:53.139 INFO:teuthology.orchestra.run.smithi190.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 311 kB/s | 18 kB 00:00 2024-09-15T03:44:53.167 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:44:53.189 INFO:teuthology.orchestra.run.smithi190.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 771 kB/s | 38 kB 00:00 2024-09-15T03:44:53.274 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:44:53.281 INFO:teuthology.orchestra.run.smithi190.stdout:(6/12): ceph-mgr-diskprediction-local-17.2.7-16 14 MB/s | 7.4 MB 00:00 2024-09-15T03:44:53.314 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T03:44:53.407 INFO:teuthology.orchestra.run.smithi190.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.0 MB/s | 3.0 MB 00:00 2024-09-15T03:44:53.450 INFO:teuthology.orchestra.run.smithi067.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.0 MB/s | 16 MB 00:04 2024-09-15T03:44:53.451 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:44:53.451 INFO:teuthology.orchestra.run.smithi067.stdout:Total 7.2 MB/s | 38 MB 00:05 2024-09-15T03:44:53.508 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T03:44:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T03:44:53.557 INFO:teuthology.orchestra.run.smithi190.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 13 MB/s | 4.9 MB 00:00 2024-09-15T03:44:53.582 INFO:teuthology.orchestra.run.smithi190.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 814 kB/s | 245 kB 00:00 2024-09-15T03:44:53.616 INFO:teuthology.orchestra.run.smithi190.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 4.2 MB/s | 250 kB 00:00 2024-09-15T03:44:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T03:44:53.691 INFO:teuthology.orchestra.run.smithi190.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 5.1 MB 00:00 2024-09-15T03:44:53.820 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T03:44:53.847 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T03:44:53.847 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T03:44:54.175 INFO:teuthology.orchestra.run.smithi190.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-09-15T03:44:54.175 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:44:54.176 INFO:teuthology.orchestra.run.smithi190.stdout:Total 20 MB/s | 38 MB 00:01 2024-09-15T03:44:54.323 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T03:44:54.372 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T03:44:54.372 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T03:44:54.436 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T03:44:54.531 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-15T03:44:54.557 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-15T03:44:54.557 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-15T03:44:55.053 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-15T03:44:55.054 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-15T03:44:55.153 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T03:44:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T03:44:55.264 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:44:55.370 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:44:55.408 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T03:44:55.601 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T03:44:55.632 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T03:44:55.719 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T03:44:55.763 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:44:55.809 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-15T03:44:55.876 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T03:44:55.916 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:44:56.015 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:44:56.049 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T03:44:56.233 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T03:44:56.265 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T03:44:56.352 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T03:44:56.415 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T03:44:56.533 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T03:44:56.849 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T03:44:56.878 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:44:57.008 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T03:44:57.127 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T03:44:57.859 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:44:58.373 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:44:58.934 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T03:44:58.963 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:44:59.413 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:44:59.413 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-15T03:44:59.413 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:44:59.413 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:44:59.413 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T03:44:59.413 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T03:44:59.413 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T03:44:59.413 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T03:44:59.413 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T03:44:59.414 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T03:44:59.414 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:44:59.414 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T03:44:59.418 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T03:44:59.448 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:44:59.789 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T03:44:59.789 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:59.789 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-15T03:44:59.790 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:44:59.790 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T03:44:59.790 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T03:44:59.790 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T03:44:59.790 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T03:44:59.790 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T03:44:59.790 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T03:44:59.790 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T03:44:59.790 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T03:44:59.790 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T03:44:59.790 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T03:44:59.790 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T03:44:59.790 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:44:59.791 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:45:00.161 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-rook 2024-09-15T03:45:00.754 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:15 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:45:00.893 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:45:00.894 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:00.894 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-15T03:45:00.894 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:00.894 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-15T03:45:00.894 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-15T03:45:00.894 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-15T03:45:00.894 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T03:45:00.894 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T03:45:00.895 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T03:45:00.895 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T03:45:00.895 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T03:45:00.895 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T03:45:00.895 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T03:45:00.895 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T03:45:00.895 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T03:45:00.895 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T03:45:00.895 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T03:45:00.895 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T03:45:00.895 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:00.895 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-15T03:45:00.896 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:00.896 INFO:teuthology.orchestra.run.smithi001.stdout:Install 13 Packages 2024-09-15T03:45:00.896 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:00.896 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 2.2 M 2024-09-15T03:45:00.896 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 26 M 2024-09-15T03:45:00.896 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-15T03:45:01.368 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:45:01.368 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-15T03:45:01.368 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:45:01.368 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:45:01.368 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T03:45:01.368 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T03:45:01.368 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T03:45:01.368 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T03:45:01.368 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T03:45:01.369 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T03:45:01.369 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:45:01.369 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T03:45:01.663 INFO:teuthology.orchestra.run.smithi001.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 221 kB/s | 26 kB 00:00 2024-09-15T03:45:01.688 INFO:teuthology.orchestra.run.smithi001.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 136 kB/s | 19 kB 00:00 2024-09-15T03:45:01.713 INFO:teuthology.orchestra.run.smithi001.stdout:(3/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 285 kB/s | 47 kB 00:00 2024-09-15T03:45:01.799 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T03:45:01.799 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:01.799 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T03:45:01.800 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T03:45:01.801 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:01.801 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:45:01.847 INFO:teuthology.orchestra.run.smithi001.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-09-15T03:45:01.889 INFO:teuthology.orchestra.run.smithi001.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.1 MB/s | 222 kB 00:00 2024-09-15T03:45:01.923 INFO:teuthology.orchestra.run.smithi001.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 713 kB/s | 54 kB 00:00 2024-09-15T03:45:01.956 INFO:teuthology.orchestra.run.smithi001.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 482 kB/s | 32 kB 00:00 2024-09-15T03:45:01.981 INFO:teuthology.orchestra.run.smithi001.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 251 kB/s | 14 kB 00:00 2024-09-15T03:45:02.007 INFO:teuthology.orchestra.run.smithi001.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 952 kB/s | 279 kB 00:00 2024-09-15T03:45:02.048 INFO:teuthology.orchestra.run.smithi001.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.4 MB/s | 220 kB 00:00 2024-09-15T03:45:02.082 INFO:teuthology.orchestra.run.smithi001.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 790 kB/s | 59 kB 00:00 2024-09-15T03:45:02.103 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:45:02.103 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-15T03:45:02.103 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:45:02.104 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:45:02.104 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T03:45:02.104 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T03:45:02.104 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T03:45:02.104 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T03:45:02.104 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T03:45:02.104 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T03:45:02.104 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:45:02.104 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T03:45:02.107 INFO:teuthology.orchestra.run.smithi001.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-09-15T03:45:02.149 INFO:teuthology.orchestra.run.smithi001.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-09-15T03:45:02.152 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:02.152 INFO:teuthology.orchestra.run.smithi001.stdout:Total 1.8 MB/s | 2.2 MB 00:01 2024-09-15T03:45:02.184 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-rook 2024-09-15T03:45:02.223 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-15T03:45:02.240 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-15T03:45:02.241 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-15T03:45:02.337 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-15T03:45:02.337 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-15T03:45:02.511 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T03:45:02.511 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:02.511 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-15T03:45:02.511 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:45:02.511 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T03:45:02.511 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T03:45:02.511 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T03:45:02.511 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T03:45:02.512 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T03:45:02.512 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T03:45:02.512 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T03:45:02.512 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T03:45:02.512 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T03:45:02.512 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T03:45:02.512 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T03:45:02.512 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:02.512 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:45:02.776 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:18 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:45:02.862 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-15T03:45:02.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-rook 2024-09-15T03:45:02.917 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:45:02.919 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:02.919 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-09-15T03:45:02.919 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:02.919 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T03:45:02.919 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-15T03:45:02.919 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-09-15T03:45:02.919 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T03:45:02.919 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T03:45:02.919 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T03:45:02.919 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T03:45:02.920 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T03:45:02.920 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T03:45:02.920 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T03:45:02.920 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T03:45:02.920 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T03:45:02.920 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T03:45:02.920 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T03:45:02.920 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T03:45:02.920 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:02.920 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T03:45:02.920 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:02.920 INFO:teuthology.orchestra.run.smithi067.stdout:Install 13 Packages 2024-09-15T03:45:02.921 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:02.921 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 2.2 M 2024-09-15T03:45:02.921 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 26 M 2024-09-15T03:45:02.921 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T03:45:03.038 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T03:45:03.094 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T03:45:03.150 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T03:45:03.185 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T03:45:03.221 INFO:teuthology.orchestra.run.smithi067.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 184 kB/s | 26 kB 00:00 2024-09-15T03:45:03.227 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T03:45:03.237 INFO:teuthology.orchestra.run.smithi067.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 123 kB/s | 19 kB 00:00 2024-09-15T03:45:03.254 INFO:teuthology.orchestra.run.smithi067.stdout:(3/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 274 kB/s | 47 kB 00:00 2024-09-15T03:45:03.346 INFO:teuthology.orchestra.run.smithi067.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 222 kB 00:00 2024-09-15T03:45:03.350 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T03:45:03.372 INFO:teuthology.orchestra.run.smithi067.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-15T03:45:03.405 INFO:teuthology.orchestra.run.smithi067.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 968 kB/s | 32 kB 00:00 2024-09-15T03:45:03.422 INFO:teuthology.orchestra.run.smithi067.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 709 kB/s | 54 kB 00:00 2024-09-15T03:45:03.439 INFO:teuthology.orchestra.run.smithi067.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 422 kB/s | 14 kB 00:00 2024-09-15T03:45:03.460 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T03:45:03.472 INFO:teuthology.orchestra.run.smithi067.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 279 kB 00:00 2024-09-15T03:45:03.491 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:13 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:45:03.515 INFO:teuthology.orchestra.run.smithi067.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.3 MB/s | 220 kB 00:00 2024-09-15T03:45:03.521 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T03:45:03.532 INFO:teuthology.orchestra.run.smithi067.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-09-15T03:45:03.557 INFO:teuthology.orchestra.run.smithi067.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 2.2 MB/s | 90 kB 00:00 2024-09-15T03:45:03.599 INFO:teuthology.orchestra.run.smithi067.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.7 MB/s | 1.0 MB 00:00 2024-09-15T03:45:03.601 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:03.601 INFO:teuthology.orchestra.run.smithi067.stdout:Total 3.2 MB/s | 2.2 MB 00:00 2024-09-15T03:45:03.624 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:45:03.625 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:03.625 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-15T03:45:03.625 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:03.626 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-15T03:45:03.626 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-15T03:45:03.626 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-15T03:45:03.626 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T03:45:03.626 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T03:45:03.626 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T03:45:03.626 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T03:45:03.626 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T03:45:03.626 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T03:45:03.626 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T03:45:03.626 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T03:45:03.626 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T03:45:03.626 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T03:45:03.627 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T03:45:03.627 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T03:45:03.627 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:03.627 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-15T03:45:03.627 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:03.627 INFO:teuthology.orchestra.run.smithi190.stdout:Install 13 Packages 2024-09-15T03:45:03.627 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:03.629 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 2.2 M 2024-09-15T03:45:03.629 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 26 M 2024-09-15T03:45:03.629 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-15T03:45:03.667 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T03:45:03.683 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T03:45:03.683 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T03:45:03.779 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T03:45:03.779 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T03:45:03.849 INFO:teuthology.orchestra.run.smithi190.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 143 kB/s | 19 kB 00:00 2024-09-15T03:45:03.882 INFO:teuthology.orchestra.run.smithi190.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 152 kB/s | 26 kB 00:00 2024-09-15T03:45:03.908 INFO:teuthology.orchestra.run.smithi190.stdout:(3/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 246 kB/s | 47 kB 00:00 2024-09-15T03:45:04.042 INFO:teuthology.orchestra.run.smithi190.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-09-15T03:45:04.067 INFO:teuthology.orchestra.run.smithi190.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.0 MB/s | 222 kB 00:00 2024-09-15T03:45:04.100 INFO:teuthology.orchestra.run.smithi190.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 919 kB/s | 54 kB 00:00 2024-09-15T03:45:04.134 INFO:teuthology.orchestra.run.smithi190.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-15T03:45:04.159 INFO:teuthology.orchestra.run.smithi190.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 279 kB 00:00 2024-09-15T03:45:04.184 INFO:teuthology.orchestra.run.smithi190.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 171 kB/s | 14 kB 00:00 2024-09-15T03:45:04.218 INFO:teuthology.orchestra.run.smithi190.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-15T03:45:04.243 INFO:teuthology.orchestra.run.smithi190.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-09-15T03:45:04.269 INFO:teuthology.orchestra.run.smithi190.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-15T03:45:04.292 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T03:45:04.299 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T03:45:04.319 INFO:teuthology.orchestra.run.smithi190.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.4 MB/s | 1.0 MB 00:00 2024-09-15T03:45:04.321 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:04.322 INFO:teuthology.orchestra.run.smithi190.stdout:Total 3.2 MB/s | 2.2 MB 00:00 2024-09-15T03:45:04.333 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T03:45:04.370 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T03:45:04.387 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-15T03:45:04.403 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-15T03:45:04.403 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-15T03:45:04.414 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T03:45:04.437 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:45:04.473 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T03:45:04.490 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-15T03:45:04.491 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-15T03:45:04.536 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T03:45:04.590 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T03:45:04.627 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T03:45:04.669 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T03:45:04.776 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T03:45:04.894 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T03:45:04.954 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T03:45:04.990 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-15T03:45:05.165 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T03:45:05.212 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T03:45:05.268 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:45:05.268 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-15T03:45:05.268 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T03:45:05.268 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T03:45:05.268 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T03:45:05.268 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T03:45:05.268 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T03:45:05.268 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T03:45:05.269 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T03:45:05.269 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T03:45:05.269 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T03:45:05.269 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T03:45:05.269 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T03:45:05.276 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T03:45:05.313 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T03:45:05.355 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T03:45:05.457 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T03:45:05.558 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T03:45:05.606 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T03:45:05.639 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T03:45:05.639 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:05.639 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-15T03:45:05.639 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:45:05.639 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T03:45:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T03:45:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T03:45:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T03:45:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T03:45:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T03:45:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T03:45:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T03:45:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T03:45:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T03:45:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T03:45:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T03:45:05.640 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:05.640 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:45:05.719 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T03:45:05.775 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T03:45:05.809 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T03:45:05.855 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T03:45:05.872 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T03:45:05.877 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:45:06.312 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T03:45:06.370 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T03:45:06.404 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T03:45:06.458 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T03:45:06.460 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:21 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:45:06.490 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:45:06.599 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:45:06.600 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:06.600 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-15T03:45:06.600 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:06.600 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-15T03:45:06.600 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-15T03:45:06.601 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-15T03:45:06.601 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T03:45:06.601 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T03:45:06.601 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T03:45:06.601 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T03:45:06.601 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T03:45:06.601 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:06.601 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-15T03:45:06.601 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:06.601 INFO:teuthology.orchestra.run.smithi001.stdout:Install 6 Packages 2024-09-15T03:45:06.601 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:06.602 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 7.0 M 2024-09-15T03:45:06.602 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 33 M 2024-09-15T03:45:06.602 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-15T03:45:06.643 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:45:06.643 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-15T03:45:06.643 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T03:45:06.643 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T03:45:06.643 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T03:45:06.643 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T03:45:06.644 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T03:45:06.644 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T03:45:06.644 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T03:45:06.644 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T03:45:06.644 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T03:45:06.644 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T03:45:06.644 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T03:45:06.988 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T03:45:06.988 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:06.988 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T03:45:06.988 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:45:06.988 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T03:45:06.989 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T03:45:06.989 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T03:45:06.989 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T03:45:06.989 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T03:45:06.989 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T03:45:06.989 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T03:45:06.989 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T03:45:06.989 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T03:45:06.989 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T03:45:06.989 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T03:45:06.989 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T03:45:06.989 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:06.990 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:45:07.214 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T03:45:07.316 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:45:07.316 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-15T03:45:07.316 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T03:45:07.316 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T03:45:07.316 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T03:45:07.316 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T03:45:07.316 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T03:45:07.316 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T03:45:07.316 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T03:45:07.316 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T03:45:07.317 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T03:45:07.317 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T03:45:07.317 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T03:45:07.501 INFO:teuthology.orchestra.run.smithi001.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 268 kB/s | 124 kB 00:00 2024-09-15T03:45:07.560 INFO:teuthology.orchestra.run.smithi001.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-15T03:45:07.576 INFO:teuthology.orchestra.run.smithi001.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 3.4 MB/s | 58 kB 00:00 2024-09-15T03:45:07.602 INFO:teuthology.orchestra.run.smithi001.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-15T03:45:07.643 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T03:45:07.643 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:07.643 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-15T03:45:07.643 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:45:07.643 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T03:45:07.643 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T03:45:07.643 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T03:45:07.643 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T03:45:07.643 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T03:45:07.643 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T03:45:07.643 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T03:45:07.644 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T03:45:07.644 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T03:45:07.644 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T03:45:07.644 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T03:45:07.644 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T03:45:07.644 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:07.644 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:45:07.803 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:23 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:45:07.852 INFO:teuthology.orchestra.run.smithi001.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 305 kB/s | 248 kB 00:00 2024-09-15T03:45:07.894 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T03:45:07.943 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:45:07.943 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:07.943 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-09-15T03:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T03:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-15T03:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-09-15T03:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T03:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T03:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T03:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T03:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T03:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T03:45:07.944 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:07.945 INFO:teuthology.orchestra.run.smithi067.stdout:Install 6 Packages 2024-09-15T03:45:07.945 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:07.945 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 7.0 M 2024-09-15T03:45:07.945 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 33 M 2024-09-15T03:45:07.945 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T03:45:08.202 INFO:teuthology.orchestra.run.smithi001.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.1 MB/s | 6.0 MB 00:01 2024-09-15T03:45:08.205 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:08.205 INFO:teuthology.orchestra.run.smithi001.stdout:Total 4.4 MB/s | 7.0 MB 00:01 2024-09-15T03:45:08.282 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-15T03:45:08.296 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-15T03:45:08.296 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-15T03:45:08.383 INFO:teuthology.orchestra.run.smithi067.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 470 kB/s | 124 kB 00:00 2024-09-15T03:45:08.404 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-15T03:45:08.404 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-15T03:45:08.408 INFO:teuthology.orchestra.run.smithi067.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 860 kB/s | 248 kB 00:00 2024-09-15T03:45:08.462 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:18 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:45:08.467 INFO:teuthology.orchestra.run.smithi067.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 991 kB/s | 58 kB 00:00 2024-09-15T03:45:08.492 INFO:teuthology.orchestra.run.smithi067.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.9 MB/s | 548 kB 00:00 2024-09-15T03:45:08.517 INFO:teuthology.orchestra.run.smithi067.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-15T03:45:08.597 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:45:08.597 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:08.597 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-15T03:45:08.597 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:08.598 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-15T03:45:08.598 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-15T03:45:08.598 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-15T03:45:08.598 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T03:45:08.598 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T03:45:08.598 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T03:45:08.598 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T03:45:08.598 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T03:45:08.598 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:08.598 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-15T03:45:08.598 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:08.599 INFO:teuthology.orchestra.run.smithi190.stdout:Install 6 Packages 2024-09-15T03:45:08.599 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:08.599 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 7.0 M 2024-09-15T03:45:08.599 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 33 M 2024-09-15T03:45:08.599 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-15T03:45:08.727 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-15T03:45:08.902 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T03:45:08.930 INFO:teuthology.orchestra.run.smithi190.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-15T03:45:08.963 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T03:45:08.981 INFO:teuthology.orchestra.run.smithi190.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-15T03:45:09.006 INFO:teuthology.orchestra.run.smithi190.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-15T03:45:09.031 INFO:teuthology.orchestra.run.smithi190.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-15T03:45:09.140 INFO:teuthology.orchestra.run.smithi190.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-09-15T03:45:09.181 INFO:teuthology.orchestra.run.smithi190.stdout:(6/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 275 kB/s | 124 kB 00:00 2024-09-15T03:45:09.183 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:09.183 INFO:teuthology.orchestra.run.smithi190.stdout:Total 12 MB/s | 7.0 MB 00:00 2024-09-15T03:45:09.254 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-15T03:45:09.265 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-15T03:45:09.265 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-15T03:45:09.364 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-15T03:45:09.372 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-15T03:45:09.527 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T03:45:09.610 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T03:45:09.673 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T03:45:09.678 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-15T03:45:09.700 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:45:09.856 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T03:45:09.900 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T03:45:10.134 INFO:teuthology.orchestra.run.smithi067.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.0 MB/s | 6.0 MB 00:02 2024-09-15T03:45:10.136 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:10.137 INFO:teuthology.orchestra.run.smithi067.stdout:Total 3.2 MB/s | 7.0 MB 00:02 2024-09-15T03:45:10.211 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T03:45:10.224 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T03:45:10.224 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T03:45:10.328 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T03:45:10.329 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T03:45:10.427 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T03:45:10.448 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:45:10.448 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-15T03:45:10.448 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T03:45:10.448 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T03:45:10.448 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T03:45:10.448 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T03:45:10.520 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T03:45:10.575 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T03:45:10.604 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:45:10.643 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T03:45:10.738 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T03:45:10.738 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:10.738 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-15T03:45:10.738 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:45:10.738 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T03:45:10.738 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T03:45:10.739 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T03:45:10.739 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T03:45:10.739 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T03:45:10.739 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:10.739 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:45:10.833 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T03:45:10.887 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T03:45:10.967 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-fuse 2024-09-15T03:45:11.361 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:45:11.361 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-15T03:45:11.361 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T03:45:11.361 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T03:45:11.361 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T03:45:11.361 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T03:45:11.442 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T03:45:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T03:45:11.562 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:26 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:45:11.597 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T03:45:11.623 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:45:11.650 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T03:45:11.650 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:11.650 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-15T03:45:11.650 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:45:11.651 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T03:45:11.651 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T03:45:11.651 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T03:45:11.651 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T03:45:11.651 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T03:45:11.651 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:11.652 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:45:11.701 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:45:11.701 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:11.701 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-15T03:45:11.701 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:11.701 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-15T03:45:11.702 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-15T03:45:11.702 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-15T03:45:11.702 INFO:teuthology.orchestra.run.smithi001.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T03:45:11.702 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:11.702 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-15T03:45:11.702 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:11.702 INFO:teuthology.orchestra.run.smithi001.stdout:Install 2 Packages 2024-09-15T03:45:11.702 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:11.703 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 893 k 2024-09-15T03:45:11.703 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 2.6 M 2024-09-15T03:45:11.703 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-15T03:45:11.904 INFO:teuthology.orchestra.run.smithi001.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 562 kB/s | 80 kB 00:00 2024-09-15T03:45:11.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-fuse 2024-09-15T03:45:12.037 INFO:teuthology.orchestra.run.smithi001.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.9 MB/s | 813 kB 00:00 2024-09-15T03:45:12.037 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:12.038 INFO:teuthology.orchestra.run.smithi001.stdout:Total 2.6 MB/s | 893 kB 00:00 2024-09-15T03:45:12.050 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-15T03:45:12.072 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-15T03:45:12.073 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-15T03:45:12.155 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-15T03:45:12.156 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-15T03:45:12.352 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-15T03:45:12.372 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:45:12.373 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-15T03:45:12.373 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T03:45:12.373 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T03:45:12.373 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T03:45:12.373 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T03:45:12.479 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:22 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:45:12.489 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T03:45:12.556 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:45:12.612 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:45:12.613 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:12.613 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-15T03:45:12.613 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:12.613 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-15T03:45:12.613 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-15T03:45:12.614 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-15T03:45:12.614 INFO:teuthology.orchestra.run.smithi190.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T03:45:12.614 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:12.614 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-15T03:45:12.614 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:12.614 INFO:teuthology.orchestra.run.smithi190.stdout:Install 2 Packages 2024-09-15T03:45:12.614 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:12.614 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 893 k 2024-09-15T03:45:12.614 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 2.6 M 2024-09-15T03:45:12.614 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-15T03:45:12.645 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T03:45:12.645 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:12.645 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T03:45:12.645 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:45:12.645 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T03:45:12.645 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T03:45:12.645 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T03:45:12.645 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T03:45:12.646 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T03:45:12.646 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:12.646 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:45:12.858 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-fuse 2024-09-15T03:45:13.159 INFO:teuthology.orchestra.run.smithi190.stdout:(1/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 3.0 MB/s | 813 kB 00:00 2024-09-15T03:45:13.224 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:45:13.224 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-15T03:45:13.350 INFO:teuthology.orchestra.run.smithi190.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 174 kB/s | 80 kB 00:00 2024-09-15T03:45:13.351 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:13.351 INFO:teuthology.orchestra.run.smithi190.stdout:Total 1.2 MB/s | 893 kB 00:00 2024-09-15T03:45:13.363 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-15T03:45:13.384 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-15T03:45:13.384 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-15T03:45:13.453 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:29 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:45:13.462 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-15T03:45:13.462 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-15T03:45:13.555 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T03:45:13.555 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:13.555 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-15T03:45:13.555 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T03:45:13.555 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:13.556 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:45:13.592 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:45:13.592 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:13.592 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-09-15T03:45:13.592 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:13.593 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T03:45:13.593 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-15T03:45:13.593 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-09-15T03:45:13.593 INFO:teuthology.orchestra.run.smithi067.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T03:45:13.593 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:13.593 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T03:45:13.593 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:13.593 INFO:teuthology.orchestra.run.smithi067.stdout:Install 2 Packages 2024-09-15T03:45:13.593 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:13.594 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 893 k 2024-09-15T03:45:13.594 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 2.6 M 2024-09-15T03:45:13.594 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T03:45:13.644 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-15T03:45:13.777 INFO:teuthology.orchestra.run.smithi067.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 590 kB/s | 80 kB 00:00 2024-09-15T03:45:13.801 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-volume 2024-09-15T03:45:13.814 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T03:45:13.882 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:45:14.396 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:29 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:45:14.533 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:45:14.534 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:14.534 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-15T03:45:14.534 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:14.534 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-15T03:45:14.534 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-15T03:45:14.534 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:14.534 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-15T03:45:14.534 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:14.534 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-15T03:45:14.535 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:14.535 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 270 k 2024-09-15T03:45:14.535 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 1.3 M 2024-09-15T03:45:14.535 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-15T03:45:14.574 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:45:14.574 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-15T03:45:14.759 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.2 MB/s | 270 kB 00:00 2024-09-15T03:45:14.759 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:14.759 INFO:teuthology.orchestra.run.smithi001.stdout:Total 1.2 MB/s | 270 kB 00:00 2024-09-15T03:45:14.760 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-15T03:45:14.769 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-15T03:45:14.769 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-15T03:45:14.803 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-15T03:45:14.804 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-15T03:45:14.858 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T03:45:14.858 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:14.858 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-15T03:45:14.858 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T03:45:14.858 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:14.858 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:45:15.033 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-15T03:45:15.112 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-volume 2024-09-15T03:45:15.128 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:45:15.145 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:45:15.145 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:45:15.146 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T03:45:15.146 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:15.687 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:25 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:45:15.826 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:45:15.826 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:15.827 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-15T03:45:15.827 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:15.827 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-15T03:45:15.827 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-15T03:45:15.827 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:15.827 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-15T03:45:15.827 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:15.827 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-15T03:45:15.827 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:15.827 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 270 k 2024-09-15T03:45:15.827 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 1.3 M 2024-09-15T03:45:15.828 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-15T03:45:16.049 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.2 MB/s | 270 kB 00:00 2024-09-15T03:45:16.050 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:16.050 INFO:teuthology.orchestra.run.smithi190.stdout:Total 1.2 MB/s | 270 kB 00:00 2024-09-15T03:45:16.050 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-15T03:45:16.059 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-15T03:45:16.059 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-15T03:45:16.083 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-15T03:45:16.083 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-15T03:45:16.087 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:45:16.087 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:16.087 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-15T03:45:16.087 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:45:16.087 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:16.087 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:45:16.299 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-15T03:45:16.357 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:45:16.366 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librados-devel 2024-09-15T03:45:16.373 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:45:16.373 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:45:16.373 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T03:45:16.373 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:16.964 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:31 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:45:17.102 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:45:17.102 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:17.102 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-09-15T03:45:17.102 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:17.102 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-15T03:45:17.103 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-15T03:45:17.103 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:17.103 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-15T03:45:17.103 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:17.103 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-15T03:45:17.103 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:17.104 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 126 k 2024-09-15T03:45:17.104 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 448 k 2024-09-15T03:45:17.104 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-15T03:45:17.297 INFO:teuthology.orchestra.run.smithi001.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 654 kB/s | 126 kB 00:00 2024-09-15T03:45:17.297 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:17.298 INFO:teuthology.orchestra.run.smithi001.stdout:Total 649 kB/s | 126 kB 00:00 2024-09-15T03:45:17.298 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-15T03:45:17.304 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-15T03:45:17.304 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-15T03:45:17.370 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-15T03:45:17.371 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-15T03:45:17.376 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:45:17.376 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:17.376 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-15T03:45:17.376 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:45:17.376 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:17.376 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:45:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-15T03:45:17.639 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:17.660 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados-devel 2024-09-15T03:45:18.037 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:18.239 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:28 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:45:18.295 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:18.295 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:18.295 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-15T03:45:18.296 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:45:18.296 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:18.296 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:45:18.377 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:45:18.378 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:18.378 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-09-15T03:45:18.378 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:18.378 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-15T03:45:18.378 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-15T03:45:18.378 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:18.378 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-15T03:45:18.379 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:18.379 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-15T03:45:18.379 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:18.379 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 126 k 2024-09-15T03:45:18.379 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 448 k 2024-09-15T03:45:18.379 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-15T03:45:18.548 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install libcephfs2 2024-09-15T03:45:18.569 INFO:teuthology.orchestra.run.smithi190.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 664 kB/s | 126 kB 00:00 2024-09-15T03:45:18.570 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:18.570 INFO:teuthology.orchestra.run.smithi190.stdout:Total 658 kB/s | 126 kB 00:00 2024-09-15T03:45:18.570 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-15T03:45:18.576 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-15T03:45:18.576 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-15T03:45:18.639 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-15T03:45:18.640 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-15T03:45:18.785 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-15T03:45:18.892 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:19.141 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:34 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:45:19.234 INFO:teuthology.orchestra.run.smithi001.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:45:19.280 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:45:19.281 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-15T03:45:19.281 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:45:19.342 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:19.346 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install libcephfs-devel 2024-09-15T03:45:19.601 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:19.601 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:19.601 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-15T03:45:19.601 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:45:19.602 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:19.602 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:45:19.838 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs2 2024-09-15T03:45:19.937 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:34 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:45:20.076 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:45:20.077 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:20.077 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-09-15T03:45:20.077 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:20.077 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-15T03:45:20.077 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:45:20.077 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:20.077 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-15T03:45:20.077 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:20.078 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-15T03:45:20.078 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:20.078 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 25 k 2024-09-15T03:45:20.078 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 110 k 2024-09-15T03:45:20.078 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-15T03:45:20.085 INFO:teuthology.orchestra.run.smithi067.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 126 kB/s | 813 kB 00:06 2024-09-15T03:45:20.085 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:20.086 INFO:teuthology.orchestra.run.smithi067.stdout:Total 138 kB/s | 893 kB 00:06 2024-09-15T03:45:20.100 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T03:45:20.126 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T03:45:20.126 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T03:45:20.215 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T03:45:20.215 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T03:45:20.283 INFO:teuthology.orchestra.run.smithi001.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 121 kB/s | 25 kB 00:00 2024-09-15T03:45:20.284 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:20.284 INFO:teuthology.orchestra.run.smithi001.stdout:Total 121 kB/s | 25 kB 00:00 2024-09-15T03:45:20.284 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-15T03:45:20.288 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-15T03:45:20.288 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-15T03:45:20.290 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-15T03:45:20.290 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-15T03:45:20.374 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-15T03:45:20.412 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:30 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:45:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T03:45:20.491 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:20.503 INFO:teuthology.orchestra.run.smithi190.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:45:20.544 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T03:45:20.546 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:45:20.547 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-15T03:45:20.547 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:45:20.620 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs-devel 2024-09-15T03:45:20.621 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:45:20.822 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:21.071 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:21.071 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:21.071 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-15T03:45:21.071 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:45:21.071 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:21.071 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:45:21.184 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:31 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:45:21.251 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librados2 2024-09-15T03:45:21.277 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:45:21.277 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-15T03:45:21.316 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:45:21.317 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:21.317 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-09-15T03:45:21.317 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:21.317 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-15T03:45:21.317 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:45:21.317 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:21.317 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-15T03:45:21.317 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:21.317 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-15T03:45:21.317 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:21.318 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 25 k 2024-09-15T03:45:21.318 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 110 k 2024-09-15T03:45:21.318 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-15T03:45:21.457 INFO:teuthology.orchestra.run.smithi190.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 179 kB/s | 25 kB 00:00 2024-09-15T03:45:21.458 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:21.458 INFO:teuthology.orchestra.run.smithi190.stdout:Total 177 kB/s | 25 kB 00:00 2024-09-15T03:45:21.458 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-15T03:45:21.465 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-15T03:45:21.465 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-15T03:45:21.465 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-15T03:45:21.465 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-15T03:45:21.543 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T03:45:21.543 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:21.543 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T03:45:21.543 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T03:45:21.543 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:21.543 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:45:21.546 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-15T03:45:21.653 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:21.827 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-volume 2024-09-15T03:45:21.845 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:36 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:45:21.939 INFO:teuthology.orchestra.run.smithi001.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:45:21.984 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:45:21.986 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-15T03:45:21.986 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:45:22.019 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:22.055 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librbd1 2024-09-15T03:45:22.257 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:22.257 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:22.257 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-15T03:45:22.258 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:45:22.258 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:22.258 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:45:22.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados2 2024-09-15T03:45:22.423 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:38 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:45:22.562 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:45:22.562 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:22.562 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-09-15T03:45:22.563 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:22.563 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T03:45:22.563 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-15T03:45:22.563 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:22.563 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T03:45:22.563 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:22.563 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-09-15T03:45:22.563 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:22.563 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 270 k 2024-09-15T03:45:22.563 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 1.3 M 2024-09-15T03:45:22.563 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T03:45:22.645 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:37 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:45:22.741 INFO:teuthology.orchestra.run.smithi001.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:45:22.785 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:45:22.786 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-15T03:45:22.786 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:45:22.789 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.2 MB/s | 270 kB 00:00 2024-09-15T03:45:22.790 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:22.790 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.2 MB/s | 270 kB 00:00 2024-09-15T03:45:22.790 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T03:45:22.799 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T03:45:22.799 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T03:45:22.831 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T03:45:22.831 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T03:45:22.856 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rados 2024-09-15T03:45:22.988 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:32 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:45:23.055 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T03:45:23.078 INFO:teuthology.orchestra.run.smithi190.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:45:23.121 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:45:23.121 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-15T03:45:23.122 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:45:23.124 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:45:23.141 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:45:23.141 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:45:23.141 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T03:45:23.141 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:23.180 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librbd1 2024-09-15T03:45:23.446 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:38 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:45:23.539 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:45:23.584 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:45:23.585 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-15T03:45:23.585 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:45:23.654 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rgw 2024-09-15T03:45:23.745 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:33 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:45:23.835 INFO:teuthology.orchestra.run.smithi190.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:45:23.878 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:45:23.879 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-15T03:45:23.879 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:45:23.954 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rados 2024-09-15T03:45:24.144 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:45:24.144 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:24.145 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T03:45:24.145 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:45:24.145 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:24.145 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:45:24.239 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:39 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:45:24.330 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:45:24.375 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:45:24.376 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-15T03:45:24.376 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:45:24.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librados-devel 2024-09-15T03:45:24.449 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-cephfs 2024-09-15T03:45:24.512 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:34 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:45:24.601 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:45:24.644 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:45:24.645 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-15T03:45:24.645 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:45:24.718 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rgw 2024-09-15T03:45:25.039 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:40 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:45:25.041 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:41 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:45:25.134 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:45:25.179 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:45:25.180 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:45:25.181 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-15T03:45:25.181 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:45:25.181 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:25.181 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-09-15T03:45:25.181 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:25.182 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T03:45:25.182 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-15T03:45:25.182 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:25.182 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T03:45:25.182 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:25.182 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-09-15T03:45:25.182 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:25.182 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 126 k 2024-09-15T03:45:25.182 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 448 k 2024-09-15T03:45:25.182 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T03:45:25.245 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rbd 2024-09-15T03:45:25.284 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:35 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:45:25.375 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:45:25.393 INFO:teuthology.orchestra.run.smithi067.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 598 kB/s | 126 kB 00:00 2024-09-15T03:45:25.393 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:25.394 INFO:teuthology.orchestra.run.smithi067.stdout:Total 594 kB/s | 126 kB 00:00 2024-09-15T03:45:25.394 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T03:45:25.400 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T03:45:25.400 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T03:45:25.417 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:45:25.418 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-15T03:45:25.418 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:45:25.467 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T03:45:25.468 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T03:45:25.492 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-cephfs 2024-09-15T03:45:25.621 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T03:45:25.710 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:25.835 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:40 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:45:25.930 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:45:25.975 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:45:25.976 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-15T03:45:25.976 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:45:26.055 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:36 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:45:26.056 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-fuse 2024-09-15T03:45:26.145 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:26.146 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:45:26.188 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:45:26.189 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-15T03:45:26.190 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:45:26.262 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rbd 2024-09-15T03:45:26.368 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:26.368 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:26.368 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T03:45:26.368 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:45:26.368 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:26.368 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:45:26.571 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs2 2024-09-15T03:45:26.650 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:41 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:45:26.791 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:45:26.792 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:26.792 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-09-15T03:45:26.792 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:26.792 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-15T03:45:26.792 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-15T03:45:26.792 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:26.792 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-15T03:45:26.792 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:26.792 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-15T03:45:26.793 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:26.793 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 82 k 2024-09-15T03:45:26.793 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 222 k 2024-09-15T03:45:26.793 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-15T03:45:26.831 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:36 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:45:26.921 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:45:26.964 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:45:26.965 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-15T03:45:26.965 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:45:26.987 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 422 kB/s | 82 kB 00:00 2024-09-15T03:45:26.988 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:26.988 INFO:teuthology.orchestra.run.smithi001.stdout:Total 418 kB/s | 82 kB 00:00 2024-09-15T03:45:26.988 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-15T03:45:26.996 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-15T03:45:26.996 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-15T03:45:27.038 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-fuse 2024-09-15T03:45:27.062 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-15T03:45:27.062 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-15T03:45:27.170 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:43 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:45:27.210 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-15T03:45:27.264 INFO:teuthology.orchestra.run.smithi067.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:45:27.309 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:45:27.310 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-09-15T03:45:27.311 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:45:27.330 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:27.367 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs-devel 2024-09-15T03:45:27.604 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:37 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:45:27.738 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:45:27.739 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:27.739 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-09-15T03:45:27.739 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:27.739 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-15T03:45:27.739 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-15T03:45:27.739 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:27.739 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-15T03:45:27.739 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:27.740 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-15T03:45:27.740 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:27.740 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 82 k 2024-09-15T03:45:27.740 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 222 k 2024-09-15T03:45:27.740 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-15T03:45:27.762 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:27.952 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 386 kB/s | 82 kB 00:00 2024-09-15T03:45:27.952 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:27.953 INFO:teuthology.orchestra.run.smithi190.stdout:Total 384 kB/s | 82 kB 00:00 2024-09-15T03:45:27.953 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-15T03:45:27.959 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-15T03:45:27.960 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-15T03:45:27.961 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:43 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:45:28.011 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:28.012 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:28.012 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-15T03:45:28.012 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:45:28.012 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:28.012 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:45:28.022 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-15T03:45:28.022 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-15T03:45:28.099 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:45:28.100 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:28.100 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-09-15T03:45:28.100 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:28.100 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T03:45:28.100 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:45:28.100 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:28.100 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T03:45:28.101 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:28.101 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-09-15T03:45:28.101 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:28.101 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 25 k 2024-09-15T03:45:28.101 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 110 k 2024-09-15T03:45:28.101 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T03:45:28.165 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-15T03:45:28.246 INFO:teuthology.orchestra.run.smithi067.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 172 kB/s | 25 kB 00:00 2024-09-15T03:45:28.246 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:28.247 INFO:teuthology.orchestra.run.smithi067.stdout:Total 171 kB/s | 25 kB 00:00 2024-09-15T03:45:28.247 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T03:45:28.249 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-mirror 2024-09-15T03:45:28.250 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T03:45:28.250 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T03:45:28.252 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T03:45:28.252 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T03:45:28.265 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:28.337 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T03:45:28.445 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:28.658 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:28.796 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:28.847 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:43 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:45:28.917 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:28.917 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:28.917 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-15T03:45:28.917 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:45:28.917 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:28.917 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:45:28.987 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:45:28.988 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:28.988 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-09-15T03:45:28.988 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:28.988 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-15T03:45:28.988 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:45:28.988 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:28.989 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-15T03:45:28.989 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:28.989 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-15T03:45:28.989 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:28.989 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 2.9 M 2024-09-15T03:45:28.989 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 12 M 2024-09-15T03:45:28.989 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-15T03:45:29.109 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:29.109 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:29.109 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T03:45:29.109 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:45:29.109 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:29.109 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:45:29.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-mirror 2024-09-15T03:45:29.260 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librados2 2024-09-15T03:45:29.315 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 9.0 MB/s | 2.9 MB 00:00 2024-09-15T03:45:29.316 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:29.316 INFO:teuthology.orchestra.run.smithi001.stdout:Total 9.0 MB/s | 2.9 MB 00:00 2024-09-15T03:45:29.316 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-15T03:45:29.326 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-15T03:45:29.327 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-15T03:45:29.411 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-15T03:45:29.411 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-15T03:45:29.725 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:39 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:45:29.844 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-15T03:45:29.858 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:45:29.859 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:29.859 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-09-15T03:45:29.859 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:29.859 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-15T03:45:29.859 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:45:29.859 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:29.859 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-15T03:45:29.859 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:29.859 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-15T03:45:29.860 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:29.860 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 2.9 M 2024-09-15T03:45:29.860 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 12 M 2024-09-15T03:45:29.860 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-15T03:45:29.862 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:45 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:45:29.900 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:29.927 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:29.927 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:45:29.927 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T03:45:29.927 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:45:29.927 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:45:29.927 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:29.959 INFO:teuthology.orchestra.run.smithi067.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:45:30.006 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:45:30.007 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-09-15T03:45:30.007 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:45:30.070 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librbd1 2024-09-15T03:45:30.198 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 8.7 MB/s | 2.9 MB 00:00 2024-09-15T03:45:30.198 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:30.199 INFO:teuthology.orchestra.run.smithi190.stdout:Total 8.7 MB/s | 2.9 MB 00:00 2024-09-15T03:45:30.199 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-15T03:45:30.208 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-15T03:45:30.208 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-15T03:45:30.288 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-15T03:45:30.288 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-15T03:45:30.673 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:46 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:45:30.696 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-15T03:45:30.749 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:30.769 INFO:teuthology.orchestra.run.smithi067.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:45:30.773 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:30.773 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:45:30.773 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T03:45:30.773 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:45:30.773 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:45:30.773 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:30.817 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:45:30.818 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-09-15T03:45:30.818 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:45:30.886 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rados 2024-09-15T03:45:31.045 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:31.045 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:31.045 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-15T03:45:31.045 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:45:31.045 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:31.045 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:45:31.305 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-nbd 2024-09-15T03:45:31.475 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:47 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:45:31.568 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:45:31.613 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:45:31.615 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-09-15T03:45:31.615 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:45:31.677 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rgw 2024-09-15T03:45:31.867 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:31.867 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:31.867 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-15T03:45:31.867 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:45:31.867 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:31.867 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:45:31.900 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:46 ago on Sun 15 Sep 2024 03:43:45 AM UTC. 2024-09-15T03:45:32.043 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-15T03:45:32.043 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-09-15T03:45:32.044 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:32.044 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-15T03:45:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-15T03:45:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:32.044 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-15T03:45:32.044 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-15T03:45:32.044 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-15T03:45:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:32.044 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 165 k 2024-09-15T03:45:32.044 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 477 k 2024-09-15T03:45:32.045 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-15T03:45:32.139 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-nbd 2024-09-15T03:45:32.260 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 765 kB/s | 165 kB 00:00 2024-09-15T03:45:32.261 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:32.261 INFO:teuthology.orchestra.run.smithi001.stdout:Total 759 kB/s | 165 kB 00:00 2024-09-15T03:45:32.261 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-15T03:45:32.266 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:48 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:45:32.270 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-15T03:45:32.270 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-15T03:45:32.338 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-15T03:45:32.338 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-15T03:45:32.358 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:45:32.404 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:45:32.405 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-09-15T03:45:32.405 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:45:32.469 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-cephfs 2024-09-15T03:45:32.493 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-15T03:45:32.594 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:32.718 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:42 ago on Sun 15 Sep 2024 03:43:50 AM UTC. 2024-09-15T03:45:32.853 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-15T03:45:32.854 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:32.854 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-09-15T03:45:32.854 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:32.855 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-15T03:45:32.855 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-15T03:45:32.855 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:32.855 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-15T03:45:32.855 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-15T03:45:32.855 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-15T03:45:32.855 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:32.855 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 165 k 2024-09-15T03:45:32.855 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 477 k 2024-09-15T03:45:32.855 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-15T03:45:32.997 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:33.060 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:49 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:45:33.069 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 773 kB/s | 165 kB 00:00 2024-09-15T03:45:33.069 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:33.069 INFO:teuthology.orchestra.run.smithi190.stdout:Total 769 kB/s | 165 kB 00:00 2024-09-15T03:45:33.070 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-15T03:45:33.077 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-15T03:45:33.077 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-15T03:45:33.140 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-15T03:45:33.140 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-15T03:45:33.154 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:45:33.199 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:45:33.200 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-09-15T03:45:33.200 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:45:33.226 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:33.226 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:33.226 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-15T03:45:33.226 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:45:33.226 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:33.226 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-15T03:45:33.261 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rbd 2024-09-15T03:45:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-15T03:45:33.408 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:33.492 DEBUG:teuthology.parallel:result is None 2024-09-15T03:45:33.837 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:33.851 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:49 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:45:33.944 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:45:33.989 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:45:33.990 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-09-15T03:45:33.990 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:45:34.050 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-fuse 2024-09-15T03:45:34.100 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:34.100 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:34.100 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-15T03:45:34.100 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:45:34.101 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:34.101 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-15T03:45:34.361 DEBUG:teuthology.parallel:result is None 2024-09-15T03:45:34.643 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:50 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:45:34.782 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:45:34.783 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:34.784 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-09-15T03:45:34.784 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:34.784 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T03:45:34.784 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-15T03:45:34.784 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:34.784 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T03:45:34.784 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:34.784 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-09-15T03:45:34.784 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:34.785 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 82 k 2024-09-15T03:45:34.785 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 222 k 2024-09-15T03:45:34.785 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T03:45:34.974 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 431 kB/s | 82 kB 00:00 2024-09-15T03:45:34.975 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:34.975 INFO:teuthology.orchestra.run.smithi067.stdout:Total 427 kB/s | 82 kB 00:00 2024-09-15T03:45:34.975 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T03:45:34.983 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T03:45:34.983 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T03:45:35.050 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T03:45:35.050 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T03:45:35.205 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T03:45:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:35.691 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:35.933 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:35.933 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:35.933 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T03:45:35.933 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:45:35.933 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:35.933 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:45:36.154 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-mirror 2024-09-15T03:45:36.741 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:52 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:45:36.879 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:45:36.879 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:36.880 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-09-15T03:45:36.880 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:36.880 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T03:45:36.880 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:45:36.880 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:36.880 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T03:45:36.880 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:36.880 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-09-15T03:45:36.880 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:36.880 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 2.9 M 2024-09-15T03:45:36.880 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 12 M 2024-09-15T03:45:36.880 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T03:45:37.227 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 8.5 MB/s | 2.9 MB 00:00 2024-09-15T03:45:37.228 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:37.228 INFO:teuthology.orchestra.run.smithi067.stdout:Total 8.4 MB/s | 2.9 MB 00:00 2024-09-15T03:45:37.228 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T03:45:37.238 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T03:45:37.238 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T03:45:37.322 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T03:45:37.322 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T03:45:37.756 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T03:45:37.811 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:37.838 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:37.839 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:45:37.839 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T03:45:37.839 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:45:37.839 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:45:37.839 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:38.934 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:38.934 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:38.934 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T03:45:38.934 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:45:38.934 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:38.934 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:45:39.181 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-nbd 2024-09-15T03:45:39.779 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:55 ago on Sun 15 Sep 2024 03:43:44 AM UTC. 2024-09-15T03:45:39.919 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T03:45:39.919 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-09-15T03:45:39.920 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:39.920 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T03:45:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-15T03:45:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:39.920 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T03:45:39.920 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-09-15T03:45:39.920 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-09-15T03:45:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:39.920 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 165 k 2024-09-15T03:45:39.920 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 477 k 2024-09-15T03:45:39.920 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T03:45:40.139 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 757 kB/s | 165 kB 00:00 2024-09-15T03:45:40.139 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:45:40.140 INFO:teuthology.orchestra.run.smithi067.stdout:Total 751 kB/s | 165 kB 00:00 2024-09-15T03:45:40.140 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T03:45:40.148 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T03:45:40.148 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T03:45:40.217 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T03:45:40.217 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T03:45:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T03:45:40.489 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:40.939 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:41.202 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:45:41.202 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:41.202 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T03:45:41.202 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:45:41.202 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:41.202 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T03:45:41.466 DEBUG:teuthology.parallel:result is None 2024-09-15T03:45:41.466 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:45:41.612 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T03:45:41.638 INFO:teuthology.orchestra.run.smithi001.stdout:17.2.7-1643.gd75f2382.el9 2024-09-15T03:45:41.639 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-15T03:45:41.639 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-15T03:45:41.641 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:45:41.781 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T03:45:41.807 INFO:teuthology.orchestra.run.smithi067.stdout:17.2.7-1643.gd75f2382.el9 2024-09-15T03:45:41.807 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-15T03:45:41.807 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-15T03:45:41.809 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:45:41.952 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T03:45:41.977 INFO:teuthology.orchestra.run.smithi190.stdout:17.2.7-1643.gd75f2382.el9 2024-09-15T03:45:41.978 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-15T03:45:41.978 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-15T03:45:41.979 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-15T03:45:41.980 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-15T03:45:41.980 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T03:45:42.012 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T03:45:42.012 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T03:45:42.043 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-15T03:45:42.043 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T03:45:42.073 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-15T03:45:42.073 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-15T03:45:42.074 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T03:45:42.105 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T03:45:42.175 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T03:45:42.175 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T03:45:42.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T03:45:42.277 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-15T03:45:42.277 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T03:45:42.306 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T03:45:42.374 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-15T03:45:42.375 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-15T03:45:42.375 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T03:45:42.406 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T03:45:42.475 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T03:45:42.476 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T03:45:42.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T03:45:42.578 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-15T03:45:42.578 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T03:45:42.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T03:45:42.669 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-15T03:45:42.669 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-15T03:45:42.669 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T03:45:42.700 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T03:45:42.770 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T03:45:42.771 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T03:45:42.802 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T03:45:42.872 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-15T03:45:42.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T03:45:42.897 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T03:45:42.964 INFO:teuthology.run_tasks:Running task ceph... 2024-09-15T03:45:43.048 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-15T03:45:43.049 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 777 /var/log/ceph 2024-09-15T03:45:43.051 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /var/log/ceph 2024-09-15T03:45:43.054 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /var/log/ceph 2024-09-15T03:45:43.081 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-15T03:45:43.082 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T03:45:43.122 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T03:45:43.125 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T03:45:43.152 INFO:tasks.ceph:Creating extra log directories... 2024-09-15T03:45:43.152 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T03:45:43.194 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T03:45:43.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T03:45:43.234 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-15T03:45:43.234 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd sloppy crc': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-15T03:45:43.234 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905308', 'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'description': 'rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7905308', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd sloppy crc': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'filestore': True, 'fs': 'xfs'}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.a'], ['osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.b', 'mgr.x'], ['osd.8', 'osd.9', 'osd.10', 'osd.11', 'mon.c']], 'seed': 2126, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'targets': {'smithi001.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA3rs2DhrBM6++QsY9hBrsnj2CCrIknFhDwNIAqmuhKmjPHgxiztD+Dz4jK+ZoK6FDABuTtWCHD1q6uc80jpEp0=', 'smithi067.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN8WEoelaeFizgFkIx1baxTd76J3XY71ZvQNlmPsKUT5z2biyJkZL/BKq9asql2W4jeU3y3/zCxL7zXxwxaJ6IE=', 'smithi190.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD3ndSH9u14htsEoJRyYt+lnfEDhk3s2fNJubG0o7+k5SrAhVk1wVqw7iFgVz5tUGkQ3QzULLej4O8T9lgemhj8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd sloppy crc': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'cluster': 'ceph'}}, {'thrashosds': {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'crush-failure-domain': 'osd', 'k': 4, 'l': 3, 'm': 2, 'name': 'lrcprofile', 'plugin': 'lrc'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_00:54:39', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-15T03:45:43.235 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T03:45:43.266 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T03:45:43.268 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T03:45:43.283 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T03:45:43.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T03:45:43.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T03:45:43.354 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-15T03:45:43.354 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T03:45:43.412 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T03:45:43.412 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_1 2024-09-15T03:45:43.469 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T03:45:43.470 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:45:43.470 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-15T03:45:43.470 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:45:43.470 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:45:43.470 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-09-15 03:45:30.201003513 +0000 2024-09-15T03:45:43.470 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-09-15 03:43:05.855241907 +0000 2024-09-15T03:45:43.470 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-09-15 03:43:05.855241907 +0000 2024-09-15T03:45:43.470 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-09-15 03:43:05.855241907 +0000 2024-09-15T03:45:43.470 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T03:45:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-09-15T03:45:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-09-15T03:45:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.00010557 s, 4.8 MB/s 2024-09-15T03:45:43.537 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T03:45:43.595 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_2 2024-09-15T03:45:43.652 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T03:45:43.652 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:45:43.652 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-15T03:45:43.652 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:45:43.652 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:45:43.652 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-09-15 03:45:30.202003500 +0000 2024-09-15T03:45:43.652 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-09-15 03:43:06.183237875 +0000 2024-09-15T03:45:43.652 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-09-15 03:43:06.183237875 +0000 2024-09-15T03:45:43.652 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-09-15 03:43:06.183237875 +0000 2024-09-15T03:45:43.653 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T03:45:43.717 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-09-15T03:45:43.717 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-09-15T03:45:43.717 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 8.4139e-05 s, 6.1 MB/s 2024-09-15T03:45:43.718 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T03:45:43.775 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_3 2024-09-15T03:45:43.831 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T03:45:43.831 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:45:43.831 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-15T03:45:43.831 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:45:43.831 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:45:43.831 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-09-15 03:45:30.202003500 +0000 2024-09-15T03:45:43.831 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-09-15 03:43:06.505233917 +0000 2024-09-15T03:45:43.831 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-09-15 03:43:06.505233917 +0000 2024-09-15T03:45:43.832 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-09-15 03:43:06.505233917 +0000 2024-09-15T03:45:43.832 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T03:45:43.895 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-09-15T03:45:43.895 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-09-15T03:45:43.895 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000102067 s, 5.0 MB/s 2024-09-15T03:45:43.896 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T03:45:43.953 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_4 2024-09-15T03:45:44.012 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T03:45:44.012 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:45:44.012 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-15T03:45:44.012 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:45:44.012 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:45:44.012 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-09-15 03:45:30.203003488 +0000 2024-09-15T03:45:44.012 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-09-15 03:43:06.827229960 +0000 2024-09-15T03:45:44.013 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-09-15 03:43:06.827229960 +0000 2024-09-15T03:45:44.013 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-09-15 03:43:06.827229960 +0000 2024-09-15T03:45:44.013 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T03:45:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-09-15T03:45:44.076 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-09-15T03:45:44.077 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 9.9183e-05 s, 5.2 MB/s 2024-09-15T03:45:44.077 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T03:45:44.135 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T03:45:44.135 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T03:45:44.135 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T03:45:44.155 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T03:45:44.155 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2024-09-15T03:45:44.215 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T03:45:44.215 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:45:44.216 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-15T03:45:44.216 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:45:44.216 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:45:44.216 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-09-15 03:45:38.103718172 +0000 2024-09-15T03:45:44.216 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-09-15 03:43:05.206094353 +0000 2024-09-15T03:45:44.216 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-09-15 03:43:05.206094353 +0000 2024-09-15T03:45:44.216 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-09-15 03:43:05.206094353 +0000 2024-09-15T03:45:44.216 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T03:45:44.288 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-09-15T03:45:44.288 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-09-15T03:45:44.289 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000279184 s, 1.8 MB/s 2024-09-15T03:45:44.289 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T03:45:44.348 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2024-09-15T03:45:44.406 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T03:45:44.407 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:45:44.407 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-15T03:45:44.407 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:45:44.407 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:45:44.407 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-09-15 03:45:38.103718172 +0000 2024-09-15T03:45:44.407 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-09-15 03:43:05.539086978 +0000 2024-09-15T03:45:44.407 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-09-15 03:43:05.539086978 +0000 2024-09-15T03:45:44.407 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-09-15 03:43:05.539086978 +0000 2024-09-15T03:45:44.407 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T03:45:44.474 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-09-15T03:45:44.474 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-09-15T03:45:44.474 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000223244 s, 2.3 MB/s 2024-09-15T03:45:44.475 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T03:45:44.534 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2024-09-15T03:45:44.590 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T03:45:44.591 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:45:44.591 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-15T03:45:44.591 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:45:44.591 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:45:44.591 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-09-15 03:45:38.104718150 +0000 2024-09-15T03:45:44.591 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-09-15 03:43:05.862079826 +0000 2024-09-15T03:45:44.591 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-09-15 03:43:05.862079826 +0000 2024-09-15T03:45:44.591 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-09-15 03:43:05.862079826 +0000 2024-09-15T03:45:44.591 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T03:45:44.657 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-09-15T03:45:44.657 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-09-15T03:45:44.657 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000193612 s, 2.6 MB/s 2024-09-15T03:45:44.658 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T03:45:44.717 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2024-09-15T03:45:44.774 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T03:45:44.774 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:45:44.774 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-15T03:45:44.774 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:45:44.774 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:45:44.774 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-09-15 03:45:38.104718150 +0000 2024-09-15T03:45:44.774 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-09-15 03:43:06.187072628 +0000 2024-09-15T03:45:44.774 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-09-15 03:43:06.187072628 +0000 2024-09-15T03:45:44.774 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-09-15 03:43:06.187072628 +0000 2024-09-15T03:45:44.775 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T03:45:44.839 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-09-15T03:45:44.839 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-09-15T03:45:44.840 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000226569 s, 2.3 MB/s 2024-09-15T03:45:44.841 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T03:45:44.898 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-15T03:45:44.898 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-15T03:45:44.898 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T03:45:44.918 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T03:45:44.918 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_1 2024-09-15T03:45:44.977 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T03:45:44.978 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:45:44.978 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-15T03:45:44.978 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:45:44.978 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:45:44.978 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-15 03:45:31.032895232 +0000 2024-09-15T03:45:44.978 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-15 03:43:01.018846320 +0000 2024-09-15T03:45:44.978 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-15 03:43:01.018846320 +0000 2024-09-15T03:45:44.978 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-09-15 03:43:01.018846320 +0000 2024-09-15T03:45:44.978 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T03:45:45.044 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-15T03:45:45.044 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-15T03:45:45.044 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000217556 s, 2.4 MB/s 2024-09-15T03:45:45.046 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T03:45:45.103 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_2 2024-09-15T03:45:45.160 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T03:45:45.161 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:45:45.161 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-15T03:45:45.161 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:45:45.161 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:45:45.161 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-15 03:45:31.032895232 +0000 2024-09-15T03:45:45.161 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-15 03:43:01.352836933 +0000 2024-09-15T03:45:45.161 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-15 03:43:01.352836933 +0000 2024-09-15T03:45:45.161 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-09-15 03:43:01.352836933 +0000 2024-09-15T03:45:45.161 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T03:45:45.225 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-15T03:45:45.225 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-15T03:45:45.225 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.00028901 s, 1.8 MB/s 2024-09-15T03:45:45.227 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T03:45:45.283 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_3 2024-09-15T03:45:45.339 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T03:45:45.340 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:45:45.340 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-15T03:45:45.340 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:45:45.340 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:45:45.340 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-15 03:45:31.033895204 +0000 2024-09-15T03:45:45.340 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-15 03:43:01.683827631 +0000 2024-09-15T03:45:45.340 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-15 03:43:01.683827631 +0000 2024-09-15T03:45:45.340 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-09-15 03:43:01.683827631 +0000 2024-09-15T03:45:45.340 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T03:45:45.404 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-15T03:45:45.404 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-15T03:45:45.404 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000207746 s, 2.5 MB/s 2024-09-15T03:45:45.405 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T03:45:45.461 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_4 2024-09-15T03:45:45.516 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T03:45:45.516 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:45:45.516 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-15T03:45:45.516 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:45:45.516 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:45:45.516 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-15 03:45:31.033895204 +0000 2024-09-15T03:45:45.516 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-15 03:43:01.998818778 +0000 2024-09-15T03:45:45.516 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-15 03:43:01.998818778 +0000 2024-09-15T03:45:45.516 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-09-15 03:43:01.998818778 +0000 2024-09-15T03:45:45.517 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T03:45:45.578 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-15T03:45:45.579 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-15T03:45:45.579 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000191101 s, 2.7 MB/s 2024-09-15T03:45:45.580 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T03:45:45.635 INFO:tasks.ceph:osd dev map: {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'} 2024-09-15T03:45:45.635 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'}} 2024-09-15T03:45:45.635 INFO:tasks.ceph:Generating config... 2024-09-15T03:45:45.637 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-15T03:45:45.637 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-15T03:45:45.637 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-15T03:45:45.637 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-15T03:45:45.637 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-09-15T03:45:45.637 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-15T03:45:45.637 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-15T03:45:45.637 INFO:tasks.ceph:[osd] osd max backfills = 6 2024-09-15T03:45:45.638 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-15T03:45:45.638 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-15T03:45:45.638 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-15T03:45:45.638 INFO:tasks.ceph:[osd] osd objectstore = filestore 2024-09-15T03:45:45.638 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-15T03:45:45.638 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-15T03:45:45.638 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-09-15T03:45:45.638 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-09-15T03:45:45.638 INFO:tasks.ceph:[osd] osd sloppy crc = True 2024-09-15T03:45:45.638 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-15T03:45:45.638 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-15T03:45:45.638 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-15T03:45:45.639 INFO:tasks.ceph:[global] ms tcp read timeout = 5 2024-09-15T03:45:45.639 INFO:tasks.ceph:[global] osd_async_recovery_min_cost = 1 2024-09-15T03:45:45.639 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-15T03:45:45.639 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-15T03:45:45.639 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-15T03:45:45.639 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-15T03:45:45.639 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-15T03:45:45.639 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-15T03:45:45.639 INFO:tasks.ceph:Setting up mon.a... 2024-09-15T03:45:45.640 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-15T03:45:45.691 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/ceph.keyring 2024-09-15T03:45:45.694 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-15T03:45:45.738 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T03:45:45.806 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.1'), ('mon.b', '172.21.15.67'), ('mon.c', '172.21.15.190')] 2024-09-15T03:45:45.806 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.1,172.21.15.67,172.21.15.190', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd sloppy crc': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-15T03:45:45.807 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-15T03:45:45.807 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T03:45:45.863 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.1 --add b 172.21.15.67 --add c 172.21.15.190 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:45:45.948 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:45:45.949 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: generated fsid f27b540e-e5e3-4e95-afe8-c95ee66b0a62 2024-09-15T03:45:45.949 INFO:teuthology.orchestra.run.smithi001.stdout:setting min_mon_release = octopus 2024-09-15T03:45:45.949 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 0 2024-09-15T03:45:45.949 INFO:teuthology.orchestra.run.smithi001.stdout:fsid f27b540e-e5e3-4e95-afe8-c95ee66b0a62 2024-09-15T03:45:45.949 INFO:teuthology.orchestra.run.smithi001.stdout:last_changed 2024-09-15T03:45:45.948542+0000 2024-09-15T03:45:45.949 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-09-15T03:45:45.948542+0000 2024-09-15T03:45:45.949 INFO:teuthology.orchestra.run.smithi001.stdout:min_mon_release 15 (octopus) 2024-09-15T03:45:45.949 INFO:teuthology.orchestra.run.smithi001.stdout:election_strategy: 1 2024-09-15T03:45:45.949 INFO:teuthology.orchestra.run.smithi001.stdout:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-09-15T03:45:45.949 INFO:teuthology.orchestra.run.smithi001.stdout:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T03:45:45.949 INFO:teuthology.orchestra.run.smithi001.stdout:2: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.c 2024-09-15T03:45:45.950 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-15T03:45:45.951 DEBUG:teuthology.orchestra.run.smithi001:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T03:45:45.967 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f27b540e-e5e3-4e95-afe8-c95ee66b0a62... 2024-09-15T03:45:45.968 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T03:45:46.010 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T03:45:46.012 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T03:45:46.053 INFO:teuthology.orchestra.run.smithi001.stdout:[global] 2024-09-15T03:45:46.053 INFO:teuthology.orchestra.run.smithi001.stdout: chdir = "" 2024-09-15T03:45:46.053 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T03:45:46.053 INFO:teuthology.orchestra.run.smithi001.stdout: auth supported = cephx 2024-09-15T03:45:46.054 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.054 INFO:teuthology.orchestra.run.smithi001.stdout: filestore xattr use omap = true 2024-09-15T03:45:46.054 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.054 INFO:teuthology.orchestra.run.smithi001.stdout: mon clock drift allowed = 1.000 2024-09-15T03:45:46.054 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.054 INFO:teuthology.orchestra.run.smithi001.stdout: osd crush chooseleaf type = 0 2024-09-15T03:45:46.054 INFO:teuthology.orchestra.run.smithi001.stdout: auth debug = true 2024-09-15T03:45:46.054 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.054 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on old message = true 2024-09-15T03:45:46.054 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on bug = true 2024-09-15T03:45:46.054 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.054 INFO:teuthology.orchestra.run.smithi001.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T03:45:46.054 INFO:teuthology.orchestra.run.smithi001.stdout: mon pg warn max object skew = 0 2024-09-15T03:45:46.054 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.055 INFO:teuthology.orchestra.run.smithi001.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T03:45:46.055 INFO:teuthology.orchestra.run.smithi001.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T03:45:46.055 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.055 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default size = 2 2024-09-15T03:45:46.055 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.055 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow primary affinity = true 2024-09-15T03:45:46.055 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow pg remap = true 2024-09-15T03:45:46.055 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on legacy crush tunables = false 2024-09-15T03:45:46.055 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on crush straw calc version zero = false 2024-09-15T03:45:46.055 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on no sortbitwise = false 2024-09-15T03:45:46.055 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on osd down out interval zero = false 2024-09-15T03:45:46.055 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on too few osds = false 2024-09-15T03:45:46.055 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T03:45:46.056 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T03:45:46.056 INFO:teuthology.orchestra.run.smithi001.stdout: mon_allow_pool_size_one = true 2024-09-15T03:45:46.056 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.056 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T03:45:46.056 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.056 INFO:teuthology.orchestra.run.smithi001.stdout: osd default data pool replay window = 5 2024-09-15T03:45:46.056 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.056 INFO:teuthology.orchestra.run.smithi001.stdout: mon allow pool delete = true 2024-09-15T03:45:46.056 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.056 INFO:teuthology.orchestra.run.smithi001.stdout: mon cluster log file level = debug 2024-09-15T03:45:46.056 INFO:teuthology.orchestra.run.smithi001.stdout: debug asserts on shutdown = true 2024-09-15T03:45:46.056 INFO:teuthology.orchestra.run.smithi001.stdout: mon health detail to clog = false 2024-09-15T03:45:46.056 INFO:teuthology.orchestra.run.smithi001.stdout: mon host = "172.21.15.1,172.21.15.67,172.21.15.190" 2024-09-15T03:45:46.056 INFO:teuthology.orchestra.run.smithi001.stdout: mon client directed command retry = 5 2024-09-15T03:45:46.057 INFO:teuthology.orchestra.run.smithi001.stdout: mon election default strategy = 1 2024-09-15T03:45:46.057 INFO:teuthology.orchestra.run.smithi001.stdout: ms inject socket failures = 2500 2024-09-15T03:45:46.057 INFO:teuthology.orchestra.run.smithi001.stdout: ms tcp read timeout = 5 2024-09-15T03:45:46.057 INFO:teuthology.orchestra.run.smithi001.stdout: osd_async_recovery_min_cost = 1 2024-09-15T03:45:46.057 INFO:teuthology.orchestra.run.smithi001.stdout: fsid = f27b540e-e5e3-4e95-afe8-c95ee66b0a62 2024-09-15T03:45:46.057 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.057 INFO:teuthology.orchestra.run.smithi001.stdout:[osd] 2024-09-15T03:45:46.057 INFO:teuthology.orchestra.run.smithi001.stdout: osd journal size = 100 2024-09-15T03:45:46.057 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.057 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub load threshold = 5.0 2024-09-15T03:45:46.057 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub max interval = 120 2024-09-15T03:45:46.057 INFO:teuthology.orchestra.run.smithi001.stdout: osd mclock profile = high_recovery_ops 2024-09-15T03:45:46.058 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.058 INFO:teuthology.orchestra.run.smithi001.stdout: osd recover clone overlap = true 2024-09-15T03:45:46.058 INFO:teuthology.orchestra.run.smithi001.stdout: osd recovery max chunk = 1048576 2024-09-15T03:45:46.058 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.058 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug shutdown = true 2024-09-15T03:45:46.058 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug op order = true 2024-09-15T03:45:46.058 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug verify stray on activate = true 2024-09-15T03:45:46.058 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.058 INFO:teuthology.orchestra.run.smithi001.stdout: osd open classes on start = true 2024-09-15T03:45:46.058 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug pg log writeout = true 2024-09-15T03:45:46.058 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.059 INFO:teuthology.orchestra.run.smithi001.stdout: osd deep scrub update digest min age = 30 2024-09-15T03:45:46.059 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.059 INFO:teuthology.orchestra.run.smithi001.stdout: osd map max advance = 10 2024-09-15T03:45:46.059 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.059 INFO:teuthology.orchestra.run.smithi001.stdout: journal zero on create = true 2024-09-15T03:45:46.059 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.059 INFO:teuthology.orchestra.run.smithi001.stdout: filestore ondisk finisher threads = 3 2024-09-15T03:45:46.059 INFO:teuthology.orchestra.run.smithi001.stdout: filestore apply finisher threads = 3 2024-09-15T03:45:46.059 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.059 INFO:teuthology.orchestra.run.smithi001.stdout: bdev debug aio = true 2024-09-15T03:45:46.059 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug misdirected ops = true 2024-09-15T03:45:46.060 INFO:teuthology.orchestra.run.smithi001.stdout: debug monc = 20 2024-09-15T03:45:46.060 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore zero block detection = True 2024-09-15T03:45:46.060 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-09-15T03:45:46.060 INFO:teuthology.orchestra.run.smithi001.stdout: debug osd = 20 2024-09-15T03:45:46.060 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug reject backfill probability = 0.3 2024-09-15T03:45:46.060 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug verify cached snaps = True 2024-09-15T03:45:46.060 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug verify missing on start = True 2024-09-15T03:45:46.060 INFO:teuthology.orchestra.run.smithi001.stdout: osd max backfills = 6 2024-09-15T03:45:46.060 INFO:teuthology.orchestra.run.smithi001.stdout: osd max markdown count = 1000 2024-09-15T03:45:46.060 INFO:teuthology.orchestra.run.smithi001.stdout: osd mclock override recovery settings = True 2024-09-15T03:45:46.061 INFO:teuthology.orchestra.run.smithi001.stdout: osd objectstore = filestore 2024-09-15T03:45:46.061 INFO:teuthology.orchestra.run.smithi001.stdout: osd op queue = debug_random 2024-09-15T03:45:46.061 INFO:teuthology.orchestra.run.smithi001.stdout: osd op queue cut off = debug_random 2024-09-15T03:45:46.061 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub min interval = 60 2024-09-15T03:45:46.061 INFO:teuthology.orchestra.run.smithi001.stdout: osd sloppy crc = True 2024-09-15T03:45:46.061 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.061 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr] 2024-09-15T03:45:46.061 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-09-15T03:45:46.061 INFO:teuthology.orchestra.run.smithi001.stdout: debug mgr = 20 2024-09-15T03:45:46.061 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-09-15T03:45:46.062 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-09-15T03:45:46.062 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:45:46.062 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:45:46.062 INFO:teuthology.orchestra.run.smithi001.stdout: mgr/telemetry/nag = false 2024-09-15T03:45:46.062 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.062 INFO:teuthology.orchestra.run.smithi001.stdout:[mon] 2024-09-15T03:45:46.062 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-09-15T03:45:46.062 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-09-15T03:45:46.062 INFO:teuthology.orchestra.run.smithi001.stdout: debug paxos = 20 2024-09-15T03:45:46.062 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-09-15T03:45:46.062 INFO:teuthology.orchestra.run.smithi001.stdout: mon data avail warn = 5 2024-09-15T03:45:46.063 INFO:teuthology.orchestra.run.smithi001.stdout: mon mgr mkfs grace = 240 2024-09-15T03:45:46.063 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:45:46.063 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd reporter subtree level = osd 2024-09-15T03:45:46.063 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd prime pg temp = true 2024-09-15T03:45:46.063 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:45:46.063 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.063 INFO:teuthology.orchestra.run.smithi001.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T03:45:46.063 INFO:teuthology.orchestra.run.smithi001.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T03:45:46.063 INFO:teuthology.orchestra.run.smithi001.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T03:45:46.063 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.063 INFO:teuthology.orchestra.run.smithi001.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T03:45:46.063 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T03:45:46.064 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T03:45:46.064 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.064 INFO:teuthology.orchestra.run.smithi001.stdout: # 1m isn't quite enough 2024-09-15T03:45:46.064 INFO:teuthology.orchestra.run.smithi001.stdout: mon_down_mkfs_grace = 2m 2024-09-15T03:45:46.064 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.064 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_filestore_osds = false 2024-09-15T03:45:46.064 INFO:teuthology.orchestra.run.smithi001.stdout: mon scrub interval = 300 2024-09-15T03:45:46.064 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:45:46.064 INFO:teuthology.orchestra.run.smithi001.stdout:[client] 2024-09-15T03:45:46.064 INFO:teuthology.orchestra.run.smithi001.stdout: rgw cache enabled = true 2024-09-15T03:45:46.064 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable ops log = true 2024-09-15T03:45:46.064 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable usage log = true 2024-09-15T03:45:46.064 INFO:teuthology.orchestra.run.smithi001.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T03:45:46.065 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T03:45:46.065 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.a] 2024-09-15T03:45:46.065 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.b] 2024-09-15T03:45:46.065 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.c] 2024-09-15T03:45:46.066 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-09-15T03:45:46.066 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2024-09-15T03:45:46.066 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T03:45:46.066 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2024-09-15T03:45:46.066 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.066 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2024-09-15T03:45:46.066 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.066 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2024-09-15T03:45:46.067 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.067 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2024-09-15T03:45:46.067 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2024-09-15T03:45:46.067 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.067 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2024-09-15T03:45:46.067 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2024-09-15T03:45:46.067 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.067 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T03:45:46.067 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2024-09-15T03:45:46.067 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.067 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T03:45:46.068 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T03:45:46.068 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.068 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2024-09-15T03:45:46.068 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.068 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2024-09-15T03:45:46.068 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2024-09-15T03:45:46.068 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2024-09-15T03:45:46.068 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2024-09-15T03:45:46.068 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2024-09-15T03:45:46.068 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2024-09-15T03:45:46.068 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2024-09-15T03:45:46.068 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T03:45:46.069 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T03:45:46.069 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2024-09-15T03:45:46.069 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.069 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T03:45:46.069 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.069 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2024-09-15T03:45:46.069 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.069 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2024-09-15T03:45:46.069 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.069 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2024-09-15T03:45:46.070 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2024-09-15T03:45:46.070 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2024-09-15T03:45:46.070 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = "172.21.15.1,172.21.15.67,172.21.15.190" 2024-09-15T03:45:46.070 INFO:teuthology.orchestra.run.smithi067.stdout: mon client directed command retry = 5 2024-09-15T03:45:46.070 INFO:teuthology.orchestra.run.smithi067.stdout: mon election default strategy = 1 2024-09-15T03:45:46.070 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject socket failures = 2500 2024-09-15T03:45:46.070 INFO:teuthology.orchestra.run.smithi067.stdout: ms tcp read timeout = 5 2024-09-15T03:45:46.070 INFO:teuthology.orchestra.run.smithi067.stdout: osd_async_recovery_min_cost = 1 2024-09-15T03:45:46.070 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = f27b540e-e5e3-4e95-afe8-c95ee66b0a62 2024-09-15T03:45:46.070 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.070 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2024-09-15T03:45:46.070 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2024-09-15T03:45:46.070 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.071 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2024-09-15T03:45:46.071 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 120 2024-09-15T03:45:46.071 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2024-09-15T03:45:46.071 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.071 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2024-09-15T03:45:46.071 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2024-09-15T03:45:46.071 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.071 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2024-09-15T03:45:46.071 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2024-09-15T03:45:46.071 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2024-09-15T03:45:46.071 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.072 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2024-09-15T03:45:46.072 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2024-09-15T03:45:46.072 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.072 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2024-09-15T03:45:46.072 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.072 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2024-09-15T03:45:46.072 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.072 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2024-09-15T03:45:46.072 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.072 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2024-09-15T03:45:46.072 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2024-09-15T03:45:46.072 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.072 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2024-09-15T03:45:46.073 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2024-09-15T03:45:46.073 INFO:teuthology.orchestra.run.smithi067.stdout: debug monc = 20 2024-09-15T03:45:46.073 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore zero block detection = True 2024-09-15T03:45:46.073 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-09-15T03:45:46.073 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2024-09-15T03:45:46.073 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug reject backfill probability = 0.3 2024-09-15T03:45:46.073 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify cached snaps = True 2024-09-15T03:45:46.073 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify missing on start = True 2024-09-15T03:45:46.073 INFO:teuthology.orchestra.run.smithi067.stdout: osd max backfills = 6 2024-09-15T03:45:46.073 INFO:teuthology.orchestra.run.smithi067.stdout: osd max markdown count = 1000 2024-09-15T03:45:46.073 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock override recovery settings = True 2024-09-15T03:45:46.073 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = filestore 2024-09-15T03:45:46.074 INFO:teuthology.orchestra.run.smithi067.stdout: osd op queue = debug_random 2024-09-15T03:45:46.074 INFO:teuthology.orchestra.run.smithi067.stdout: osd op queue cut off = debug_random 2024-09-15T03:45:46.074 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub min interval = 60 2024-09-15T03:45:46.074 INFO:teuthology.orchestra.run.smithi067.stdout: osd sloppy crc = True 2024-09-15T03:45:46.074 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.074 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2024-09-15T03:45:46.074 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-09-15T03:45:46.074 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2024-09-15T03:45:46.074 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-09-15T03:45:46.074 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-09-15T03:45:46.074 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:45:46.074 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:45:46.074 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2024-09-15T03:45:46.074 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.075 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2024-09-15T03:45:46.075 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-09-15T03:45:46.075 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-09-15T03:45:46.075 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2024-09-15T03:45:46.075 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-09-15T03:45:46.075 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2024-09-15T03:45:46.075 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2024-09-15T03:45:46.075 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:45:46.075 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2024-09-15T03:45:46.075 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2024-09-15T03:45:46.075 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:45:46.075 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.075 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T03:45:46.075 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T03:45:46.076 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T03:45:46.076 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.076 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T03:45:46.076 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T03:45:46.076 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T03:45:46.076 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.076 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2024-09-15T03:45:46.076 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2024-09-15T03:45:46.076 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.076 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2024-09-15T03:45:46.076 INFO:teuthology.orchestra.run.smithi067.stdout: mon scrub interval = 300 2024-09-15T03:45:46.076 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T03:45:46.076 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2024-09-15T03:45:46.076 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2024-09-15T03:45:46.077 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2024-09-15T03:45:46.077 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2024-09-15T03:45:46.077 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T03:45:46.077 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T03:45:46.077 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2024-09-15T03:45:46.077 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.b] 2024-09-15T03:45:46.077 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.c] 2024-09-15T03:45:46.078 INFO:teuthology.orchestra.run.smithi190.stdout:[global] 2024-09-15T03:45:46.078 INFO:teuthology.orchestra.run.smithi190.stdout: chdir = "" 2024-09-15T03:45:46.078 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T03:45:46.078 INFO:teuthology.orchestra.run.smithi190.stdout: auth supported = cephx 2024-09-15T03:45:46.078 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.078 INFO:teuthology.orchestra.run.smithi190.stdout: filestore xattr use omap = true 2024-09-15T03:45:46.078 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.078 INFO:teuthology.orchestra.run.smithi190.stdout: mon clock drift allowed = 1.000 2024-09-15T03:45:46.078 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.078 INFO:teuthology.orchestra.run.smithi190.stdout: osd crush chooseleaf type = 0 2024-09-15T03:45:46.078 INFO:teuthology.orchestra.run.smithi190.stdout: auth debug = true 2024-09-15T03:45:46.078 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.079 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on old message = true 2024-09-15T03:45:46.079 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on bug = true 2024-09-15T03:45:46.079 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.079 INFO:teuthology.orchestra.run.smithi190.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T03:45:46.079 INFO:teuthology.orchestra.run.smithi190.stdout: mon pg warn max object skew = 0 2024-09-15T03:45:46.079 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.079 INFO:teuthology.orchestra.run.smithi190.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T03:45:46.079 INFO:teuthology.orchestra.run.smithi190.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T03:45:46.079 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.079 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default size = 2 2024-09-15T03:45:46.079 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.079 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow primary affinity = true 2024-09-15T03:45:46.079 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow pg remap = true 2024-09-15T03:45:46.080 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on legacy crush tunables = false 2024-09-15T03:45:46.080 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on crush straw calc version zero = false 2024-09-15T03:45:46.080 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on no sortbitwise = false 2024-09-15T03:45:46.080 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on osd down out interval zero = false 2024-09-15T03:45:46.080 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on too few osds = false 2024-09-15T03:45:46.080 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T03:45:46.080 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T03:45:46.080 INFO:teuthology.orchestra.run.smithi190.stdout: mon_allow_pool_size_one = true 2024-09-15T03:45:46.080 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.080 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T03:45:46.080 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.080 INFO:teuthology.orchestra.run.smithi190.stdout: osd default data pool replay window = 5 2024-09-15T03:45:46.080 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.081 INFO:teuthology.orchestra.run.smithi190.stdout: mon allow pool delete = true 2024-09-15T03:45:46.081 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.081 INFO:teuthology.orchestra.run.smithi190.stdout: mon cluster log file level = debug 2024-09-15T03:45:46.081 INFO:teuthology.orchestra.run.smithi190.stdout: debug asserts on shutdown = true 2024-09-15T03:45:46.081 INFO:teuthology.orchestra.run.smithi190.stdout: mon health detail to clog = false 2024-09-15T03:45:46.081 INFO:teuthology.orchestra.run.smithi190.stdout: mon host = "172.21.15.1,172.21.15.67,172.21.15.190" 2024-09-15T03:45:46.081 INFO:teuthology.orchestra.run.smithi190.stdout: mon client directed command retry = 5 2024-09-15T03:45:46.081 INFO:teuthology.orchestra.run.smithi190.stdout: mon election default strategy = 1 2024-09-15T03:45:46.081 INFO:teuthology.orchestra.run.smithi190.stdout: ms inject socket failures = 2500 2024-09-15T03:45:46.081 INFO:teuthology.orchestra.run.smithi190.stdout: ms tcp read timeout = 5 2024-09-15T03:45:46.081 INFO:teuthology.orchestra.run.smithi190.stdout: osd_async_recovery_min_cost = 1 2024-09-15T03:45:46.081 INFO:teuthology.orchestra.run.smithi190.stdout: fsid = f27b540e-e5e3-4e95-afe8-c95ee66b0a62 2024-09-15T03:45:46.081 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.081 INFO:teuthology.orchestra.run.smithi190.stdout:[osd] 2024-09-15T03:45:46.082 INFO:teuthology.orchestra.run.smithi190.stdout: osd journal size = 100 2024-09-15T03:45:46.082 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.082 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub load threshold = 5.0 2024-09-15T03:45:46.082 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub max interval = 120 2024-09-15T03:45:46.082 INFO:teuthology.orchestra.run.smithi190.stdout: osd mclock profile = high_recovery_ops 2024-09-15T03:45:46.082 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.082 INFO:teuthology.orchestra.run.smithi190.stdout: osd recover clone overlap = true 2024-09-15T03:45:46.082 INFO:teuthology.orchestra.run.smithi190.stdout: osd recovery max chunk = 1048576 2024-09-15T03:45:46.082 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.082 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug shutdown = true 2024-09-15T03:45:46.082 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug op order = true 2024-09-15T03:45:46.082 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify stray on activate = true 2024-09-15T03:45:46.083 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.083 INFO:teuthology.orchestra.run.smithi190.stdout: osd open classes on start = true 2024-09-15T03:45:46.083 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug pg log writeout = true 2024-09-15T03:45:46.083 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.083 INFO:teuthology.orchestra.run.smithi190.stdout: osd deep scrub update digest min age = 30 2024-09-15T03:45:46.083 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.083 INFO:teuthology.orchestra.run.smithi190.stdout: osd map max advance = 10 2024-09-15T03:45:46.083 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.083 INFO:teuthology.orchestra.run.smithi190.stdout: journal zero on create = true 2024-09-15T03:45:46.084 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.084 INFO:teuthology.orchestra.run.smithi190.stdout: filestore ondisk finisher threads = 3 2024-09-15T03:45:46.084 INFO:teuthology.orchestra.run.smithi190.stdout: filestore apply finisher threads = 3 2024-09-15T03:45:46.084 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.084 INFO:teuthology.orchestra.run.smithi190.stdout: bdev debug aio = true 2024-09-15T03:45:46.084 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug misdirected ops = true 2024-09-15T03:45:46.084 INFO:teuthology.orchestra.run.smithi190.stdout: debug monc = 20 2024-09-15T03:45:46.084 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore zero block detection = True 2024-09-15T03:45:46.084 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-09-15T03:45:46.084 INFO:teuthology.orchestra.run.smithi190.stdout: debug osd = 20 2024-09-15T03:45:46.084 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug reject backfill probability = 0.3 2024-09-15T03:45:46.084 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify cached snaps = True 2024-09-15T03:45:46.085 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify missing on start = True 2024-09-15T03:45:46.085 INFO:teuthology.orchestra.run.smithi190.stdout: osd max backfills = 6 2024-09-15T03:45:46.085 INFO:teuthology.orchestra.run.smithi190.stdout: osd max markdown count = 1000 2024-09-15T03:45:46.085 INFO:teuthology.orchestra.run.smithi190.stdout: osd mclock override recovery settings = True 2024-09-15T03:45:46.085 INFO:teuthology.orchestra.run.smithi190.stdout: osd objectstore = filestore 2024-09-15T03:45:46.085 INFO:teuthology.orchestra.run.smithi190.stdout: osd op queue = debug_random 2024-09-15T03:45:46.085 INFO:teuthology.orchestra.run.smithi190.stdout: osd op queue cut off = debug_random 2024-09-15T03:45:46.085 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub min interval = 60 2024-09-15T03:45:46.085 INFO:teuthology.orchestra.run.smithi190.stdout: osd sloppy crc = True 2024-09-15T03:45:46.085 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.085 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr] 2024-09-15T03:45:46.085 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-09-15T03:45:46.085 INFO:teuthology.orchestra.run.smithi190.stdout: debug mgr = 20 2024-09-15T03:45:46.086 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2024-09-15T03:45:46.086 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2024-09-15T03:45:46.086 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:45:46.086 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:45:46.086 INFO:teuthology.orchestra.run.smithi190.stdout: mgr/telemetry/nag = false 2024-09-15T03:45:46.086 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.086 INFO:teuthology.orchestra.run.smithi190.stdout:[mon] 2024-09-15T03:45:46.086 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-09-15T03:45:46.086 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2024-09-15T03:45:46.086 INFO:teuthology.orchestra.run.smithi190.stdout: debug paxos = 20 2024-09-15T03:45:46.086 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2024-09-15T03:45:46.086 INFO:teuthology.orchestra.run.smithi190.stdout: mon data avail warn = 5 2024-09-15T03:45:46.086 INFO:teuthology.orchestra.run.smithi190.stdout: mon mgr mkfs grace = 240 2024-09-15T03:45:46.086 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:45:46.086 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd reporter subtree level = osd 2024-09-15T03:45:46.087 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd prime pg temp = true 2024-09-15T03:45:46.087 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:45:46.087 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.087 INFO:teuthology.orchestra.run.smithi190.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T03:45:46.087 INFO:teuthology.orchestra.run.smithi190.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T03:45:46.087 INFO:teuthology.orchestra.run.smithi190.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T03:45:46.087 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.087 INFO:teuthology.orchestra.run.smithi190.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T03:45:46.087 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T03:45:46.087 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T03:45:46.087 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.087 INFO:teuthology.orchestra.run.smithi190.stdout: # 1m isn't quite enough 2024-09-15T03:45:46.087 INFO:teuthology.orchestra.run.smithi190.stdout: mon_down_mkfs_grace = 2m 2024-09-15T03:45:46.087 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.088 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_filestore_osds = false 2024-09-15T03:45:46.088 INFO:teuthology.orchestra.run.smithi190.stdout: mon scrub interval = 300 2024-09-15T03:45:46.088 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-15T03:45:46.088 INFO:teuthology.orchestra.run.smithi190.stdout:[client] 2024-09-15T03:45:46.088 INFO:teuthology.orchestra.run.smithi190.stdout: rgw cache enabled = true 2024-09-15T03:45:46.088 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable ops log = true 2024-09-15T03:45:46.088 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable usage log = true 2024-09-15T03:45:46.088 INFO:teuthology.orchestra.run.smithi190.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T03:45:46.088 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T03:45:46.088 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.a] 2024-09-15T03:45:46.088 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.b] 2024-09-15T03:45:46.088 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.c] 2024-09-15T03:45:46.090 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-15T03:45:46.090 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-15T03:45:46.181 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-15T03:45:46.181 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-15T03:45:46.182 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-15T03:45:46.197 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-15T03:45:46.197 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-15T03:45:46.252 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi001.front.sepia.ceph.com 2024-09-15T03:45:46.252 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-15T03:45:46.252 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T03:45:46.252 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T03:45:46.328 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-15T03:45:46.328 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:45:46.385 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi067.front.sepia.ceph.com 2024-09-15T03:45:46.385 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T03:45:46.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T03:45:46.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T03:45:46.428 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T03:45:46.428 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:45:46.487 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi190.front.sepia.ceph.com 2024-09-15T03:45:46.487 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-15T03:45:46.487 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T03:45:46.487 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T03:45:46.522 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-15T03:45:46.522 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:45:46.578 INFO:tasks.ceph:Setting up mon nodes... 2024-09-15T03:45:46.578 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-15T03:45:46.578 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T03:45:46.647 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T03:45:46.649 INFO:tasks.ceph:Setting up mds nodes... 2024-09-15T03:45:46.650 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-15T03:45:46.650 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-15T03:45:46.690 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-15T03:45:46.702 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-15T03:45:46.703 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'}} 2024-09-15T03:45:46.703 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-15T03:45:46.769 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T03:45:46.769 INFO:tasks.ceph:role: osd.0 2024-09-15T03:45:46.769 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-15T03:45:46.769 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T03:45:46.834 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:45:46.834 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:45:46.834 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:45:46.834 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:45:46.834 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:45:46.834 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:45:46.834 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:45:46.834 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:45:46.834 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:45:46.834 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:45:46.866 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-09-15T03:45:46.868 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-09-15T03:45:46.868 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-15T03:45:46.938 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-15T03:45:47.009 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-15T03:45:47.076 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T03:45:47.076 INFO:tasks.ceph:role: osd.1 2024-09-15T03:45:47.076 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-15T03:45:47.076 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T03:45:47.141 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:45:47.141 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:45:47.141 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:45:47.141 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:45:47.141 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:45:47.141 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:45:47.142 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:45:47.142 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:45:47.142 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:45:47.142 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:45:47.174 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-09-15T03:45:47.176 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-09-15T03:45:47.176 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-15T03:45:47.248 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-15T03:45:47.318 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-15T03:45:47.384 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T03:45:47.384 INFO:tasks.ceph:role: osd.2 2024-09-15T03:45:47.384 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-15T03:45:47.384 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T03:45:47.448 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:45:47.449 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:45:47.449 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:45:47.449 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:45:47.449 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:45:47.449 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:45:47.449 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:45:47.449 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:45:47.449 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:45:47.449 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:45:47.481 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-09-15T03:45:47.483 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-09-15T03:45:47.484 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-15T03:45:47.553 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-15T03:45:47.623 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-15T03:45:47.690 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T03:45:47.690 INFO:tasks.ceph:role: osd.3 2024-09-15T03:45:47.690 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-15T03:45:47.690 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-15T03:45:47.756 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:45:47.756 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:45:47.756 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:45:47.756 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:45:47.757 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:45:47.757 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:45:47.757 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:45:47.757 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:45:47.757 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:45:47.757 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:45:47.789 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-09-15T03:45:47.791 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-09-15T03:45:47.791 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-15T03:45:47.861 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-15T03:45:47.932 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:45:48.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.034+0000 7f2c6425a440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-15T03:45:48.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.034+0000 7f2c6425a440 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-15T03:45:48.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.038+0000 7f2c6425a440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:48.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.038+0000 7f2c6425a440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-15T03:45:48.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.095+0000 7f2c6425a440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:48.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.095+0000 7f2c6425a440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:48.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.095+0000 7f2c6425a440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:48.102 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-15T03:45:48.128 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:45:48.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.209+0000 7f13aab2d440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-15T03:45:48.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.209+0000 7f13aab2d440 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-15T03:45:48.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.214+0000 7f13aab2d440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:48.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.214+0000 7f13aab2d440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-15T03:45:48.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.269+0000 7f13aab2d440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:48.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.269+0000 7f13aab2d440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:48.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.269+0000 7f13aab2d440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:48.277 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-15T03:45:48.343 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:45:48.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.424+0000 7fd01ec1c440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-15T03:45:48.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.424+0000 7fd01ec1c440 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-15T03:45:48.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.428+0000 7fd01ec1c440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:48.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.428+0000 7fd01ec1c440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-15T03:45:48.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.485+0000 7fd01ec1c440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:48.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.485+0000 7fd01ec1c440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:48.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.485+0000 7fd01ec1c440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:48.493 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-15T03:45:48.559 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:45:48.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.639+0000 7f46e2409440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-15T03:45:48.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.639+0000 7f46e2409440 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-15T03:45:48.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.644+0000 7f46e2409440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:48.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.644+0000 7f46e2409440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-15T03:45:48.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.701+0000 7f46e2409440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:48.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.701+0000 7f46e2409440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:48.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:45:48.701+0000 7f46e2409440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:48.709 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-15T03:45:48.775 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-15T03:45:48.807 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-15T03:45:48.807 INFO:tasks.ceph:role: osd.4 2024-09-15T03:45:48.807 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-15T03:45:48.807 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T03:45:48.883 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:45:48.884 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:45:48.884 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:45:48.884 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:45:48.884 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:45:48.884 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:45:48.884 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:45:48.884 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:45:48.884 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:45:48.885 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:45:49.381 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-09-15T03:45:49.384 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-09-15T03:45:49.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-15T03:45:49.498 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-15T03:45:49.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-15T03:45:49.602 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-15T03:45:49.602 INFO:tasks.ceph:role: osd.5 2024-09-15T03:45:49.602 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-15T03:45:49.602 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T03:45:49.679 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:45:49.679 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:45:49.680 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:45:49.680 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:45:49.680 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:45:49.680 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:45:49.680 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:45:49.680 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:45:49.680 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:45:49.680 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:45:50.141 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-09-15T03:45:50.143 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-09-15T03:45:50.143 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-15T03:45:50.227 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-15T03:45:50.297 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-15T03:45:50.364 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-15T03:45:50.365 INFO:tasks.ceph:role: osd.6 2024-09-15T03:45:50.365 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-15T03:45:50.365 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T03:45:50.438 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:45:50.438 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:45:50.438 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:45:50.438 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:45:50.438 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:45:50.439 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:45:50.439 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:45:50.439 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:45:50.439 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:45:50.439 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:45:50.956 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-09-15T03:45:50.959 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-09-15T03:45:50.959 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-15T03:45:51.083 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-15T03:45:51.114 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-15T03:45:51.179 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-15T03:45:51.180 INFO:tasks.ceph:role: osd.7 2024-09-15T03:45:51.180 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-15T03:45:51.180 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-15T03:45:51.253 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:45:51.253 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:45:51.253 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:45:51.254 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:45:51.254 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:45:51.254 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:45:51.254 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:45:51.254 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:45:51.254 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:45:51.254 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:45:51.782 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-09-15T03:45:51.784 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-09-15T03:45:51.784 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-15T03:45:51.909 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-15T03:45:51.944 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:45:52.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:52.048+0000 7f4eddc3f440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-15T03:45:52.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:52.048+0000 7f4eddc3f440 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-15T03:45:52.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:52.055+0000 7f4eddc3f440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:52.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:52.055+0000 7f4eddc3f440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-15T03:45:52.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:52.250+0000 7f4eddc3f440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:52.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:52.253+0000 7f4eddc3f440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:52.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:52.253+0000 7f4eddc3f440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:52.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-15T03:45:52.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:45:52.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:52.386+0000 7f4800121440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-15T03:45:52.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:52.386+0000 7f4800121440 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-15T03:45:52.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:52.391+0000 7f4800121440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:52.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:52.391+0000 7f4800121440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-15T03:45:52.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:52.540+0000 7f4800121440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:52.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:52.540+0000 7f4800121440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:52.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:52.540+0000 7f4800121440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:52.549 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-15T03:45:52.615 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:45:52.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:52.697+0000 7f2ee9010440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-15T03:45:52.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:52.697+0000 7f2ee9010440 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-15T03:45:52.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:52.702+0000 7f2ee9010440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:52.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:52.702+0000 7f2ee9010440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-15T03:45:52.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:52.846+0000 7f2ee9010440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:52.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:52.846+0000 7f2ee9010440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:52.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:52.846+0000 7f2ee9010440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:52.855 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-15T03:45:52.920 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:45:53.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:53.002+0000 7f376be65440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-15T03:45:53.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:53.002+0000 7f376be65440 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-15T03:45:53.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:53.007+0000 7f376be65440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:53.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:53.007+0000 7f376be65440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-15T03:45:53.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:53.137+0000 7f376be65440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:53.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:53.138+0000 7f376be65440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:53.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T03:45:53.138+0000 7f376be65440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:53.146 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-15T03:45:53.212 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-09-15T03:45:53.241 INFO:tasks.ceph:roles_to_devs: {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'} 2024-09-15T03:45:53.241 INFO:tasks.ceph:role: osd.8 2024-09-15T03:45:53.241 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-15T03:45:53.241 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T03:45:53.316 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:45:53.316 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:45:53.316 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:45:53.316 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:45:53.316 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:45:53.316 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:45:53.316 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:45:53.316 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:45:53.316 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:45:53.316 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:45:53.835 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-09-15T03:45:53.837 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-09-15T03:45:53.837 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-8 2024-09-15T03:45:53.985 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2024-09-15T03:45:54.019 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-09-15T03:45:54.084 INFO:tasks.ceph:roles_to_devs: {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'} 2024-09-15T03:45:54.084 INFO:tasks.ceph:role: osd.9 2024-09-15T03:45:54.084 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-15T03:45:54.085 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T03:45:54.157 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:45:54.157 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:45:54.157 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:45:54.157 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:45:54.157 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:45:54.157 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:45:54.157 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:45:54.158 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:45:54.158 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:45:54.158 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:45:54.609 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-09-15T03:45:54.611 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-09-15T03:45:54.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-9 2024-09-15T03:45:54.705 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2024-09-15T03:45:54.734 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2024-09-15T03:45:54.801 INFO:tasks.ceph:roles_to_devs: {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'} 2024-09-15T03:45:54.801 INFO:tasks.ceph:role: osd.10 2024-09-15T03:45:54.801 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-15T03:45:54.801 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T03:45:54.873 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:45:54.873 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:45:54.873 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:45:54.873 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:45:54.873 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:45:54.873 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:45:54.873 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:45:54.874 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:45:54.874 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:45:54.874 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:45:55.362 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-09-15T03:45:55.365 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-09-15T03:45:55.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2024-09-15T03:45:55.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2024-09-15T03:45:55.517 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2024-09-15T03:45:55.582 INFO:tasks.ceph:roles_to_devs: {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'} 2024-09-15T03:45:55.582 INFO:tasks.ceph:role: osd.11 2024-09-15T03:45:55.582 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-15T03:45:55.582 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-15T03:45:55.653 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:45:55.653 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:45:55.653 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:45:55.653 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:45:55.654 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:45:55.654 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:45:55.654 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:45:55.654 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:45:55.654 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:45:55.654 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:45:56.158 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-09-15T03:45:56.160 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-09-15T03:45:56.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-11 2024-09-15T03:45:56.299 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2024-09-15T03:45:56.329 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:45:56.422 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:56.421+0000 7f3764125440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-09-15T03:45:56.422 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:56.421+0000 7f3764125440 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-09-15T03:45:56.429 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:56.428+0000 7f3764125440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:56.429 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:56.428+0000 7f3764125440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-15T03:45:56.657 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:56.656+0000 7f3764125440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:56.657 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:56.656+0000 7f3764125440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:56.657 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:56.656+0000 7f3764125440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:56.683 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-09-15T03:45:56.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:45:56.788 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:56.787+0000 7f72d8923440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-09-15T03:45:56.788 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:56.787+0000 7f72d8923440 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-09-15T03:45:56.792 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:56.791+0000 7f72d8923440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:56.792 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:56.791+0000 7f72d8923440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-15T03:45:56.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:56.955+0000 7f72d8923440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:56.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:56.955+0000 7f72d8923440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:56.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:56.955+0000 7f72d8923440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:56.964 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-09-15T03:45:57.030 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:45:57.111 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:57.110+0000 7f672251d440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2024-09-15T03:45:57.111 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:57.110+0000 7f672251d440 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2024-09-15T03:45:57.115 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:57.115+0000 7f672251d440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:57.116 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:57.115+0000 7f672251d440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-15T03:45:57.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:57.247+0000 7f672251d440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:57.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:57.247+0000 7f672251d440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:57.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:57.247+0000 7f672251d440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:57.255 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-09-15T03:45:57.320 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:45:57.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:57.400+0000 7f1efd087440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2024-09-15T03:45:57.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:57.400+0000 7f1efd087440 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2024-09-15T03:45:57.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:57.405+0000 7f1efd087440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:57.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:57.405+0000 7f1efd087440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-15T03:45:57.542 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:57.542+0000 7f1efd087440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:45:57.545 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:57.544+0000 7f1efd087440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:57.545 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:45:57.544+0000 7f1efd087440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:45:57.554 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-09-15T03:45:57.618 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-15T03:45:57.619 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-15T03:45:57.619 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-15T03:45:57.650 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-15T03:45:57.650 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-15T03:45:57.721 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-15T03:45:57.721 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-15T03:45:57.791 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-15T03:45:57.792 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-15T03:45:57.863 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T03:45:57.863 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-15T03:45:57.897 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T03:45:57.897 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-15T03:45:57.968 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T03:45:57.969 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-15T03:45:58.039 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T03:45:58.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-15T03:45:58.108 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T03:45:58.108 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-15T03:45:58.175 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-15T03:45:58.175 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-09-15T03:45:58.203 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-15T03:45:58.203 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-09-15T03:45:58.269 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-15T03:45:58.270 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2024-09-15T03:45:58.335 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-15T03:45:58.336 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2024-09-15T03:45:58.402 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-15T03:45:58.402 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-15T03:45:58.418 INFO:tasks.ceph:Adding keys to all mons... 2024-09-15T03:45:58.419 DEBUG:teuthology.orchestra.run.smithi001:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T03:45:58.461 DEBUG:teuthology.orchestra.run.smithi067:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T03:45:58.464 DEBUG:teuthology.orchestra.run.smithi190:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T03:45:58.483 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.0] 2024-09-15T03:45:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBsWOZmFKsUAhAARME4SOq9k7cyqFN89He1dA== 2024-09-15T03:45:58.483 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.1] 2024-09-15T03:45:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBsWOZmZwGMDBAAEZUbwh7fTZoZTMvasMVnMA== 2024-09-15T03:45:58.483 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.2] 2024-09-15T03:45:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBsWOZmhopTGRAAPB9mDZ5Hb2ZBrDGUlZI/5Q== 2024-09-15T03:45:58.483 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.3] 2024-09-15T03:45:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBsWOZmVSIwJhAAhD7NZW4hepak0XnjsSIMiA== 2024-09-15T03:45:58.484 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr.x] 2024-09-15T03:45:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBqWOZmSEmUJhAA6NFyAsmBTONTsP16n9UrgQ== 2024-09-15T03:45:58.484 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.4] 2024-09-15T03:45:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBwWOZmZvHqAhAAyE3Yo3VjGaXdTiwPo9+Cbg== 2024-09-15T03:45:58.484 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.5] 2024-09-15T03:45:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBwWOZmcFQQFxAA2HOd1BQiwPxqtbesH4HG4g== 2024-09-15T03:45:58.484 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.6] 2024-09-15T03:45:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBwWOZmnkefKRAAw8XsnkD61sxjBuCK52qntQ== 2024-09-15T03:45:58.484 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.7] 2024-09-15T03:45:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBxWOZm1C0vABAAQIKlIyObWf+ql6q4YYj+Bg== 2024-09-15T03:45:58.484 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.8] 2024-09-15T03:45:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQB0WOZmAxMmGRAAyMMlznwCSMnBnEJbjdd76w== 2024-09-15T03:45:58.485 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.9] 2024-09-15T03:45:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQB0WOZm81T1LhAAHDmWZOJyPDJI0jcAZSoYVQ== 2024-09-15T03:45:58.485 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.10] 2024-09-15T03:45:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQB1WOZmrVqkBhAAx4CbchvSdQrn+huRgFc8kQ== 2024-09-15T03:45:58.485 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.11] 2024-09-15T03:45:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQB1WOZmRN3rFxAAr8aj/ao/aLsaqF1mQsVw6Q== 2024-09-15T03:45:58.485 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-09-15T03:45:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBqWOZmwJogKRAAHyJyzDGAXG554J1PC4r8aA== 2024-09-15T03:45:58.487 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.0] 2024-09-15T03:45:58.488 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBsWOZmFKsUAhAARME4SOq9k7cyqFN89He1dA== 2024-09-15T03:45:58.488 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.1] 2024-09-15T03:45:58.488 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBsWOZmZwGMDBAAEZUbwh7fTZoZTMvasMVnMA== 2024-09-15T03:45:58.488 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.2] 2024-09-15T03:45:58.488 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBsWOZmhopTGRAAPB9mDZ5Hb2ZBrDGUlZI/5Q== 2024-09-15T03:45:58.488 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.3] 2024-09-15T03:45:58.488 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBsWOZmVSIwJhAAhD7NZW4hepak0XnjsSIMiA== 2024-09-15T03:45:58.488 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr.x] 2024-09-15T03:45:58.488 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBqWOZmSEmUJhAA6NFyAsmBTONTsP16n9UrgQ== 2024-09-15T03:45:58.488 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.4] 2024-09-15T03:45:58.488 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBwWOZmZvHqAhAAyE3Yo3VjGaXdTiwPo9+Cbg== 2024-09-15T03:45:58.488 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.5] 2024-09-15T03:45:58.489 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBwWOZmcFQQFxAA2HOd1BQiwPxqtbesH4HG4g== 2024-09-15T03:45:58.489 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.6] 2024-09-15T03:45:58.489 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBwWOZmnkefKRAAw8XsnkD61sxjBuCK52qntQ== 2024-09-15T03:45:58.489 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.7] 2024-09-15T03:45:58.489 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBxWOZm1C0vABAAQIKlIyObWf+ql6q4YYj+Bg== 2024-09-15T03:45:58.489 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.8] 2024-09-15T03:45:58.489 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQB0WOZmAxMmGRAAyMMlznwCSMnBnEJbjdd76w== 2024-09-15T03:45:58.489 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.9] 2024-09-15T03:45:58.489 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQB0WOZm81T1LhAAHDmWZOJyPDJI0jcAZSoYVQ== 2024-09-15T03:45:58.489 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.10] 2024-09-15T03:45:58.489 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQB1WOZmrVqkBhAAx4CbchvSdQrn+huRgFc8kQ== 2024-09-15T03:45:58.489 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.11] 2024-09-15T03:45:58.489 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQB1WOZmRN3rFxAAr8aj/ao/aLsaqF1mQsVw6Q== 2024-09-15T03:45:58.489 INFO:teuthology.orchestra.run.smithi190.stdout:[client.0] 2024-09-15T03:45:58.490 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBqWOZmwJogKRAAHyJyzDGAXG554J1PC4r8aA== 2024-09-15T03:45:58.490 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.0] 2024-09-15T03:45:58.491 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBsWOZmFKsUAhAARME4SOq9k7cyqFN89He1dA== 2024-09-15T03:45:58.491 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.1] 2024-09-15T03:45:58.491 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBsWOZmZwGMDBAAEZUbwh7fTZoZTMvasMVnMA== 2024-09-15T03:45:58.491 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.2] 2024-09-15T03:45:58.491 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBsWOZmhopTGRAAPB9mDZ5Hb2ZBrDGUlZI/5Q== 2024-09-15T03:45:58.491 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.3] 2024-09-15T03:45:58.491 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBsWOZmVSIwJhAAhD7NZW4hepak0XnjsSIMiA== 2024-09-15T03:45:58.491 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.x] 2024-09-15T03:45:58.491 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBqWOZmSEmUJhAA6NFyAsmBTONTsP16n9UrgQ== 2024-09-15T03:45:58.491 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.4] 2024-09-15T03:45:58.491 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBwWOZmZvHqAhAAyE3Yo3VjGaXdTiwPo9+Cbg== 2024-09-15T03:45:58.491 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.5] 2024-09-15T03:45:58.491 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBwWOZmcFQQFxAA2HOd1BQiwPxqtbesH4HG4g== 2024-09-15T03:45:58.492 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.6] 2024-09-15T03:45:58.492 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBwWOZmnkefKRAAw8XsnkD61sxjBuCK52qntQ== 2024-09-15T03:45:58.492 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.7] 2024-09-15T03:45:58.492 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBxWOZm1C0vABAAQIKlIyObWf+ql6q4YYj+Bg== 2024-09-15T03:45:58.492 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.8] 2024-09-15T03:45:58.492 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB0WOZmAxMmGRAAyMMlznwCSMnBnEJbjdd76w== 2024-09-15T03:45:58.492 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.9] 2024-09-15T03:45:58.492 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB0WOZm81T1LhAAHDmWZOJyPDJI0jcAZSoYVQ== 2024-09-15T03:45:58.492 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.10] 2024-09-15T03:45:58.492 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB1WOZmrVqkBhAAx4CbchvSdQrn+huRgFc8kQ== 2024-09-15T03:45:58.492 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.11] 2024-09-15T03:45:58.492 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB1WOZmRN3rFxAAr8aj/ao/aLsaqF1mQsVw6Q== 2024-09-15T03:45:58.492 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-09-15T03:45:58.492 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBqWOZmwJogKRAAHyJyzDGAXG554J1PC4r8aA== 2024-09-15T03:45:58.493 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.528 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.536 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.587 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.590 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.640 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.645 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.693 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.695 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.744 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T03:45:58.746 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T03:45:58.748 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T03:45:58.795 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.798 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.801 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.848 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.850 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.852 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.898 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.900 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.903 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.949 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.952 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:58.954 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:59.001 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:59.003 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:59.005 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:59.051 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:59.054 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:59.056 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:59.101 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:59.104 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:59.106 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:59.152 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:59.154 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:59.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:45:59.202 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T03:45:59.204 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T03:45:59.206 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T03:45:59.255 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-15T03:45:59.255 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-15T03:45:59.280 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T03:45:59.372 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-15T03:45:59.397 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-15T03:45:59.423 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T03:45:59.516 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-15T03:45:59.541 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-15T03:45:59.566 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T03:45:59.658 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-15T03:45:59.682 DEBUG:teuthology.orchestra.run.smithi001:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:45:59.685 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:45:59.687 DEBUG:teuthology.orchestra.run.smithi190:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:45:59.737 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-15T03:45:59.738 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T03:45:59.738 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T03:45:59.741 INFO:tasks.ceph.mon.a:Started 2024-09-15T03:45:59.741 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T03:45:59.741 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T03:45:59.743 INFO:tasks.ceph.mon.b:Started 2024-09-15T03:45:59.743 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T03:45:59.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T03:45:59.780 INFO:tasks.ceph.mon.c:Started 2024-09-15T03:45:59.780 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-15T03:45:59.780 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-15T03:45:59.781 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-15T03:45:59.783 INFO:tasks.ceph.mgr.x:Started 2024-09-15T03:45:59.783 INFO:tasks.ceph:Setting crush tunables to default 2024-09-15T03:45:59.783 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster ceph osd crush tunables default 2024-09-15T03:46:09.936 INFO:teuthology.orchestra.run.smithi001.stderr:adjusted tunables profile to default 2024-09-15T03:46:09.948 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-15T03:46:09.948 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-15T03:46:09.948 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-15T03:46:09.975 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-15T03:46:09.975 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-15T03:46:09.979 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:09.978+0000 7fe0ae7bfec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T03:46:10.022 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:10.021+0000 7fe0ae7bfec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T03:46:10.042 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-15T03:46:10.042 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-15T03:46:10.074 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:10.073+0000 7fe0ae7bfec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T03:46:10.109 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-15T03:46:10.109 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-15T03:46:10.175 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T03:46:10.175 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-15T03:46:10.202 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T03:46:10.202 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-15T03:46:10.267 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T03:46:10.267 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-15T03:46:10.334 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T03:46:10.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-15T03:46:10.393 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:10.392+0000 7fe0ae7bfec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T03:46:10.401 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-15T03:46:10.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-09-15T03:46:10.431 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-15T03:46:10.431 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-09-15T03:46:10.498 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-15T03:46:10.499 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2024-09-15T03:46:10.505 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:10.505+0000 7fe0ae7bfec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T03:46:10.565 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-15T03:46:10.565 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2024-09-15T03:46:10.632 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new cd433a6a-c875-4e6e-b89c-938caecbae3a 0 2024-09-15T03:46:10.823 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:10.822+0000 7fe0ae7bfec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T03:46:10.849 INFO:teuthology.orchestra.run.smithi190.stdout:0 2024-09-15T03:46:10.860 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 789930f9-42bf-476b-9f09-a21bffe33a79 1 2024-09-15T03:46:11.015 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:11.013+0000 7fe0ae7bfec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T03:46:11.034 INFO:teuthology.orchestra.run.smithi190.stdout:1 2024-09-15T03:46:11.045 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 9a83fec5-0d63-461c-b516-289443b3882b 2 2024-09-15T03:46:11.155 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:11.153+0000 7fe0ae7bfec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T03:46:11.219 INFO:teuthology.orchestra.run.smithi190.stdout:2 2024-09-15T03:46:11.230 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new d944bf89-e183-496a-9ca3-7949741b4be4 3 2024-09-15T03:46:11.397 INFO:teuthology.orchestra.run.smithi190.stdout:3 2024-09-15T03:46:11.408 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 15ec2dbd-d099-4aff-94b2-1bf9dbb00b56 4 2024-09-15T03:46:11.576 INFO:teuthology.orchestra.run.smithi190.stdout:4 2024-09-15T03:46:11.586 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 1e45de07-746b-4612-9a52-d05509cae9a3 5 2024-09-15T03:46:11.757 INFO:teuthology.orchestra.run.smithi190.stdout:5 2024-09-15T03:46:11.767 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new ccf785f8-751d-473e-b17c-3df9346be9f8 6 2024-09-15T03:46:11.936 INFO:teuthology.orchestra.run.smithi190.stdout:6 2024-09-15T03:46:11.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new f9f59987-869d-42eb-9eb6-08c39e3afe6d 7 2024-09-15T03:46:12.117 INFO:teuthology.orchestra.run.smithi190.stdout:7 2024-09-15T03:46:12.127 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 69ddfe52-11df-4608-9a10-e8907c4cd87f 8 2024-09-15T03:46:12.215 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:12.213+0000 7fe0ae7bfec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T03:46:12.268 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:12.266+0000 7fe0ae7bfec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T03:46:12.297 INFO:teuthology.orchestra.run.smithi190.stdout:8 2024-09-15T03:46:12.308 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new f92e1609-2a1a-4814-9ac0-44a73373d229 9 2024-09-15T03:46:12.475 INFO:teuthology.orchestra.run.smithi190.stdout:9 2024-09-15T03:46:12.486 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 5f2e8ca3-8c65-4595-8e82-27dc3768ab99 10 2024-09-15T03:46:12.654 INFO:teuthology.orchestra.run.smithi190.stdout:10 2024-09-15T03:46:12.655 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:12.653+0000 7fe0ae7bfec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T03:46:12.665 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new cedf7df4-f875-42f5-a8cf-13cb82f05cc2 11 2024-09-15T03:46:12.740 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:12.739+0000 7fe0ae7bfec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T03:46:12.833 INFO:teuthology.orchestra.run.smithi190.stdout:11 2024-09-15T03:46:12.834 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:12.832+0000 7fe0ae7bfec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T03:46:12.844 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-15T03:46:12.844 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-15T03:46:12.846 INFO:tasks.ceph.osd.0:Started 2024-09-15T03:46:12.847 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-15T03:46:12.847 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-15T03:46:12.849 INFO:tasks.ceph.osd.1:Started 2024-09-15T03:46:12.849 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-15T03:46:12.849 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-15T03:46:12.851 INFO:tasks.ceph.osd.2:Started 2024-09-15T03:46:12.852 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-15T03:46:12.852 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-15T03:46:12.854 INFO:tasks.ceph.osd.3:Started 2024-09-15T03:46:12.854 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-15T03:46:12.854 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-15T03:46:12.857 INFO:tasks.ceph.osd.4:Started 2024-09-15T03:46:12.857 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-15T03:46:12.857 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-15T03:46:12.859 INFO:tasks.ceph.osd.5:Started 2024-09-15T03:46:12.859 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-15T03:46:12.859 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-15T03:46:12.861 INFO:tasks.ceph.osd.6:Started 2024-09-15T03:46:12.862 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-15T03:46:12.862 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-15T03:46:12.864 INFO:tasks.ceph.osd.7:Started 2024-09-15T03:46:12.864 INFO:tasks.ceph.osd.8:Restarting daemon 2024-09-15T03:46:12.864 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-09-15T03:46:12.867 INFO:tasks.ceph.osd.8:Started 2024-09-15T03:46:12.867 INFO:tasks.ceph.osd.9:Restarting daemon 2024-09-15T03:46:12.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-09-15T03:46:12.870 INFO:tasks.ceph.osd.9:Started 2024-09-15T03:46:12.870 INFO:tasks.ceph.osd.10:Restarting daemon 2024-09-15T03:46:12.870 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-09-15T03:46:12.872 INFO:tasks.ceph.osd.10:Started 2024-09-15T03:46:12.872 INFO:tasks.ceph.osd.11:Restarting daemon 2024-09-15T03:46:12.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-09-15T03:46:12.874 INFO:tasks.ceph.osd.11:Started 2024-09-15T03:46:12.875 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:46:12.889 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:12.887+0000 7fe0ae7bfec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T03:46:12.936 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:12.935+0000 7f624e325440 -1 Falling back to public interface 2024-09-15T03:46:12.938 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:46:12.936+0000 7f6fa660a440 -1 Falling back to public interface 2024-09-15T03:46:12.939 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:12.937+0000 7f63ced2a440 -1 Falling back to public interface 2024-09-15T03:46:12.941 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:12.939+0000 7f38d4f0f440 -1 Falling back to public interface 2024-09-15T03:46:12.941 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:12.940+0000 7f433e323440 -1 Falling back to public interface 2024-09-15T03:46:12.941 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:12.941+0000 7f522af26440 -1 Falling back to public interface 2024-09-15T03:46:12.943 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:12.942+0000 7f0a96b1f440 -1 Falling back to public interface 2024-09-15T03:46:12.944 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:12.943+0000 7f977a04f440 -1 Falling back to public interface 2024-09-15T03:46:12.948 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:12.947+0000 7fcd20e17440 -1 Falling back to public interface 2024-09-15T03:46:12.950 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:12.949+0000 7fac91d1d440 -1 Falling back to public interface 2024-09-15T03:46:12.952 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:12.952+0000 7f234412b440 -1 Falling back to public interface 2024-09-15T03:46:12.953 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:12.952+0000 7f433e323440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:46:12.953 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:12.952+0000 7f433e323440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.953 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:12.952+0000 7f433e323440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.954 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:12.953+0000 7f624e325440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:46:12.954 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:12.953+0000 7f624e325440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.954 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:12.953+0000 7f624e325440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.956 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:12.955+0000 7f522af26440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:46:12.956 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:12.955+0000 7f522af26440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.956 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:12.955+0000 7f977a04f440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:46:12.957 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:12.955+0000 7f522af26440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.957 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:12.955+0000 7f977a04f440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.957 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:12.956+0000 7f977a04f440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.958 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:46:12.956+0000 7f6fa660a440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:46:12.958 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:46:12.956+0000 7f6fa660a440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.958 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:46:12.956+0000 7f6fa660a440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.958 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:12.957+0000 7f433e323440 -1 osd.9 0 log_to_monitors true 2024-09-15T03:46:12.959 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:12.957+0000 7f38d4f0f440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:46:12.959 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:12.957+0000 7f38d4f0f440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.959 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:12.957+0000 7f38d4f0f440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.960 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:12.957+0000 7f63ced2a440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:46:12.960 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:12.957+0000 7f63ced2a440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.960 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:12.957+0000 7f63ced2a440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.960 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:12.959+0000 7f0a96b1f440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:46:12.961 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:12.960+0000 7f0a96b1f440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.961 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:12.960+0000 7f0a96b1f440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.961 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:12.961+0000 7f624e325440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:46:12.962 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:12.961+0000 7f977a04f440 -1 osd.10 0 log_to_monitors true 2024-09-15T03:46:12.964 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:12.963+0000 7fcd20e17440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:46:12.964 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:12.963+0000 7fcd20e17440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.965 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:12.963+0000 7fcd20e17440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.967 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:46:12.965+0000 7f6fa660a440 -1 osd.6 0 log_to_monitors true 2024-09-15T03:46:12.967 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:12.965+0000 7f6793d29440 -1 Falling back to public interface 2024-09-15T03:46:12.968 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:12.967+0000 7f522af26440 -1 osd.8 0 log_to_monitors true 2024-09-15T03:46:12.968 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:12.968+0000 7f0a96b1f440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:46:12.970 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:12.969+0000 7fcd20e17440 -1 osd.3 0 log_to_monitors true 2024-09-15T03:46:12.971 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:12.971+0000 7f234412b440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:46:12.972 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:12.971+0000 7f234412b440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.972 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:12.971+0000 7f234412b440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.972 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:12.971+0000 7fac91d1d440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:46:12.972 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:12.971+0000 7fac91d1d440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.972 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:12.971+0000 7fac91d1d440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.973 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:12.971+0000 7f63ced2a440 -1 osd.4 0 log_to_monitors true 2024-09-15T03:46:12.973 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:12.971+0000 7f38d4f0f440 -1 osd.7 0 log_to_monitors true 2024-09-15T03:46:12.976 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:12.975+0000 7f234412b440 -1 osd.11 0 log_to_monitors true 2024-09-15T03:46:12.985 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:12.984+0000 7fac91d1d440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:46:12.991 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:12.989+0000 7f6793d29440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:46:12.991 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:12.990+0000 7f6793d29440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.991 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:12.990+0000 7f6793d29440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:46:12.998 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:12.996+0000 7f6793d29440 -1 osd.5 0 log_to_monitors true 2024-09-15T03:46:13.028 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:13.026+0000 7fe0ae7bfec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T03:46:13.082 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:13.080+0000 7fe0ae7bfec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T03:46:13.082 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:46:13.083 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":14,"fsid":"f27b540e-e5e3-4e95-afe8-c95ee66b0a62","created":"2024-09-15T03:46:09.843094+0000","modified":"2024-09-15T03:46:12.830218+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T03:46:12.830218+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"cd433a6a-c875-4e6e-b89c-938caecbae3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"789930f9-42bf-476b-9f09-a21bffe33a79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9a83fec5-0d63-461c-b516-289443b3882b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d944bf89-e183-496a-9ca3-7949741b4be4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"15ec2dbd-d099-4aff-94b2-1bf9dbb00b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e45de07-746b-4612-9a52-d05509cae9a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ccf785f8-751d-473e-b17c-3df9346be9f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f9f59987-869d-42eb-9eb6-08c39e3afe6d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"69ddfe52-11df-4608-9a10-e8907c4cd87f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"f92e1609-2a1a-4814-9ac0-44a73373d229","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"5f2e8ca3-8c65-4595-8e82-27dc3768ab99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"cedf7df4-f875-42f5-a8cf-13cb82f05cc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:46:13.093 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-15T03:46:13.093 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-15T03:46:13.127 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:13.126+0000 7fe0ae7bfec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T03:46:13.225 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:13.223+0000 7fe0ae7bfec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T03:46:13.275 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:13.274+0000 7fe0ae7bfec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T03:46:13.328 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:13.326+0000 7fe0ae7bfec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T03:46:13.397 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:13.396+0000 7fe0ae7bfec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T03:46:13.442 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:13.440+0000 7fe0ae7bfec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T03:46:13.487 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:13.485+0000 7fe0ae7bfec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T03:46:14.133 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:14.131+0000 7fe0ae7bfec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T03:46:14.304 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T03:46:14.473 INFO:teuthology.misc.health.smithi001.stdout: 2024-09-15T03:46:14.473 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":15,"fsid":"f27b540e-e5e3-4e95-afe8-c95ee66b0a62","created":"2024-09-15T03:46:09.843094+0000","modified":"2024-09-15T03:46:13.859110+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T03:46:12.830218+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"cd433a6a-c875-4e6e-b89c-938caecbae3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"789930f9-42bf-476b-9f09-a21bffe33a79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9a83fec5-0d63-461c-b516-289443b3882b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d944bf89-e183-496a-9ca3-7949741b4be4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"15ec2dbd-d099-4aff-94b2-1bf9dbb00b56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1e45de07-746b-4612-9a52-d05509cae9a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ccf785f8-751d-473e-b17c-3df9346be9f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f9f59987-869d-42eb-9eb6-08c39e3afe6d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"69ddfe52-11df-4608-9a10-e8907c4cd87f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"f92e1609-2a1a-4814-9ac0-44a73373d229","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"5f2e8ca3-8c65-4595-8e82-27dc3768ab99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"cedf7df4-f875-42f5-a8cf-13cb82f05cc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:46:14.482 DEBUG:teuthology.misc:0 of 12 OSDs are up 2024-09-15T03:46:14.868 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:14.866+0000 7f678eff7640 -1 osd.5 0 waiting for initial osdmap 2024-09-15T03:46:14.871 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:14.870+0000 7f678b68c640 -1 osd.5 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:46:14.875 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:14.874+0000 7f62495f7640 -1 osd.0 0 waiting for initial osdmap 2024-09-15T03:46:14.876 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:14.874+0000 7f43395f7640 -1 osd.9 0 waiting for initial osdmap 2024-09-15T03:46:14.876 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:14.874+0000 7f0a91df7640 -1 osd.1 0 waiting for initial osdmap 2024-09-15T03:46:14.876 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:14.874+0000 7fcd1bdf7640 -1 osd.3 0 waiting for initial osdmap 2024-09-15T03:46:14.876 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:14.874+0000 7fac8cff7640 -1 osd.2 0 waiting for initial osdmap 2024-09-15T03:46:14.877 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:14.873+0000 7f63c9ff7640 -1 osd.4 0 waiting for initial osdmap 2024-09-15T03:46:14.877 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:14.876+0000 7f52261f7640 -1 osd.8 0 waiting for initial osdmap 2024-09-15T03:46:14.878 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:46:14.873+0000 7f6fa15f7640 -1 osd.6 0 waiting for initial osdmap 2024-09-15T03:46:14.878 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:14.876+0000 7f233f3f7640 -1 osd.11 0 waiting for initial osdmap 2024-09-15T03:46:14.878 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:14.875+0000 7f38d07f7640 -1 osd.7 0 waiting for initial osdmap 2024-09-15T03:46:14.878 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:14.876+0000 7f97759f7640 -1 osd.10 0 waiting for initial osdmap 2024-09-15T03:46:14.879 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:14.878+0000 7f38ccff0640 -1 osd.7 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:46:14.880 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:14.878+0000 7f63c67f0640 -1 osd.4 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:46:14.880 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:46:14.878+0000 7f6f9ddf0640 -1 osd.6 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:46:14.881 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:14.879+0000 7f6245c8c640 -1 osd.0 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:46:14.881 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:14.879+0000 7fcd185f0640 -1 osd.3 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:46:14.881 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:14.879+0000 7f0a8e5f0640 -1 osd.1 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:46:14.881 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:14.880+0000 7fac897f0640 -1 osd.2 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:46:14.882 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:14.881+0000 7f97721f0640 -1 osd.10 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:46:14.882 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:14.881+0000 7f522288c640 -1 osd.8 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:46:14.882 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:14.881+0000 7f233bbf0640 -1 osd.11 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:46:14.883 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:14.881+0000 7f4335df0640 -1 osd.9 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:46:15.150 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:15.148+0000 7fe092e38640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-15T03:46:15.151 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-09-15T03:46:15.149+0000 7fe092e38640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-15T03:46:21.694 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T03:46:21.973 INFO:teuthology.misc.health.smithi001.stdout: 2024-09-15T03:46:21.973 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":20,"fsid":"f27b540e-e5e3-4e95-afe8-c95ee66b0a62","created":"2024-09-15T03:46:09.843094+0000","modified":"2024-09-15T03:46:18.897983+0000","last_up_change":"2024-09-15T03:46:15.871985+0000","last_in_change":"2024-09-15T03:46:12.830218+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:46:16.185530+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"cd433a6a-c875-4e6e-b89c-938caecbae3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":28908},{"type":"v1","addr":"172.21.15.1:6801","nonce":28908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":28908},{"type":"v1","addr":"172.21.15.1:6803","nonce":28908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":28908},{"type":"v1","addr":"172.21.15.1:6807","nonce":28908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":28908},{"type":"v1","addr":"172.21.15.1:6805","nonce":28908}]},"public_addr":"172.21.15.1:6801/28908","cluster_addr":"172.21.15.1:6803/28908","heartbeat_back_addr":"172.21.15.1:6807/28908","heartbeat_front_addr":"172.21.15.1:6805/28908","state":["exists","up"]},{"osd":1,"uuid":"789930f9-42bf-476b-9f09-a21bffe33a79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":28913},{"type":"v1","addr":"172.21.15.1:6809","nonce":28913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":28913},{"type":"v1","addr":"172.21.15.1:6811","nonce":28913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":28913},{"type":"v1","addr":"172.21.15.1:6815","nonce":28913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":28913},{"type":"v1","addr":"172.21.15.1:6813","nonce":28913}]},"public_addr":"172.21.15.1:6809/28913","cluster_addr":"172.21.15.1:6811/28913","heartbeat_back_addr":"172.21.15.1:6815/28913","heartbeat_front_addr":"172.21.15.1:6813/28913","state":["exists","up"]},{"osd":2,"uuid":"9a83fec5-0d63-461c-b516-289443b3882b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":28911},{"type":"v1","addr":"172.21.15.1:6823","nonce":28911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":28911},{"type":"v1","addr":"172.21.15.1:6827","nonce":28911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":28911},{"type":"v1","addr":"172.21.15.1:6831","nonce":28911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":28911},{"type":"v1","addr":"172.21.15.1:6829","nonce":28911}]},"public_addr":"172.21.15.1:6823/28911","cluster_addr":"172.21.15.1:6827/28911","heartbeat_back_addr":"172.21.15.1:6831/28911","heartbeat_front_addr":"172.21.15.1:6829/28911","state":["exists","up"]},{"osd":3,"uuid":"d944bf89-e183-496a-9ca3-7949741b4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":28914},{"type":"v1","addr":"172.21.15.1:6817","nonce":28914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":28914},{"type":"v1","addr":"172.21.15.1:6819","nonce":28914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":28914},{"type":"v1","addr":"172.21.15.1:6826","nonce":28914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":28914},{"type":"v1","addr":"172.21.15.1:6822","nonce":28914}]},"public_addr":"172.21.15.1:6817/28914","cluster_addr":"172.21.15.1:6819/28914","heartbeat_back_addr":"172.21.15.1:6826/28914","heartbeat_front_addr":"172.21.15.1:6822/28914","state":["exists","up"]},{"osd":4,"uuid":"15ec2dbd-d099-4aff-94b2-1bf9dbb00b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":28468},{"type":"v1","addr":"172.21.15.67:6805","nonce":28468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6807","nonce":28468},{"type":"v1","addr":"172.21.15.67:6809","nonce":28468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":28468},{"type":"v1","addr":"172.21.15.67:6815","nonce":28468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":28468},{"type":"v1","addr":"172.21.15.67:6813","nonce":28468}]},"public_addr":"172.21.15.67:6805/28468","cluster_addr":"172.21.15.67:6809/28468","heartbeat_back_addr":"172.21.15.67:6815/28468","heartbeat_front_addr":"172.21.15.67:6813/28468","state":["exists","up"]},{"osd":5,"uuid":"1e45de07-746b-4612-9a52-d05509cae9a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":28495},{"type":"v1","addr":"172.21.15.67:6825","nonce":28495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":28495},{"type":"v1","addr":"172.21.15.67:6827","nonce":28495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":28495},{"type":"v1","addr":"172.21.15.67:6831","nonce":28495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":28495},{"type":"v1","addr":"172.21.15.67:6829","nonce":28495}]},"public_addr":"172.21.15.67:6825/28495","cluster_addr":"172.21.15.67:6827/28495","heartbeat_back_addr":"172.21.15.67:6831/28495","heartbeat_front_addr":"172.21.15.67:6829/28495","state":["exists","up"]},{"osd":6,"uuid":"ccf785f8-751d-473e-b17c-3df9346be9f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":28469},{"type":"v1","addr":"172.21.15.67:6801","nonce":28469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":28469},{"type":"v1","addr":"172.21.15.67:6804","nonce":28469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":28469},{"type":"v1","addr":"172.21.15.67:6812","nonce":28469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":28469},{"type":"v1","addr":"172.21.15.67:6808","nonce":28469}]},"public_addr":"172.21.15.67:6801/28469","cluster_addr":"172.21.15.67:6804/28469","heartbeat_back_addr":"172.21.15.67:6812/28469","heartbeat_front_addr":"172.21.15.67:6808/28469","state":["exists","up"]},{"osd":7,"uuid":"f9f59987-869d-42eb-9eb6-08c39e3afe6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":28470},{"type":"v1","addr":"172.21.15.67:6817","nonce":28470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":28470},{"type":"v1","addr":"172.21.15.67:6819","nonce":28470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":28470},{"type":"v1","addr":"172.21.15.67:6823","nonce":28470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":28470},{"type":"v1","addr":"172.21.15.67:6821","nonce":28470}]},"public_addr":"172.21.15.67:6817/28470","cluster_addr":"172.21.15.67:6819/28470","heartbeat_back_addr":"172.21.15.67:6823/28470","heartbeat_front_addr":"172.21.15.67:6821/28470","state":["exists","up"]},{"osd":8,"uuid":"69ddfe52-11df-4608-9a10-e8907c4cd87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":29472},{"type":"v1","addr":"172.21.15.190:6809","nonce":29472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":29472},{"type":"v1","addr":"172.21.15.190:6811","nonce":29472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":29472},{"type":"v1","addr":"172.21.15.190:6815","nonce":29472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":29472},{"type":"v1","addr":"172.21.15.190:6813","nonce":29472}]},"public_addr":"172.21.15.190:6809/29472","cluster_addr":"172.21.15.190:6811/29472","heartbeat_back_addr":"172.21.15.190:6815/29472","heartbeat_front_addr":"172.21.15.190:6813/29472","state":["exists","up"]},{"osd":9,"uuid":"f92e1609-2a1a-4814-9ac0-44a73373d229","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":29473},{"type":"v1","addr":"172.21.15.190:6801","nonce":29473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":29473},{"type":"v1","addr":"172.21.15.190:6803","nonce":29473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":29473},{"type":"v1","addr":"172.21.15.190:6807","nonce":29473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":29473},{"type":"v1","addr":"172.21.15.190:6805","nonce":29473}]},"public_addr":"172.21.15.190:6801/29473","cluster_addr":"172.21.15.190:6803/29473","heartbeat_back_addr":"172.21.15.190:6807/29473","heartbeat_front_addr":"172.21.15.190:6805/29473","state":["exists","up"]},{"osd":10,"uuid":"5f2e8ca3-8c65-4595-8e82-27dc3768ab99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":29474},{"type":"v1","addr":"172.21.15.190:6817","nonce":29474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":29474},{"type":"v1","addr":"172.21.15.190:6819","nonce":29474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":29474},{"type":"v1","addr":"172.21.15.190:6823","nonce":29474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":29474},{"type":"v1","addr":"172.21.15.190:6821","nonce":29474}]},"public_addr":"172.21.15.190:6817/29474","cluster_addr":"172.21.15.190:6819/29474","heartbeat_back_addr":"172.21.15.190:6823/29474","heartbeat_front_addr":"172.21.15.190:6821/29474","state":["exists","up"]},{"osd":11,"uuid":"cedf7df4-f875-42f5-a8cf-13cb82f05cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":29475},{"type":"v1","addr":"172.21.15.190:6825","nonce":29475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":29475},{"type":"v1","addr":"172.21.15.190:6827","nonce":29475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":29475},{"type":"v1","addr":"172.21.15.190:6831","nonce":29475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":29475},{"type":"v1","addr":"172.21.15.190:6829","nonce":29475}]},"public_addr":"172.21.15.190:6825/29475","cluster_addr":"172.21.15.190:6827/29475","heartbeat_back_addr":"172.21.15.190:6831/29475","heartbeat_front_addr":"172.21.15.190:6829/29475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.943165+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.946049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.981045+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:14.019932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.936197+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.982429+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.953833+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.967611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.996438+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.926618+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.993934+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:14.022484+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:46:21.983 DEBUG:teuthology.misc:12 of 12 OSDs are up 2024-09-15T03:46:21.983 INFO:tasks.ceph:Creating RBD pool 2024-09-15T03:46:21.984 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-15T03:46:23.032 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'rbd' created 2024-09-15T03:46:23.047 DEBUG:teuthology.orchestra.run.smithi001:> rbd --cluster ceph pool init rbd 2024-09-15T03:46:26.050 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-15T03:46:26.050 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-15T03:46:26.050 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-15T03:46:26.051 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-15T03:46:26.051 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-15T03:46:26.383 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:46:26.394 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":28283},{"type":"v1","addr":"172.21.15.67:6833","nonce":28283}]},"active_addr":"172.21.15.67:6833/28283","active_change":"2024-09-15T03:46:14.135626+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":2921885263}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":632876599}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":4175073138}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":3220944830}]}]} 2024-09-15T03:46:26.397 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-15T03:46:26.397 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-15T03:46:26.397 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:46:26.668 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:46:26.668 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":24,"fsid":"f27b540e-e5e3-4e95-afe8-c95ee66b0a62","created":"2024-09-15T03:46:09.843094+0000","modified":"2024-09-15T03:46:26.038725+0000","last_up_change":"2024-09-15T03:46:15.871985+0000","last_in_change":"2024-09-15T03:46:12.830218+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:46:16.185530+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:46:22.284406+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":24,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"cd433a6a-c875-4e6e-b89c-938caecbae3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":28908},{"type":"v1","addr":"172.21.15.1:6801","nonce":28908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":28908},{"type":"v1","addr":"172.21.15.1:6803","nonce":28908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":28908},{"type":"v1","addr":"172.21.15.1:6807","nonce":28908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":28908},{"type":"v1","addr":"172.21.15.1:6805","nonce":28908}]},"public_addr":"172.21.15.1:6801/28908","cluster_addr":"172.21.15.1:6803/28908","heartbeat_back_addr":"172.21.15.1:6807/28908","heartbeat_front_addr":"172.21.15.1:6805/28908","state":["exists","up"]},{"osd":1,"uuid":"789930f9-42bf-476b-9f09-a21bffe33a79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":28913},{"type":"v1","addr":"172.21.15.1:6809","nonce":28913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":28913},{"type":"v1","addr":"172.21.15.1:6811","nonce":28913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":28913},{"type":"v1","addr":"172.21.15.1:6815","nonce":28913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":28913},{"type":"v1","addr":"172.21.15.1:6813","nonce":28913}]},"public_addr":"172.21.15.1:6809/28913","cluster_addr":"172.21.15.1:6811/28913","heartbeat_back_addr":"172.21.15.1:6815/28913","heartbeat_front_addr":"172.21.15.1:6813/28913","state":["exists","up"]},{"osd":2,"uuid":"9a83fec5-0d63-461c-b516-289443b3882b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":28911},{"type":"v1","addr":"172.21.15.1:6823","nonce":28911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":28911},{"type":"v1","addr":"172.21.15.1:6827","nonce":28911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":28911},{"type":"v1","addr":"172.21.15.1:6831","nonce":28911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":28911},{"type":"v1","addr":"172.21.15.1:6829","nonce":28911}]},"public_addr":"172.21.15.1:6823/28911","cluster_addr":"172.21.15.1:6827/28911","heartbeat_back_addr":"172.21.15.1:6831/28911","heartbeat_front_addr":"172.21.15.1:6829/28911","state":["exists","up"]},{"osd":3,"uuid":"d944bf89-e183-496a-9ca3-7949741b4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":28914},{"type":"v1","addr":"172.21.15.1:6817","nonce":28914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":28914},{"type":"v1","addr":"172.21.15.1:6819","nonce":28914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":28914},{"type":"v1","addr":"172.21.15.1:6826","nonce":28914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":28914},{"type":"v1","addr":"172.21.15.1:6822","nonce":28914}]},"public_addr":"172.21.15.1:6817/28914","cluster_addr":"172.21.15.1:6819/28914","heartbeat_back_addr":"172.21.15.1:6826/28914","heartbeat_front_addr":"172.21.15.1:6822/28914","state":["exists","up"]},{"osd":4,"uuid":"15ec2dbd-d099-4aff-94b2-1bf9dbb00b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":28468},{"type":"v1","addr":"172.21.15.67:6805","nonce":28468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6807","nonce":28468},{"type":"v1","addr":"172.21.15.67:6809","nonce":28468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":28468},{"type":"v1","addr":"172.21.15.67:6815","nonce":28468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":28468},{"type":"v1","addr":"172.21.15.67:6813","nonce":28468}]},"public_addr":"172.21.15.67:6805/28468","cluster_addr":"172.21.15.67:6809/28468","heartbeat_back_addr":"172.21.15.67:6815/28468","heartbeat_front_addr":"172.21.15.67:6813/28468","state":["exists","up"]},{"osd":5,"uuid":"1e45de07-746b-4612-9a52-d05509cae9a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":28495},{"type":"v1","addr":"172.21.15.67:6825","nonce":28495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":28495},{"type":"v1","addr":"172.21.15.67:6827","nonce":28495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":28495},{"type":"v1","addr":"172.21.15.67:6831","nonce":28495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":28495},{"type":"v1","addr":"172.21.15.67:6829","nonce":28495}]},"public_addr":"172.21.15.67:6825/28495","cluster_addr":"172.21.15.67:6827/28495","heartbeat_back_addr":"172.21.15.67:6831/28495","heartbeat_front_addr":"172.21.15.67:6829/28495","state":["exists","up"]},{"osd":6,"uuid":"ccf785f8-751d-473e-b17c-3df9346be9f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":28469},{"type":"v1","addr":"172.21.15.67:6801","nonce":28469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":28469},{"type":"v1","addr":"172.21.15.67:6804","nonce":28469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":28469},{"type":"v1","addr":"172.21.15.67:6812","nonce":28469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":28469},{"type":"v1","addr":"172.21.15.67:6808","nonce":28469}]},"public_addr":"172.21.15.67:6801/28469","cluster_addr":"172.21.15.67:6804/28469","heartbeat_back_addr":"172.21.15.67:6812/28469","heartbeat_front_addr":"172.21.15.67:6808/28469","state":["exists","up"]},{"osd":7,"uuid":"f9f59987-869d-42eb-9eb6-08c39e3afe6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":28470},{"type":"v1","addr":"172.21.15.67:6817","nonce":28470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":28470},{"type":"v1","addr":"172.21.15.67:6819","nonce":28470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":28470},{"type":"v1","addr":"172.21.15.67:6823","nonce":28470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":28470},{"type":"v1","addr":"172.21.15.67:6821","nonce":28470}]},"public_addr":"172.21.15.67:6817/28470","cluster_addr":"172.21.15.67:6819/28470","heartbeat_back_addr":"172.21.15.67:6823/28470","heartbeat_front_addr":"172.21.15.67:6821/28470","state":["exists","up"]},{"osd":8,"uuid":"69ddfe52-11df-4608-9a10-e8907c4cd87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":29472},{"type":"v1","addr":"172.21.15.190:6809","nonce":29472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":29472},{"type":"v1","addr":"172.21.15.190:6811","nonce":29472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":29472},{"type":"v1","addr":"172.21.15.190:6815","nonce":29472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":29472},{"type":"v1","addr":"172.21.15.190:6813","nonce":29472}]},"public_addr":"172.21.15.190:6809/29472","cluster_addr":"172.21.15.190:6811/29472","heartbeat_back_addr":"172.21.15.190:6815/29472","heartbeat_front_addr":"172.21.15.190:6813/29472","state":["exists","up"]},{"osd":9,"uuid":"f92e1609-2a1a-4814-9ac0-44a73373d229","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":29473},{"type":"v1","addr":"172.21.15.190:6801","nonce":29473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":29473},{"type":"v1","addr":"172.21.15.190:6803","nonce":29473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":29473},{"type":"v1","addr":"172.21.15.190:6807","nonce":29473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":29473},{"type":"v1","addr":"172.21.15.190:6805","nonce":29473}]},"public_addr":"172.21.15.190:6801/29473","cluster_addr":"172.21.15.190:6803/29473","heartbeat_back_addr":"172.21.15.190:6807/29473","heartbeat_front_addr":"172.21.15.190:6805/29473","state":["exists","up"]},{"osd":10,"uuid":"5f2e8ca3-8c65-4595-8e82-27dc3768ab99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":29474},{"type":"v1","addr":"172.21.15.190:6817","nonce":29474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":29474},{"type":"v1","addr":"172.21.15.190:6819","nonce":29474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":29474},{"type":"v1","addr":"172.21.15.190:6823","nonce":29474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":29474},{"type":"v1","addr":"172.21.15.190:6821","nonce":29474}]},"public_addr":"172.21.15.190:6817/29474","cluster_addr":"172.21.15.190:6819/29474","heartbeat_back_addr":"172.21.15.190:6823/29474","heartbeat_front_addr":"172.21.15.190:6821/29474","state":["exists","up"]},{"osd":11,"uuid":"cedf7df4-f875-42f5-a8cf-13cb82f05cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":29475},{"type":"v1","addr":"172.21.15.190:6825","nonce":29475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":29475},{"type":"v1","addr":"172.21.15.190:6827","nonce":29475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":29475},{"type":"v1","addr":"172.21.15.190:6831","nonce":29475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":29475},{"type":"v1","addr":"172.21.15.190:6829","nonce":29475}]},"public_addr":"172.21.15.190:6825/29475","cluster_addr":"172.21.15.190:6827/29475","heartbeat_back_addr":"172.21.15.190:6831/29475","heartbeat_front_addr":"172.21.15.190:6829/29475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.943165+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.946049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.981045+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:14.019932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.936197+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.982429+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.953833+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.967611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.996438+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.926618+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.993934+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:14.022484+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:46:26.680 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-15T03:46:26.680 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:46:26.953 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:46:26.954 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":24,"fsid":"f27b540e-e5e3-4e95-afe8-c95ee66b0a62","created":"2024-09-15T03:46:09.843094+0000","modified":"2024-09-15T03:46:26.038725+0000","last_up_change":"2024-09-15T03:46:15.871985+0000","last_in_change":"2024-09-15T03:46:12.830218+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:46:16.185530+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:46:22.284406+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":24,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"cd433a6a-c875-4e6e-b89c-938caecbae3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":28908},{"type":"v1","addr":"172.21.15.1:6801","nonce":28908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":28908},{"type":"v1","addr":"172.21.15.1:6803","nonce":28908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":28908},{"type":"v1","addr":"172.21.15.1:6807","nonce":28908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":28908},{"type":"v1","addr":"172.21.15.1:6805","nonce":28908}]},"public_addr":"172.21.15.1:6801/28908","cluster_addr":"172.21.15.1:6803/28908","heartbeat_back_addr":"172.21.15.1:6807/28908","heartbeat_front_addr":"172.21.15.1:6805/28908","state":["exists","up"]},{"osd":1,"uuid":"789930f9-42bf-476b-9f09-a21bffe33a79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":28913},{"type":"v1","addr":"172.21.15.1:6809","nonce":28913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":28913},{"type":"v1","addr":"172.21.15.1:6811","nonce":28913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":28913},{"type":"v1","addr":"172.21.15.1:6815","nonce":28913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":28913},{"type":"v1","addr":"172.21.15.1:6813","nonce":28913}]},"public_addr":"172.21.15.1:6809/28913","cluster_addr":"172.21.15.1:6811/28913","heartbeat_back_addr":"172.21.15.1:6815/28913","heartbeat_front_addr":"172.21.15.1:6813/28913","state":["exists","up"]},{"osd":2,"uuid":"9a83fec5-0d63-461c-b516-289443b3882b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":28911},{"type":"v1","addr":"172.21.15.1:6823","nonce":28911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":28911},{"type":"v1","addr":"172.21.15.1:6827","nonce":28911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":28911},{"type":"v1","addr":"172.21.15.1:6831","nonce":28911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":28911},{"type":"v1","addr":"172.21.15.1:6829","nonce":28911}]},"public_addr":"172.21.15.1:6823/28911","cluster_addr":"172.21.15.1:6827/28911","heartbeat_back_addr":"172.21.15.1:6831/28911","heartbeat_front_addr":"172.21.15.1:6829/28911","state":["exists","up"]},{"osd":3,"uuid":"d944bf89-e183-496a-9ca3-7949741b4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":28914},{"type":"v1","addr":"172.21.15.1:6817","nonce":28914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":28914},{"type":"v1","addr":"172.21.15.1:6819","nonce":28914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":28914},{"type":"v1","addr":"172.21.15.1:6826","nonce":28914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":28914},{"type":"v1","addr":"172.21.15.1:6822","nonce":28914}]},"public_addr":"172.21.15.1:6817/28914","cluster_addr":"172.21.15.1:6819/28914","heartbeat_back_addr":"172.21.15.1:6826/28914","heartbeat_front_addr":"172.21.15.1:6822/28914","state":["exists","up"]},{"osd":4,"uuid":"15ec2dbd-d099-4aff-94b2-1bf9dbb00b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":28468},{"type":"v1","addr":"172.21.15.67:6805","nonce":28468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6807","nonce":28468},{"type":"v1","addr":"172.21.15.67:6809","nonce":28468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":28468},{"type":"v1","addr":"172.21.15.67:6815","nonce":28468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":28468},{"type":"v1","addr":"172.21.15.67:6813","nonce":28468}]},"public_addr":"172.21.15.67:6805/28468","cluster_addr":"172.21.15.67:6809/28468","heartbeat_back_addr":"172.21.15.67:6815/28468","heartbeat_front_addr":"172.21.15.67:6813/28468","state":["exists","up"]},{"osd":5,"uuid":"1e45de07-746b-4612-9a52-d05509cae9a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":28495},{"type":"v1","addr":"172.21.15.67:6825","nonce":28495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":28495},{"type":"v1","addr":"172.21.15.67:6827","nonce":28495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":28495},{"type":"v1","addr":"172.21.15.67:6831","nonce":28495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":28495},{"type":"v1","addr":"172.21.15.67:6829","nonce":28495}]},"public_addr":"172.21.15.67:6825/28495","cluster_addr":"172.21.15.67:6827/28495","heartbeat_back_addr":"172.21.15.67:6831/28495","heartbeat_front_addr":"172.21.15.67:6829/28495","state":["exists","up"]},{"osd":6,"uuid":"ccf785f8-751d-473e-b17c-3df9346be9f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":28469},{"type":"v1","addr":"172.21.15.67:6801","nonce":28469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":28469},{"type":"v1","addr":"172.21.15.67:6804","nonce":28469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":28469},{"type":"v1","addr":"172.21.15.67:6812","nonce":28469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":28469},{"type":"v1","addr":"172.21.15.67:6808","nonce":28469}]},"public_addr":"172.21.15.67:6801/28469","cluster_addr":"172.21.15.67:6804/28469","heartbeat_back_addr":"172.21.15.67:6812/28469","heartbeat_front_addr":"172.21.15.67:6808/28469","state":["exists","up"]},{"osd":7,"uuid":"f9f59987-869d-42eb-9eb6-08c39e3afe6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":28470},{"type":"v1","addr":"172.21.15.67:6817","nonce":28470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":28470},{"type":"v1","addr":"172.21.15.67:6819","nonce":28470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":28470},{"type":"v1","addr":"172.21.15.67:6823","nonce":28470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":28470},{"type":"v1","addr":"172.21.15.67:6821","nonce":28470}]},"public_addr":"172.21.15.67:6817/28470","cluster_addr":"172.21.15.67:6819/28470","heartbeat_back_addr":"172.21.15.67:6823/28470","heartbeat_front_addr":"172.21.15.67:6821/28470","state":["exists","up"]},{"osd":8,"uuid":"69ddfe52-11df-4608-9a10-e8907c4cd87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":29472},{"type":"v1","addr":"172.21.15.190:6809","nonce":29472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":29472},{"type":"v1","addr":"172.21.15.190:6811","nonce":29472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":29472},{"type":"v1","addr":"172.21.15.190:6815","nonce":29472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":29472},{"type":"v1","addr":"172.21.15.190:6813","nonce":29472}]},"public_addr":"172.21.15.190:6809/29472","cluster_addr":"172.21.15.190:6811/29472","heartbeat_back_addr":"172.21.15.190:6815/29472","heartbeat_front_addr":"172.21.15.190:6813/29472","state":["exists","up"]},{"osd":9,"uuid":"f92e1609-2a1a-4814-9ac0-44a73373d229","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":29473},{"type":"v1","addr":"172.21.15.190:6801","nonce":29473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":29473},{"type":"v1","addr":"172.21.15.190:6803","nonce":29473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":29473},{"type":"v1","addr":"172.21.15.190:6807","nonce":29473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":29473},{"type":"v1","addr":"172.21.15.190:6805","nonce":29473}]},"public_addr":"172.21.15.190:6801/29473","cluster_addr":"172.21.15.190:6803/29473","heartbeat_back_addr":"172.21.15.190:6807/29473","heartbeat_front_addr":"172.21.15.190:6805/29473","state":["exists","up"]},{"osd":10,"uuid":"5f2e8ca3-8c65-4595-8e82-27dc3768ab99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":29474},{"type":"v1","addr":"172.21.15.190:6817","nonce":29474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":29474},{"type":"v1","addr":"172.21.15.190:6819","nonce":29474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":29474},{"type":"v1","addr":"172.21.15.190:6823","nonce":29474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":29474},{"type":"v1","addr":"172.21.15.190:6821","nonce":29474}]},"public_addr":"172.21.15.190:6817/29474","cluster_addr":"172.21.15.190:6819/29474","heartbeat_back_addr":"172.21.15.190:6823/29474","heartbeat_front_addr":"172.21.15.190:6821/29474","state":["exists","up"]},{"osd":11,"uuid":"cedf7df4-f875-42f5-a8cf-13cb82f05cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":29475},{"type":"v1","addr":"172.21.15.190:6825","nonce":29475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":29475},{"type":"v1","addr":"172.21.15.190:6827","nonce":29475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":29475},{"type":"v1","addr":"172.21.15.190:6831","nonce":29475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":29475},{"type":"v1","addr":"172.21.15.190:6829","nonce":29475}]},"public_addr":"172.21.15.190:6825/29475","cluster_addr":"172.21.15.190:6827/29475","heartbeat_back_addr":"172.21.15.190:6831/29475","heartbeat_front_addr":"172.21.15.190:6829/29475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.943165+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.946049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.981045+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:14.019932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.936197+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.982429+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.953833+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.967611+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.996438+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.926618+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.993934+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:14.022484+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:46:26.966 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T03:46:26.967 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-15T03:46:26.967 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T03:46:26.968 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T03:46:26.968 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-15T03:46:26.968 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-15T03:46:26.969 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-15T03:46:26.969 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-15T03:46:26.970 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-15T03:46:26.970 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-15T03:46:26.970 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-15T03:46:26.971 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-15T03:46:27.272 INFO:teuthology.orchestra.run.smithi001.stdout:73014444036 2024-09-15T03:46:27.273 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-15T03:46:27.284 INFO:teuthology.orchestra.run.smithi001.stdout:73014444036 2024-09-15T03:46:27.284 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-15T03:46:27.290 INFO:teuthology.orchestra.run.smithi001.stdout:73014444036 2024-09-15T03:46:27.290 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-15T03:46:27.294 INFO:teuthology.orchestra.run.smithi001.stdout:73014444036 2024-09-15T03:46:27.296 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:46:27.300 INFO:teuthology.orchestra.run.smithi001.stdout:73014444036 2024-09-15T03:46:27.300 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-15T03:46:27.320 INFO:teuthology.orchestra.run.smithi001.stdout:73014444036 2024-09-15T03:46:27.321 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-15T03:46:27.322 INFO:teuthology.orchestra.run.smithi001.stdout:73014444036 2024-09-15T03:46:27.322 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T03:46:27.329 INFO:teuthology.orchestra.run.smithi001.stdout:73014444036 2024-09-15T03:46:27.329 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-15T03:46:27.342 INFO:teuthology.orchestra.run.smithi001.stdout:73014444036 2024-09-15T03:46:27.342 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:46:27.349 INFO:teuthology.orchestra.run.smithi001.stdout:73014444036 2024-09-15T03:46:27.349 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-15T03:46:27.350 INFO:teuthology.orchestra.run.smithi001.stdout:73014444036 2024-09-15T03:46:27.351 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:46:27.354 INFO:teuthology.orchestra.run.smithi001.stdout:73014444037 2024-09-15T03:46:27.354 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T03:46:27.974 INFO:teuthology.orchestra.run.smithi001.stdout:73014444034 2024-09-15T03:46:27.994 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444034 for osd.4 2024-09-15T03:46:28.015 INFO:teuthology.orchestra.run.smithi001.stdout:73014444034 2024-09-15T03:46:28.034 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444034 for osd.0 2024-09-15T03:46:28.037 INFO:teuthology.orchestra.run.smithi001.stdout:73014444034 2024-09-15T03:46:28.037 INFO:teuthology.orchestra.run.smithi001.stdout:73014444034 2024-09-15T03:46:28.063 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444034 for osd.5 2024-09-15T03:46:28.076 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444034 for osd.8 2024-09-15T03:46:28.081 INFO:teuthology.orchestra.run.smithi001.stdout:73014444034 2024-09-15T03:46:28.098 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444034 for osd.7 2024-09-15T03:46:28.112 INFO:teuthology.orchestra.run.smithi001.stdout:73014444034 2024-09-15T03:46:28.117 INFO:teuthology.orchestra.run.smithi001.stdout:73014444034 2024-09-15T03:46:28.126 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444034 for osd.9 2024-09-15T03:46:28.135 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444034 for osd.6 2024-09-15T03:46:28.143 INFO:teuthology.orchestra.run.smithi001.stdout:73014444034 2024-09-15T03:46:28.157 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444034 for osd.3 2024-09-15T03:46:28.159 INFO:teuthology.orchestra.run.smithi001.stdout:73014444034 2024-09-15T03:46:28.177 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444034 for osd.10 2024-09-15T03:46:28.185 INFO:teuthology.orchestra.run.smithi001.stdout:73014444034 2024-09-15T03:46:28.198 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444034 for osd.2 2024-09-15T03:46:28.217 INFO:teuthology.orchestra.run.smithi001.stdout:73014444035 2024-09-15T03:46:28.233 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444037 got 73014444035 for osd.1 2024-09-15T03:46:28.234 INFO:teuthology.orchestra.run.smithi001.stdout:73014444034 2024-09-15T03:46:28.246 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444034 for osd.11 2024-09-15T03:46:28.995 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T03:46:29.036 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:46:29.064 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-15T03:46:29.076 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-15T03:46:29.099 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-15T03:46:29.127 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-15T03:46:29.136 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-15T03:46:29.158 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:46:29.177 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-15T03:46:29.199 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:46:29.234 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T03:46:29.247 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-15T03:46:29.536 INFO:teuthology.orchestra.run.smithi001.stdout:73014444036 2024-09-15T03:46:29.576 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444036 for osd.4 2024-09-15T03:46:29.576 DEBUG:teuthology.parallel:result is None 2024-09-15T03:46:29.581 INFO:teuthology.orchestra.run.smithi001.stdout:73014444036 2024-09-15T03:46:29.598 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444036 for osd.0 2024-09-15T03:46:29.599 DEBUG:teuthology.parallel:result is None 2024-09-15T03:46:29.719 INFO:teuthology.orchestra.run.smithi001.stdout:73014444036 2024-09-15T03:46:29.742 INFO:teuthology.orchestra.run.smithi001.stdout:73014444036 2024-09-15T03:46:29.744 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444036 for osd.9 2024-09-15T03:46:29.744 DEBUG:teuthology.parallel:result is None 2024-09-15T03:46:29.776 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444036 for osd.8 2024-09-15T03:46:29.776 DEBUG:teuthology.parallel:result is None 2024-09-15T03:46:29.795 INFO:teuthology.orchestra.run.smithi001.stdout:73014444036 2024-09-15T03:46:29.814 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444036 for osd.5 2024-09-15T03:46:29.814 DEBUG:teuthology.parallel:result is None 2024-09-15T03:46:29.820 INFO:teuthology.orchestra.run.smithi001.stdout:73014444036 2024-09-15T03:46:29.839 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444036 for osd.7 2024-09-15T03:46:29.839 DEBUG:teuthology.parallel:result is None 2024-09-15T03:46:29.849 INFO:teuthology.orchestra.run.smithi001.stdout:73014444036 2024-09-15T03:46:29.866 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444036 for osd.3 2024-09-15T03:46:29.867 DEBUG:teuthology.parallel:result is None 2024-09-15T03:46:29.910 INFO:teuthology.orchestra.run.smithi001.stdout:73014444036 2024-09-15T03:46:29.911 INFO:teuthology.orchestra.run.smithi001.stdout:73014444036 2024-09-15T03:46:29.914 INFO:teuthology.orchestra.run.smithi001.stdout:73014444037 2024-09-15T03:46:29.930 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444036 for osd.10 2024-09-15T03:46:29.931 DEBUG:teuthology.parallel:result is None 2024-09-15T03:46:29.931 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444036 for osd.2 2024-09-15T03:46:29.931 DEBUG:teuthology.parallel:result is None 2024-09-15T03:46:29.932 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444037 got 73014444037 for osd.1 2024-09-15T03:46:29.932 DEBUG:teuthology.parallel:result is None 2024-09-15T03:46:29.977 INFO:teuthology.orchestra.run.smithi001.stdout:73014444036 2024-09-15T03:46:29.981 INFO:teuthology.orchestra.run.smithi001.stdout:73014444036 2024-09-15T03:46:29.990 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444036 for osd.6 2024-09-15T03:46:29.990 DEBUG:teuthology.parallel:result is None 2024-09-15T03:46:29.994 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444036 for osd.11 2024-09-15T03:46:29.995 DEBUG:teuthology.parallel:result is None 2024-09-15T03:46:29.995 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T03:46:29.995 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:46:30.310 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:46:30.310 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-15T03:46:30.322 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-15T03:46:30.145382+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":125,"num_write_kb":2152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":12,"num_per_pool_osds":2,"num_per_pool_omap_osds":0,"kb":1124155392,"kb_used":9554148,"kb_used_data":1114602196,"kb_used_omap":701,"kb_used_meta":0,"kb_avail":1114601244,"statfs":{"total":1151135121408,"available":1141351673856,"internally_reserved":974848,"allocated":1141352648704,"data_stored":1141352648704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":718129,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001523"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.044552+0000","last_change":"2024-09-15T03:46:26.044851+0000","last_active":"2024-09-15T03:46:26.044552+0000","last_peered":"2024-09-15T03:46:26.044552+0000","last_clean":"2024-09-15T03:46:26.044552+0000","last_became_active":"2024-09-15T03:46:24.063335+0000","last_became_peered":"2024-09-15T03:46:24.063335+0000","last_unstale":"2024-09-15T03:46:26.044552+0000","last_undegraded":"2024-09-15T03:46:26.044552+0000","last_fullsized":"2024-09-15T03:46:26.044552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:34.631264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065928000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.776177+0000","last_change":"2024-09-15T03:46:26.776405+0000","last_active":"2024-09-15T03:46:26.776177+0000","last_peered":"2024-09-15T03:46:26.776177+0000","last_clean":"2024-09-15T03:46:26.776177+0000","last_became_active":"2024-09-15T03:46:24.063380+0000","last_became_peered":"2024-09-15T03:46:24.063380+0000","last_unstale":"2024-09-15T03:46:26.776177+0000","last_undegraded":"2024-09-15T03:46:26.776177+0000","last_fullsized":"2024-09-15T03:46:26.776177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:24.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034490600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.041904+0000","last_change":"2024-09-15T03:46:26.042111+0000","last_active":"2024-09-15T03:46:26.041904+0000","last_peered":"2024-09-15T03:46:26.041904+0000","last_clean":"2024-09-15T03:46:26.041904+0000","last_became_active":"2024-09-15T03:46:24.038945+0000","last_became_peered":"2024-09-15T03:46:24.038945+0000","last_unstale":"2024-09-15T03:46:26.041904+0000","last_undegraded":"2024-09-15T03:46:26.041904+0000","last_fullsized":"2024-09-15T03:46:26.041904+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:27.950416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032787499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.041373+0000","last_change":"2024-09-15T03:46:26.041613+0000","last_active":"2024-09-15T03:46:26.041373+0000","last_peered":"2024-09-15T03:46:26.041373+0000","last_clean":"2024-09-15T03:46:26.041373+0000","last_became_active":"2024-09-15T03:46:24.063070+0000","last_became_peered":"2024-09-15T03:46:24.063070+0000","last_unstale":"2024-09-15T03:46:26.041373+0000","last_undegraded":"2024-09-15T03:46:26.041373+0000","last_fullsized":"2024-09-15T03:46:26.041373+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:35.057577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034123999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"24'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.045048+0000","last_change":"2024-09-15T03:46:26.045048+0000","last_active":"2024-09-15T03:46:26.045048+0000","last_peered":"2024-09-15T03:46:26.045048+0000","last_clean":"2024-09-15T03:46:26.045048+0000","last_became_active":"2024-09-15T03:46:24.062930+0000","last_became_peered":"2024-09-15T03:46:24.062930+0000","last_unstale":"2024-09-15T03:46:26.045048+0000","last_undegraded":"2024-09-15T03:46:26.045048+0000","last_fullsized":"2024-09-15T03:46:26.045048+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:24.188285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000944677,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.042552+0000","last_change":"2024-09-15T03:46:26.042740+0000","last_active":"2024-09-15T03:46:26.042552+0000","last_peered":"2024-09-15T03:46:26.042552+0000","last_clean":"2024-09-15T03:46:26.042552+0000","last_became_active":"2024-09-15T03:46:24.037471+0000","last_became_peered":"2024-09-15T03:46:24.037471+0000","last_unstale":"2024-09-15T03:46:26.042552+0000","last_undegraded":"2024-09-15T03:46:26.042552+0000","last_fullsized":"2024-09-15T03:46:26.042552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:29.654520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026490299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.044278+0000","last_change":"2024-09-15T03:46:26.044503+0000","last_active":"2024-09-15T03:46:26.044278+0000","last_peered":"2024-09-15T03:46:26.044278+0000","last_clean":"2024-09-15T03:46:26.044278+0000","last_became_active":"2024-09-15T03:46:24.063106+0000","last_became_peered":"2024-09-15T03:46:24.063106+0000","last_unstale":"2024-09-15T03:46:26.044278+0000","last_undegraded":"2024-09-15T03:46:26.044278+0000","last_fullsized":"2024-09-15T03:46:26.044278+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:24.101828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000393075,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"22'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.776443+0000","last_change":"2024-09-15T03:46:26.776623+0000","last_active":"2024-09-15T03:46:26.776443+0000","last_peered":"2024-09-15T03:46:26.776443+0000","last_clean":"2024-09-15T03:46:26.776443+0000","last_became_active":"2024-09-15T03:46:24.038633+0000","last_became_peered":"2024-09-15T03:46:24.038633+0000","last_unstale":"2024-09-15T03:46:26.776443+0000","last_undegraded":"2024-09-15T03:46:26.776443+0000","last_fullsized":"2024-09-15T03:46:26.776443+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:45.125997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030178200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"20'92","reported_seq":126,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-15T03:46:25.041684+0000","last_change":"2024-09-15T03:46:18.011841+0000","last_active":"2024-09-15T03:46:25.041684+0000","last_peered":"2024-09-15T03:46:25.041684+0000","last_clean":"2024-09-15T03:46:25.041684+0000","last_became_active":"2024-09-15T03:46:18.011130+0000","last_became_peered":"2024-09-15T03:46:18.011130+0000","last_unstale":"2024-09-15T03:46:25.041684+0000","last_undegraded":"2024-09-15T03:46:25.041684+0000","last_fullsized":"2024-09-15T03:46:25.041684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_clean_scrub_stamp":"2024-09-15T03:46:16.886982+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:32.556840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":11,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795992,"kb_used_data":92883700,"kb_used_omap":46,"kb_used_meta":0,"kb_avail":92883624,"statfs":{"total":95927926784,"available":95112830976,"internally_reserved":77824,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47699,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]}]},{"osd":10,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795836,"kb_used_data":92883828,"kb_used_omap":40,"kb_used_meta":0,"kb_avail":92883780,"statfs":{"total":95927926784,"available":95112990720,"internally_reserved":49152,"allocated":95113039872,"data_stored":95113039872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41303,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]}]},{"osd":9,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795964,"kb_used_data":92883724,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92883652,"statfs":{"total":95927926784,"available":95112859648,"internally_reserved":73728,"allocated":95112933376,"data_stored":95112933376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48511,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":8,"up_from":17,"seq":73014444036,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797288,"kb_used_data":92882456,"kb_used_omap":116,"kb_used_meta":0,"kb_avail":92882328,"statfs":{"total":95927926784,"available":95111503872,"internally_reserved":131072,"allocated":95111634944,"data_stored":95111634944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119759,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]}]},{"osd":7,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795988,"kb_used_data":92883700,"kb_used_omap":46,"kb_used_meta":0,"kb_avail":92883628,"statfs":{"total":95927926784,"available":95112835072,"internally_reserved":73728,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47192,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94599999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]}]},{"osd":6,"up_from":17,"seq":73014444036,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":796036,"kb_used_data":92883676,"kb_used_omap":55,"kb_used_meta":0,"kb_avail":92883580,"statfs":{"total":95927926784,"available":95112785920,"internally_reserved":98304,"allocated":95112884224,"data_stored":95112884224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":56376,"internal_metadata":0},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]}]},{"osd":5,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":796008,"kb_used_data":92883700,"kb_used_omap":53,"kb_used_meta":0,"kb_avail":92883608,"statfs":{"total":95927926784,"available":95112814592,"internally_reserved":94208,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":54440,"internal_metadata":0},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0149999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0549999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.177}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.236}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]}]},{"osd":4,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":796008,"kb_used_data":92883700,"kb_used_omap":53,"kb_used_meta":0,"kb_avail":92883608,"statfs":{"total":95927926784,"available":95112814592,"internally_reserved":94208,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":54440,"internal_metadata":0},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]}]},{"osd":3,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795988,"kb_used_data":92883700,"kb_used_omap":46,"kb_used_meta":0,"kb_avail":92883628,"statfs":{"total":95927926784,"available":95112835072,"internally_reserved":73728,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47192,"internal_metadata":0},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]}]},{"osd":2,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795836,"kb_used_data":92883828,"kb_used_omap":40,"kb_used_meta":0,"kb_avail":92883780,"statfs":{"total":95927926784,"available":95112990720,"internally_reserved":49152,"allocated":95113039872,"data_stored":95113039872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41296,"internal_metadata":0},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]}]},{"osd":1,"up_from":17,"seq":73014444037,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795984,"kb_used_data":92883700,"kb_used_omap":46,"kb_used_meta":0,"kb_avail":92883632,"statfs":{"total":95927926784,"available":95112839168,"internally_reserved":69632,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47846,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.258}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2410000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2809999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797220,"kb_used_data":92882484,"kb_used_omap":109,"kb_used_meta":0,"kb_avail":92882396,"statfs":{"total":95927926784,"available":95111573504,"internally_reserved":90112,"allocated":95111663616,"data_stored":95111663616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112075,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]}]}],"pool_statfs":[]}} 2024-09-15T03:46:30.325 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:46:30.596 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:46:30.596 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-15T03:46:30.607 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-15T03:46:30.145382+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":125,"num_write_kb":2152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":12,"num_per_pool_osds":2,"num_per_pool_omap_osds":0,"kb":1124155392,"kb_used":9554148,"kb_used_data":1114602196,"kb_used_omap":701,"kb_used_meta":0,"kb_avail":1114601244,"statfs":{"total":1151135121408,"available":1141351673856,"internally_reserved":974848,"allocated":1141352648704,"data_stored":1141352648704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":718129,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001523"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.044552+0000","last_change":"2024-09-15T03:46:26.044851+0000","last_active":"2024-09-15T03:46:26.044552+0000","last_peered":"2024-09-15T03:46:26.044552+0000","last_clean":"2024-09-15T03:46:26.044552+0000","last_became_active":"2024-09-15T03:46:24.063335+0000","last_became_peered":"2024-09-15T03:46:24.063335+0000","last_unstale":"2024-09-15T03:46:26.044552+0000","last_undegraded":"2024-09-15T03:46:26.044552+0000","last_fullsized":"2024-09-15T03:46:26.044552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:34.631264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065928000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.776177+0000","last_change":"2024-09-15T03:46:26.776405+0000","last_active":"2024-09-15T03:46:26.776177+0000","last_peered":"2024-09-15T03:46:26.776177+0000","last_clean":"2024-09-15T03:46:26.776177+0000","last_became_active":"2024-09-15T03:46:24.063380+0000","last_became_peered":"2024-09-15T03:46:24.063380+0000","last_unstale":"2024-09-15T03:46:26.776177+0000","last_undegraded":"2024-09-15T03:46:26.776177+0000","last_fullsized":"2024-09-15T03:46:26.776177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:24.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034490600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.041904+0000","last_change":"2024-09-15T03:46:26.042111+0000","last_active":"2024-09-15T03:46:26.041904+0000","last_peered":"2024-09-15T03:46:26.041904+0000","last_clean":"2024-09-15T03:46:26.041904+0000","last_became_active":"2024-09-15T03:46:24.038945+0000","last_became_peered":"2024-09-15T03:46:24.038945+0000","last_unstale":"2024-09-15T03:46:26.041904+0000","last_undegraded":"2024-09-15T03:46:26.041904+0000","last_fullsized":"2024-09-15T03:46:26.041904+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:27.950416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032787499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.041373+0000","last_change":"2024-09-15T03:46:26.041613+0000","last_active":"2024-09-15T03:46:26.041373+0000","last_peered":"2024-09-15T03:46:26.041373+0000","last_clean":"2024-09-15T03:46:26.041373+0000","last_became_active":"2024-09-15T03:46:24.063070+0000","last_became_peered":"2024-09-15T03:46:24.063070+0000","last_unstale":"2024-09-15T03:46:26.041373+0000","last_undegraded":"2024-09-15T03:46:26.041373+0000","last_fullsized":"2024-09-15T03:46:26.041373+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:35.057577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034123999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"24'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.045048+0000","last_change":"2024-09-15T03:46:26.045048+0000","last_active":"2024-09-15T03:46:26.045048+0000","last_peered":"2024-09-15T03:46:26.045048+0000","last_clean":"2024-09-15T03:46:26.045048+0000","last_became_active":"2024-09-15T03:46:24.062930+0000","last_became_peered":"2024-09-15T03:46:24.062930+0000","last_unstale":"2024-09-15T03:46:26.045048+0000","last_undegraded":"2024-09-15T03:46:26.045048+0000","last_fullsized":"2024-09-15T03:46:26.045048+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:24.188285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000944677,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.042552+0000","last_change":"2024-09-15T03:46:26.042740+0000","last_active":"2024-09-15T03:46:26.042552+0000","last_peered":"2024-09-15T03:46:26.042552+0000","last_clean":"2024-09-15T03:46:26.042552+0000","last_became_active":"2024-09-15T03:46:24.037471+0000","last_became_peered":"2024-09-15T03:46:24.037471+0000","last_unstale":"2024-09-15T03:46:26.042552+0000","last_undegraded":"2024-09-15T03:46:26.042552+0000","last_fullsized":"2024-09-15T03:46:26.042552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:29.654520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026490299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.044278+0000","last_change":"2024-09-15T03:46:26.044503+0000","last_active":"2024-09-15T03:46:26.044278+0000","last_peered":"2024-09-15T03:46:26.044278+0000","last_clean":"2024-09-15T03:46:26.044278+0000","last_became_active":"2024-09-15T03:46:24.063106+0000","last_became_peered":"2024-09-15T03:46:24.063106+0000","last_unstale":"2024-09-15T03:46:26.044278+0000","last_undegraded":"2024-09-15T03:46:26.044278+0000","last_fullsized":"2024-09-15T03:46:26.044278+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:24.101828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000393075,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"22'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.776443+0000","last_change":"2024-09-15T03:46:26.776623+0000","last_active":"2024-09-15T03:46:26.776443+0000","last_peered":"2024-09-15T03:46:26.776443+0000","last_clean":"2024-09-15T03:46:26.776443+0000","last_became_active":"2024-09-15T03:46:24.038633+0000","last_became_peered":"2024-09-15T03:46:24.038633+0000","last_unstale":"2024-09-15T03:46:26.776443+0000","last_undegraded":"2024-09-15T03:46:26.776443+0000","last_fullsized":"2024-09-15T03:46:26.776443+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:45.125997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030178200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"20'92","reported_seq":126,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-15T03:46:25.041684+0000","last_change":"2024-09-15T03:46:18.011841+0000","last_active":"2024-09-15T03:46:25.041684+0000","last_peered":"2024-09-15T03:46:25.041684+0000","last_clean":"2024-09-15T03:46:25.041684+0000","last_became_active":"2024-09-15T03:46:18.011130+0000","last_became_peered":"2024-09-15T03:46:18.011130+0000","last_unstale":"2024-09-15T03:46:25.041684+0000","last_undegraded":"2024-09-15T03:46:25.041684+0000","last_fullsized":"2024-09-15T03:46:25.041684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_clean_scrub_stamp":"2024-09-15T03:46:16.886982+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:32.556840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":11,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795992,"kb_used_data":92883700,"kb_used_omap":46,"kb_used_meta":0,"kb_avail":92883624,"statfs":{"total":95927926784,"available":95112830976,"internally_reserved":77824,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47699,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]}]},{"osd":10,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795836,"kb_used_data":92883828,"kb_used_omap":40,"kb_used_meta":0,"kb_avail":92883780,"statfs":{"total":95927926784,"available":95112990720,"internally_reserved":49152,"allocated":95113039872,"data_stored":95113039872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41303,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]}]},{"osd":9,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795964,"kb_used_data":92883724,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92883652,"statfs":{"total":95927926784,"available":95112859648,"internally_reserved":73728,"allocated":95112933376,"data_stored":95112933376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48511,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":8,"up_from":17,"seq":73014444036,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797288,"kb_used_data":92882456,"kb_used_omap":116,"kb_used_meta":0,"kb_avail":92882328,"statfs":{"total":95927926784,"available":95111503872,"internally_reserved":131072,"allocated":95111634944,"data_stored":95111634944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119759,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]}]},{"osd":7,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795988,"kb_used_data":92883700,"kb_used_omap":46,"kb_used_meta":0,"kb_avail":92883628,"statfs":{"total":95927926784,"available":95112835072,"internally_reserved":73728,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47192,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94599999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]}]},{"osd":6,"up_from":17,"seq":73014444036,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":796036,"kb_used_data":92883676,"kb_used_omap":55,"kb_used_meta":0,"kb_avail":92883580,"statfs":{"total":95927926784,"available":95112785920,"internally_reserved":98304,"allocated":95112884224,"data_stored":95112884224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":56376,"internal_metadata":0},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]}]},{"osd":5,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":796008,"kb_used_data":92883700,"kb_used_omap":53,"kb_used_meta":0,"kb_avail":92883608,"statfs":{"total":95927926784,"available":95112814592,"internally_reserved":94208,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":54440,"internal_metadata":0},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0149999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0549999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.177}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.236}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]}]},{"osd":4,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":796008,"kb_used_data":92883700,"kb_used_omap":53,"kb_used_meta":0,"kb_avail":92883608,"statfs":{"total":95927926784,"available":95112814592,"internally_reserved":94208,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":54440,"internal_metadata":0},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]}]},{"osd":3,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795988,"kb_used_data":92883700,"kb_used_omap":46,"kb_used_meta":0,"kb_avail":92883628,"statfs":{"total":95927926784,"available":95112835072,"internally_reserved":73728,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47192,"internal_metadata":0},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]}]},{"osd":2,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795836,"kb_used_data":92883828,"kb_used_omap":40,"kb_used_meta":0,"kb_avail":92883780,"statfs":{"total":95927926784,"available":95112990720,"internally_reserved":49152,"allocated":95113039872,"data_stored":95113039872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41296,"internal_metadata":0},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]}]},{"osd":1,"up_from":17,"seq":73014444037,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795984,"kb_used_data":92883700,"kb_used_omap":46,"kb_used_meta":0,"kb_avail":92883632,"statfs":{"total":95927926784,"available":95112839168,"internally_reserved":69632,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47846,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.258}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2410000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2809999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797220,"kb_used_data":92882484,"kb_used_omap":109,"kb_used_meta":0,"kb_avail":92882396,"statfs":{"total":95927926784,"available":95111573504,"internally_reserved":90112,"allocated":95111663616,"data_stored":95111663616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112075,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]}]}],"pool_statfs":[]}} 2024-09-15T03:46:30.609 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T03:46:30.609 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-15T03:46:30.609 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-15T03:46:30.609 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-15T03:46:30.913 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:46:30.914 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-15T03:46:30.926 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-15T03:46:30.926 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-15T03:46:30.938 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-15T03:46:30.938 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd sloppy crc': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'filestore': True, 'fs': 'xfs'}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-15T03:46:30.938 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-15T03:46:30.938 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-15T03:46:30.938 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T03:46:30.938 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:46:31.213 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:46:31.214 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-15T03:46:31.226 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-15T03:46:30.145382+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":125,"num_write_kb":2152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":12,"num_per_pool_osds":2,"num_per_pool_omap_osds":0,"kb":1124155392,"kb_used":9554148,"kb_used_data":1114602196,"kb_used_omap":701,"kb_used_meta":0,"kb_avail":1114601244,"statfs":{"total":1151135121408,"available":1141351673856,"internally_reserved":974848,"allocated":1141352648704,"data_stored":1141352648704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":718129,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001523"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.044552+0000","last_change":"2024-09-15T03:46:26.044851+0000","last_active":"2024-09-15T03:46:26.044552+0000","last_peered":"2024-09-15T03:46:26.044552+0000","last_clean":"2024-09-15T03:46:26.044552+0000","last_became_active":"2024-09-15T03:46:24.063335+0000","last_became_peered":"2024-09-15T03:46:24.063335+0000","last_unstale":"2024-09-15T03:46:26.044552+0000","last_undegraded":"2024-09-15T03:46:26.044552+0000","last_fullsized":"2024-09-15T03:46:26.044552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:34.631264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065928000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.776177+0000","last_change":"2024-09-15T03:46:26.776405+0000","last_active":"2024-09-15T03:46:26.776177+0000","last_peered":"2024-09-15T03:46:26.776177+0000","last_clean":"2024-09-15T03:46:26.776177+0000","last_became_active":"2024-09-15T03:46:24.063380+0000","last_became_peered":"2024-09-15T03:46:24.063380+0000","last_unstale":"2024-09-15T03:46:26.776177+0000","last_undegraded":"2024-09-15T03:46:26.776177+0000","last_fullsized":"2024-09-15T03:46:26.776177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:24.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034490600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.041904+0000","last_change":"2024-09-15T03:46:26.042111+0000","last_active":"2024-09-15T03:46:26.041904+0000","last_peered":"2024-09-15T03:46:26.041904+0000","last_clean":"2024-09-15T03:46:26.041904+0000","last_became_active":"2024-09-15T03:46:24.038945+0000","last_became_peered":"2024-09-15T03:46:24.038945+0000","last_unstale":"2024-09-15T03:46:26.041904+0000","last_undegraded":"2024-09-15T03:46:26.041904+0000","last_fullsized":"2024-09-15T03:46:26.041904+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:27.950416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032787499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.041373+0000","last_change":"2024-09-15T03:46:26.041613+0000","last_active":"2024-09-15T03:46:26.041373+0000","last_peered":"2024-09-15T03:46:26.041373+0000","last_clean":"2024-09-15T03:46:26.041373+0000","last_became_active":"2024-09-15T03:46:24.063070+0000","last_became_peered":"2024-09-15T03:46:24.063070+0000","last_unstale":"2024-09-15T03:46:26.041373+0000","last_undegraded":"2024-09-15T03:46:26.041373+0000","last_fullsized":"2024-09-15T03:46:26.041373+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:35.057577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034123999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"24'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.045048+0000","last_change":"2024-09-15T03:46:26.045048+0000","last_active":"2024-09-15T03:46:26.045048+0000","last_peered":"2024-09-15T03:46:26.045048+0000","last_clean":"2024-09-15T03:46:26.045048+0000","last_became_active":"2024-09-15T03:46:24.062930+0000","last_became_peered":"2024-09-15T03:46:24.062930+0000","last_unstale":"2024-09-15T03:46:26.045048+0000","last_undegraded":"2024-09-15T03:46:26.045048+0000","last_fullsized":"2024-09-15T03:46:26.045048+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:24.188285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000944677,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.042552+0000","last_change":"2024-09-15T03:46:26.042740+0000","last_active":"2024-09-15T03:46:26.042552+0000","last_peered":"2024-09-15T03:46:26.042552+0000","last_clean":"2024-09-15T03:46:26.042552+0000","last_became_active":"2024-09-15T03:46:24.037471+0000","last_became_peered":"2024-09-15T03:46:24.037471+0000","last_unstale":"2024-09-15T03:46:26.042552+0000","last_undegraded":"2024-09-15T03:46:26.042552+0000","last_fullsized":"2024-09-15T03:46:26.042552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:29.654520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026490299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.044278+0000","last_change":"2024-09-15T03:46:26.044503+0000","last_active":"2024-09-15T03:46:26.044278+0000","last_peered":"2024-09-15T03:46:26.044278+0000","last_clean":"2024-09-15T03:46:26.044278+0000","last_became_active":"2024-09-15T03:46:24.063106+0000","last_became_peered":"2024-09-15T03:46:24.063106+0000","last_unstale":"2024-09-15T03:46:26.044278+0000","last_undegraded":"2024-09-15T03:46:26.044278+0000","last_fullsized":"2024-09-15T03:46:26.044278+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:24.101828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000393075,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"22'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.776443+0000","last_change":"2024-09-15T03:46:26.776623+0000","last_active":"2024-09-15T03:46:26.776443+0000","last_peered":"2024-09-15T03:46:26.776443+0000","last_clean":"2024-09-15T03:46:26.776443+0000","last_became_active":"2024-09-15T03:46:24.038633+0000","last_became_peered":"2024-09-15T03:46:24.038633+0000","last_unstale":"2024-09-15T03:46:26.776443+0000","last_undegraded":"2024-09-15T03:46:26.776443+0000","last_fullsized":"2024-09-15T03:46:26.776443+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:45.125997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030178200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"20'92","reported_seq":126,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-15T03:46:25.041684+0000","last_change":"2024-09-15T03:46:18.011841+0000","last_active":"2024-09-15T03:46:25.041684+0000","last_peered":"2024-09-15T03:46:25.041684+0000","last_clean":"2024-09-15T03:46:25.041684+0000","last_became_active":"2024-09-15T03:46:18.011130+0000","last_became_peered":"2024-09-15T03:46:18.011130+0000","last_unstale":"2024-09-15T03:46:25.041684+0000","last_undegraded":"2024-09-15T03:46:25.041684+0000","last_fullsized":"2024-09-15T03:46:25.041684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_clean_scrub_stamp":"2024-09-15T03:46:16.886982+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:32.556840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":11,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795992,"kb_used_data":92883700,"kb_used_omap":46,"kb_used_meta":0,"kb_avail":92883624,"statfs":{"total":95927926784,"available":95112830976,"internally_reserved":77824,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47699,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]}]},{"osd":10,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795836,"kb_used_data":92883828,"kb_used_omap":40,"kb_used_meta":0,"kb_avail":92883780,"statfs":{"total":95927926784,"available":95112990720,"internally_reserved":49152,"allocated":95113039872,"data_stored":95113039872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41303,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]}]},{"osd":9,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795964,"kb_used_data":92883724,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92883652,"statfs":{"total":95927926784,"available":95112859648,"internally_reserved":73728,"allocated":95112933376,"data_stored":95112933376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48511,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":8,"up_from":17,"seq":73014444036,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797288,"kb_used_data":92882456,"kb_used_omap":116,"kb_used_meta":0,"kb_avail":92882328,"statfs":{"total":95927926784,"available":95111503872,"internally_reserved":131072,"allocated":95111634944,"data_stored":95111634944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119759,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]}]},{"osd":7,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795988,"kb_used_data":92883700,"kb_used_omap":46,"kb_used_meta":0,"kb_avail":92883628,"statfs":{"total":95927926784,"available":95112835072,"internally_reserved":73728,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47192,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94599999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]}]},{"osd":6,"up_from":17,"seq":73014444036,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":796036,"kb_used_data":92883676,"kb_used_omap":55,"kb_used_meta":0,"kb_avail":92883580,"statfs":{"total":95927926784,"available":95112785920,"internally_reserved":98304,"allocated":95112884224,"data_stored":95112884224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":56376,"internal_metadata":0},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]}]},{"osd":5,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":796008,"kb_used_data":92883700,"kb_used_omap":53,"kb_used_meta":0,"kb_avail":92883608,"statfs":{"total":95927926784,"available":95112814592,"internally_reserved":94208,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":54440,"internal_metadata":0},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0149999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0549999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.177}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.236}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]}]},{"osd":4,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":796008,"kb_used_data":92883700,"kb_used_omap":53,"kb_used_meta":0,"kb_avail":92883608,"statfs":{"total":95927926784,"available":95112814592,"internally_reserved":94208,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":54440,"internal_metadata":0},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]}]},{"osd":3,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795988,"kb_used_data":92883700,"kb_used_omap":46,"kb_used_meta":0,"kb_avail":92883628,"statfs":{"total":95927926784,"available":95112835072,"internally_reserved":73728,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47192,"internal_metadata":0},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]}]},{"osd":2,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795836,"kb_used_data":92883828,"kb_used_omap":40,"kb_used_meta":0,"kb_avail":92883780,"statfs":{"total":95927926784,"available":95112990720,"internally_reserved":49152,"allocated":95113039872,"data_stored":95113039872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41296,"internal_metadata":0},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]}]},{"osd":1,"up_from":17,"seq":73014444037,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795984,"kb_used_data":92883700,"kb_used_omap":46,"kb_used_meta":0,"kb_avail":92883632,"statfs":{"total":95927926784,"available":95112839168,"internally_reserved":69632,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47846,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.258}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2410000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2809999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797220,"kb_used_data":92882484,"kb_used_omap":109,"kb_used_meta":0,"kb_avail":92882396,"statfs":{"total":95927926784,"available":95111573504,"internally_reserved":90112,"allocated":95111663616,"data_stored":95111663616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112075,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]}]}],"pool_statfs":[]}} 2024-09-15T03:46:31.228 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:46:31.503 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:46:31.504 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-15T03:46:31.515 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-15T03:46:30.145382+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":125,"num_write_kb":2152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":12,"num_per_pool_osds":2,"num_per_pool_omap_osds":0,"kb":1124155392,"kb_used":9554148,"kb_used_data":1114602196,"kb_used_omap":701,"kb_used_meta":0,"kb_avail":1114601244,"statfs":{"total":1151135121408,"available":1141351673856,"internally_reserved":974848,"allocated":1141352648704,"data_stored":1141352648704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":718129,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001523"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.044552+0000","last_change":"2024-09-15T03:46:26.044851+0000","last_active":"2024-09-15T03:46:26.044552+0000","last_peered":"2024-09-15T03:46:26.044552+0000","last_clean":"2024-09-15T03:46:26.044552+0000","last_became_active":"2024-09-15T03:46:24.063335+0000","last_became_peered":"2024-09-15T03:46:24.063335+0000","last_unstale":"2024-09-15T03:46:26.044552+0000","last_undegraded":"2024-09-15T03:46:26.044552+0000","last_fullsized":"2024-09-15T03:46:26.044552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:34.631264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065928000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.776177+0000","last_change":"2024-09-15T03:46:26.776405+0000","last_active":"2024-09-15T03:46:26.776177+0000","last_peered":"2024-09-15T03:46:26.776177+0000","last_clean":"2024-09-15T03:46:26.776177+0000","last_became_active":"2024-09-15T03:46:24.063380+0000","last_became_peered":"2024-09-15T03:46:24.063380+0000","last_unstale":"2024-09-15T03:46:26.776177+0000","last_undegraded":"2024-09-15T03:46:26.776177+0000","last_fullsized":"2024-09-15T03:46:26.776177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:24.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034490600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.041904+0000","last_change":"2024-09-15T03:46:26.042111+0000","last_active":"2024-09-15T03:46:26.041904+0000","last_peered":"2024-09-15T03:46:26.041904+0000","last_clean":"2024-09-15T03:46:26.041904+0000","last_became_active":"2024-09-15T03:46:24.038945+0000","last_became_peered":"2024-09-15T03:46:24.038945+0000","last_unstale":"2024-09-15T03:46:26.041904+0000","last_undegraded":"2024-09-15T03:46:26.041904+0000","last_fullsized":"2024-09-15T03:46:26.041904+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:27.950416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032787499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.041373+0000","last_change":"2024-09-15T03:46:26.041613+0000","last_active":"2024-09-15T03:46:26.041373+0000","last_peered":"2024-09-15T03:46:26.041373+0000","last_clean":"2024-09-15T03:46:26.041373+0000","last_became_active":"2024-09-15T03:46:24.063070+0000","last_became_peered":"2024-09-15T03:46:24.063070+0000","last_unstale":"2024-09-15T03:46:26.041373+0000","last_undegraded":"2024-09-15T03:46:26.041373+0000","last_fullsized":"2024-09-15T03:46:26.041373+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:35.057577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034123999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"24'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.045048+0000","last_change":"2024-09-15T03:46:26.045048+0000","last_active":"2024-09-15T03:46:26.045048+0000","last_peered":"2024-09-15T03:46:26.045048+0000","last_clean":"2024-09-15T03:46:26.045048+0000","last_became_active":"2024-09-15T03:46:24.062930+0000","last_became_peered":"2024-09-15T03:46:24.062930+0000","last_unstale":"2024-09-15T03:46:26.045048+0000","last_undegraded":"2024-09-15T03:46:26.045048+0000","last_fullsized":"2024-09-15T03:46:26.045048+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:24.188285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000944677,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.042552+0000","last_change":"2024-09-15T03:46:26.042740+0000","last_active":"2024-09-15T03:46:26.042552+0000","last_peered":"2024-09-15T03:46:26.042552+0000","last_clean":"2024-09-15T03:46:26.042552+0000","last_became_active":"2024-09-15T03:46:24.037471+0000","last_became_peered":"2024-09-15T03:46:24.037471+0000","last_unstale":"2024-09-15T03:46:26.042552+0000","last_undegraded":"2024-09-15T03:46:26.042552+0000","last_fullsized":"2024-09-15T03:46:26.042552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:29.654520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026490299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.044278+0000","last_change":"2024-09-15T03:46:26.044503+0000","last_active":"2024-09-15T03:46:26.044278+0000","last_peered":"2024-09-15T03:46:26.044278+0000","last_clean":"2024-09-15T03:46:26.044278+0000","last_became_active":"2024-09-15T03:46:24.063106+0000","last_became_peered":"2024-09-15T03:46:24.063106+0000","last_unstale":"2024-09-15T03:46:26.044278+0000","last_undegraded":"2024-09-15T03:46:26.044278+0000","last_fullsized":"2024-09-15T03:46:26.044278+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:24.101828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000393075,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"22'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-15T03:46:26.776443+0000","last_change":"2024-09-15T03:46:26.776623+0000","last_active":"2024-09-15T03:46:26.776443+0000","last_peered":"2024-09-15T03:46:26.776443+0000","last_clean":"2024-09-15T03:46:26.776443+0000","last_became_active":"2024-09-15T03:46:24.038633+0000","last_became_peered":"2024-09-15T03:46:24.038633+0000","last_unstale":"2024-09-15T03:46:26.776443+0000","last_undegraded":"2024-09-15T03:46:26.776443+0000","last_fullsized":"2024-09-15T03:46:26.776443+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:45.125997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030178200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"20'92","reported_seq":126,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-15T03:46:25.041684+0000","last_change":"2024-09-15T03:46:18.011841+0000","last_active":"2024-09-15T03:46:25.041684+0000","last_peered":"2024-09-15T03:46:25.041684+0000","last_clean":"2024-09-15T03:46:25.041684+0000","last_became_active":"2024-09-15T03:46:18.011130+0000","last_became_peered":"2024-09-15T03:46:18.011130+0000","last_unstale":"2024-09-15T03:46:25.041684+0000","last_undegraded":"2024-09-15T03:46:25.041684+0000","last_fullsized":"2024-09-15T03:46:25.041684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_clean_scrub_stamp":"2024-09-15T03:46:16.886982+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:32.556840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":11,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795992,"kb_used_data":92883700,"kb_used_omap":46,"kb_used_meta":0,"kb_avail":92883624,"statfs":{"total":95927926784,"available":95112830976,"internally_reserved":77824,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47699,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]}]},{"osd":10,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795836,"kb_used_data":92883828,"kb_used_omap":40,"kb_used_meta":0,"kb_avail":92883780,"statfs":{"total":95927926784,"available":95112990720,"internally_reserved":49152,"allocated":95113039872,"data_stored":95113039872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41303,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]}]},{"osd":9,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795964,"kb_used_data":92883724,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92883652,"statfs":{"total":95927926784,"available":95112859648,"internally_reserved":73728,"allocated":95112933376,"data_stored":95112933376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48511,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":8,"up_from":17,"seq":73014444036,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797288,"kb_used_data":92882456,"kb_used_omap":116,"kb_used_meta":0,"kb_avail":92882328,"statfs":{"total":95927926784,"available":95111503872,"internally_reserved":131072,"allocated":95111634944,"data_stored":95111634944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":119759,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]}]},{"osd":7,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795988,"kb_used_data":92883700,"kb_used_omap":46,"kb_used_meta":0,"kb_avail":92883628,"statfs":{"total":95927926784,"available":95112835072,"internally_reserved":73728,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47192,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94599999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]}]},{"osd":6,"up_from":17,"seq":73014444036,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":796036,"kb_used_data":92883676,"kb_used_omap":55,"kb_used_meta":0,"kb_avail":92883580,"statfs":{"total":95927926784,"available":95112785920,"internally_reserved":98304,"allocated":95112884224,"data_stored":95112884224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":56376,"internal_metadata":0},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]}]},{"osd":5,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":796008,"kb_used_data":92883700,"kb_used_omap":53,"kb_used_meta":0,"kb_avail":92883608,"statfs":{"total":95927926784,"available":95112814592,"internally_reserved":94208,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":54440,"internal_metadata":0},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0149999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0549999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.177}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.236}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]}]},{"osd":4,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":796008,"kb_used_data":92883700,"kb_used_omap":53,"kb_used_meta":0,"kb_avail":92883608,"statfs":{"total":95927926784,"available":95112814592,"internally_reserved":94208,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":54440,"internal_metadata":0},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]}]},{"osd":3,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795988,"kb_used_data":92883700,"kb_used_omap":46,"kb_used_meta":0,"kb_avail":92883628,"statfs":{"total":95927926784,"available":95112835072,"internally_reserved":73728,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47192,"internal_metadata":0},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]}]},{"osd":2,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795836,"kb_used_data":92883828,"kb_used_omap":40,"kb_used_meta":0,"kb_avail":92883780,"statfs":{"total":95927926784,"available":95112990720,"internally_reserved":49152,"allocated":95113039872,"data_stored":95113039872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41296,"internal_metadata":0},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]}]},{"osd":1,"up_from":17,"seq":73014444037,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795984,"kb_used_data":92883700,"kb_used_omap":46,"kb_used_meta":0,"kb_avail":92883632,"statfs":{"total":95927926784,"available":95112839168,"internally_reserved":69632,"allocated":95112908800,"data_stored":95112908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47846,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.258}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2410000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2809999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797220,"kb_used_data":92882484,"kb_used_omap":109,"kb_used_meta":0,"kb_avail":92882396,"statfs":{"total":95927926784,"available":95111573504,"internally_reserved":90112,"allocated":95111663616,"data_stored":95111663616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112075,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]}]}],"pool_statfs":[]}} 2024-09-15T03:46:31.517 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T03:46:31.517 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-15T03:46:31.788 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 25 2024-09-15T03:46:31.788 INFO:teuthology.orchestra.run.smithi001.stdout:fsid f27b540e-e5e3-4e95-afe8-c95ee66b0a62 2024-09-15T03:46:31.788 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-09-15T03:46:09.843094+0000 2024-09-15T03:46:31.788 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-09-15T03:46:29.822704+0000 2024-09-15T03:46:31.789 INFO:teuthology.orchestra.run.smithi001.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-15T03:46:31.789 INFO:teuthology.orchestra.run.smithi001.stdout:crush_version 4 2024-09-15T03:46:31.789 INFO:teuthology.orchestra.run.smithi001.stdout:full_ratio 0.95 2024-09-15T03:46:31.789 INFO:teuthology.orchestra.run.smithi001.stdout:backfillfull_ratio 0.9 2024-09-15T03:46:31.789 INFO:teuthology.orchestra.run.smithi001.stdout:nearfull_ratio 0.85 2024-09-15T03:46:31.789 INFO:teuthology.orchestra.run.smithi001.stdout:require_min_compat_client luminous 2024-09-15T03:46:31.789 INFO:teuthology.orchestra.run.smithi001.stdout:min_compat_client jewel 2024-09-15T03:46:31.789 INFO:teuthology.orchestra.run.smithi001.stdout:require_osd_release quincy 2024-09-15T03:46:31.789 INFO:teuthology.orchestra.run.smithi001.stdout:stretch_mode_enabled false 2024-09-15T03:46:31.789 INFO:teuthology.orchestra.run.smithi001.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 20 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr 2024-09-15T03:46:31.789 INFO:teuthology.orchestra.run.smithi001.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 24 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd 2024-09-15T03:46:31.789 INFO:teuthology.orchestra.run.smithi001.stdout:max_osd 12 2024-09-15T03:46:31.790 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.1:6800/28908,v1:172.21.15.1:6801/28908] [v2:172.21.15.1:6802/28908,v1:172.21.15.1:6803/28908] exists,up cd433a6a-c875-4e6e-b89c-938caecbae3a 2024-09-15T03:46:31.790 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.1:6808/28913,v1:172.21.15.1:6809/28913] [v2:172.21.15.1:6810/28913,v1:172.21.15.1:6811/28913] exists,up 789930f9-42bf-476b-9f09-a21bffe33a79 2024-09-15T03:46:31.790 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.1:6821/28911,v1:172.21.15.1:6823/28911] [v2:172.21.15.1:6825/28911,v1:172.21.15.1:6827/28911] exists,up 9a83fec5-0d63-461c-b516-289443b3882b 2024-09-15T03:46:31.790 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 up in weight 1 up_from 17 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.1:6816/28914,v1:172.21.15.1:6817/28914] [v2:172.21.15.1:6818/28914,v1:172.21.15.1:6819/28914] exists,up d944bf89-e183-496a-9ca3-7949741b4be4 2024-09-15T03:46:31.790 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 up in weight 1 up_from 17 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6803/28468,v1:172.21.15.67:6805/28468] [v2:172.21.15.67:6807/28468,v1:172.21.15.67:6809/28468] exists,up 15ec2dbd-d099-4aff-94b2-1bf9dbb00b56 2024-09-15T03:46:31.790 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 up in weight 1 up_from 17 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6824/28495,v1:172.21.15.67:6825/28495] [v2:172.21.15.67:6826/28495,v1:172.21.15.67:6827/28495] exists,up 1e45de07-746b-4612-9a52-d05509cae9a3 2024-09-15T03:46:31.790 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6800/28469,v1:172.21.15.67:6801/28469] [v2:172.21.15.67:6802/28469,v1:172.21.15.67:6804/28469] exists,up ccf785f8-751d-473e-b17c-3df9346be9f8 2024-09-15T03:46:31.790 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7 up in weight 1 up_from 17 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6816/28470,v1:172.21.15.67:6817/28470] [v2:172.21.15.67:6818/28470,v1:172.21.15.67:6819/28470] exists,up f9f59987-869d-42eb-9eb6-08c39e3afe6d 2024-09-15T03:46:31.790 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8 up in weight 1 up_from 17 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6808/29472,v1:172.21.15.190:6809/29472] [v2:172.21.15.190:6810/29472,v1:172.21.15.190:6811/29472] exists,up 69ddfe52-11df-4608-9a10-e8907c4cd87f 2024-09-15T03:46:31.790 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6800/29473,v1:172.21.15.190:6801/29473] [v2:172.21.15.190:6802/29473,v1:172.21.15.190:6803/29473] exists,up f92e1609-2a1a-4814-9ac0-44a73373d229 2024-09-15T03:46:31.790 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6816/29474,v1:172.21.15.190:6817/29474] [v2:172.21.15.190:6818/29474,v1:172.21.15.190:6819/29474] exists,up 5f2e8ca3-8c65-4595-8e82-27dc3768ab99 2024-09-15T03:46:31.790 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11 up in weight 1 up_from 17 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6824/29475,v1:172.21.15.190:6825/29475] [v2:172.21.15.190:6826/29475,v1:172.21.15.190:6827/29475] exists,up cedf7df4-f875-42f5-a8cf-13cb82f05cc2 2024-09-15T03:46:31.800 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.1:6800/28908,v1:172.21.15.1:6801/28908] [v2:172.21.15.1:6802/28908,v1:172.21.15.1:6803/28908] exists,up cd433a6a-c875-4e6e-b89c-938caecbae3a', 'osd.1 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.1:6808/28913,v1:172.21.15.1:6809/28913] [v2:172.21.15.1:6810/28913,v1:172.21.15.1:6811/28913] exists,up 789930f9-42bf-476b-9f09-a21bffe33a79', 'osd.2 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.1:6821/28911,v1:172.21.15.1:6823/28911] [v2:172.21.15.1:6825/28911,v1:172.21.15.1:6827/28911] exists,up 9a83fec5-0d63-461c-b516-289443b3882b', 'osd.3 up in weight 1 up_from 17 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.1:6816/28914,v1:172.21.15.1:6817/28914] [v2:172.21.15.1:6818/28914,v1:172.21.15.1:6819/28914] exists,up d944bf89-e183-496a-9ca3-7949741b4be4', 'osd.4 up in weight 1 up_from 17 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6803/28468,v1:172.21.15.67:6805/28468] [v2:172.21.15.67:6807/28468,v1:172.21.15.67:6809/28468] exists,up 15ec2dbd-d099-4aff-94b2-1bf9dbb00b56', 'osd.5 up in weight 1 up_from 17 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6824/28495,v1:172.21.15.67:6825/28495] [v2:172.21.15.67:6826/28495,v1:172.21.15.67:6827/28495] exists,up 1e45de07-746b-4612-9a52-d05509cae9a3', 'osd.6 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6800/28469,v1:172.21.15.67:6801/28469] [v2:172.21.15.67:6802/28469,v1:172.21.15.67:6804/28469] exists,up ccf785f8-751d-473e-b17c-3df9346be9f8', 'osd.7 up in weight 1 up_from 17 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6816/28470,v1:172.21.15.67:6817/28470] [v2:172.21.15.67:6818/28470,v1:172.21.15.67:6819/28470] exists,up f9f59987-869d-42eb-9eb6-08c39e3afe6d', 'osd.8 up in weight 1 up_from 17 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6808/29472,v1:172.21.15.190:6809/29472] [v2:172.21.15.190:6810/29472,v1:172.21.15.190:6811/29472] exists,up 69ddfe52-11df-4608-9a10-e8907c4cd87f', 'osd.9 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6800/29473,v1:172.21.15.190:6801/29473] [v2:172.21.15.190:6802/29473,v1:172.21.15.190:6803/29473] exists,up f92e1609-2a1a-4814-9ac0-44a73373d229', 'osd.10 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6816/29474,v1:172.21.15.190:6817/29474] [v2:172.21.15.190:6818/29474,v1:172.21.15.190:6819/29474] exists,up 5f2e8ca3-8c65-4595-8e82-27dc3768ab99', 'osd.11 up in weight 1 up_from 17 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6824/29475,v1:172.21.15.190:6825/29475] [v2:172.21.15.190:6826/29475,v1:172.21.15.190:6827/29475] exists,up cedf7df4-f875-42f5-a8cf-13cb82f05cc2'] 2024-09-15T03:46:31.801 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-15T03:46:31.898 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:46:31.899 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "mon.a", 2024-09-15T03:46:31.899 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster": "ceph", 2024-09-15T03:46:31.899 INFO:teuthology.orchestra.run.smithi001.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-15T03:46:31.899 INFO:teuthology.orchestra.run.smithi001.stdout: "admin_socket_mode": "", 2024-09-15T03:46:31.899 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-15T03:46:31.899 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_client_required": "cephx, none", 2024-09-15T03:46:31.899 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_cluster_required": "cephx", 2024-09-15T03:46:31.899 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_debug": "true", 2024-09-15T03:46:31.899 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-15T03:46:31.900 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-15T03:46:31.900 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_service_required": "cephx", 2024-09-15T03:46:31.900 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-15T03:46:31.900 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_supported": "cephx", 2024-09-15T03:46:31.900 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_aio": "true", 2024-09-15T03:46:31.900 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-15T03:46:31.900 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_aio_poll_ms": "250", 2024-09-15T03:46:31.900 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_aio_reap_max": "16", 2024-09-15T03:46:31.900 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_async_discard": "false", 2024-09-15T03:46:31.901 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_block_size": "4096", 2024-09-15T03:46:31.901 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_debug_aio": "false", 2024-09-15T03:46:31.901 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-15T03:46:31.901 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-15T03:46:31.901 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_debug_inflight_ios": "false", 2024-09-15T03:46:31.901 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_enable_discard": "false", 2024-09-15T03:46:31.901 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_flock_retry": "3", 2024-09-15T03:46:31.901 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-15T03:46:31.901 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_inject_crash": "0", 2024-09-15T03:46:31.901 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-15T03:46:31.901 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_ioring": "false", 2024-09-15T03:46:31.901 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_ioring_hipri": "false", 2024-09-15T03:46:31.902 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-15T03:46:31.902 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-15T03:46:31.902 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-15T03:46:31.902 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-15T03:46:31.902 INFO:teuthology.orchestra.run.smithi001.stdout: "bdev_type": "", 2024-09-15T03:46:31.902 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_alloc_size": "1048576", 2024-09-15T03:46:31.902 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_allocator": "hybrid", 2024-09-15T03:46:31.902 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_buffered_io": "true", 2024-09-15T03:46:31.902 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_check_for_zeros": "false", 2024-09-15T03:46:31.902 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-15T03:46:31.902 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-15T03:46:31.903 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_compact_log_sync": "false", 2024-09-15T03:46:31.903 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-15T03:46:31.903 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-15T03:46:31.903 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-15T03:46:31.903 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-15T03:46:31.903 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_max_log_runway": "4194304", 2024-09-15T03:46:31.903 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_max_prefetch": "1048576", 2024-09-15T03:46:31.903 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_min_flush_size": "524288", 2024-09-15T03:46:31.903 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_min_log_runway": "1048576", 2024-09-15T03:46:31.903 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_replay_recovery": "false", 2024-09-15T03:46:31.903 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-15T03:46:31.904 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-15T03:46:31.904 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_sync_write": "false", 2024-09-15T03:46:31.904 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-15T03:46:31.904 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-15T03:46:31.904 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-15T03:46:31.904 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_allocation_from_file": "true", 2024-09-15T03:46:31.904 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_allocator": "hybrid", 2024-09-15T03:46:31.904 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_async_db_compaction": "true", 2024-09-15T03:46:31.904 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-15T03:46:31.904 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-15T03:46:31.904 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-15T03:46:31.904 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-15T03:46:31.904 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-15T03:46:31.905 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-15T03:46:31.905 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-15T03:46:31.905 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_block_create": "true", 2024-09-15T03:46:31.905 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_block_db_create": "false", 2024-09-15T03:46:31.905 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_block_db_path": "", 2024-09-15T03:46:31.905 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_block_db_size": "0", 2024-09-15T03:46:31.905 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_block_path": "", 2024-09-15T03:46:31.905 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_block_preallocate_file": "false", 2024-09-15T03:46:31.905 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_block_size": "107374182400", 2024-09-15T03:46:31.905 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_block_wal_create": "false", 2024-09-15T03:46:31.905 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_block_wal_path": "", 2024-09-15T03:46:31.906 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_block_wal_size": "100663296", 2024-09-15T03:46:31.906 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_bluefs": "true", 2024-09-15T03:46:31.906 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-15T03:46:31.906 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-15T03:46:31.906 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-15T03:46:31.906 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-15T03:46:31.906 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-15T03:46:31.906 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-15T03:46:31.906 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-15T03:46:31.906 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-15T03:46:31.906 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_autotune": "true", 2024-09-15T03:46:31.906 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-15T03:46:31.907 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-15T03:46:31.907 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-15T03:46:31.907 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-15T03:46:31.907 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_size": "0", 2024-09-15T03:46:31.907 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-15T03:46:31.907 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-15T03:46:31.907 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-15T03:46:31.907 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-15T03:46:31.907 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_type": "2q", 2024-09-15T03:46:31.907 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-15T03:46:31.907 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_clone_cow": "true", 2024-09-15T03:46:31.907 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-15T03:46:31.907 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-15T03:46:31.908 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-15T03:46:31.908 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-15T03:46:31.908 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-15T03:46:31.908 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-15T03:46:31.908 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-15T03:46:31.908 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_compression_mode": "none", 2024-09-15T03:46:31.908 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-15T03:46:31.908 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_csum_type": "crc32c", 2024-09-15T03:46:31.908 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-15T03:46:31.908 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_freelist": "false", 2024-09-15T03:46:31.909 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-15T03:46:31.909 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-15T03:46:31.909 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_inject_bug21040": "false", 2024-09-15T03:46:31.909 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-15T03:46:31.909 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-15T03:46:31.909 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-15T03:46:31.909 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-15T03:46:31.909 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-15T03:46:31.909 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-15T03:46:31.909 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-15T03:46:31.909 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_prefill": "0.000000", 2024-09-15T03:46:31.909 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-15T03:46:31.909 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-15T03:46:31.910 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-15T03:46:31.910 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_small_allocations": "0", 2024-09-15T03:46:31.910 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-15T03:46:31.910 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_default_buffered_read": "true", 2024-09-15T03:46:31.910 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_default_buffered_write": "false", 2024-09-15T03:46:31.910 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-15T03:46:31.910 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-15T03:46:31.910 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-15T03:46:31.910 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-15T03:46:31.910 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-15T03:46:31.910 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-15T03:46:31.910 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-15T03:46:31.910 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-15T03:46:31.910 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fail_eio": "false", 2024-09-15T03:46:31.911 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-15T03:46:31.911 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-15T03:46:31.911 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-15T03:46:31.911 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-15T03:46:31.911 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-15T03:46:31.911 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-15T03:46:31.911 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_on_mount": "false", 2024-09-15T03:46:31.911 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-15T03:46:31.911 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_on_umount": "false", 2024-09-15T03:46:31.911 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-15T03:46:31.911 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-15T03:46:31.911 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-15T03:46:31.911 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-15T03:46:31.911 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-15T03:46:31.912 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-15T03:46:31.912 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-15T03:46:31.912 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-15T03:46:31.912 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_ignore_data_csum": "false", 2024-09-15T03:46:31.912 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-15T03:46:31.912 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-15T03:46:31.912 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-15T03:46:31.912 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-15T03:46:31.912 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_log_op_age": "5.000000", 2024-09-15T03:46:31.912 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_max_alloc_size": "0", 2024-09-15T03:46:31.912 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_max_blob_size": "0", 2024-09-15T03:46:31.912 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-15T03:46:31.912 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-15T03:46:31.913 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-15T03:46:31.913 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_max_deferred_txc": "32", 2024-09-15T03:46:31.913 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_min_alloc_size": "0", 2024-09-15T03:46:31.913 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-15T03:46:31.913 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-15T03:46:31.913 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_nid_prealloc": "1024", 2024-09-15T03:46:31.913 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-15T03:46:31.913 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-15T03:46:31.913 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-15T03:46:31.913 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-15T03:46:31.913 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_retry_disk_reads": "3", 2024-09-15T03:46:31.913 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_rocksdb_cf": "true", 2024-09-15T03:46:31.913 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2024-09-15T03:46:31.913 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", 2024-09-15T03:46:31.914 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-15T03:46:31.914 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-15T03:46:31.914 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-15T03:46:31.914 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-15T03:46:31.914 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_spdk_mem": "512", 2024-09-15T03:46:31.914 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-15T03:46:31.914 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-15T03:46:31.914 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-15T03:46:31.914 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-15T03:46:31.914 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-15T03:46:31.914 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-15T03:46:31.914 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-15T03:46:31.914 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_tracing": "false", 2024-09-15T03:46:31.915 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-15T03:46:31.915 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-15T03:46:31.915 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-15T03:46:31.915 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-15T03:46:31.915 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-15T03:46:31.915 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-15T03:46:31.915 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-15T03:46:31.915 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-15T03:46:31.915 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-15T03:46:31.915 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_zero_block_detection": "false", 2024-09-15T03:46:31.915 INFO:teuthology.orchestra.run.smithi001.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-15T03:46:31.915 INFO:teuthology.orchestra.run.smithi001.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-15T03:46:31.915 INFO:teuthology.orchestra.run.smithi001.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-15T03:46:31.916 INFO:teuthology.orchestra.run.smithi001.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-15T03:46:31.916 INFO:teuthology.orchestra.run.smithi001.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-15T03:46:31.916 INFO:teuthology.orchestra.run.smithi001.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-15T03:46:31.916 INFO:teuthology.orchestra.run.smithi001.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-15T03:46:31.916 INFO:teuthology.orchestra.run.smithi001.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-15T03:46:31.916 INFO:teuthology.orchestra.run.smithi001.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-15T03:46:31.916 INFO:teuthology.orchestra.run.smithi001.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-15T03:46:31.916 INFO:teuthology.orchestra.run.smithi001.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-15T03:46:31.916 INFO:teuthology.orchestra.run.smithi001.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-15T03:46:31.916 INFO:teuthology.orchestra.run.smithi001.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-15T03:46:31.916 INFO:teuthology.orchestra.run.smithi001.stdout: "cephx_cluster_require_signatures": "false", 2024-09-15T03:46:31.916 INFO:teuthology.orchestra.run.smithi001.stdout: "cephx_cluster_require_version": "2", 2024-09-15T03:46:31.916 INFO:teuthology.orchestra.run.smithi001.stdout: "cephx_require_signatures": "false", 2024-09-15T03:46:31.917 INFO:teuthology.orchestra.run.smithi001.stdout: "cephx_require_version": "2", 2024-09-15T03:46:31.917 INFO:teuthology.orchestra.run.smithi001.stdout: "cephx_service_require_signatures": "false", 2024-09-15T03:46:31.917 INFO:teuthology.orchestra.run.smithi001.stdout: "cephx_service_require_version": "2", 2024-09-15T03:46:31.917 INFO:teuthology.orchestra.run.smithi001.stdout: "cephx_sign_messages": "true", 2024-09-15T03:46:31.917 INFO:teuthology.orchestra.run.smithi001.stdout: "chdir": "", 2024-09-15T03:46:31.917 INFO:teuthology.orchestra.run.smithi001.stdout: "client_acl_type": "", 2024-09-15T03:46:31.917 INFO:teuthology.orchestra.run.smithi001.stdout: "client_asio_thread_count": "2", 2024-09-15T03:46:31.917 INFO:teuthology.orchestra.run.smithi001.stdout: "client_cache_mid": "0.750000", 2024-09-15T03:46:31.917 INFO:teuthology.orchestra.run.smithi001.stdout: "client_cache_size": "16384", 2024-09-15T03:46:31.917 INFO:teuthology.orchestra.run.smithi001.stdout: "client_caps_release_delay": "5", 2024-09-15T03:46:31.917 INFO:teuthology.orchestra.run.smithi001.stdout: "client_check_pool_perm": "true", 2024-09-15T03:46:31.917 INFO:teuthology.orchestra.run.smithi001.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-15T03:46:31.917 INFO:teuthology.orchestra.run.smithi001.stdout: "client_debug_force_sync_read": "false", 2024-09-15T03:46:31.917 INFO:teuthology.orchestra.run.smithi001.stdout: "client_debug_getattr_caps": "false", 2024-09-15T03:46:31.918 INFO:teuthology.orchestra.run.smithi001.stdout: "client_debug_inject_features": "", 2024-09-15T03:46:31.918 INFO:teuthology.orchestra.run.smithi001.stdout: "client_debug_inject_tick_delay": "0", 2024-09-15T03:46:31.918 INFO:teuthology.orchestra.run.smithi001.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-15T03:46:31.918 INFO:teuthology.orchestra.run.smithi001.stdout: "client_die_on_failed_remount": "false", 2024-09-15T03:46:31.918 INFO:teuthology.orchestra.run.smithi001.stdout: "client_dirsize_rbytes": "true", 2024-09-15T03:46:31.918 INFO:teuthology.orchestra.run.smithi001.stdout: "client_force_lazyio": "false", 2024-09-15T03:46:31.918 INFO:teuthology.orchestra.run.smithi001.stdout: "client_fs": "", 2024-09-15T03:46:31.918 INFO:teuthology.orchestra.run.smithi001.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-15T03:46:31.918 INFO:teuthology.orchestra.run.smithi001.stdout: "client_inject_release_failure": "false", 2024-09-15T03:46:31.918 INFO:teuthology.orchestra.run.smithi001.stdout: "client_max_inline_size": "4096", 2024-09-15T03:46:31.918 INFO:teuthology.orchestra.run.smithi001.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-15T03:46:31.918 INFO:teuthology.orchestra.run.smithi001.stdout: "client_mds_namespace": "", 2024-09-15T03:46:31.918 INFO:teuthology.orchestra.run.smithi001.stdout: "client_metadata": "", 2024-09-15T03:46:31.919 INFO:teuthology.orchestra.run.smithi001.stdout: "client_mount_gid": "-1", 2024-09-15T03:46:31.919 INFO:teuthology.orchestra.run.smithi001.stdout: "client_mount_timeout": "300", 2024-09-15T03:46:31.919 INFO:teuthology.orchestra.run.smithi001.stdout: "client_mount_uid": "-1", 2024-09-15T03:46:31.919 INFO:teuthology.orchestra.run.smithi001.stdout: "client_mountpoint": "/", 2024-09-15T03:46:31.919 INFO:teuthology.orchestra.run.smithi001.stdout: "client_notify_timeout": "10", 2024-09-15T03:46:31.919 INFO:teuthology.orchestra.run.smithi001.stdout: "client_oc": "true", 2024-09-15T03:46:31.919 INFO:teuthology.orchestra.run.smithi001.stdout: "client_oc_max_dirty": "104857600", 2024-09-15T03:46:31.919 INFO:teuthology.orchestra.run.smithi001.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-15T03:46:31.919 INFO:teuthology.orchestra.run.smithi001.stdout: "client_oc_max_objects": "1000", 2024-09-15T03:46:31.919 INFO:teuthology.orchestra.run.smithi001.stdout: "client_oc_size": "209715200", 2024-09-15T03:46:31.919 INFO:teuthology.orchestra.run.smithi001.stdout: "client_oc_target_dirty": "8388608", 2024-09-15T03:46:31.919 INFO:teuthology.orchestra.run.smithi001.stdout: "client_permissions": "true", 2024-09-15T03:46:31.919 INFO:teuthology.orchestra.run.smithi001.stdout: "client_quota": "true", 2024-09-15T03:46:31.920 INFO:teuthology.orchestra.run.smithi001.stdout: "client_quota_df": "true", 2024-09-15T03:46:31.920 INFO:teuthology.orchestra.run.smithi001.stdout: "client_readahead_max_bytes": "0", 2024-09-15T03:46:31.920 INFO:teuthology.orchestra.run.smithi001.stdout: "client_readahead_max_periods": "4", 2024-09-15T03:46:31.920 INFO:teuthology.orchestra.run.smithi001.stdout: "client_readahead_min": "131072", 2024-09-15T03:46:31.920 INFO:teuthology.orchestra.run.smithi001.stdout: "client_reconnect_stale": "false", 2024-09-15T03:46:31.920 INFO:teuthology.orchestra.run.smithi001.stdout: "client_shutdown_timeout": "30", 2024-09-15T03:46:31.920 INFO:teuthology.orchestra.run.smithi001.stdout: "client_snapdir": ".snap", 2024-09-15T03:46:31.920 INFO:teuthology.orchestra.run.smithi001.stdout: "client_tick_interval": "1", 2024-09-15T03:46:31.920 INFO:teuthology.orchestra.run.smithi001.stdout: "client_trace": "", 2024-09-15T03:46:31.920 INFO:teuthology.orchestra.run.smithi001.stdout: "client_try_dentry_invalidate": "false", 2024-09-15T03:46:31.920 INFO:teuthology.orchestra.run.smithi001.stdout: "client_use_faked_inos": "false", 2024-09-15T03:46:31.920 INFO:teuthology.orchestra.run.smithi001.stdout: "client_use_random_mds": "false", 2024-09-15T03:46:31.920 INFO:teuthology.orchestra.run.smithi001.stdout: "clog_to_graylog": "false", 2024-09-15T03:46:31.921 INFO:teuthology.orchestra.run.smithi001.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-15T03:46:31.921 INFO:teuthology.orchestra.run.smithi001.stdout: "clog_to_graylog_port": "12201", 2024-09-15T03:46:31.921 INFO:teuthology.orchestra.run.smithi001.stdout: "clog_to_monitors": "default=true", 2024-09-15T03:46:31.921 INFO:teuthology.orchestra.run.smithi001.stdout: "clog_to_syslog": "false", 2024-09-15T03:46:31.921 INFO:teuthology.orchestra.run.smithi001.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-15T03:46:31.921 INFO:teuthology.orchestra.run.smithi001.stdout: "clog_to_syslog_level": "info", 2024-09-15T03:46:31.921 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "-", 2024-09-15T03:46:31.921 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_network": "", 2024-09-15T03:46:31.921 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_network_interface": "", 2024-09-15T03:46:31.921 INFO:teuthology.orchestra.run.smithi001.stdout: "compressor_zlib_isal": "false", 2024-09-15T03:46:31.921 INFO:teuthology.orchestra.run.smithi001.stdout: "compressor_zlib_level": "5", 2024-09-15T03:46:31.921 INFO:teuthology.orchestra.run.smithi001.stdout: "compressor_zlib_winsize": "-15", 2024-09-15T03:46:31.921 INFO:teuthology.orchestra.run.smithi001.stdout: "compressor_zstd_level": "1", 2024-09-15T03:46:31.922 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-15T03:46:31.922 INFO:teuthology.orchestra.run.smithi001.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-15T03:46:31.922 INFO:teuthology.orchestra.run.smithi001.stdout: "crimson_alien_op_num_threads": "6", 2024-09-15T03:46:31.922 INFO:teuthology.orchestra.run.smithi001.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-15T03:46:31.922 INFO:teuthology.orchestra.run.smithi001.stdout: "crimson_osd_obc_lru_size": "10", 2024-09-15T03:46:31.922 INFO:teuthology.orchestra.run.smithi001.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-15T03:46:31.922 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "", 2024-09-15T03:46:31.922 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location_hook": "", 2024-09-15T03:46:31.922 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location_hook_timeout": "10", 2024-09-15T03:46:31.922 INFO:teuthology.orchestra.run.smithi001.stdout: "daemonize": "false", 2024-09-15T03:46:31.922 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_alienstore": "0/5", 2024-09-15T03:46:31.922 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_allow_any_pool_priority": "false", 2024-09-15T03:46:31.922 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_asok": "1/5", 2024-09-15T03:46:31.923 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_asok_assert_abort": "false", 2024-09-15T03:46:31.923 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_asserts_on_shutdown": "true", 2024-09-15T03:46:31.923 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_auth": "20/20", 2024-09-15T03:46:31.923 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_bdev": "1/3", 2024-09-15T03:46:31.923 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_bluefs": "1/5", 2024-09-15T03:46:31.923 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_bluestore": "1/5", 2024-09-15T03:46:31.923 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_buffer": "0/1", 2024-09-15T03:46:31.923 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_ceph_exporter": "1/5", 2024-09-15T03:46:31.923 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_cephfs_mirror": "0/5", 2024-09-15T03:46:31.923 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_cephsqlite": "0/5", 2024-09-15T03:46:31.923 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_civetweb": "1/10", 2024-09-15T03:46:31.923 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_client": "0/5", 2024-09-15T03:46:31.923 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_compressor": "1/5", 2024-09-15T03:46:31.924 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_context": "0/1", 2024-09-15T03:46:31.924 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_crush": "1/1", 2024-09-15T03:46:31.924 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_crypto": "1/5", 2024-09-15T03:46:31.924 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_deliberately_leak_memory": "false", 2024-09-15T03:46:31.924 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_disable_randomized_ping": "false", 2024-09-15T03:46:31.924 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_dpdk": "1/5", 2024-09-15T03:46:31.924 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_eventtrace": "1/5", 2024-09-15T03:46:31.924 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_filer": "0/1", 2024-09-15T03:46:31.924 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_filestore": "1/3", 2024-09-15T03:46:31.924 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_finisher": "1/1", 2024-09-15T03:46:31.924 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_fuse": "1/5", 2024-09-15T03:46:31.924 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_heartbeat_testing_span": "0", 2024-09-15T03:46:31.924 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_heartbeatmap": "1/5", 2024-09-15T03:46:31.925 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-15T03:46:31.925 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_javaclient": "1/5", 2024-09-15T03:46:31.925 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_journal": "1/3", 2024-09-15T03:46:31.925 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_journaler": "0/5", 2024-09-15T03:46:31.925 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_kstore": "1/5", 2024-09-15T03:46:31.925 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_leveldb": "4/5", 2024-09-15T03:46:31.925 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_lockdep": "0/1", 2024-09-15T03:46:31.925 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_mclock": "1/5", 2024-09-15T03:46:31.925 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_mds": "1/5", 2024-09-15T03:46:31.925 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_mds_balancer": "1/5", 2024-09-15T03:46:31.925 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_mds_locker": "1/5", 2024-09-15T03:46:31.925 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_mds_log": "1/5", 2024-09-15T03:46:31.925 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_mds_log_expire": "1/5", 2024-09-15T03:46:31.925 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_mds_migrator": "1/5", 2024-09-15T03:46:31.926 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_memdb": "4/5", 2024-09-15T03:46:31.926 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_mgr": "2/5", 2024-09-15T03:46:31.926 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_mgrc": "1/5", 2024-09-15T03:46:31.926 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_mon": "20/20", 2024-09-15T03:46:31.926 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_monc": "0/10", 2024-09-15T03:46:31.926 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_ms": "1/1", 2024-09-15T03:46:31.926 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_none": "0/5", 2024-09-15T03:46:31.926 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_objclass": "0/5", 2024-09-15T03:46:31.926 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_objectcacher": "0/5", 2024-09-15T03:46:31.926 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_objecter": "0/1", 2024-09-15T03:46:31.926 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_optracker": "0/5", 2024-09-15T03:46:31.926 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_osd": "1/5", 2024-09-15T03:46:31.926 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_paxos": "20/20", 2024-09-15T03:46:31.926 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_perfcounter": "1/5", 2024-09-15T03:46:31.927 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_prioritycache": "1/5", 2024-09-15T03:46:31.927 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rados": "0/5", 2024-09-15T03:46:31.927 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rbd": "0/5", 2024-09-15T03:46:31.927 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rbd_mirror": "0/5", 2024-09-15T03:46:31.927 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rbd_pwl": "0/5", 2024-09-15T03:46:31.927 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rbd_replay": "0/5", 2024-09-15T03:46:31.927 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_refs": "0/0", 2024-09-15T03:46:31.927 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_reserver": "1/1", 2024-09-15T03:46:31.927 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rgw": "1/5", 2024-09-15T03:46:31.927 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rgw_datacache": "1/5", 2024-09-15T03:46:31.927 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rgw_sync": "1/5", 2024-09-15T03:46:31.927 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_rocksdb": "4/5", 2024-09-15T03:46:31.927 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore": "0/5", 2024-09-15T03:46:31.928 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore_cache": "0/5", 2024-09-15T03:46:31.928 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore_cleaner": "0/5", 2024-09-15T03:46:31.928 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore_device": "0/5", 2024-09-15T03:46:31.928 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore_journal": "0/5", 2024-09-15T03:46:31.928 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore_lba": "0/5", 2024-09-15T03:46:31.928 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore_odata": "0/5", 2024-09-15T03:46:31.928 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore_omap": "0/5", 2024-09-15T03:46:31.928 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore_onode": "0/5", 2024-09-15T03:46:31.928 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_seastore_tm": "0/5", 2024-09-15T03:46:31.928 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_striper": "0/1", 2024-09-15T03:46:31.928 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_test": "0/5", 2024-09-15T03:46:31.928 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_throttle": "1/1", 2024-09-15T03:46:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_timer": "0/1", 2024-09-15T03:46:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: "debug_tp": "0/5", 2024-09-15T03:46:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-15T03:46:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: "device_failure_prediction_mode": "none", 2024-09-15T03:46:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-15T03:46:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-15T03:46:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: "err_to_graylog": "false", 2024-09-15T03:46:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: "err_to_journald": "false", 2024-09-15T03:46:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: "err_to_stderr": "true", 2024-09-15T03:46:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: "err_to_syslog": "false", 2024-09-15T03:46:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: "event_tracing": "false", 2024-09-15T03:46:31.929 INFO:teuthology.orchestra.run.smithi001.stdout: "exporter_addr": "0.0.0.0", 2024-09-15T03:46:31.930 INFO:teuthology.orchestra.run.smithi001.stdout: "exporter_http_port": "9926", 2024-09-15T03:46:31.930 INFO:teuthology.orchestra.run.smithi001.stdout: "exporter_prio_limit": "5", 2024-09-15T03:46:31.930 INFO:teuthology.orchestra.run.smithi001.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-15T03:46:31.930 INFO:teuthology.orchestra.run.smithi001.stdout: "exporter_sort_metrics": "true", 2024-09-15T03:46:31.930 INFO:teuthology.orchestra.run.smithi001.stdout: "exporter_stats_period": "5", 2024-09-15T03:46:31.930 INFO:teuthology.orchestra.run.smithi001.stdout: "fake_statfs_for_testing": "0", 2024-09-15T03:46:31.930 INFO:teuthology.orchestra.run.smithi001.stdout: "fatal_signal_handlers": "true", 2024-09-15T03:46:31.930 INFO:teuthology.orchestra.run.smithi001.stdout: "filer_max_purge_ops": "10", 2024-09-15T03:46:31.930 INFO:teuthology.orchestra.run.smithi001.stdout: "filer_max_truncate_ops": "128", 2024-09-15T03:46:31.930 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_apply_finisher_threads": "1", 2024-09-15T03:46:31.930 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_blackhole": "false", 2024-09-15T03:46:31.930 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_btrfs_clone_range": "true", 2024-09-15T03:46:31.930 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_btrfs_snap": "true", 2024-09-15T03:46:31.931 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_caller_concurrency": "10", 2024-09-15T03:46:31.931 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_collect_device_partition_information": "true", 2024-09-15T03:46:31.931 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_commit_timeout": "600.000000", 2024-09-15T03:46:31.931 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_debug_inject_read_err": "false", 2024-09-15T03:46:31.931 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_debug_omap_check": "false", 2024-09-15T03:46:31.931 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-15T03:46:31.931 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_debug_verify_split": "false", 2024-09-15T03:46:31.931 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_dump_file": "", 2024-09-15T03:46:31.931 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-15T03:46:31.931 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-15T03:46:31.931 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_fadvise": "true", 2024-09-15T03:46:31.931 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_fail_eio": "true", 2024-09-15T03:46:31.931 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_fd_cache_shards": "16", 2024-09-15T03:46:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_fd_cache_size": "128", 2024-09-15T03:46:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_fiemap": "false", 2024-09-15T03:46:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_fiemap_threshold": "4096", 2024-09-15T03:46:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-15T03:46:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-15T03:46:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_inject_stall": "0", 2024-09-15T03:46:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_journal_parallel": "false", 2024-09-15T03:46:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_journal_trailing": "false", 2024-09-15T03:46:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_journal_writeahead": "false", 2024-09-15T03:46:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_kill_at": "0", 2024-09-15T03:46:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-15T03:46:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-15T03:46:31.932 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-15T03:46:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-15T03:46:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-15T03:46:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_inline_xattrs": "0", 2024-09-15T03:46:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-15T03:46:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-15T03:46:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-15T03:46:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-15T03:46:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_xattr_value_size": "0", 2024-09-15T03:46:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-15T03:46:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-15T03:46:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-15T03:46:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_merge_threshold": "-10", 2024-09-15T03:46:31.933 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-15T03:46:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_odsync_write": "false", 2024-09-15T03:46:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_omap_backend": "rocksdb", 2024-09-15T03:46:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_omap_backend_path": "", 2024-09-15T03:46:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-15T03:46:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-15T03:46:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-15T03:46:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_op_thread_timeout": "60", 2024-09-15T03:46:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_op_threads": "2", 2024-09-15T03:46:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_punch_hole": "false", 2024-09-15T03:46:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-15T03:46:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-15T03:46:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-15T03:46:31.934 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-15T03:46:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-15T03:46:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-15T03:46:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-15T03:46:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-15T03:46:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-15T03:46:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_queue_max_ops": "50", 2024-09-15T03:46:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-15T03:46:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_seek_data_hole": "false", 2024-09-15T03:46:31.935 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_sloppy_crc": "false", 2024-09-15T03:46:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-15T03:46:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_splice": "false", 2024-09-15T03:46:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_split_multiple": "2", 2024-09-15T03:46:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_split_rand_factor": "20", 2024-09-15T03:46:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_update_to": "1000", 2024-09-15T03:46:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-15T03:46:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-15T03:46:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-15T03:46:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-15T03:46:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-15T03:46:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-15T03:46:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_enable": "true", 2024-09-15T03:46:31.936 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-15T03:46:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-15T03:46:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-15T03:46:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-15T03:46:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-15T03:46:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-15T03:46:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_xfs_extsize": "false", 2024-09-15T03:46:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: "filestore_zfs_snap": "false", 2024-09-15T03:46:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: "fio_dir": "/tmp/fio", 2024-09-15T03:46:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "f27b540e-e5e3-4e95-afe8-c95ee66b0a62", 2024-09-15T03:46:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_allow_other": "true", 2024-09-15T03:46:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_atomic_o_trunc": "true", 2024-09-15T03:46:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_big_writes": "true", 2024-09-15T03:46:31.937 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_debug": "false", 2024-09-15T03:46:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_default_permissions": "false", 2024-09-15T03:46:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_disable_pagecache": "false", 2024-09-15T03:46:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_max_write": "0", 2024-09-15T03:46:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_multithreaded": "true", 2024-09-15T03:46:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_require_active_mds": "true", 2024-09-15T03:46:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_set_user_groups": "true", 2024-09-15T03:46:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_splice_move": "true", 2024-09-15T03:46:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_splice_read": "true", 2024-09-15T03:46:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_splice_write": "true", 2024-09-15T03:46:31.938 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-15T03:46:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: "fuse_use_invalidate_cb": "true", 2024-09-15T03:46:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-15T03:46:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: "gss_target_name": "ceph", 2024-09-15T03:46:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_file": "", 2024-09-15T03:46:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_inject_failure": "0", 2024-09-15T03:46:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_interval": "5", 2024-09-15T03:46:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: "host": "smithi001", 2024-09-15T03:46:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-15T03:46:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-15T03:46:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-15T03:46:31.939 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-15T03:46:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-15T03:46:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-15T03:46:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-15T03:46:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-15T03:46:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-15T03:46:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-15T03:46:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-15T03:46:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-15T03:46:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-15T03:46:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: "inject_early_sigterm": "false", 2024-09-15T03:46:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: "jaeger_tracing_enable": "false", 2024-09-15T03:46:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_aio": "true", 2024-09-15T03:46:31.940 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_align_min_size": "65536", 2024-09-15T03:46:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_block_align": "true", 2024-09-15T03:46:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_block_size": "4096", 2024-09-15T03:46:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_dio": "true", 2024-09-15T03:46:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_discard": "false", 2024-09-15T03:46:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_force_aio": "false", 2024-09-15T03:46:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_ignore_corruption": "false", 2024-09-15T03:46:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_max_write_bytes": "10485760", 2024-09-15T03:46:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_max_write_entries": "100", 2024-09-15T03:46:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_replay_from": "0", 2024-09-15T03:46:31.941 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-15T03:46:31.942 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-15T03:46:31.942 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-15T03:46:31.942 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-15T03:46:31.942 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_write_header_frequency": "0", 2024-09-15T03:46:31.942 INFO:teuthology.orchestra.run.smithi001.stdout: "journal_zero_on_create": "false", 2024-09-15T03:46:31.942 INFO:teuthology.orchestra.run.smithi001.stdout: "journaler_prefetch_periods": "10", 2024-09-15T03:46:31.943 INFO:teuthology.orchestra.run.smithi001.stdout: "journaler_prezero_periods": "5", 2024-09-15T03:46:31.943 INFO:teuthology.orchestra.run.smithi001.stdout: "journaler_write_head_interval": "15", 2024-09-15T03:46:31.943 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "", 2024-09-15T03:46:31.943 INFO:teuthology.orchestra.run.smithi001.stdout: "keyfile": "", 2024-09-15T03:46:31.943 INFO:teuthology.orchestra.run.smithi001.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-15T03:46:31.943 INFO:teuthology.orchestra.run.smithi001.stdout: "kstore_backend": "rocksdb", 2024-09-15T03:46:31.943 INFO:teuthology.orchestra.run.smithi001.stdout: "kstore_default_stripe_size": "65536", 2024-09-15T03:46:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: "kstore_fsck_on_mount": "false", 2024-09-15T03:46:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-15T03:46:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: "kstore_max_bytes": "67108864", 2024-09-15T03:46:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: "kstore_max_ops": "512", 2024-09-15T03:46:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: "kstore_nid_prealloc": "1024", 2024-09-15T03:46:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: "kstore_onode_map_size": "1024", 2024-09-15T03:46:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-15T03:46:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: "kstore_sync_submit_transaction": "false", 2024-09-15T03:46:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: "kstore_sync_transaction": "false", 2024-09-15T03:46:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: "leveldb_block_size": "65536", 2024-09-15T03:46:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: "leveldb_bloom_size": "0", 2024-09-15T03:46:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: "leveldb_cache_size": "536870912", 2024-09-15T03:46:31.944 INFO:teuthology.orchestra.run.smithi001.stdout: "leveldb_compact_on_mount": "false", 2024-09-15T03:46:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: "leveldb_compression": "false", 2024-09-15T03:46:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: "leveldb_log": "", 2024-09-15T03:46:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: "leveldb_log_to_ceph_log": "true", 2024-09-15T03:46:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: "leveldb_max_open_files": "0", 2024-09-15T03:46:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: "leveldb_paranoid": "false", 2024-09-15T03:46:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: "leveldb_write_buffer_size": "33554432", 2024-09-15T03:46:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: "librados_thread_count": "2", 2024-09-15T03:46:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: "lockdep": "false", 2024-09-15T03:46:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: "lockdep_force_backtrace": "false", 2024-09-15T03:46:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: "log_coarse_timestamps": "true", 2024-09-15T03:46:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-15T03:46:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: "log_flush_on_exit": "false", 2024-09-15T03:46:31.945 INFO:teuthology.orchestra.run.smithi001.stdout: "log_graylog_host": "127.0.0.1", 2024-09-15T03:46:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: "log_graylog_port": "12201", 2024-09-15T03:46:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: "log_max_new": "1000", 2024-09-15T03:46:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: "log_max_recent": "10000", 2024-09-15T03:46:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: "log_stderr_prefix": "", 2024-09-15T03:46:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: "log_stop_at_utilization": "0.970000", 2024-09-15T03:46:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": "true", 2024-09-15T03:46:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_graylog": "false", 2024-09-15T03:46:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_journald": "false", 2024-09-15T03:46:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_stderr": "false", 2024-09-15T03:46:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_syslog": "false", 2024-09-15T03:46:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: "max_rotating_auth_attempts": "10", 2024-09-15T03:46:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-15T03:46:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_action_on_write_error": "1", 2024-09-15T03:46:31.946 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_alternate_name_max": "8192", 2024-09-15T03:46:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_asio_thread_count": "2", 2024-09-15T03:46:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_export_pin": "true", 2024-09-15T03:46:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_fragment_dirs": "true", 2024-09-15T03:46:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-15T03:46:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_fragment_interval": "5", 2024-09-15T03:46:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-15T03:46:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-15T03:46:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_interval": "10", 2024-09-15T03:46:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_max": "-1", 2024-09-15T03:46:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_max_until": "-1", 2024-09-15T03:46:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_merge_size": "50", 2024-09-15T03:46:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_midchunk": "0.300000", 2024-09-15T03:46:31.947 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-15T03:46:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_min_start": "0.200000", 2024-09-15T03:46:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_minchunk": "0.001000", 2024-09-15T03:46:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_mode": "0", 2024-09-15T03:46:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_need_max": "1.200000", 2024-09-15T03:46:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_need_min": "0.800000", 2024-09-15T03:46:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-15T03:46:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-15T03:46:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_split_bits": "3", 2024-09-15T03:46:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-15T03:46:31.948 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_split_size": "10000", 2024-09-15T03:46:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-15T03:46:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_target_decay": "10.000000", 2024-09-15T03:46:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-15T03:46:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_beacon_grace": "15.000000", 2024-09-15T03:46:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_beacon_interval": "4.000000", 2024-09-15T03:46:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-15T03:46:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-15T03:46:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cache_mid": "0.700000", 2024-09-15T03:46:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cache_release_free_interval": "10", 2024-09-15T03:46:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cache_reservation": "0.050000", 2024-09-15T03:46:31.949 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-15T03:46:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cache_trim_interval": "1", 2024-09-15T03:46:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cache_trim_threshold": "262144", 2024-09-15T03:46:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-15T03:46:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-15T03:46:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-15T03:46:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_client_prealloc_inos": "1000", 2024-09-15T03:46:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-15T03:46:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_connect_bootstrapping": "false", 2024-09-15T03:46:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_damage_table_max_entries": "10000", 2024-09-15T03:46:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-15T03:46:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_debug_auth_pins": "false", 2024-09-15T03:46:31.950 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_debug_frag": "false", 2024-09-15T03:46:31.951 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_debug_scatterstat": "false", 2024-09-15T03:46:31.951 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_debug_subtrees": "false", 2024-09-15T03:46:31.951 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_decay_halflife": "5.000000", 2024-09-15T03:46:31.951 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_default_dir_hash": "2", 2024-09-15T03:46:31.951 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_defer_session_stale": "true", 2024-09-15T03:46:31.951 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_deny_all_reconnect": "false", 2024-09-15T03:46:31.951 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_dir_keys_per_op": "16384", 2024-09-15T03:46:31.951 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_dir_max_commit_size": "10", 2024-09-15T03:46:31.951 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_dir_max_entries": "0", 2024-09-15T03:46:31.951 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-15T03:46:31.951 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-15T03:46:31.951 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_dump_cache_on_map": "false", 2024-09-15T03:46:31.951 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-15T03:46:31.951 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-15T03:46:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_early_reply": "true", 2024-09-15T03:46:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_enable_op_tracker": "true", 2024-09-15T03:46:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_enforce_unique_name": "true", 2024-09-15T03:46:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-15T03:46:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-15T03:46:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_export_ephemeral_random": "true", 2024-09-15T03:46:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-15T03:46:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-15T03:46:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-15T03:46:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-15T03:46:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-15T03:46:31.952 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-15T03:46:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_health_summarize_threshold": "10", 2024-09-15T03:46:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-15T03:46:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-15T03:46:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_inject_health_dummy": "false", 2024-09-15T03:46:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-15T03:46:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_inject_migrator_session_race": "false", 2024-09-15T03:46:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-15T03:46:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-15T03:46:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-15T03:46:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_join_fs": "", 2024-09-15T03:46:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_journal_format": "1", 2024-09-15T03:46:31.953 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_create_at": "0", 2024-09-15T03:46:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_export_at": "0", 2024-09-15T03:46:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_import_at": "0", 2024-09-15T03:46:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_journal_at": "0", 2024-09-15T03:46:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_journal_expire_at": "0", 2024-09-15T03:46:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_journal_replay_at": "0", 2024-09-15T03:46:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_link_at": "0", 2024-09-15T03:46:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_mdstable_at": "0", 2024-09-15T03:46:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_openc_at": "0", 2024-09-15T03:46:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_rename_at": "0", 2024-09-15T03:46:31.954 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-09-15T03:46:31.955 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_log_events_per_segment": "1024", 2024-09-15T03:46:31.955 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_log_max_events": "-1", 2024-09-15T03:46:31.955 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_log_max_segments": "128", 2024-09-15T03:46:31.955 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_log_pause": "false", 2024-09-15T03:46:31.955 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_log_segment_size": "0", 2024-09-15T03:46:31.955 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-15T03:46:31.955 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_log_warn_factor": "2.000000", 2024-09-15T03:46:31.955 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_max_caps_per_client": "1048576", 2024-09-15T03:46:31.955 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_max_completed_flushes": "100000", 2024-09-15T03:46:31.955 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_max_completed_requests": "100000", 2024-09-15T03:46:31.956 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_max_export_size": "20971520", 2024-09-15T03:46:31.956 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_max_file_recover": "32", 2024-09-15T03:46:31.956 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_max_purge_files": "64", 2024-09-15T03:46:31.956 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_max_purge_ops": "8192", 2024-09-15T03:46:31.956 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-15T03:46:31.956 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-15T03:46:31.956 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_max_snaps_per_dir": "100", 2024-09-15T03:46:31.956 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_max_xattr_pairs_size": "65536", 2024-09-15T03:46:31.956 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_metrics_update_interval": "2", 2024-09-15T03:46:31.956 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_min_caps_per_client": "100", 2024-09-15T03:46:31.956 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_min_caps_working_set": "10000", 2024-09-15T03:46:31.956 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-15T03:46:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_numa_node": "-1", 2024-09-15T03:46:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-15T03:46:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_op_complaint_time": "30.000000", 2024-09-15T03:46:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_op_history_duration": "600", 2024-09-15T03:46:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_op_history_size": "20", 2024-09-15T03:46:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_op_history_slow_op_size": "20", 2024-09-15T03:46:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-15T03:46:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_op_log_threshold": "5", 2024-09-15T03:46:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_ping_grace": "15", 2024-09-15T03:46:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_ping_interval": "5", 2024-09-15T03:46:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-15T03:46:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-15T03:46:31.957 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_recall_max_caps": "30000", 2024-09-15T03:46:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-15T03:46:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-15T03:46:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-15T03:46:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_recall_warning_threshold": "262144", 2024-09-15T03:46:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-15T03:46:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_replay_interval": "1.000000", 2024-09-15T03:46:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-15T03:46:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-15T03:46:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_root_ino_gid": "0", 2024-09-15T03:46:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_root_ino_uid": "0", 2024-09-15T03:46:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-15T03:46:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-15T03:46:31.958 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-15T03:46:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-15T03:46:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-15T03:46:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-15T03:46:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-15T03:46:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-15T03:46:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-15T03:46:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-15T03:46:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_shutdown_check": "0", 2024-09-15T03:46:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_skip_ino": "0", 2024-09-15T03:46:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-15T03:46:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_snap_max_uid": "4294967294", 2024-09-15T03:46:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_snap_min_uid": "0", 2024-09-15T03:46:31.959 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_snap_rstat": "false", 2024-09-15T03:46:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_standby_replay_damaged": "false", 2024-09-15T03:46:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_symlink_recovery": "true", 2024-09-15T03:46:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-15T03:46:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_thrash_exports": "0", 2024-09-15T03:46:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_thrash_fragments": "0", 2024-09-15T03:46:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_tick_interval": "5.000000", 2024-09-15T03:46:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_valgrind_exit": "false", 2024-09-15T03:46:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_verify_backtrace": "1", 2024-09-15T03:46:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_verify_scatter": "false", 2024-09-15T03:46:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-15T03:46:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_wipe_sessions": "false", 2024-09-15T03:46:31.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mempool_debug": "false", 2024-09-15T03:46:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-15T03:46:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: "memstore_device_bytes": "1073741824", 2024-09-15T03:46:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: "memstore_page_set": "false", 2024-09-15T03:46:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: "memstore_page_size": "65536", 2024-09-15T03:46:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_client_bytes": "134217728", 2024-09-15T03:46:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_client_messages": "512", 2024-09-15T03:46:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-15T03:46:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-15T03:46:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-15T03:46:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-15T03:46:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_disabled_modules": "diskprediction_local", 2024-09-15T03:46:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-15T03:46:31.961 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_max_pg_num_change": "128", 2024-09-15T03:46:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_mds_bytes": "134217728", 2024-09-15T03:46:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_mds_messages": "128", 2024-09-15T03:46:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-15T03:46:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_mon_bytes": "134217728", 2024-09-15T03:46:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_mon_messages": "128", 2024-09-15T03:46:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_osd_bytes": "536870912", 2024-09-15T03:46:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_osd_messages": "8192", 2024-09-15T03:46:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_pool": "true", 2024-09-15T03:46:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-15T03:46:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_standby_modules": "true", 2024-09-15T03:46:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_stats_period": "5", 2024-09-15T03:46:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_stats_threshold": "5", 2024-09-15T03:46:31.962 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_tick_period": "2", 2024-09-15T03:46:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-15T03:46:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-15T03:46:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_allow_pool_delete": "true", 2024-09-15T03:46:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_allow_pool_size_one": "true", 2024-09-15T03:46:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_auth_validate_all_caps": "true", 2024-09-15T03:46:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-15T03:46:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-15T03:46:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_bytes": "104857600", 2024-09-15T03:46:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_directed_command_retry": "5", 2024-09-15T03:46:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-15T03:46:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-15T03:46:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-15T03:46:31.963 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-15T03:46:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_hunt_parallel": "3", 2024-09-15T03:46:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_log_interval": "1.000000", 2024-09-15T03:46:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-15T03:46:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_ping_interval": "10.000000", 2024-09-15T03:46:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-15T03:46:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-15T03:46:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-15T03:46:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-15T03:46:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cluster_log_level": "debug", 2024-09-15T03:46:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cluster_log_to_file": "true", 2024-09-15T03:46:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-15T03:46:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-15T03:46:31.964 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-15T03:46:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cluster_log_to_journald": "false", 2024-09-15T03:46:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-15T03:46:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-15T03:46:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-15T03:46:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_compact_on_bootstrap": "false", 2024-09-15T03:46:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_compact_on_start": "false", 2024-09-15T03:46:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_compact_on_trim": "true", 2024-09-15T03:46:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-15T03:46:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-15T03:46:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-15T03:46:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_cpu_threads": "4", 2024-09-15T03:46:31.965 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_crush_min_required_version": "hammer", 2024-09-15T03:46:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_daemon_bytes": "419430400", 2024-09-15T03:46:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-15T03:46:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_data_avail_crit": "5", 2024-09-15T03:46:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_data_avail_warn": "5", 2024-09-15T03:46:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_data_size_warn": "16106127360", 2024-09-15T03:46:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-15T03:46:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-15T03:46:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_debug_dump_json": "false", 2024-09-15T03:46:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-15T03:46:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_debug_dump_transactions": "false", 2024-09-15T03:46:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_debug_extra_checks": "false", 2024-09-15T03:46:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-15T03:46:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-15T03:46:31.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_debug_no_require_pacific": "false", 2024-09-15T03:46:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_debug_no_require_quincy": "false", 2024-09-15T03:46:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-15T03:46:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-15T03:46:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-15T03:46:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_down_mkfs_grace": "120", 2024-09-15T03:46:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_election_default_strategy": "1", 2024-09-15T03:46:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_election_timeout": "5.000000", 2024-09-15T03:46:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-15T03:46:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_elector_ping_divisor": "2", 2024-09-15T03:46:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-15T03:46:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_enable_op_tracker": "true", 2024-09-15T03:46:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_fake_pool_delete": "false", 2024-09-15T03:46:31.967 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_force_quorum_join": "false", 2024-09-15T03:46:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_globalid_prealloc": "10000", 2024-09-15T03:46:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_health_detail_to_clog": "false", 2024-09-15T03:46:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_health_log_update_period": "5", 2024-09-15T03:46:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_health_max_detail": "50", 2024-09-15T03:46:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_health_to_clog": "true", 2024-09-15T03:46:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_health_to_clog_interval": "600", 2024-09-15T03:46:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-15T03:46:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_host": "172.21.15.1,172.21.15.67,172.21.15.190", 2024-09-15T03:46:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_host_override": "", 2024-09-15T03:46:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_initial_members": "", 2024-09-15T03:46:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-15T03:46:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-15T03:46:31.968 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-15T03:46:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-15T03:46:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-15T03:46:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_lease": "5.000000", 2024-09-15T03:46:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-15T03:46:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-15T03:46:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_log_full_interval": "50", 2024-09-15T03:46:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_log_max": "10000", 2024-09-15T03:46:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_log_max_summary": "50", 2024-09-15T03:46:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-15T03:46:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_max_log_epochs": "500", 2024-09-15T03:46:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-15T03:46:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-15T03:46:31.969 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_max_osd": "10000", 2024-09-15T03:46:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_max_pg_per_osd": "10000", 2024-09-15T03:46:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_max_pool_pg_num": "65536", 2024-09-15T03:46:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-15T03:46:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-15T03:46:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_mds_force_trim_to": "0", 2024-09-15T03:46:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_mds_skip_sanity": "false", 2024-09-15T03:46:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_memory_autotune": "true", 2024-09-15T03:46:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_memory_target": "2147483648", 2024-09-15T03:46:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_mgr_beacon_grace": "30", 2024-09-15T03:46:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-15T03:46:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_mgr_digest_period": "5", 2024-09-15T03:46:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_mgr_inactive_grace": "60", 2024-09-15T03:46:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-15T03:46:31.970 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-15T03:46:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_min_osdmap_epochs": "500", 2024-09-15T03:46:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_op_complaint_time": "30", 2024-09-15T03:46:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_op_history_duration": "600", 2024-09-15T03:46:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_op_history_size": "20", 2024-09-15T03:46:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_op_history_slow_op_size": "20", 2024-09-15T03:46:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-15T03:46:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_op_log_threshold": "5", 2024-09-15T03:46:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-15T03:46:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-15T03:46:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-15T03:46:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_auto_mark_in": "false", 2024-09-15T03:46:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-15T03:46:31.971 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-15T03:46:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-15T03:46:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_cache_size": "500", 2024-09-15T03:46:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-15T03:46:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-15T03:46:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-15T03:46:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_down_out_interval": "600", 2024-09-15T03:46:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-15T03:46:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-15T03:46:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_force_trim_to": "0", 2024-09-15T03:46:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-15T03:46:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-15T03:46:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-15T03:46:31.972 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-15T03:46:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-15T03:46:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-15T03:46:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-15T03:46:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-15T03:46:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_min_down_reporters": "2", 2024-09-15T03:46:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-15T03:46:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-15T03:46:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-15T03:46:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-15T03:46:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-15T03:46:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-15T03:46:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_report_timeout": "900", 2024-09-15T03:46:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-15T03:46:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-15T03:46:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-15T03:46:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-15T03:46:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-15T03:46:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-15T03:46:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-15T03:46:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-15T03:46:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-15T03:46:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_pg_stuck_threshold": "60", 2024-09-15T03:46:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-15T03:46:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-15T03:46:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-15T03:46:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-15T03:46:31.974 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-15T03:46:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-15T03:46:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_probe_timeout": "2.000000", 2024-09-15T03:46:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_reweight_max_change": "0.050000", 2024-09-15T03:46:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_reweight_max_osds": "4", 2024-09-15T03:46:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-15T03:46:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-15T03:46:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-15T03:46:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-15T03:46:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-15T03:46:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_scrub_interval": "300", 2024-09-15T03:46:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_scrub_max_keys": "100", 2024-09-15T03:46:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_scrub_timeout": "300", 2024-09-15T03:46:31.975 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_session_timeout": "300", 2024-09-15T03:46:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_smart_report_timeout": "5", 2024-09-15T03:46:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_stat_smooth_intervals": "6", 2024-09-15T03:46:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-15T03:46:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_stretch_pool_min_size": "2", 2024-09-15T03:46:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_stretch_pool_size": "4", 2024-09-15T03:46:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-15T03:46:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-15T03:46:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_sync_debug": "false", 2024-09-15T03:46:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-15T03:46:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-15T03:46:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_sync_provider_kill_at": "0", 2024-09-15T03:46:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_sync_requester_kill_at": "0", 2024-09-15T03:46:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_sync_timeout": "60.000000", 2024-09-15T03:46:31.976 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_target_pg_per_osd": "100", 2024-09-15T03:46:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_tick_interval": "5", 2024-09-15T03:46:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_timecheck_interval": "300.000000", 2024-09-15T03:46:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-15T03:46:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_older_version_delay": "604800", 2024-09-15T03:46:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-15T03:46:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-15T03:46:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-15T03:46:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-15T03:46:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-15T03:46:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-15T03:46:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-15T03:46:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_misplaced": "false", 2024-09-15T03:46:31.977 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-15T03:46:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_older_version": "true", 2024-09-15T03:46:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-15T03:46:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-15T03:46:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-15T03:46:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-15T03:46:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-15T03:46:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-15T03:46:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-15T03:46:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-15T03:46:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-15T03:46:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": "", 2024-09-15T03:46:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_op_threads": "3", 2024-09-15T03:46:31.978 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-15T03:46:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_cm": "false", 2024-09-15T03:46:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_device_name": "", 2024-09-15T03:46:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_dscp": "96", 2024-09-15T03:46:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-15T03:46:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-15T03:46:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_local_gid": "", 2024-09-15T03:46:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-15T03:46:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_port_num": "1", 2024-09-15T03:46:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-15T03:46:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-15T03:46:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-15T03:46:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-15T03:46:31.979 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_sl": "3", 2024-09-15T03:46:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_support_srq": "true", 2024-09-15T03:46:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_rdma_type": "ib", 2024-09-15T03:46:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_async_reap_threshold": "5", 2024-09-15T03:46:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_bind_before_connect": "false", 2024-09-15T03:46:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_bind_ipv4": "true", 2024-09-15T03:46:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_bind_ipv6": "false", 2024-09-15T03:46:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_bind_msgr1": "true", 2024-09-15T03:46:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_bind_msgr2": "true", 2024-09-15T03:46:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_bind_port_max": "7568", 2024-09-15T03:46:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_bind_port_min": "6800", 2024-09-15T03:46:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-15T03:46:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_bind_retry_count": "3", 2024-09-15T03:46:31.980 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_bind_retry_delay": "5", 2024-09-15T03:46:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_blackhole_client": "false", 2024-09-15T03:46:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_blackhole_mds": "false", 2024-09-15T03:46:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_blackhole_mgr": "false", 2024-09-15T03:46:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_blackhole_mon": "false", 2024-09-15T03:46:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_blackhole_osd": "false", 2024-09-15T03:46:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_client_mode": "crc secure", 2024-09-15T03:46:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_cluster_mode": "crc secure", 2024-09-15T03:46:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_cluster_type": "", 2024-09-15T03:46:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_compress_secure": "false", 2024-09-15T03:46:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_connection_idle_timeout": "900", 2024-09-15T03:46:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_connection_ready_timeout": "10", 2024-09-15T03:46:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_crc_data": "true", 2024-09-15T03:46:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_crc_header": "true", 2024-09-15T03:46:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_die_on_bad_msg": "false", 2024-09-15T03:46:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_die_on_bug": "true", 2024-09-15T03:46:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_die_on_old_message": "true", 2024-09-15T03:46:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_die_on_skipped_message": "false", 2024-09-15T03:46:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-15T03:46:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-15T03:46:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_coremask": "0xF", 2024-09-15T03:46:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-15T03:46:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-15T03:46:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_enable_tso": "true", 2024-09-15T03:46:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-15T03:46:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-15T03:46:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_hugepages": "", 2024-09-15T03:46:31.982 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-15T03:46:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-15T03:46:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_lro": "true", 2024-09-15T03:46:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_memory_channel": "4", 2024-09-15T03:46:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-15T03:46:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_pmd": "", 2024-09-15T03:46:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_port_id": "0", 2024-09-15T03:46:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-15T03:46:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-15T03:46:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_dump_on_send": "false", 2024-09-15T03:46:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_initial_backoff": "0.200000", 2024-09-15T03:46:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_inject_delay_max": "1.000000", 2024-09-15T03:46:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-15T03:46:31.983 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_inject_delay_type": "", 2024-09-15T03:46:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-15T03:46:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_inject_socket_failures": "2500", 2024-09-15T03:46:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_learn_addr_from_peer": "true", 2024-09-15T03:46:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_max_accept_failures": "4", 2024-09-15T03:46:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_max_backoff": "15.000000", 2024-09-15T03:46:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_mon_client_mode": "secure crc", 2024-09-15T03:46:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-15T03:46:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_mon_service_mode": "secure crc", 2024-09-15T03:46:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_osd_compress_min_size": "1024", 2024-09-15T03:46:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_osd_compress_mode": "none", 2024-09-15T03:46:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_osd_compression_algorithm": "snappy zlib zstd lz4", 2024-09-15T03:46:31.984 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-15T03:46:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_pq_min_cost": "65536", 2024-09-15T03:46:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_public_type": "", 2024-09-15T03:46:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_service_mode": "crc secure", 2024-09-15T03:46:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_tcp_listen_backlog": "512", 2024-09-15T03:46:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_tcp_nodelay": "true", 2024-09-15T03:46:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-15T03:46:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_tcp_rcvbuf": "0", 2024-09-15T03:46:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "ms_type": "async+posix", 2024-09-15T03:46:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "no_config_file": "false", 2024-09-15T03:46:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "objecter_completion_locks_per_session": "32", 2024-09-15T03:46:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-15T03:46:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-15T03:46:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "objecter_inflight_ops": "1024", 2024-09-15T03:46:31.985 INFO:teuthology.orchestra.run.smithi001.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-15T03:46:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-15T03:46:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: "objecter_tick_interval": "5.000000", 2024-09-15T03:46:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: "objecter_timeout": "10.000000", 2024-09-15T03:46:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: "objectstore_blackhole": "false", 2024-09-15T03:46:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: "openssl_engine_opts": "", 2024-09-15T03:46:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_agent_delay_time": "5.000000", 2024-09-15T03:46:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_agent_hist_halflife": "1000", 2024-09-15T03:46:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_agent_max_low_ops": "2", 2024-09-15T03:46:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_agent_max_ops": "4", 2024-09-15T03:46:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-15T03:46:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-15T03:46:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_agent_slop": "0.020000", 2024-09-15T03:46:31.986 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-15T03:46:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-15T03:46:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_asio_thread_count": "2", 2024-09-15T03:46:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_async_recovery_min_cost": "1", 2024-09-15T03:46:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-15T03:46:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-15T03:46:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_backfill_scan_max": "512", 2024-09-15T03:46:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_backfill_scan_min": "64", 2024-09-15T03:46:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_backoff_on_degraded": "false", 2024-09-15T03:46:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_backoff_on_peering": "false", 2024-09-15T03:46:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_backoff_on_unfound": "true", 2024-09-15T03:46:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_beacon_report_interval": "300", 2024-09-15T03:46:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_bench_duration": "30", 2024-09-15T03:46:31.987 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-15T03:46:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_bench_max_block_size": "67108864", 2024-09-15T03:46:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-15T03:46:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_blkin_trace_all": "false", 2024-09-15T03:46:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-15T03:46:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-15T03:46:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_check_for_log_corruption": "false", 2024-09-15T03:46:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-15T03:46:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-15T03:46:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-15T03:46:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-15T03:46:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_class_update_on_start": "true", 2024-09-15T03:46:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_client_message_cap": "256", 2024-09-15T03:46:31.988 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_client_message_size_cap": "524288000", 2024-09-15T03:46:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_client_op_priority": "63", 2024-09-15T03:46:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_client_watch_timeout": "30", 2024-09-15T03:46:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_command_max_records": "256", 2024-09-15T03:46:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_compact_on_start": "false", 2024-09-15T03:46:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-15T03:46:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-15T03:46:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-15T03:46:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_crush_update_on_start": "true", 2024-09-15T03:46:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_crush_update_weight_set": "true", 2024-09-15T03:46:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-15T03:46:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-15T03:46:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-15T03:46:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-15T03:46:31.989 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-15T03:46:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_feed_pullee": "-1", 2024-09-15T03:46:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-15T03:46:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-15T03:46:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-15T03:46:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_misdirected_ops": "false", 2024-09-15T03:46:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_no_acting_change": "false", 2024-09-15T03:46:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_no_purge_strays": "false", 2024-09-15T03:46:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_op_order": "false", 2024-09-15T03:46:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-15T03:46:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-15T03:46:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-15T03:46:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-15T03:46:31.990 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_shutdown": "false", 2024-09-15T03:46:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-15T03:46:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-15T03:46:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-15T03:46:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-15T03:46:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_verify_snaps": "false", 2024-09-15T03:46:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-15T03:46:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-15T03:46:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_deep_scrub_keys": "1024", 2024-09-15T03:46:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-15T03:46:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-15T03:46:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-15T03:46:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_deep_scrub_stride": "524288", 2024-09-15T03:46:31.991 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-15T03:46:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-15T03:46:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_default_notify_timeout": "30", 2024-09-15T03:46:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_delete_sleep": "0.000000", 2024-09-15T03:46:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-15T03:46:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-15T03:46:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-15T03:46:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_discard_disconnected_ops": "true", 2024-09-15T03:46:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_enable_op_tracker": "true", 2024-09-15T03:46:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-15T03:46:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-15T03:46:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-15T03:46:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_fast_info": "true", 2024-09-15T03:46:31.992 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_fast_shutdown": "true", 2024-09-15T03:46:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-15T03:46:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-15T03:46:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-15T03:46:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-15T03:46:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-15T03:46:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_function_tracing": "false", 2024-09-15T03:46:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_heartbeat_grace": "20", 2024-09-15T03:46:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_heartbeat_interval": "6", 2024-09-15T03:46:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-15T03:46:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_heartbeat_min_peers": "10", 2024-09-15T03:46:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_heartbeat_min_size": "2000", 2024-09-15T03:46:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_heartbeat_stale": "600", 2024-09-15T03:46:31.993 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-15T03:46:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_hit_set_max_size": "100000", 2024-09-15T03:46:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_hit_set_min_size": "1000", 2024-09-15T03:46:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-15T03:46:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-15T03:46:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-15T03:46:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-15T03:46:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-15T03:46:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-15T03:46:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_journal_size": "5120", 2024-09-15T03:46:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_kill_backfill_at": "0", 2024-09-15T03:46:31.994 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-15T03:46:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_map_cache_size": "50", 2024-09-15T03:46:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_map_dedup": "true", 2024-09-15T03:46:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_map_message_max": "40", 2024-09-15T03:46:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-15T03:46:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_map_share_max_epochs": "40", 2024-09-15T03:46:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_attr_name_len": "100", 2024-09-15T03:46:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_attr_size": "0", 2024-09-15T03:46:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_backfills": "1", 2024-09-15T03:46:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_markdown_count": "5", 2024-09-15T03:46:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_markdown_period": "600", 2024-09-15T03:46:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_object_name_len": "2048", 2024-09-15T03:46:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_object_namespace_len": "256", 2024-09-15T03:46:31.995 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_object_size": "134217728", 2024-09-15T03:46:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-15T03:46:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-15T03:46:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_pg_blocked_by": "16", 2024-09-15T03:46:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_pg_log_entries": "10000", 2024-09-15T03:46:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-15T03:46:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_pgls": "1024", 2024-09-15T03:46:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_push_cost": "8388608", 2024-09-15T03:46:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_push_objects": "10", 2024-09-15T03:46:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_scrubs": "3", 2024-09-15T03:46:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-15T03:46:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_trimming_pgs": "2", 2024-09-15T03:46:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_write_op_reply_len": "64", 2024-09-15T03:46:31.996 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_max_write_size": "90", 2024-09-15T03:46:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-15T03:46:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-15T03:46:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-15T03:46:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-15T03:46:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-15T03:46:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-15T03:46:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-15T03:46:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-15T03:46:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_profile": "balanced", 2024-09-15T03:46:31.997 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-15T03:46:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-15T03:46:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-15T03:46:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-15T03:46:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-15T03:46:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-15T03:46:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-15T03:46:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-15T03:46:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-15T03:46:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-15T03:46:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-15T03:46:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_memory_base": "805306368", 2024-09-15T03:46:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_memory_cache_min": "134217728", 2024-09-15T03:46:31.998 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-15T03:46:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-15T03:46:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_memory_target": "4294967296", 2024-09-15T03:46:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_memory_target_autotune": "false", 2024-09-15T03:46:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-15T03:46:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_min_pg_log_entries": "250", 2024-09-15T03:46:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_min_recovery_priority": "0", 2024-09-15T03:46:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-15T03:46:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-15T03:46:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mon_report_interval": "5", 2024-09-15T03:46:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-15T03:46:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-15T03:46:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_num_cache_shards": "32", 2024-09-15T03:46:31.999 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_num_op_tracker_shard": "32", 2024-09-15T03:46:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_numa_auto_affinity": "true", 2024-09-15T03:46:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_numa_node": "-1", 2024-09-15T03:46:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_numa_prefer_iface": "true", 2024-09-15T03:46:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-15T03:46:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_objecter_finishers": "1", 2024-09-15T03:46:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:46:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_objectstore_fuse": "false", 2024-09-15T03:46:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_objectstore_tracing": "false", 2024-09-15T03:46:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_complaint_time": "30.000000", 2024-09-15T03:46:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_history_duration": "600", 2024-09-15T03:46:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_history_size": "20", 2024-09-15T03:46:32.000 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_history_slow_op_size": "20", 2024-09-15T03:46:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-15T03:46:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_log_threshold": "5", 2024-09-15T03:46:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_num_shards": "0", 2024-09-15T03:46:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_num_shards_hdd": "5", 2024-09-15T03:46:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_num_shards_ssd": "8", 2024-09-15T03:46:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-15T03:46:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-15T03:46:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-15T03:46:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-15T03:46:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_pq_min_cost": "65536", 2024-09-15T03:46:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-15T03:46:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_queue_cut_off": "high", 2024-09-15T03:46:32.001 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-15T03:46:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_op_thread_timeout": "15", 2024-09-15T03:46:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_open_classes_on_start": "true", 2024-09-15T03:46:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_os_flags": "0", 2024-09-15T03:46:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_peering_op_priority": "255", 2024-09-15T03:46:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pg_delete_cost": "1048576", 2024-09-15T03:46:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pg_delete_priority": "5", 2024-09-15T03:46:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-15T03:46:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-15T03:46:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-15T03:46:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pg_log_trim_max": "10000", 2024-09-15T03:46:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pg_log_trim_min": "100", 2024-09-15T03:46:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-15T03:46:32.002 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-15T03:46:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pg_stat_report_interval_max": "500", 2024-09-15T03:46:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-15T03:46:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-15T03:46:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-15T03:46:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-15T03:46:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-15T03:46:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-15T03:46:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-15T03:46:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-15T03:46:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-15T03:46:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-15T03:46:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-15T03:46:32.003 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-15T03:46:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-15T03:46:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-15T03:46:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_flags": "0", 2024-09-15T03:46:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-15T03:46:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_min_size": "0", 2024-09-15T03:46:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-15T03:46:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_pg_num": "32", 2024-09-15T03:46:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_pgp_num": "0", 2024-09-15T03:46:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-15T03:46:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_size": "2", 2024-09-15T03:46:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_default_type": "replicated", 2024-09-15T03:46:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-15T03:46:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-15T03:46:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_push_per_object_cost": "1000", 2024-09-15T03:46:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-15T03:46:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recover_clone_overlap": "true", 2024-09-15T03:46:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-15T03:46:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_cost": "20971520", 2024-09-15T03:46:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-15T03:46:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_max_active": "0", 2024-09-15T03:46:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-15T03:46:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-15T03:46:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-15T03:46:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-15T03:46:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_max_single_start": "1", 2024-09-15T03:46:32.005 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_op_priority": "3", 2024-09-15T03:46:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-15T03:46:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_priority": "5", 2024-09-15T03:46:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-15T03:46:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_sleep": "0.000000", 2024-09-15T03:46:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-15T03:46:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-15T03:46:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-15T03:46:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_repair_during_recovery": "false", 2024-09-15T03:46:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_requested_scrub_priority": "120", 2024-09-15T03:46:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-15T03:46:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-15T03:46:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_auto_repair": "false", 2024-09-15T03:46:32.006 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-15T03:46:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-15T03:46:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_begin_hour": "0", 2024-09-15T03:46:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_begin_week_day": "0", 2024-09-15T03:46:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_chunk_max": "25", 2024-09-15T03:46:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_chunk_min": "5", 2024-09-15T03:46:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_cost": "52428800", 2024-09-15T03:46:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_during_recovery": "false", 2024-09-15T03:46:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_end_hour": "0", 2024-09-15T03:46:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_end_week_day": "0", 2024-09-15T03:46:32.007 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_event_cost": "4096", 2024-09-15T03:46:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-15T03:46:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-15T03:46:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_invalid_stats": "true", 2024-09-15T03:46:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-15T03:46:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-15T03:46:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_max_preemptions": "5", 2024-09-15T03:46:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-15T03:46:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_priority": "5", 2024-09-15T03:46:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_scrub_sleep": "0.000000", 2024-09-15T03:46:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-15T03:46:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_skip_data_digest": "false", 2024-09-15T03:46:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_smart_report_timeout": "5", 2024-09-15T03:46:32.008 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_snap_trim_cost": "1048576", 2024-09-15T03:46:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_snap_trim_priority": "5", 2024-09-15T03:46:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-15T03:46:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-15T03:46:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-15T03:46:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-15T03:46:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-15T03:46:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_target_transaction_size": "30", 2024-09-15T03:46:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-15T03:46:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-15T03:46:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-15T03:46:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-15T03:46:32.009 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-15T03:46:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-15T03:46:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-15T03:46:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-15T03:46:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-15T03:46:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-15T03:46:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_tracing": "false", 2024-09-15T03:46:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_use_stale_snap": "false", 2024-09-15T03:46:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-15T03:46:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "osdc_blkin_trace_all": "false", 2024-09-15T03:46:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "paxos_kill_at": "0", 2024-09-15T03:46:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "paxos_max_join_drift": "10", 2024-09-15T03:46:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "paxos_min": "500", 2024-09-15T03:46:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "paxos_min_wait": "0.050000", 2024-09-15T03:46:32.010 INFO:teuthology.orchestra.run.smithi001.stdout: "paxos_propose_interval": "1.000000", 2024-09-15T03:46:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "paxos_service_trim_max": "500", 2024-09-15T03:46:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-15T03:46:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "paxos_service_trim_min": "250", 2024-09-15T03:46:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "paxos_stash_full_interval": "25", 2024-09-15T03:46:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "paxos_trim_max": "500", 2024-09-15T03:46:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "paxos_trim_min": "250", 2024-09-15T03:46:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "perf": "true", 2024-09-15T03:46:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-15T03:46:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-15T03:46:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-15T03:46:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "-", 2024-09-15T03:46:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrv": "", 2024-09-15T03:46:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "public_bind_addr": "-", 2024-09-15T03:46:32.011 INFO:teuthology.orchestra.run.smithi001.stdout: "public_network": "", 2024-09-15T03:46:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "public_network_interface": "", 2024-09-15T03:46:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "qat_compressor_enabled": "false", 2024-09-15T03:46:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "rados_mon_op_timeout": "0", 2024-09-15T03:46:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "rados_osd_op_timeout": "0", 2024-09-15T03:46:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "rados_tracing": "false", 2024-09-15T03:46:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_atime_update_interval": "60", 2024-09-15T03:46:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-15T03:46:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_balance_parent_reads": "false", 2024-09-15T03:46:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_balance_snap_reads": "false", 2024-09-15T03:46:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_blkin_trace_all": "false", 2024-09-15T03:46:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-15T03:46:32.012 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-15T03:46:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_cache": "true", 2024-09-15T03:46:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-15T03:46:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-15T03:46:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-15T03:46:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-15T03:46:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_cache_policy": "writearound", 2024-09-15T03:46:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_cache_size": "33554432", 2024-09-15T03:46:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-15T03:46:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-15T03:46:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_clone_copy_on_read": "false", 2024-09-15T03:46:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_compression_hint": "none", 2024-09-15T03:46:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_concurrent_management_ops": "10", 2024-09-15T03:46:32.013 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-15T03:46:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_default_clone_format": "auto", 2024-09-15T03:46:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_default_data_pool": "", 2024-09-15T03:46:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_default_features": "61", 2024-09-15T03:46:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_default_format": "2", 2024-09-15T03:46:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_default_map_options": "", 2024-09-15T03:46:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_default_order": "22", 2024-09-15T03:46:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_default_pool": "rbd", 2024-09-15T03:46:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-15T03:46:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_default_stripe_count": "0", 2024-09-15T03:46:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_default_stripe_unit": "0", 2024-09-15T03:46:32.014 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-15T03:46:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-15T03:46:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-15T03:46:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_enable_alloc_hint": "true", 2024-09-15T03:46:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-15T03:46:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_io_scheduler": "simple", 2024-09-15T03:46:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-15T03:46:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-15T03:46:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-15T03:46:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-15T03:46:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-15T03:46:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-15T03:46:32.015 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-15T03:46:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-15T03:46:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-15T03:46:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_journal_order": "24", 2024-09-15T03:46:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_journal_pool": "", 2024-09-15T03:46:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_journal_splay_width": "4", 2024-09-15T03:46:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_localize_parent_reads": "false", 2024-09-15T03:46:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_localize_snap_reads": "false", 2024-09-15T03:46:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-15T03:46:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-15T03:46:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-15T03:46:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-15T03:46:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-15T03:46:32.016 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-15T03:46:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-15T03:46:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-15T03:46:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-15T03:46:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-15T03:46:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-15T03:46:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-15T03:46:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-15T03:46:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-15T03:46:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-15T03:46:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-15T03:46:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-15T03:46:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-15T03:46:32.017 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-15T03:46:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-15T03:46:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-15T03:46:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-15T03:46:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-15T03:46:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-15T03:46:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-15T03:46:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-15T03:46:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-15T03:46:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-15T03:46:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-15T03:46:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-15T03:46:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-15T03:46:32.018 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_mtime_update_interval": "60", 2024-09-15T03:46:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_non_blocking_aio": "true", 2024-09-15T03:46:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_op_thread_timeout": "60", 2024-09-15T03:46:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_op_threads": "1", 2024-09-15T03:46:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_parent_cache_enabled": "false", 2024-09-15T03:46:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-15T03:46:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-15T03:46:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-15T03:46:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_plugins": "", 2024-09-15T03:46:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_bps_burst": "0", 2024-09-15T03:46:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-15T03:46:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_bps_limit": "0", 2024-09-15T03:46:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_exclude_ops": "0", 2024-09-15T03:46:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_iops_burst": "0", 2024-09-15T03:46:32.019 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-15T03:46:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_iops_limit": "0", 2024-09-15T03:46:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-15T03:46:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-15T03:46:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-15T03:46:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-15T03:46:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-15T03:46:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-15T03:46:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-15T03:46:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-15T03:46:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-15T03:46:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-15T03:46:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-15T03:46:32.020 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-15T03:46:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-15T03:46:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-15T03:46:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_read_from_replica_policy": "default", 2024-09-15T03:46:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-15T03:46:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-15T03:46:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-15T03:46:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-15T03:46:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_skip_partial_discard": "true", 2024-09-15T03:46:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-15T03:46:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_tracing": "false", 2024-09-15T03:46:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_validate_names": "true", 2024-09-15T03:46:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_validate_pool": "true", 2024-09-15T03:46:32.021 INFO:teuthology.orchestra.run.smithi001.stdout: "restapi_base_url": "", 2024-09-15T03:46:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: "restapi_log_level": "", 2024-09-15T03:46:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_acl_grants_max_num": "100", 2024-09-15T03:46:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_admin_entry": "admin", 2024-09-15T03:46:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_backend_store": "rados", 2024-09-15T03:46:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_barbican_url": "", 2024-09-15T03:46:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_beast_enable_async": "true", 2024-09-15T03:46:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-15T03:46:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-15T03:46:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-15T03:46:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-15T03:46:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-15T03:46:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_cache_enabled": "true", 2024-09-15T03:46:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_cache_expiry_interval": "900", 2024-09-15T03:46:32.022 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_cache_lru_size": "10000", 2024-09-15T03:46:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_content_length_compat": "false", 2024-09-15T03:46:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_copy_obj_progress": "true", 2024-09-15T03:46:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-15T03:46:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_cors_rules_max_num": "100", 2024-09-15T03:46:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_cross_domain_policy": "", 2024-09-15T03:46:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-15T03:46:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_kmip_addr": "", 2024-09-15T03:46:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-15T03:46:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-15T03:46:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-15T03:46:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-15T03:46:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_kmip_password": "", 2024-09-15T03:46:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-15T03:46:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_kmip_username": "", 2024-09-15T03:46:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_require_ssl": "true", 2024-09-15T03:46:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-15T03:46:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-15T03:46:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-15T03:46:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-15T03:46:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-15T03:46:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-15T03:46:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-15T03:46:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-15T03:46:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-15T03:46:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-15T03:46:32.024 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-15T03:46:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-15T03:46:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-15T03:46:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-15T03:46:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-15T03:46:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_vault_addr": "", 2024-09-15T03:46:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_vault_auth": "token", 2024-09-15T03:46:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_vault_namespace": "", 2024-09-15T03:46:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_vault_prefix": "", 2024-09-15T03:46:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-15T03:46:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-15T03:46:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-15T03:46:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-15T03:46:32.025 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_vault_token_file": "", 2024-09-15T03:46:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-15T03:46:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_curl_buffersize": "524288", 2024-09-15T03:46:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-15T03:46:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_curl_low_speed_time": "300", 2024-09-15T03:46:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-15T03:46:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-15T03:46:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-15T03:46:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-15T03:46:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-15T03:46:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-15T03:46:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-15T03:46:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-15T03:46:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-15T03:46:32.026 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-15T03:46:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_data_log_changes_size": "1000", 2024-09-15T03:46:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_data_log_num_shards": "128", 2024-09-15T03:46:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-15T03:46:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_data_log_window": "30", 2024-09-15T03:46:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_data_notify_interval_msec": "200", 2024-09-15T03:46:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-15T03:46:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-15T03:46:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-15T03:46:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-15T03:46:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-15T03:46:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-15T03:46:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-15T03:46:32.027 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-15T03:46:32.028 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-15T03:46:32.028 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-15T03:46:32.028 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-15T03:46:32.028 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-15T03:46:32.028 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-15T03:46:32.028 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-15T03:46:32.028 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-15T03:46:32.028 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-15T03:46:32.028 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-15T03:46:32.028 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-15T03:46:32.028 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-15T03:46:32.028 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-15T03:46:32.028 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-15T03:46:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-15T03:46:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dns_name": "", 2024-09-15T03:46:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dns_s3website_name": "", 2024-09-15T03:46:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_dynamic_resharding": "true", 2024-09-15T03:46:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-15T03:46:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_enable_gc_threads": "true", 2024-09-15T03:46:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_enable_lc_threads": "true", 2024-09-15T03:46:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_enable_ops_log": "false", 2024-09-15T03:46:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_enable_quota_threads": "true", 2024-09-15T03:46:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_enable_static_website": "false", 2024-09-15T03:46:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_enable_usage_log": "false", 2024-09-15T03:46:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_enforce_swift_acls": "true", 2024-09-15T03:46:32.029 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_exit_timeout_secs": "120", 2024-09-15T03:46:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_expose_bucket": "false", 2024-09-15T03:46:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_extended_http_attrs": "", 2024-09-15T03:46:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-15T03:46:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_frontends": "beast port=7480", 2024-09-15T03:46:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-15T03:46:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_gc_max_deferred": "50", 2024-09-15T03:46:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-15T03:46:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_gc_max_objs": "32", 2024-09-15T03:46:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-15T03:46:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-15T03:46:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-15T03:46:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-15T03:46:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_gc_processor_period": "3600", 2024-09-15T03:46:32.030 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-15T03:46:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-15T03:46:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-15T03:46:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-15T03:46:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_init_timeout": "300", 2024-09-15T03:46:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-15T03:46:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_kafka_connection_idle": "300", 2024-09-15T03:46:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-15T03:46:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-15T03:46:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-15T03:46:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_admin_domain": "", 2024-09-15T03:46:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_admin_password": "", 2024-09-15T03:46:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_admin_password_path": "", 2024-09-15T03:46:32.031 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_admin_project": "", 2024-09-15T03:46:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_admin_tenant": "", 2024-09-15T03:46:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_admin_token": "", 2024-09-15T03:46:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_admin_token_path": "", 2024-09-15T03:46:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_admin_user": "", 2024-09-15T03:46:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_api_version": "2", 2024-09-15T03:46:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_barbican_domain": "", 2024-09-15T03:46:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_barbican_password": "", 2024-09-15T03:46:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_barbican_project": "", 2024-09-15T03:46:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-15T03:46:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_barbican_user": "", 2024-09-15T03:46:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-15T03:46:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-15T03:46:32.032 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-15T03:46:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-15T03:46:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-15T03:46:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_url": "", 2024-09-15T03:46:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-15T03:46:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_lc_debug_interval": "-1", 2024-09-15T03:46:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_lc_lock_max_time": "90", 2024-09-15T03:46:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_lc_max_objs": "32", 2024-09-15T03:46:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_lc_max_rules": "1000", 2024-09-15T03:46:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_lc_max_worker": "3", 2024-09-15T03:46:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-15T03:46:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_lc_thread_delay": "0", 2024-09-15T03:46:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-15T03:46:32.033 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_ldap_dnattr": "uid", 2024-09-15T03:46:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-15T03:46:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_ldap_searchfilter": "", 2024-09-15T03:46:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-15T03:46:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_ldap_uri": "", 2024-09-15T03:46:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-15T03:46:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-15T03:46:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-15T03:46:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_log_http_headers": "", 2024-09-15T03:46:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-15T03:46:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-15T03:46:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_log_object_name_utc": "false", 2024-09-15T03:46:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-09-15T03:46:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_attr_name_len": "0", 2024-09-15T03:46:32.034 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_attr_size": "0", 2024-09-15T03:46:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-15T03:46:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_chunk_size": "4194304", 2024-09-15T03:46:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-15T03:46:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-15T03:46:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_listing_results": "1000", 2024-09-15T03:46:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_notify_retries": "10", 2024-09-15T03:46:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-15T03:46:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_put_param_size": "1048576", 2024-09-15T03:46:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_put_size": "5368709120", 2024-09-15T03:46:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_max_slo_entries": "1000", 2024-09-15T03:46:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_md_log_max_shards": "64", 2024-09-15T03:46:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-15T03:46:32.035 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-15T03:46:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_mp_lock_max_time": "600", 2024-09-15T03:46:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-15T03:46:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-15T03:46:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-15T03:46:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-15T03:46:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-15T03:46:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-15T03:46:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-15T03:46:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_max_gc": "300", 2024-09-15T03:46:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-15T03:46:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-15T03:46:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-15T03:46:32.036 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-15T03:46:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-15T03:46:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-15T03:46:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-15T03:46:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_num_async_rados_threads": "32", 2024-09-15T03:46:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_num_control_oids": "8", 2024-09-15T03:46:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_numa_node": "-1", 2024-09-15T03:46:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-15T03:46:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-15T03:46:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_objexp_chunk_size": "100", 2024-09-15T03:46:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_objexp_gc_interval": "600", 2024-09-15T03:46:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-15T03:46:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-15T03:46:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-15T03:46:32.037 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_op_thread_timeout": "600", 2024-09-15T03:46:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_op_tracing": "false", 2024-09-15T03:46:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_opa_token": "", 2024-09-15T03:46:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_opa_url": "", 2024-09-15T03:46:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_opa_verify_ssl": "true", 2024-09-15T03:46:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-15T03:46:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_ops_log_file_path": "", 2024-09-15T03:46:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_ops_log_rados": "true", 2024-09-15T03:46:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_ops_log_socket_path": "", 2024-09-15T03:46:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-15T03:46:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-15T03:46:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_period_push_interval": "2.000000", 2024-09-15T03:46:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-15T03:46:32.038 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-15T03:46:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_print_continue": "true", 2024-09-15T03:46:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-15T03:46:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-15T03:46:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-15T03:46:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-15T03:46:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-15T03:46:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_rados_tracing": "false", 2024-09-15T03:46:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_realm": "", 2024-09-15T03:46:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_realm_id": "", 2024-09-15T03:46:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-15T03:46:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_region": "", 2024-09-15T03:46:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-15T03:46:32.039 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-15T03:46:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-15T03:46:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-15T03:46:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_request_uri": "", 2024-09-15T03:46:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_reshard_batch_size": "64", 2024-09-15T03:46:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-15T03:46:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_reshard_max_aio": "128", 2024-09-15T03:46:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_reshard_num_logs": "16", 2024-09-15T03:46:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_reshard_thread_interval": "600", 2024-09-15T03:46:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_resolve_cname": "false", 2024-09-15T03:46:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-15T03:46:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_run_sync_thread": "true", 2024-09-15T03:46:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-15T03:46:32.040 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-15T03:46:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-15T03:46:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-15T03:46:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-15T03:46:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-15T03:46:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-15T03:46:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-15T03:46:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-15T03:46:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_scheduler_type": "throttler", 2024-09-15T03:46:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_script_uri": "", 2024-09-15T03:46:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_service_provider_name": "", 2024-09-15T03:46:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-15T03:46:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sts_client_id": "", 2024-09-15T03:46:32.041 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sts_client_secret": "", 2024-09-15T03:46:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sts_entry": "sts", 2024-09-15T03:46:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sts_key": "sts", 2024-09-15T03:46:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-15T03:46:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sts_min_session_duration": "900", 2024-09-15T03:46:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sts_token_introspection_url": "", 2024-09-15T03:46:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_swift_account_in_url": "false", 2024-09-15T03:46:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_swift_auth_entry": "auth", 2024-09-15T03:46:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_swift_auth_url": "", 2024-09-15T03:46:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_swift_custom_header": "", 2024-09-15T03:46:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-15T03:46:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_swift_need_stats": "true", 2024-09-15T03:46:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_swift_tenant_name": "", 2024-09-15T03:46:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_swift_token_expiration": "86400", 2024-09-15T03:46:32.042 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_swift_url": "", 2024-09-15T03:46:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_swift_url_prefix": "swift", 2024-09-15T03:46:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-15T03:46:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-15T03:46:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_lease_period": "120", 2024-09-15T03:46:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-15T03:46:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-15T03:46:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-15T03:46:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-15T03:46:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-15T03:46:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-15T03:46:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-15T03:46:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-15T03:46:32.043 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-15T03:46:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_thread_pool_size": "512", 2024-09-15T03:46:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_torrent_comment": "", 2024-09-15T03:46:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_torrent_createby": "", 2024-09-15T03:46:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_torrent_encoding": "", 2024-09-15T03:46:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_torrent_flag": "false", 2024-09-15T03:46:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_torrent_origin": "", 2024-09-15T03:46:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-15T03:46:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_torrent_tracker": "", 2024-09-15T03:46:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_trust_forwarded_https": "false", 2024-09-15T03:46:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-15T03:46:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-15T03:46:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_usage_max_shards": "32", 2024-09-15T03:46:32.044 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_usage_max_user_shards": "1", 2024-09-15T03:46:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_use_opa_authz": "false", 2024-09-15T03:46:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-15T03:46:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-15T03:46:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_user_max_buckets": "1000", 2024-09-15T03:46:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-15T03:46:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-15T03:46:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-15T03:46:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-15T03:46:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_user_unique_email": "true", 2024-09-15T03:46:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_verify_ssl": "true", 2024-09-15T03:46:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-15T03:46:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_zone": "", 2024-09-15T03:46:32.045 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_zone_id": "", 2024-09-15T03:46:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-15T03:46:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_zonegroup": "", 2024-09-15T03:46:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_zonegroup_id": "", 2024-09-15T03:46:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-15T03:46:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_block_size": "4096", 2024-09-15T03:46:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-15T03:46:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-15T03:46:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-15T03:46:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-15T03:46:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-15T03:46:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_cache_size": "536870912", 2024-09-15T03:46:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-15T03:46:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-15T03:46:32.046 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-15T03:46:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-15T03:46:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-15T03:46:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-15T03:46:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-15T03:46:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-15T03:46:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_index_type": "binary_search", 2024-09-15T03:46:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-15T03:46:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-15T03:46:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_partition_filters": "false", 2024-09-15T03:46:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_perf": "false", 2024-09-15T03:46:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-15T03:46:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-15T03:46:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-15T03:46:32.047 INFO:teuthology.orchestra.run.smithi001.stdout: "run_dir": "/var/run/ceph", 2024-09-15T03:46:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_block_create": "true", 2024-09-15T03:46:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_cache_lru_size": "67108864", 2024-09-15T03:46:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_default_max_object_size": "16777216", 2024-09-15T03:46:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-15T03:46:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_device_size": "107374182400", 2024-09-15T03:46:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_init_rewrite_segments_num_per_device": "6", 2024-09-15T03:46:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_journal_batch_capacity": "16", 2024-09-15T03:46:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-15T03:46:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-15T03:46:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-15T03:46:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: "seastore_segment_size": "67108864", 2024-09-15T03:46:32.048 INFO:teuthology.orchestra.run.smithi001.stdout: "setgroup": "", 2024-09-15T03:46:32.049 INFO:teuthology.orchestra.run.smithi001.stdout: "setuser": "", 2024-09-15T03:46:32.049 INFO:teuthology.orchestra.run.smithi001.stdout: "setuser_match_path": "", 2024-09-15T03:46:32.049 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-15T03:46:32.049 INFO:teuthology.orchestra.run.smithi001.stdout: "thp": "false", 2024-09-15T03:46:32.049 INFO:teuthology.orchestra.run.smithi001.stdout: "threadpool_default_timeout": "60", 2024-09-15T03:46:32.049 INFO:teuthology.orchestra.run.smithi001.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-15T03:46:32.049 INFO:teuthology.orchestra.run.smithi001.stdout: "throttler_perf_counter": "true" 2024-09-15T03:46:32.049 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:46:32.051 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-15T03:46:32.162 INFO:teuthology.orchestra.run.smithi001.stdout:mon.a: {} 2024-09-15T03:46:32.162 INFO:teuthology.orchestra.run.smithi001.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-15T03:46:32.172 INFO:teuthology.orchestra.run.smithi001.stdout:mon.b: {} 2024-09-15T03:46:32.172 INFO:teuthology.orchestra.run.smithi001.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-15T03:46:32.183 INFO:teuthology.orchestra.run.smithi001.stdout:mon.c: {} 2024-09-15T03:46:32.183 INFO:teuthology.orchestra.run.smithi001.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-15T03:46:32.192 DEBUG:teuthology.orchestra.run.smithi001:> type ceph-objectstore-tool 2024-09-15T03:46:32.208 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-15T03:46:32.208 DEBUG:teuthology.orchestra.run.smithi190:> type ceph-objectstore-tool 2024-09-15T03:46:32.227 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-15T03:46:32.227 DEBUG:teuthology.orchestra.run.smithi067:> type ceph-objectstore-tool 2024-09-15T03:46:32.247 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-15T03:46:32.247 INFO:teuthology.run_tasks:Running task rados... 2024-09-15T03:46:32.260 INFO:tasks.rados:Beginning rados... 2024-09-15T03:46:32.260 INFO:tasks.rados:config is {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'crush-failure-domain': 'osd', 'k': 4, 'l': 3, 'm': 2, 'name': 'lrcprofile', 'plugin': 'lrc'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False} 2024-09-15T03:46:32.260 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd sloppy crc': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'filestore': True, 'fs': 'xfs'}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-15T03:46:32.261 INFO:tasks.rados:config is {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'crush-failure-domain': 'osd', 'k': 4, 'l': 3, 'm': 2, 'name': 'lrcprofile', 'plugin': 'lrc'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False} 2024-09-15T03:46:32.261 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-09-15T03:46:32.278 INFO:tasks.rados:joining rados 2024-09-15T03:46:32.278 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-15T03:46:32.278 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-15T03:46:32.278 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T03:46:32.278 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-15T03:46:32.279 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-15T03:46:32.279 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:46:32.279 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-15T03:46:32.280 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:46:32.280 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-15T03:46:32.280 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T03:46:32.280 INFO:tasks.rados:clients are ['client.0'] 2024-09-15T03:46:32.281 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set lrcprofile crush-failure-domain=osd k=4 l=3 m=2 name=lrcprofile plugin=lrc 2024-09-15T03:46:32.282 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:32.279+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:46:32.380 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:32.380+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:46:32.390 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:46:32.390 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:46:32.390 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-15T03:46:32.390 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:46:32.402 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:46:32.407 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:46:32.407 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:46:32.423 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:46:32.423 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:46:32.442 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:46:32.443 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:46:32.459 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:46:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:46:32.474 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:46:32.474 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:46:32.482 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:32.481+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:46:32.490 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:46:32.490 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:46:32.505 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:46:32.505 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:46:32.514 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:32.514 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:32.520 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:46:32.520 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:46:32.523 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:46:32.535 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:46:32.536 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:46:32.551 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:46:32.551 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:46:32.570 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:46:32.570 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:46:32.583 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:32.582+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:32.594 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:46:32.594 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:46:32.639 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:32.639 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:32.648 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:46:32.684 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:32.683+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:32.785 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:32.784+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:46:32.789 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:32.789 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:32.798 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:46:32.885 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:46:32.884+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:46:32.899 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:32.899 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:32.908 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:46:32.986 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:32.986+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:33.010 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:33.010 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:33.018 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:46:33.067 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-15T03:46:33.083 INFO:tasks.rados:starting run 0 out of 1 2024-09-15T03:46:33.083 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-15T03:46:33.083 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure lrcprofile 2024-09-15T03:46:33.124 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:33.123+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:46:33.162 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:33.163 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:33.172 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:46:33.213 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:33.212+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:46:33.290 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:33.289+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:33.315 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:33.315 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:33.324 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:46:33.391 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:46:33.391+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:46:33.466 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:33.466 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:33.474 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:46:33.518 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:33.517+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:46:33.575 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:33.575 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:33.583 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:46:33.625 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:33.624+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:46:33.694 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:33.693+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:46:33.725 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:33.725 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:33.733 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:46:33.817 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:33.816+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:46:33.873 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:33.874 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:33.882 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:46:33.923 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:33.922+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:46:33.996 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:33.996 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:33.996 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:33.996+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:34.004 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:46:34.098 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:34.097+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:46:34.107 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:34.107 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:34.115 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:46:34.199 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:34.198+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:46:34.216 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:34.217 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:34.225 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:46:34.300 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:34.300+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:34.326 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:34.326 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:34.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:46:34.401 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:34.400+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:46:34.434 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:34.434 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:34.442 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:46:34.501 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:34.501+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:46:34.543 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:34.543 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:34.551 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:46:34.602 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:34.602+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:34.606 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:46:34.692 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:34.693 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:34.701 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:46:34.703 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:34.702+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:34.731 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:46:34.731 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:46:34.745 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:46:34.746 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:46:34.760 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:46:34.760 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:46:34.774 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:46:34.774 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:46:34.789 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:46:34.789 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:46:34.802 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:34.802 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:34.804 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:46:34.804 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:46:34.804 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:34.803+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:46:34.811 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:46:34.818 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:46:34.818 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:46:34.833 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:46:34.833 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:46:34.849 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:46:34.849 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:46:34.864 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:46:34.865 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:46:34.880 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:46:34.880 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:46:34.895 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:46:34.895 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:46:34.905 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:34.904+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:46:34.911 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:34.911 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:46:34.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:46:34.912 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:46:34.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.912 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4263.0:5 2.2 e23/21 2:5cea7035:::rbd_info:head v 23'1, mlcod=0'0)", 2024-09-15T03:46:34.912 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:25.042195+0000", 2024-09-15T03:46:34.912 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.8694813759999995, 2024-09-15T03:46:34.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00060975899999999997, 2024-09-15T03:46:34.912 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:34.912 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:34.912 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:34.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:34.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042195+0000", 2024-09-15T03:46:34.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:34.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:34.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042195+0000", 2024-09-15T03:46:34.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:34.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:34.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042195+0000", 2024-09-15T03:46:34.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.91e-07 2024-09-15T03:46:34.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:34.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042200+0000", 2024-09-15T03:46:34.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4e-06 2024-09-15T03:46:34.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:34.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042216+0000", 2024-09-15T03:46:34.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5204999999999999e-05 2024-09-15T03:46:34.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:34.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042224+0000", 2024-09-15T03:46:34.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7860000000000002e-06 2024-09-15T03:46:34.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:34.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042287+0000", 2024-09-15T03:46:34.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2348999999999995e-05 2024-09-15T03:46:34.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:34.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042326+0000", 2024-09-15T03:46:34.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9218000000000003e-05 2024-09-15T03:46:34.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:34.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042480+0000", 2024-09-15T03:46:34.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015405799999999999 2024-09-15T03:46:34.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:34.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042489+0000", 2024-09-15T03:46:34.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5879999999999998e-06 2024-09-15T03:46:34.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:34.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042547+0000", 2024-09-15T03:46:34.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8689e-05 2024-09-15T03:46:34.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:34.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042574+0000", 2024-09-15T03:46:34.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6478999999999998e-05 2024-09-15T03:46:34.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:34.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042804+0000", 2024-09-15T03:46:34.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023079599999999999 2024-09-15T03:46:34.918 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:34.918 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:34.918 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:34.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.919 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4263.0:7 2.2 e24/21 2:5cea7035:::rbd_info:head v 24'2, mlcod=0'0)", 2024-09-15T03:46:34.919 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:26.044758+0000", 2024-09-15T03:46:34.920 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.866918385, 2024-09-15T03:46:34.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040876500000000001, 2024-09-15T03:46:34.920 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:34.920 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:34.920 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:34.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:34.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.044758+0000", 2024-09-15T03:46:34.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:34.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:34.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.044758+0000", 2024-09-15T03:46:34.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:34.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:34.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.044758+0000", 2024-09-15T03:46:34.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.04e-07 2024-09-15T03:46:34.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:34.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.044763+0000", 2024-09-15T03:46:34.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8269999999999997e-06 2024-09-15T03:46:34.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:34.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.044774+0000", 2024-09-15T03:46:34.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1207e-05 2024-09-15T03:46:34.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:34.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.044782+0000", 2024-09-15T03:46:34.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9219999999999997e-06 2024-09-15T03:46:34.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:34.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.044843+0000", 2024-09-15T03:46:34.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1327000000000004e-05 2024-09-15T03:46:34.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:34.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.044861+0000", 2024-09-15T03:46:34.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7368999999999999e-05 2024-09-15T03:46:34.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:34.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.044943+0000", 2024-09-15T03:46:34.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2583000000000005e-05 2024-09-15T03:46:34.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:34.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.044955+0000", 2024-09-15T03:46:34.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1708e-05 2024-09-15T03:46:34.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:34.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.045004+0000", 2024-09-15T03:46:34.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9218999999999998e-05 2024-09-15T03:46:34.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:34.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.045026+0000", 2024-09-15T03:46:34.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2289999999999998e-05 2024-09-15T03:46:34.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:34.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:34.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:34.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.045166+0000", 2024-09-15T03:46:34.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000140109 2024-09-15T03:46:34.925 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:34.925 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:34.925 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:34.925 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:34.925 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:34.925 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:34.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:46:35.006 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:35.005+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:35.027 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:46:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:46:35.027 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:35.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:46:35.081 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'unique_pool_0' created 2024-09-15T03:46:35.082 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T03:46:35.099 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-15T03:46:35.106 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:35.105+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:35.140 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:35.140 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:46:35.140 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:46:35.140 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:46:35.140 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:35.149 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:46:35.207 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:35.206+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:35.249 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:46:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:46:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [] 2024-09-15T03:46:35.250 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:35.257 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:46:35.308 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:35.307+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:46:35.365 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:35.366 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:35.366 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:46:35.366 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:35.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:46:35.415 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:35.414+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:46:35.511 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:35.511 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:35.511 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:46:35.511 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:46:35.511 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:35.511 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:35.510+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:46:35.519 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:46:35.613 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:35.612+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:46:35.618 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:35.618 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:46:35.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:46:35.618 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:46:35.618 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.618 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:35.618 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:46:35.618 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 17.712366498000002, 2024-09-15T03:46:35.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10988437500000001, 2024-09-15T03:46:35.618 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:35.619 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:35.619 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:35.619 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:35.619 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:35.619 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1 2024-09-15T03:46:35.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.619 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:35.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.619 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:35.619 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:46:35.619 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:35.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:46:35.620 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:35.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902355+0000", 2024-09-15T03:46:35.620 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.429e-06 2024-09-15T03:46:35.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:35.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902362+0000", 2024-09-15T03:46:35.620 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7290000000000003e-06 2024-09-15T03:46:35.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.621 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:35.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902373+0000", 2024-09-15T03:46:35.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0533999999999999e-05 2024-09-15T03:46:35.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.621 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:35.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902379+0000", 2024-09-15T03:46:35.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4350000000000002e-06 2024-09-15T03:46:35.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:35.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902428+0000", 2024-09-15T03:46:35.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8571e-05 2024-09-15T03:46:35.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:46:35.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902434+0000", 2024-09-15T03:46:35.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5610000000000004e-06 2024-09-15T03:46:35.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:35.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011307+0000", 2024-09-15T03:46:35.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.108872306 2024-09-15T03:46:35.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:35.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011448+0000", 2024-09-15T03:46:35.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014137199999999999 2024-09-15T03:46:35.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:35.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011569+0000", 2024-09-15T03:46:35.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120925 2024-09-15T03:46:35.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:35.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011695+0000", 2024-09-15T03:46:35.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012586800000000001 2024-09-15T03:46:35.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:35.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011703+0000", 2024-09-15T03:46:35.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4190000000000002e-06 2024-09-15T03:46:35.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:35.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011739+0000", 2024-09-15T03:46:35.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5474e-05 2024-09-15T03:46:35.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:35.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011940+0000", 2024-09-15T03:46:35.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020108699999999999 2024-09-15T03:46:35.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:35.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012179+0000", 2024-09-15T03:46:35.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023950299999999999 2024-09-15T03:46:35.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:35.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012215+0000", 2024-09-15T03:46:35.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5332e-05 2024-09-15T03:46:35.627 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.627 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.627 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:35.627 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012238+0000", 2024-09-15T03:46:35.627 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3830000000000001e-05 2024-09-15T03:46:35.627 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.627 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:35.627 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.627 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.170542+0000", 2024-09-15T03:46:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 17.444178388000001, 2024-09-15T03:46:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002816753, 2024-09-15T03:46:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 46 2024-09-15T03:46:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:35.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:35.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170542+0000", 2024-09-15T03:46:35.630 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:35.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170542+0000", 2024-09-15T03:46:35.630 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170542+0000", 2024-09-15T03:46:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.34e-07 2024-09-15T03:46:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170545+0000", 2024-09-15T03:46:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6989999999999999e-06 2024-09-15T03:46:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170549+0000", 2024-09-15T03:46:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2320000000000003e-06 2024-09-15T03:46:35.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.632 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:35.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170552+0000", 2024-09-15T03:46:35.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5150000000000001e-06 2024-09-15T03:46:35.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.632 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:35.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.172570+0000", 2024-09-15T03:46:35.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020182809999999998 2024-09-15T03:46:35.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.633 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:35.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.172620+0000", 2024-09-15T03:46:35.633 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0031999999999998e-05 2024-09-15T03:46:35.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.633 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:35.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.172693+0000", 2024-09-15T03:46:35.633 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2762999999999994e-05 2024-09-15T03:46:35.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.633 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:35.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.172800+0000", 2024-09-15T03:46:35.633 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010696000000000001 2024-09-15T03:46:35.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:35.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.172810+0000", 2024-09-15T03:46:35.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8030000000000008e-06 2024-09-15T03:46:35.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:35.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.172836+0000", 2024-09-15T03:46:35.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6064e-05 2024-09-15T03:46:35.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:35.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173224+0000", 2024-09-15T03:46:35.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00038843800000000001 2024-09-15T03:46:35.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.635 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.635 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:35.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173320+0000", 2024-09-15T03:46:35.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5933999999999993e-05 2024-09-15T03:46:35.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.635 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.635 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:35.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173342+0000", 2024-09-15T03:46:35.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.207e-05 2024-09-15T03:46:35.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.636 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.636 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:35.636 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173359+0000", 2024-09-15T03:46:35.636 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6728e-05 2024-09-15T03:46:35.636 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.636 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:35.636 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.636 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.636 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:35.636 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.170563+0000", 2024-09-15T03:46:35.636 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 17.444157131000001, 2024-09-15T03:46:35.636 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0033790360000000002, 2024-09-15T03:46:35.637 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:35.637 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:35.637 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:35.637 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:35.637 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:35.637 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 47 2024-09-15T03:46:35.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.637 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:35.637 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.637 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:35.637 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170563+0000", 2024-09-15T03:46:35.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.638 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:35.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170563+0000", 2024-09-15T03:46:35.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.638 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:35.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170564+0000", 2024-09-15T03:46:35.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7099999999999998e-07 2024-09-15T03:46:35.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.638 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:35.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170658+0000", 2024-09-15T03:46:35.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4112999999999998e-05 2024-09-15T03:46:35.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:35.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170666+0000", 2024-09-15T03:46:35.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0800000000000006e-06 2024-09-15T03:46:35.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:35.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170669+0000", 2024-09-15T03:46:35.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5920000000000001e-06 2024-09-15T03:46:35.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:35.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.172869+0000", 2024-09-15T03:46:35.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021995640000000002 2024-09-15T03:46:35.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:35.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.172919+0000", 2024-09-15T03:46:35.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9907999999999997e-05 2024-09-15T03:46:35.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:35.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.172997+0000", 2024-09-15T03:46:35.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7896000000000001e-05 2024-09-15T03:46:35.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:35.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173102+0000", 2024-09-15T03:46:35.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000105269 2024-09-15T03:46:35.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:35.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173109+0000", 2024-09-15T03:46:35.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2089999999999999e-06 2024-09-15T03:46:35.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:35.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173167+0000", 2024-09-15T03:46:35.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7627999999999998e-05 2024-09-15T03:46:35.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:35.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173229+0000", 2024-09-15T03:46:35.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2039999999999996e-05 2024-09-15T03:46:35.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:35.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173897+0000", 2024-09-15T03:46:35.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00066788899999999996 2024-09-15T03:46:35.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:35.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173923+0000", 2024-09-15T03:46:35.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5871e-05 2024-09-15T03:46:35.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:35.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173942+0000", 2024-09-15T03:46:35.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9706000000000001e-05 2024-09-15T03:46:35.644 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.644 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:35.644 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.644 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:35.644 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.170673+0000", 2024-09-15T03:46:35.644 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 17.444047176000002, 2024-09-15T03:46:35.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0033540229999999998, 2024-09-15T03:46:35.645 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:35.645 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:35.645 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:35.645 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:35.645 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:35.645 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 48 2024-09-15T03:46:35.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.645 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:35.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:35.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170673+0000", 2024-09-15T03:46:35.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:35.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170673+0000", 2024-09-15T03:46:35.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:35.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170674+0000", 2024-09-15T03:46:35.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2799999999999996e-07 2024-09-15T03:46:35.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:35.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170676+0000", 2024-09-15T03:46:35.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1600000000000001e-06 2024-09-15T03:46:35.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.647 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:35.647 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170679+0000", 2024-09-15T03:46:35.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2049999999999998e-06 2024-09-15T03:46:35.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.647 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:35.647 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170681+0000", 2024-09-15T03:46:35.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.244e-06 2024-09-15T03:46:35.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.647 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:35.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173386+0000", 2024-09-15T03:46:35.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027046090000000002 2024-09-15T03:46:35.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:35.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173433+0000", 2024-09-15T03:46:35.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6983000000000001e-05 2024-09-15T03:46:35.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:35.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173506+0000", 2024-09-15T03:46:35.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3411999999999994e-05 2024-09-15T03:46:35.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:35.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173591+0000", 2024-09-15T03:46:35.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4122000000000005e-05 2024-09-15T03:46:35.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:35.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173598+0000", 2024-09-15T03:46:35.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4390000000000003e-06 2024-09-15T03:46:35.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:35.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173622+0000", 2024-09-15T03:46:35.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4351000000000001e-05 2024-09-15T03:46:35.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:35.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173962+0000", 2024-09-15T03:46:35.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034009600000000002 2024-09-15T03:46:35.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:35.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173988+0000", 2024-09-15T03:46:35.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6001999999999999e-05 2024-09-15T03:46:35.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.651 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:35.651 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174011+0000", 2024-09-15T03:46:35.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2178999999999999e-05 2024-09-15T03:46:35.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.651 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:35.651 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174027+0000", 2024-09-15T03:46:35.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6693000000000001e-05 2024-09-15T03:46:35.651 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.651 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:35.652 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.652 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:35.652 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.170687+0000", 2024-09-15T03:46:35.652 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 17.444033323999999, 2024-09-15T03:46:35.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0039080060000000003, 2024-09-15T03:46:35.652 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:35.652 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:35.652 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:35.652 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:35.652 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:35.652 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 49 2024-09-15T03:46:35.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.653 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:35.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:35.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170687+0000", 2024-09-15T03:46:35.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:35.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170687+0000", 2024-09-15T03:46:35.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:35.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170687+0000", 2024-09-15T03:46:35.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5899999999999998e-07 2024-09-15T03:46:35.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:35.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170689+0000", 2024-09-15T03:46:35.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9120000000000001e-06 2024-09-15T03:46:35.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:35.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170695+0000", 2024-09-15T03:46:35.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1550000000000004e-06 2024-09-15T03:46:35.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:35.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170698+0000", 2024-09-15T03:46:35.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2089999999999999e-06 2024-09-15T03:46:35.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:35.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173634+0000", 2024-09-15T03:46:35.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002936204 2024-09-15T03:46:35.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:35.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173684+0000", 2024-09-15T03:46:35.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9798e-05 2024-09-15T03:46:35.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:35.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173755+0000", 2024-09-15T03:46:35.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0957999999999994e-05 2024-09-15T03:46:35.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:35.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173850+0000", 2024-09-15T03:46:35.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4936e-05 2024-09-15T03:46:35.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:35.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173857+0000", 2024-09-15T03:46:35.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2830000000000004e-06 2024-09-15T03:46:35.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:35.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173883+0000", 2024-09-15T03:46:35.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5910000000000001e-05 2024-09-15T03:46:35.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:35.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173969+0000", 2024-09-15T03:46:35.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5894000000000001e-05 2024-09-15T03:46:35.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:35.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174550+0000", 2024-09-15T03:46:35.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00058144000000000002 2024-09-15T03:46:35.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:35.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174576+0000", 2024-09-15T03:46:35.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5624999999999999e-05 2024-09-15T03:46:35.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.659 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:35.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174595+0000", 2024-09-15T03:46:35.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9423e-05 2024-09-15T03:46:35.659 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.659 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:35.659 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.659 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:35.659 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.170804+0000", 2024-09-15T03:46:35.659 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 17.443916650999999, 2024-09-15T03:46:35.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0041461739999999999, 2024-09-15T03:46:35.659 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:35.660 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:35.660 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:35.660 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:35.660 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:35.660 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 50 2024-09-15T03:46:35.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.660 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:35.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.660 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:35.660 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170804+0000", 2024-09-15T03:46:35.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.660 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:35.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170804+0000", 2024-09-15T03:46:35.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:35.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170804+0000", 2024-09-15T03:46:35.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7199999999999999e-07 2024-09-15T03:46:35.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:35.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170856+0000", 2024-09-15T03:46:35.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1647000000000002e-05 2024-09-15T03:46:35.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:35.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170865+0000", 2024-09-15T03:46:35.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0829999999999993e-06 2024-09-15T03:46:35.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:35.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170869+0000", 2024-09-15T03:46:35.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8369999999999999e-06 2024-09-15T03:46:35.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:35.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174053+0000", 2024-09-15T03:46:35.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031840639999999999 2024-09-15T03:46:35.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:35.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174099+0000", 2024-09-15T03:46:35.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5578000000000003e-05 2024-09-15T03:46:35.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:35.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174178+0000", 2024-09-15T03:46:35.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9268000000000001e-05 2024-09-15T03:46:35.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:35.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174261+0000", 2024-09-15T03:46:35.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3023999999999997e-05 2024-09-15T03:46:35.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.664 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:35.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174269+0000", 2024-09-15T03:46:35.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2460000000000003e-06 2024-09-15T03:46:35.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.664 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:35.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174322+0000", 2024-09-15T03:46:35.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2957999999999997e-05 2024-09-15T03:46:35.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.664 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:35.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174614+0000", 2024-09-15T03:46:35.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029206300000000001 2024-09-15T03:46:35.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.665 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:35.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174906+0000", 2024-09-15T03:46:35.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029138299999999998 2024-09-15T03:46:35.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:35.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174931+0000", 2024-09-15T03:46:35.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5809999999999999e-05 2024-09-15T03:46:35.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:35.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174950+0000", 2024-09-15T03:46:35.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8641e-05 2024-09-15T03:46:35.666 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.666 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:35.666 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.667 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:35.667 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.170873+0000", 2024-09-15T03:46:35.667 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 17.443847442999999, 2024-09-15T03:46:35.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0041437130000000003, 2024-09-15T03:46:35.667 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:35.667 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:35.667 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:35.667 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:35.667 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:35.667 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 51 2024-09-15T03:46:35.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.668 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:35.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:35.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170873+0000", 2024-09-15T03:46:35.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:35.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170873+0000", 2024-09-15T03:46:35.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:35.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170873+0000", 2024-09-15T03:46:35.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:46:35.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:35.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170876+0000", 2024-09-15T03:46:35.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3010000000000001e-06 2024-09-15T03:46:35.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:35.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170879+0000", 2024-09-15T03:46:35.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2710000000000001e-06 2024-09-15T03:46:35.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.670 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.670 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:35.670 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170881+0000", 2024-09-15T03:46:35.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3630000000000001e-06 2024-09-15T03:46:35.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.670 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.670 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:35.670 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174304+0000", 2024-09-15T03:46:35.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0034224139999999999 2024-09-15T03:46:35.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.670 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.670 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:35.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174351+0000", 2024-09-15T03:46:35.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7638e-05 2024-09-15T03:46:35.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.671 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:35.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174421+0000", 2024-09-15T03:46:35.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9283999999999996e-05 2024-09-15T03:46:35.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.671 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:35.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174501+0000", 2024-09-15T03:46:35.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0652e-05 2024-09-15T03:46:35.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:35.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174508+0000", 2024-09-15T03:46:35.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6540000000000002e-06 2024-09-15T03:46:35.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:35.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174530+0000", 2024-09-15T03:46:35.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2473999999999999e-05 2024-09-15T03:46:35.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:35.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174621+0000", 2024-09-15T03:46:35.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0803000000000005e-05 2024-09-15T03:46:35.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.673 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:35.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174975+0000", 2024-09-15T03:46:35.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00035398800000000001 2024-09-15T03:46:35.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.673 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:35.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174999+0000", 2024-09-15T03:46:35.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3241999999999999e-05 2024-09-15T03:46:35.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.673 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:35.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.175017+0000", 2024-09-15T03:46:35.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8219e-05 2024-09-15T03:46:35.674 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.674 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:35.674 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.674 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:35.674 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.170884+0000", 2024-09-15T03:46:35.674 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 17.443836084000001, 2024-09-15T03:46:35.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042576849999999998, 2024-09-15T03:46:35.674 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:35.674 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:35.674 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:35.674 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:35.675 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:35.675 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 52 2024-09-15T03:46:35.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.675 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:35.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.675 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:35.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170884+0000", 2024-09-15T03:46:35.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.675 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:35.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170884+0000", 2024-09-15T03:46:35.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.676 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:35.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170885+0000", 2024-09-15T03:46:35.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0599999999999999e-07 2024-09-15T03:46:35.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.676 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:35.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170887+0000", 2024-09-15T03:46:35.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.17e-06 2024-09-15T03:46:35.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:35.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170890+0000", 2024-09-15T03:46:35.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9210000000000001e-06 2024-09-15T03:46:35.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:35.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170892+0000", 2024-09-15T03:46:35.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.779e-06 2024-09-15T03:46:35.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:35.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174638+0000", 2024-09-15T03:46:35.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0037452269999999998 2024-09-15T03:46:35.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:35.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174694+0000", 2024-09-15T03:46:35.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6378000000000001e-05 2024-09-15T03:46:35.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:35.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174770+0000", 2024-09-15T03:46:35.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5654000000000004e-05 2024-09-15T03:46:35.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:35.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174859+0000", 2024-09-15T03:46:35.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9074999999999997e-05 2024-09-15T03:46:35.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:35.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174865+0000", 2024-09-15T03:46:35.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.178e-06 2024-09-15T03:46:35.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:35.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174890+0000", 2024-09-15T03:46:35.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4956e-05 2024-09-15T03:46:35.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:35.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.175029+0000", 2024-09-15T03:46:35.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013951999999999999 2024-09-15T03:46:35.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:35.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.175102+0000", 2024-09-15T03:46:35.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2713000000000007e-05 2024-09-15T03:46:35.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.681 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:35.681 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.175126+0000", 2024-09-15T03:46:35.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3541e-05 2024-09-15T03:46:35.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.681 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:35.681 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.175142+0000", 2024-09-15T03:46:35.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6367000000000001e-05 2024-09-15T03:46:35.681 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.681 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:35.681 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.682 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.682 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:35.682 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.600740+0000", 2024-09-15T03:46:35.682 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 17.013980760999999, 2024-09-15T03:46:35.682 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030445950000000002, 2024-09-15T03:46:35.682 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:35.682 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:35.682 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:35.682 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:35.682 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:35.682 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 72 2024-09-15T03:46:35.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.682 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:35.682 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:35.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600740+0000", 2024-09-15T03:46:35.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:35.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600740+0000", 2024-09-15T03:46:35.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:35.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600740+0000", 2024-09-15T03:46:35.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2199999999999999e-07 2024-09-15T03:46:35.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.684 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.684 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:35.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600811+0000", 2024-09-15T03:46:35.684 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0443000000000003e-05 2024-09-15T03:46:35.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.684 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.684 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:35.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600819+0000", 2024-09-15T03:46:35.684 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7620000000000001e-06 2024-09-15T03:46:35.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.684 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.684 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:35.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600824+0000", 2024-09-15T03:46:35.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4009999999999999e-06 2024-09-15T03:46:35.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.685 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:35.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.602728+0000", 2024-09-15T03:46:35.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001904329 2024-09-15T03:46:35.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.685 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:35.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.602776+0000", 2024-09-15T03:46:35.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7605999999999998e-05 2024-09-15T03:46:35.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.686 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:35.686 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.602863+0000", 2024-09-15T03:46:35.686 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7177999999999998e-05 2024-09-15T03:46:35.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.686 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:35.686 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.602945+0000", 2024-09-15T03:46:35.686 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2269999999999997e-05 2024-09-15T03:46:35.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.686 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:35.687 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.602954+0000", 2024-09-15T03:46:35.687 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1470000000000007e-06 2024-09-15T03:46:35.687 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.687 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.687 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:35.687 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603013+0000", 2024-09-15T03:46:35.687 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8527999999999999e-05 2024-09-15T03:46:35.687 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.687 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.687 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:35.687 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603137+0000", 2024-09-15T03:46:35.687 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012378000000000001 2024-09-15T03:46:35.687 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.687 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.688 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:35.688 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603735+0000", 2024-09-15T03:46:35.688 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00059823600000000004 2024-09-15T03:46:35.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.688 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.688 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:35.688 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603764+0000", 2024-09-15T03:46:35.688 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9689999999999999e-05 2024-09-15T03:46:35.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.688 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.688 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:35.688 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603784+0000", 2024-09-15T03:46:35.688 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9803e-05 2024-09-15T03:46:35.689 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.689 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:35.689 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.689 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:35.689 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.600830+0000", 2024-09-15T03:46:35.689 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 17.013890162999999, 2024-09-15T03:46:35.689 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030580360000000001, 2024-09-15T03:46:35.689 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:35.689 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:35.689 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:35.689 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:35.689 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:35.690 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 73 2024-09-15T03:46:35.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.690 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:35.690 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.690 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:35.690 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600830+0000", 2024-09-15T03:46:35.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.690 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.690 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:35.690 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600830+0000", 2024-09-15T03:46:35.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.690 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.690 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:35.691 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600831+0000", 2024-09-15T03:46:35.691 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4900000000000005e-07 2024-09-15T03:46:35.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.691 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.691 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:35.691 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600835+0000", 2024-09-15T03:46:35.691 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3490000000000002e-06 2024-09-15T03:46:35.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.691 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.691 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:35.691 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600841+0000", 2024-09-15T03:46:35.691 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0440000000000001e-06 2024-09-15T03:46:35.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:35.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600846+0000", 2024-09-15T03:46:35.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9699999999999998e-06 2024-09-15T03:46:35.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:35.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603228+0000", 2024-09-15T03:46:35.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023815260000000001 2024-09-15T03:46:35.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:35.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603275+0000", 2024-09-15T03:46:35.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7034999999999999e-05 2024-09-15T03:46:35.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:35.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603345+0000", 2024-09-15T03:46:35.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0258000000000004e-05 2024-09-15T03:46:35.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:35.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603429+0000", 2024-09-15T03:46:35.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3290999999999999e-05 2024-09-15T03:46:35.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.694 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:35.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603435+0000", 2024-09-15T03:46:35.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.883e-06 2024-09-15T03:46:35.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.694 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:35.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603463+0000", 2024-09-15T03:46:35.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7813999999999998e-05 2024-09-15T03:46:35.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.694 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:35.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603813+0000", 2024-09-15T03:46:35.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034945700000000001 2024-09-15T03:46:35.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:35.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603847+0000", 2024-09-15T03:46:35.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4023999999999998e-05 2024-09-15T03:46:35.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:35.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603871+0000", 2024-09-15T03:46:35.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4083999999999998e-05 2024-09-15T03:46:35.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:35.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603888+0000", 2024-09-15T03:46:35.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7552000000000001e-05 2024-09-15T03:46:35.696 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.696 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:35.696 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.696 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:35.696 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.600852+0000", 2024-09-15T03:46:35.696 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 17.013868641999998, 2024-09-15T03:46:35.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003621132, 2024-09-15T03:46:35.696 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:35.697 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:35.697 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:35.697 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:35.697 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:35.697 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 74 2024-09-15T03:46:35.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.697 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:35.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:35.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600852+0000", 2024-09-15T03:46:35.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:35.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600852+0000", 2024-09-15T03:46:35.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:35.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600852+0000", 2024-09-15T03:46:35.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1399999999999997e-07 2024-09-15T03:46:35.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:35.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600856+0000", 2024-09-15T03:46:35.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7009999999999998e-06 2024-09-15T03:46:35.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:35.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600863+0000", 2024-09-15T03:46:35.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9689999999999997e-06 2024-09-15T03:46:35.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:35.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600867+0000", 2024-09-15T03:46:35.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7129999999999999e-06 2024-09-15T03:46:35.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:35.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603471+0000", 2024-09-15T03:46:35.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002604711 2024-09-15T03:46:35.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:35.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603521+0000", 2024-09-15T03:46:35.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0148999999999997e-05 2024-09-15T03:46:35.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:35.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603596+0000", 2024-09-15T03:46:35.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4144e-05 2024-09-15T03:46:35.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:35.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603690+0000", 2024-09-15T03:46:35.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4098999999999995e-05 2024-09-15T03:46:35.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:35.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603698+0000", 2024-09-15T03:46:35.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6309999999999992e-06 2024-09-15T03:46:35.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:35.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603726+0000", 2024-09-15T03:46:35.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7810999999999999e-05 2024-09-15T03:46:35.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:35.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603825+0000", 2024-09-15T03:46:35.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9321999999999995e-05 2024-09-15T03:46:35.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:35.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604424+0000", 2024-09-15T03:46:35.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00059867700000000004 2024-09-15T03:46:35.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:35.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604451+0000", 2024-09-15T03:46:35.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7277e-05 2024-09-15T03:46:35.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:35.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604473+0000", 2024-09-15T03:46:35.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1514000000000001e-05 2024-09-15T03:46:35.703 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.703 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:35.703 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.703 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:35.703 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.600872+0000", 2024-09-15T03:46:35.704 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 17.013848884000002, 2024-09-15T03:46:35.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0037887039999999999, 2024-09-15T03:46:35.704 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:35.704 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:35.704 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:35.704 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:35.704 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:35.704 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 75 2024-09-15T03:46:35.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.704 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:35.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:35.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600872+0000", 2024-09-15T03:46:35.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:35.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600872+0000", 2024-09-15T03:46:35.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:35.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600872+0000", 2024-09-15T03:46:35.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4700000000000002e-07 2024-09-15T03:46:35.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:35.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600933+0000", 2024-09-15T03:46:35.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0671999999999998e-05 2024-09-15T03:46:35.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:35.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600941+0000", 2024-09-15T03:46:35.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5779999999999999e-06 2024-09-15T03:46:35.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:35.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600946+0000", 2024-09-15T03:46:35.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8810000000000004e-06 2024-09-15T03:46:35.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:35.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603916+0000", 2024-09-15T03:46:35.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0029702359999999998 2024-09-15T03:46:35.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:35.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603962+0000", 2024-09-15T03:46:35.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5148999999999998e-05 2024-09-15T03:46:35.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:35.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604037+0000", 2024-09-15T03:46:35.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5352999999999995e-05 2024-09-15T03:46:35.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.708 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:35.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604136+0000", 2024-09-15T03:46:35.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8655999999999999e-05 2024-09-15T03:46:35.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.708 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:35.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604142+0000", 2024-09-15T03:46:35.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4060000000000002e-06 2024-09-15T03:46:35.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:35.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604199+0000", 2024-09-15T03:46:35.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6820999999999997e-05 2024-09-15T03:46:35.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:35.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604497+0000", 2024-09-15T03:46:35.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029777299999999998 2024-09-15T03:46:35.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:35.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604617+0000", 2024-09-15T03:46:35.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012054599999999999 2024-09-15T03:46:35.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:35.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604642+0000", 2024-09-15T03:46:35.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4513000000000001e-05 2024-09-15T03:46:35.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:35.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604660+0000", 2024-09-15T03:46:35.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8672999999999999e-05 2024-09-15T03:46:35.710 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.710 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:35.710 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.711 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:35.711 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.600953+0000", 2024-09-15T03:46:35.711 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 17.013767856000001, 2024-09-15T03:46:35.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003775086, 2024-09-15T03:46:35.711 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:35.711 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:35.711 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:35.712 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:35.712 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:35.712 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 76 2024-09-15T03:46:35.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.712 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:35.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.713 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:35.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600953+0000", 2024-09-15T03:46:35.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.713 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:35.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600953+0000", 2024-09-15T03:46:35.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.714 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:35.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600953+0000", 2024-09-15T03:46:35.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8100000000000002e-07 2024-09-15T03:46:35.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.714 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:35.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600957+0000", 2024-09-15T03:46:35.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1579999999999998e-06 2024-09-15T03:46:35.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:35.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600964+0000", 2024-09-15T03:46:35.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3219999999999998e-06 2024-09-15T03:46:35.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:35.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600967+0000", 2024-09-15T03:46:35.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6339999999999999e-06 2024-09-15T03:46:35.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:35.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604177+0000", 2024-09-15T03:46:35.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00320967 2024-09-15T03:46:35.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:35.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604226+0000", 2024-09-15T03:46:35.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8646000000000001e-05 2024-09-15T03:46:35.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:35.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604297+0000", 2024-09-15T03:46:35.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1390000000000006e-05 2024-09-15T03:46:35.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:35.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604383+0000", 2024-09-15T03:46:35.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5927999999999995e-05 2024-09-15T03:46:35.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:35.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604392+0000", 2024-09-15T03:46:35.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.799e-06 2024-09-15T03:46:35.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:35.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604418+0000", 2024-09-15T03:46:35.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6585999999999999e-05 2024-09-15T03:46:35.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:35.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604506+0000", 2024-09-15T03:46:35.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7806000000000006e-05 2024-09-15T03:46:35.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.719 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.719 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:35.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604683+0000", 2024-09-15T03:46:35.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000176971 2024-09-15T03:46:35.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.719 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.719 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:35.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604709+0000", 2024-09-15T03:46:35.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6044e-05 2024-09-15T03:46:35.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.719 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.720 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:35.720 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604728+0000", 2024-09-15T03:46:35.720 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8451e-05 2024-09-15T03:46:35.720 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.720 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:35.720 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.720 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:95 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-15T03:46:35.720 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:19.008511+0000", 2024-09-15T03:46:35.720 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.606209948, 2024-09-15T03:46:35.720 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0028780899999999998, 2024-09-15T03:46:35.720 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:35.720 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:35.721 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:35.721 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:35.721 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:35.721 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 95 2024-09-15T03:46:35.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.721 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:35.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:35.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008511+0000", 2024-09-15T03:46:35.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:35.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008511+0000", 2024-09-15T03:46:35.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:35.722 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008511+0000", 2024-09-15T03:46:35.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0699999999999998e-07 2024-09-15T03:46:35.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:35.722 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008513+0000", 2024-09-15T03:46:35.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6240000000000002e-06 2024-09-15T03:46:35.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:35.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008517+0000", 2024-09-15T03:46:35.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1700000000000001e-06 2024-09-15T03:46:35.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:35.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008524+0000", 2024-09-15T03:46:35.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0090000000000001e-06 2024-09-15T03:46:35.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:35.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.010474+0000", 2024-09-15T03:46:35.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019501480000000001 2024-09-15T03:46:35.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:35.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.010524+0000", 2024-09-15T03:46:35.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0047999999999999e-05 2024-09-15T03:46:35.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:35.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.010589+0000", 2024-09-15T03:46:35.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5482999999999996e-05 2024-09-15T03:46:35.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:35.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.010675+0000", 2024-09-15T03:46:35.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5211999999999997e-05 2024-09-15T03:46:35.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:35.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.010681+0000", 2024-09-15T03:46:35.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8789999999999997e-06 2024-09-15T03:46:35.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:35.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.010704+0000", 2024-09-15T03:46:35.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2296000000000001e-05 2024-09-15T03:46:35.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:35.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.010826+0000", 2024-09-15T03:46:35.726 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012271999999999999 2024-09-15T03:46:35.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.726 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:35.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011348+0000", 2024-09-15T03:46:35.726 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00052180800000000002 2024-09-15T03:46:35.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.726 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:35.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011371+0000", 2024-09-15T03:46:35.726 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3254000000000001e-05 2024-09-15T03:46:35.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:35.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011389+0000", 2024-09-15T03:46:35.727 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7132000000000001e-05 2024-09-15T03:46:35.727 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.727 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:35.727 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.727 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:96 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-15T03:46:35.727 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:19.008531+0000", 2024-09-15T03:46:35.727 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.606189440000001, 2024-09-15T03:46:35.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0034252969999999999, 2024-09-15T03:46:35.728 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:35.728 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:35.728 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:35.728 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:35.728 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:35.728 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 96 2024-09-15T03:46:35.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.728 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:35.728 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.728 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:35.728 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008531+0000", 2024-09-15T03:46:35.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.729 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:35.729 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008531+0000", 2024-09-15T03:46:35.729 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.729 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:35.729 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008531+0000", 2024-09-15T03:46:35.729 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3900000000000001e-07 2024-09-15T03:46:35.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.729 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:35.730 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008596+0000", 2024-09-15T03:46:35.730 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4475999999999997e-05 2024-09-15T03:46:35.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.730 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.730 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:35.730 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008603+0000", 2024-09-15T03:46:35.730 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3749999999999997e-06 2024-09-15T03:46:35.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.730 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.730 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:35.730 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008608+0000", 2024-09-15T03:46:35.730 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8600000000000001e-06 2024-09-15T03:46:35.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.730 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:35.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.010839+0000", 2024-09-15T03:46:35.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022307889999999999 2024-09-15T03:46:35.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:35.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.010881+0000", 2024-09-15T03:46:35.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1766e-05 2024-09-15T03:46:35.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:35.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.010945+0000", 2024-09-15T03:46:35.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4196000000000007e-05 2024-09-15T03:46:35.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.732 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:35.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011025+0000", 2024-09-15T03:46:35.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9818e-05 2024-09-15T03:46:35.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.732 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:35.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011032+0000", 2024-09-15T03:46:35.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1019999999999999e-06 2024-09-15T03:46:35.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.732 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:35.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011083+0000", 2024-09-15T03:46:35.733 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1687999999999997e-05 2024-09-15T03:46:35.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:35.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011402+0000", 2024-09-15T03:46:35.733 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00031831600000000002 2024-09-15T03:46:35.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:35.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011899+0000", 2024-09-15T03:46:35.733 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00049779100000000003 2024-09-15T03:46:35.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:35.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011924+0000", 2024-09-15T03:46:35.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4595000000000001e-05 2024-09-15T03:46:35.735 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.736 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.736 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:35.736 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011956+0000", 2024-09-15T03:46:35.736 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2286000000000002e-05 2024-09-15T03:46:35.736 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.736 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:35.736 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.736 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.736 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-15T03:46:35.736 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:19.008611+0000", 2024-09-15T03:46:35.736 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.606109095000001, 2024-09-15T03:46:35.736 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00344955, 2024-09-15T03:46:35.737 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:35.737 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:35.737 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:35.737 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:35.737 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:35.737 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 97 2024-09-15T03:46:35.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.737 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:35.737 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.737 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:35.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008611+0000", 2024-09-15T03:46:35.737 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.737 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.738 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:35.738 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008611+0000", 2024-09-15T03:46:35.738 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.738 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:35.738 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008612+0000", 2024-09-15T03:46:35.738 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.39e-07 2024-09-15T03:46:35.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.738 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:35.738 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008616+0000", 2024-09-15T03:46:35.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0350000000000003e-06 2024-09-15T03:46:35.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.739 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.739 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:35.739 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008619+0000", 2024-09-15T03:46:35.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4359999999999998e-06 2024-09-15T03:46:35.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.739 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.739 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:35.739 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008626+0000", 2024-09-15T03:46:35.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1609999999999997e-06 2024-09-15T03:46:35.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.739 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.740 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:35.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011071+0000", 2024-09-15T03:46:35.740 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024443569999999999 2024-09-15T03:46:35.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.740 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:35.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011126+0000", 2024-09-15T03:46:35.740 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5300000000000002e-05 2024-09-15T03:46:35.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.740 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:35.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011205+0000", 2024-09-15T03:46:35.741 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9221000000000006e-05 2024-09-15T03:46:35.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.741 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.741 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:35.741 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011307+0000", 2024-09-15T03:46:35.741 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010136000000000001 2024-09-15T03:46:35.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.741 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.741 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:35.741 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011313+0000", 2024-09-15T03:46:35.741 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.844e-06 2024-09-15T03:46:35.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.741 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.742 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:35.742 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011336+0000", 2024-09-15T03:46:35.742 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2993000000000002e-05 2024-09-15T03:46:35.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.742 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.742 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:35.742 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011407+0000", 2024-09-15T03:46:35.742 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1002999999999998e-05 2024-09-15T03:46:35.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.742 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.742 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:35.742 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012007+0000", 2024-09-15T03:46:35.742 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000599652 2024-09-15T03:46:35.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.743 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:35.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012029+0000", 2024-09-15T03:46:35.743 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2067999999999999e-05 2024-09-15T03:46:35.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.743 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:35.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012061+0000", 2024-09-15T03:46:35.743 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1780999999999999e-05 2024-09-15T03:46:35.743 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.743 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:35.744 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.744 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.744 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-15T03:46:35.744 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:19.008629+0000", 2024-09-15T03:46:35.744 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.606091123999999, 2024-09-15T03:46:35.744 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0035108140000000001, 2024-09-15T03:46:35.744 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:35.744 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:35.744 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:35.744 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:35.744 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:35.745 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 98 2024-09-15T03:46:35.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.745 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:35.745 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.745 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:35.745 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008629+0000", 2024-09-15T03:46:35.745 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.745 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.745 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:35.745 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008629+0000", 2024-09-15T03:46:35.745 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.746 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.746 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:35.746 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008630+0000", 2024-09-15T03:46:35.746 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2100000000000001e-07 2024-09-15T03:46:35.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.746 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.746 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:35.746 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008633+0000", 2024-09-15T03:46:35.746 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.366e-06 2024-09-15T03:46:35.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.746 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.746 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:35.747 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008636+0000", 2024-09-15T03:46:35.747 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3330000000000001e-06 2024-09-15T03:46:35.747 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.747 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.747 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:35.747 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008638+0000", 2024-09-15T03:46:35.747 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.024e-06 2024-09-15T03:46:35.747 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.747 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.747 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:35.747 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011420+0000", 2024-09-15T03:46:35.747 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027817599999999999 2024-09-15T03:46:35.748 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.748 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.748 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:35.748 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011461+0000", 2024-09-15T03:46:35.748 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1180000000000002e-05 2024-09-15T03:46:35.748 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.748 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.748 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:35.748 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011525+0000", 2024-09-15T03:46:35.748 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3917000000000005e-05 2024-09-15T03:46:35.748 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.748 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.748 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:35.748 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011605+0000", 2024-09-15T03:46:35.749 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9813000000000003e-05 2024-09-15T03:46:35.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.749 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.749 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:35.749 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011612+0000", 2024-09-15T03:46:35.749 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5880000000000003e-06 2024-09-15T03:46:35.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.749 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.749 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:35.749 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011632+0000", 2024-09-15T03:46:35.749 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0882000000000001e-05 2024-09-15T03:46:35.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.749 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.749 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:35.750 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011977+0000", 2024-09-15T03:46:35.750 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034493000000000002 2024-09-15T03:46:35.750 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.750 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.750 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:35.750 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012095+0000", 2024-09-15T03:46:35.750 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000117791 2024-09-15T03:46:35.750 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.750 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.750 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:35.750 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012121+0000", 2024-09-15T03:46:35.750 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5729e-05 2024-09-15T03:46:35.751 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.751 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.751 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:35.751 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012140+0000", 2024-09-15T03:46:35.751 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9279999999999998e-05 2024-09-15T03:46:35.751 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.751 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:35.751 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.751 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.751 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.751 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-15T03:46:35.751 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:19.008643+0000", 2024-09-15T03:46:35.751 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.606077881000001, 2024-09-15T03:46:35.751 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003782948, 2024-09-15T03:46:35.752 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:35.752 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:35.752 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:35.752 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:35.752 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:35.752 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 99 2024-09-15T03:46:35.752 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.752 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:35.752 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.752 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:35.752 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008643+0000", 2024-09-15T03:46:35.752 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.752 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.753 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.753 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:35.753 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008643+0000", 2024-09-15T03:46:35.753 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.753 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.753 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.753 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:35.753 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008643+0000", 2024-09-15T03:46:35.753 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2499999999999999e-07 2024-09-15T03:46:35.753 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.753 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.753 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:35.753 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008717+0000", 2024-09-15T03:46:35.753 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4688e-05 2024-09-15T03:46:35.754 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.754 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.754 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:35.754 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008726+0000", 2024-09-15T03:46:35.754 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2109999999999998e-06 2024-09-15T03:46:35.754 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.754 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.754 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:35.754 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008730+0000", 2024-09-15T03:46:35.754 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7169999999999999e-06 2024-09-15T03:46:35.754 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.754 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.755 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:35.755 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011646+0000", 2024-09-15T03:46:35.755 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0029156759999999999 2024-09-15T03:46:35.755 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.755 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.755 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:35.755 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011698+0000", 2024-09-15T03:46:35.755 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1918e-05 2024-09-15T03:46:35.755 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.755 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.755 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:35.755 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011768+0000", 2024-09-15T03:46:35.756 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9741999999999997e-05 2024-09-15T03:46:35.756 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.756 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.756 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:35.756 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011858+0000", 2024-09-15T03:46:35.756 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0117999999999993e-05 2024-09-15T03:46:35.756 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.756 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.756 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:35.756 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011864+0000", 2024-09-15T03:46:35.756 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5980000000000002e-06 2024-09-15T03:46:35.756 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.757 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.757 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:35.757 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011914+0000", 2024-09-15T03:46:35.757 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9428000000000002e-05 2024-09-15T03:46:35.757 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.757 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.757 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:35.757 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011987+0000", 2024-09-15T03:46:35.757 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2873999999999994e-05 2024-09-15T03:46:35.757 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.757 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.757 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:35.757 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012383+0000", 2024-09-15T03:46:35.758 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039653399999999998 2024-09-15T03:46:35.758 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.758 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.758 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:35.758 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012408+0000", 2024-09-15T03:46:35.758 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4890000000000001e-05 2024-09-15T03:46:35.758 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.758 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.758 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:35.758 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012426+0000", 2024-09-15T03:46:35.758 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7329e-05 2024-09-15T03:46:35.759 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.759 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:35.759 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.759 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.759 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.759 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-15T03:46:35.759 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:19.008737+0000", 2024-09-15T03:46:35.759 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.605983896000001, 2024-09-15T03:46:35.759 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003912993, 2024-09-15T03:46:35.759 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:35.759 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:35.759 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:35.759 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:35.759 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:35.760 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 100 2024-09-15T03:46:35.760 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.760 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:35.760 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.760 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:35.760 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008737+0000", 2024-09-15T03:46:35.760 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.760 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.760 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.760 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:35.760 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008737+0000", 2024-09-15T03:46:35.760 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.760 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.761 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.761 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:35.761 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008737+0000", 2024-09-15T03:46:35.761 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4700000000000001e-07 2024-09-15T03:46:35.761 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.761 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.761 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:35.761 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008746+0000", 2024-09-15T03:46:35.761 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7229999999999993e-06 2024-09-15T03:46:35.761 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.761 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.761 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:35.761 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008756+0000", 2024-09-15T03:46:35.761 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6800000000000005e-06 2024-09-15T03:46:35.762 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.762 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.762 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:35.762 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008760+0000", 2024-09-15T03:46:35.762 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9210000000000002e-06 2024-09-15T03:46:35.762 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.762 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.762 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:35.762 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012160+0000", 2024-09-15T03:46:35.762 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0034000469999999998 2024-09-15T03:46:35.762 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.762 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.762 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:35.762 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012205+0000", 2024-09-15T03:46:35.763 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5405000000000003e-05 2024-09-15T03:46:35.763 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.763 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.763 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:35.763 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012269+0000", 2024-09-15T03:46:35.763 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4068999999999999e-05 2024-09-15T03:46:35.763 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.763 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.763 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:35.763 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012346+0000", 2024-09-15T03:46:35.763 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6903999999999994e-05 2024-09-15T03:46:35.763 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.763 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.764 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:35.764 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012353+0000", 2024-09-15T03:46:35.764 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3100000000000003e-06 2024-09-15T03:46:35.764 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.764 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.764 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:35.764 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012377+0000", 2024-09-15T03:46:35.764 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3730000000000001e-05 2024-09-15T03:46:35.764 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.764 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.764 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:35.764 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012442+0000", 2024-09-15T03:46:35.764 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4827999999999997e-05 2024-09-15T03:46:35.764 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.765 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.765 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:35.765 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012595+0000", 2024-09-15T03:46:35.765 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015312 2024-09-15T03:46:35.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.765 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.765 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:35.765 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012618+0000", 2024-09-15T03:46:35.765 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3045e-05 2024-09-15T03:46:35.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.766 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.766 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:35.766 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012650+0000", 2024-09-15T03:46:35.766 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1563999999999998e-05 2024-09-15T03:46:35.766 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.766 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:35.766 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.766 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.766 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.766 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4263.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2024-09-15T03:46:35.766 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:46:35.766 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.570969812, 2024-09-15T03:46:35.766 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019495788999999999, 2024-09-15T03:46:35.767 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:35.767 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:35.767 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:35.767 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4263", 2024-09-15T03:46:35.767 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/646873808", 2024-09-15T03:46:35.767 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2 2024-09-15T03:46:35.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.767 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:35.767 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.767 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:35.767 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:46:35.767 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.767 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.768 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:35.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:46:35.768 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:35.768 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.768 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.768 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:35.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043752+0000", 2024-09-15T03:46:35.768 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.815e-06 2024-09-15T03:46:35.768 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.768 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.768 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:35.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043757+0000", 2024-09-15T03:46:35.768 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4240000000000003e-06 2024-09-15T03:46:35.768 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.769 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:35.769 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043766+0000", 2024-09-15T03:46:35.769 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8629999999999997e-06 2024-09-15T03:46:35.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.769 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:35.769 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043773+0000", 2024-09-15T03:46:35.769 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0099999999999998e-06 2024-09-15T03:46:35.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.769 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:35.769 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043830+0000", 2024-09-15T03:46:35.770 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7089999999999997e-05 2024-09-15T03:46:35.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.770 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.770 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:46:35.770 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043837+0000", 2024-09-15T03:46:35.770 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.08e-06 2024-09-15T03:46:35.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.770 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.770 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:35.770 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063103+0000", 2024-09-15T03:46:35.770 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019265619000000001 2024-09-15T03:46:35.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:35.770 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:35.770 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:35.771 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063246+0000", 2024-09-15T03:46:35.771 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000143888 2024-09-15T03:46:35.771 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.771 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:35.771 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.771 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:35.771 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:35.771 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:35.772 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:46:35.773 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:35.772+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:46:35.870 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:35.871 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:35.871 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:46:35.871 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:35.874 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:35.873+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:35.879 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:46:35.974 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:35.975 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:35.975 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:46:35.975 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:46:35.975 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:35.975 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:35.974+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:35.983 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:46:36.076 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:36.075+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:36.087 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-15T03:46:36.087 INFO:teuthology.orchestra.run.smithi001.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-15T03:46:36.130 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:36.130 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:46:36.131 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:46:36.131 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:46:36.131 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.131 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4263.0:1 2.3 e22/21 2:c4c92e5a:::rbd_trash:head v 22'1, mlcod=0'0)", 2024-09-15T03:46:36.131 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:24.039192+0000", 2024-09-15T03:46:36.131 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 12.091200388000001, 2024-09-15T03:46:36.131 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001176968, 2024-09-15T03:46:36.131 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:36.131 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:36.132 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:36.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.132 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:36.132 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039192+0000", 2024-09-15T03:46:36.132 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:36.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.132 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:36.132 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039192+0000", 2024-09-15T03:46:36.132 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:36.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.133 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.133 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:36.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039192+0000", 2024-09-15T03:46:36.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3999999999999997e-07 2024-09-15T03:46:36.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.133 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.133 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:36.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039199+0000", 2024-09-15T03:46:36.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2580000000000001e-06 2024-09-15T03:46:36.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:36.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039217+0000", 2024-09-15T03:46:36.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8895000000000001e-05 2024-09-15T03:46:36.134 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:36.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039226+0000", 2024-09-15T03:46:36.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4840000000000004e-06 2024-09-15T03:46:36.134 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:36.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039268+0000", 2024-09-15T03:46:36.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2104000000000002e-05 2024-09-15T03:46:36.135 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.135 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for flush", 2024-09-15T03:46:36.135 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039283+0000", 2024-09-15T03:46:36.135 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5009e-05 2024-09-15T03:46:36.135 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.135 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:36.135 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040042+0000", 2024-09-15T03:46:36.135 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00075874199999999997 2024-09-15T03:46:36.135 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.135 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:36.136 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040067+0000", 2024-09-15T03:46:36.136 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4888999999999998e-05 2024-09-15T03:46:36.136 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.136 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.136 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:36.136 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040169+0000", 2024-09-15T03:46:36.136 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010252299999999999 2024-09-15T03:46:36.136 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.136 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.136 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:36.136 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040178+0000", 2024-09-15T03:46:36.136 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7709999999999995e-06 2024-09-15T03:46:36.136 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.136 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:36.137 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040211+0000", 2024-09-15T03:46:36.137 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3552e-05 2024-09-15T03:46:36.137 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:36.137 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040226+0000", 2024-09-15T03:46:36.137 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4695000000000001e-05 2024-09-15T03:46:36.137 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:36.137 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040369+0000", 2024-09-15T03:46:36.137 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014270600000000001 2024-09-15T03:46:36.138 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:36.138 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:36.138 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:36.138 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:36.138 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:36.138 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:36.139 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:46:36.177 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:36.176+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:46:36.278 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:36.278 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:36.278 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:46:36.278 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:36.278 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:36.277+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:46:36.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:46:36.378 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:36.378+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:36.422 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:36.423 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:36.423 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:46:36.423 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:46:36.423 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:36.431 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:46:36.479 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:36.479+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:46:36.567 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:36.568 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:46:36.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:46:36.568 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [] 2024-09-15T03:46:36.568 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:36.576 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:46:36.580 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:36.579+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:36.673 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:36.673 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:36.673 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:46:36.673 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:36.681 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:46:36.682 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:36.680+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:46:36.777 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:36.778 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:36.778 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:46:36.778 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:46:36.778 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:36.782 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:36.781+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:36.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:46:36.883 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:46:36.882+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:46:36.907 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:46:36.922 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:36.923 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:46:36.923 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:46:36.923 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:46:36.923 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.923 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4263.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-15T03:46:36.923 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:24.037919+0000", 2024-09-15T03:46:36.923 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 12.884674715999999, 2024-09-15T03:46:36.923 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025392869999999999, 2024-09-15T03:46:36.923 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:36.923 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:36.923 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:36.923 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4263", 2024-09-15T03:46:36.924 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/646873808", 2024-09-15T03:46:36.924 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1 2024-09-15T03:46:36.924 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.924 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:36.924 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.924 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:36.924 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.037919+0000", 2024-09-15T03:46:36.924 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:36.924 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.924 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.924 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:36.924 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.037919+0000", 2024-09-15T03:46:36.924 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:36.925 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.925 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.925 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:36.925 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.037920+0000", 2024-09-15T03:46:36.925 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.863e-06 2024-09-15T03:46:36.925 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.925 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.925 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:36.925 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.037930+0000", 2024-09-15T03:46:36.925 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1530000000000003e-06 2024-09-15T03:46:36.925 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.925 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.925 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:36.925 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.037948+0000", 2024-09-15T03:46:36.926 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7969e-05 2024-09-15T03:46:36.926 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:36.926 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.037958+0000", 2024-09-15T03:46:36.926 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0288e-05 2024-09-15T03:46:36.926 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:36.926 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.038029+0000", 2024-09-15T03:46:36.926 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0778999999999994e-05 2024-09-15T03:46:36.926 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.927 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:46:36.927 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.038037+0000", 2024-09-15T03:46:36.927 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6179999999999995e-06 2024-09-15T03:46:36.927 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.927 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.927 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:36.927 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.038835+0000", 2024-09-15T03:46:36.927 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00079816700000000002 2024-09-15T03:46:36.927 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.927 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.927 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:36.927 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.038974+0000", 2024-09-15T03:46:36.927 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013816499999999999 2024-09-15T03:46:36.927 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.928 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.928 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 9", 2024-09-15T03:46:36.928 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039080+0000", 2024-09-15T03:46:36.928 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000105969 2024-09-15T03:46:36.928 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.928 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.928 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:36.928 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039223+0000", 2024-09-15T03:46:36.928 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014346399999999999 2024-09-15T03:46:36.928 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.928 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.928 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:36.929 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039232+0000", 2024-09-15T03:46:36.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.935e-06 2024-09-15T03:46:36.929 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.929 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.929 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:36.929 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039283+0000", 2024-09-15T03:46:36.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0807999999999999e-05 2024-09-15T03:46:36.929 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.929 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.929 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:36.929 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039475+0000", 2024-09-15T03:46:36.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019226000000000001 2024-09-15T03:46:36.929 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.930 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.930 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:36.930 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040394+0000", 2024-09-15T03:46:36.930 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00091921199999999998 2024-09-15T03:46:36.930 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.930 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.930 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:36.930 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040430+0000", 2024-09-15T03:46:36.930 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5954999999999997e-05 2024-09-15T03:46:36.930 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.930 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.930 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:36.930 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040458+0000", 2024-09-15T03:46:36.930 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7682e-05 2024-09-15T03:46:36.931 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:36.931 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:36.931 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:36.931 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.931 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.931 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.4263.0:1 2.3 e22/21 ondisk, result = 0)", 2024-09-15T03:46:36.931 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:24.040322+0000", 2024-09-15T03:46:36.931 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 12.882271413, 2024-09-15T03:46:36.931 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014034400000000001, 2024-09-15T03:46:36.931 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:36.931 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:46:36.931 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:36.932 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.932 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:36.932 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040322+0000", 2024-09-15T03:46:36.932 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:36.932 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.932 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.932 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:36.932 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040322+0000", 2024-09-15T03:46:36.932 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:36.932 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.932 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.933 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:36.933 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040323+0000", 2024-09-15T03:46:36.933 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6799999999999999e-07 2024-09-15T03:46:36.933 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.933 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.933 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:36.933 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040328+0000", 2024-09-15T03:46:36.933 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0270000000000003e-06 2024-09-15T03:46:36.933 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.933 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.933 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:36.933 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040339+0000", 2024-09-15T03:46:36.933 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1591e-05 2024-09-15T03:46:36.933 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.934 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:36.934 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040345+0000", 2024-09-15T03:46:36.934 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5380000000000002e-06 2024-09-15T03:46:36.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.934 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:36.934 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040385+0000", 2024-09-15T03:46:36.934 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0333000000000002e-05 2024-09-15T03:46:36.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.934 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:36.934 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040391+0000", 2024-09-15T03:46:36.934 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6540000000000004e-06 2024-09-15T03:46:36.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:36.935 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:36.935 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:36.935 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040462+0000", 2024-09-15T03:46:36.935 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1433000000000005e-05 2024-09-15T03:46:36.935 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:36.935 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:36.935 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:36.935 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:36.935 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:36.935 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:36.936 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:46:36.995 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:36.994+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:46:37.037 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:46:37.038 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:46:37.052 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:46:37.052 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:46:37.060 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:37.061 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:37.067 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:46:37.067 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:46:37.071 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:46:37.083 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:46:37.084 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:46:37.085 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:37.084+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:37.098 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:46:37.099 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:46:37.113 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:46:37.113 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:46:37.120 DEBUG:teuthology.orchestra.run.smithi001:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0 2024-09-15T03:46:37.129 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:46:37.129 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:46:37.145 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:46:37.145 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:46:37.146 INFO:tasks.rados.rados.0.smithi001.stdout:adding op weight read -> 100 2024-09-15T03:46:37.146 INFO:tasks.rados.rados.0.smithi001.stdout:adding op weight delete -> 50 2024-09-15T03:46:37.146 INFO:tasks.rados.rados.0.smithi001.stdout:adding op weight snap_create -> 50 2024-09-15T03:46:37.146 INFO:tasks.rados.rados.0.smithi001.stdout:adding op weight snap_remove -> 50 2024-09-15T03:46:37.146 INFO:tasks.rados.rados.0.smithi001.stdout:adding op weight rollback -> 50 2024-09-15T03:46:37.146 INFO:tasks.rados.rados.0.smithi001.stdout:adding op weight setattr -> 25 2024-09-15T03:46:37.146 INFO:tasks.rados.rados.0.smithi001.stdout:adding op weight rmattr -> 25 2024-09-15T03:46:37.146 INFO:tasks.rados.rados.0.smithi001.stdout:adding op weight copy_from -> 50 2024-09-15T03:46:37.147 INFO:tasks.rados.rados.0.smithi001.stdout:adding op weight append -> 100 2024-09-15T03:46:37.147 INFO:tasks.rados.rados.0.smithi001.stdout:ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable) 2024-09-15T03:46:37.147 INFO:tasks.rados.rados.0.smithi001.stdout:Configuration: 2024-09-15T03:46:37.147 INFO:tasks.rados.rados.0.smithi001.stdout: Number of operations: 400 2024-09-15T03:46:37.147 INFO:tasks.rados.rados.0.smithi001.stdout: Number of objects: 50 2024-09-15T03:46:37.147 INFO:tasks.rados.rados.0.smithi001.stdout: Max in flight operations: 16 2024-09-15T03:46:37.147 INFO:tasks.rados.rados.0.smithi001.stdout: Object size (in bytes): 4000000 2024-09-15T03:46:37.147 INFO:tasks.rados.rados.0.smithi001.stdout: Write stride min: 400000 2024-09-15T03:46:37.147 INFO:tasks.rados.rados.0.smithi001.stdout: Write stride max: 800000 2024-09-15T03:46:37.161 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:46:37.162 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:46:37.176 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:46:37.177 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:46:37.186 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:37.186 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:37.186 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:37.186+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:37.191 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:46:37.192 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:46:37.195 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:46:37.207 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:46:37.207 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:46:37.279 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-15T03:46:37.279 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T03:46:37.279 INFO:tasks.thrashosds.thrasher:Removing osd 10, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-15T03:46:37.279 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 10 2024-09-15T03:46:37.287 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:37.286+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:46:37.307 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:37.308 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:37.317 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:46:37.387 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:37.387+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:46:37.460 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:37.460 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:37.469 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:46:37.513 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:37.512+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:46:37.565 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-09-15T03:46:37.564+0000 7ff94fdf1640 -1 mon.a@0(leader).osd e30 definitely_dead 0 2024-09-15T03:46:37.570 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:37.570 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:37.579 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:46:37.621 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:37.620+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:37.690 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:37.689+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:37.721 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:37.721 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:37.730 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:46:37.791 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:37.790+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:37.830 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:37.831 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:37.839 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:46:37.892 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:37.892+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:46:37.939 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:37.939 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:37.947 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:46:37.993 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:37.993+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:38.048 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:38.048 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:38.056 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:46:38.096 INFO:tasks.rados.rados.0.smithi001.stdout:1: write initial oid 1 2024-09-15T03:46:38.096 INFO:tasks.rados.rados.0.smithi001.stdout:1: seq_num 0 ranges {0=32768} 2024-09-15T03:46:38.096 INFO:teuthology.orchestra.run.smithi001.stderr:marked out osd.10. 2024-09-15T03:46:38.096 INFO:tasks.rados.rados.0.smithi001.stdout:1: writing smithi00132172-1 from 0 to 32768 tid 1 2024-09-15T03:46:38.097 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:38.095+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:46:38.098 INFO:tasks.rados.rados.0.smithi001.stdout:2: write initial oid 2 2024-09-15T03:46:38.098 INFO:tasks.rados.rados.0.smithi001.stdout:2: seq_num 1 ranges {0=32768} 2024-09-15T03:46:38.098 INFO:tasks.rados.rados.0.smithi001.stdout:2: writing smithi00132172-2 from 0 to 32768 tid 1 2024-09-15T03:46:38.098 INFO:tasks.rados.rados.0.smithi001.stdout:3: write initial oid 3 2024-09-15T03:46:38.098 INFO:tasks.rados.rados.0.smithi001.stdout:3: seq_num 2 ranges {0=49152} 2024-09-15T03:46:38.098 INFO:tasks.rados.rados.0.smithi001.stdout:3: writing smithi00132172-3 from 0 to 49152 tid 1 2024-09-15T03:46:38.098 INFO:tasks.rados.rados.0.smithi001.stdout:4: write initial oid 4 2024-09-15T03:46:38.098 INFO:tasks.rados.rados.0.smithi001.stdout:4: seq_num 3 ranges {0=65536} 2024-09-15T03:46:38.098 INFO:tasks.rados.rados.0.smithi001.stdout:4: writing smithi00132172-4 from 0 to 65536 tid 1 2024-09-15T03:46:38.098 INFO:tasks.rados.rados.0.smithi001.stdout:5: write initial oid 5 2024-09-15T03:46:38.098 INFO:tasks.rados.rados.0.smithi001.stdout:5: seq_num 4 ranges {0=81920} 2024-09-15T03:46:38.099 INFO:tasks.rados.rados.0.smithi001.stdout:5: writing smithi00132172-5 from 0 to 81920 tid 1 2024-09-15T03:46:38.099 INFO:tasks.rados.rados.0.smithi001.stdout:6: write initial oid 6 2024-09-15T03:46:38.099 INFO:tasks.rados.rados.0.smithi001.stdout:6: seq_num 5 ranges {0=98304} 2024-09-15T03:46:38.099 INFO:tasks.rados.rados.0.smithi001.stdout:6: writing smithi00132172-6 from 0 to 98304 tid 1 2024-09-15T03:46:38.099 INFO:tasks.rados.rados.0.smithi001.stdout:7: write initial oid 7 2024-09-15T03:46:38.099 INFO:tasks.rados.rados.0.smithi001.stdout:7: seq_num 6 ranges {0=114688} 2024-09-15T03:46:38.101 INFO:tasks.rados.rados.0.smithi001.stdout:7: writing smithi00132172-7 from 0 to 114688 tid 1 2024-09-15T03:46:38.101 INFO:tasks.rados.rados.0.smithi001.stdout:8: write initial oid 8 2024-09-15T03:46:38.101 INFO:tasks.rados.rados.0.smithi001.stdout:8: seq_num 7 ranges {0=131072} 2024-09-15T03:46:38.101 INFO:tasks.rados.rados.0.smithi001.stdout:8: writing smithi00132172-8 from 0 to 131072 tid 1 2024-09-15T03:46:38.101 INFO:tasks.rados.rados.0.smithi001.stdout:9: write initial oid 9 2024-09-15T03:46:38.101 INFO:tasks.rados.rados.0.smithi001.stdout:9: seq_num 8 ranges {0=147456} 2024-09-15T03:46:38.103 INFO:tasks.rados.rados.0.smithi001.stdout:9: writing smithi00132172-9 from 0 to 147456 tid 1 2024-09-15T03:46:38.103 INFO:tasks.rados.rados.0.smithi001.stdout:10: write initial oid 10 2024-09-15T03:46:38.104 INFO:tasks.rados.rados.0.smithi001.stdout:10: seq_num 9 ranges {0=163840} 2024-09-15T03:46:38.105 INFO:tasks.rados.rados.0.smithi001.stdout:10: writing smithi00132172-10 from 0 to 163840 tid 1 2024-09-15T03:46:38.105 INFO:tasks.rados.rados.0.smithi001.stdout:11: write initial oid 11 2024-09-15T03:46:38.105 INFO:tasks.rados.rados.0.smithi001.stdout:11: seq_num 10 ranges {0=180224} 2024-09-15T03:46:38.106 INFO:tasks.rados.rados.0.smithi001.stdout:11: writing smithi00132172-11 from 0 to 180224 tid 1 2024-09-15T03:46:38.106 INFO:tasks.rados.rados.0.smithi001.stdout:12: write initial oid 12 2024-09-15T03:46:38.106 INFO:tasks.rados.rados.0.smithi001.stdout:12: seq_num 11 ranges {0=196608} 2024-09-15T03:46:38.107 INFO:tasks.rados.rados.0.smithi001.stdout:12: writing smithi00132172-12 from 0 to 196608 tid 1 2024-09-15T03:46:38.107 INFO:tasks.rados.rados.0.smithi001.stdout:13: write initial oid 13 2024-09-15T03:46:38.108 INFO:tasks.rados.rados.0.smithi001.stdout:13: seq_num 12 ranges {0=212992} 2024-09-15T03:46:38.109 INFO:tasks.rados.rados.0.smithi001.stdout:13: writing smithi00132172-13 from 0 to 212992 tid 1 2024-09-15T03:46:38.109 INFO:tasks.rados.rados.0.smithi001.stdout:14: write initial oid 14 2024-09-15T03:46:38.109 INFO:tasks.rados.rados.0.smithi001.stdout:14: seq_num 13 ranges {0=229376} 2024-09-15T03:46:38.111 INFO:tasks.rados.rados.0.smithi001.stdout:14: writing smithi00132172-14 from 0 to 229376 tid 1 2024-09-15T03:46:38.111 INFO:tasks.rados.rados.0.smithi001.stdout:15: write initial oid 15 2024-09-15T03:46:38.111 INFO:tasks.rados.rados.0.smithi001.stdout:15: seq_num 14 ranges {0=245760} 2024-09-15T03:46:38.113 INFO:tasks.rados.rados.0.smithi001.stdout:15: writing smithi00132172-15 from 0 to 245760 tid 1 2024-09-15T03:46:38.114 INFO:tasks.rados.rados.0.smithi001.stdout:16: write initial oid 16 2024-09-15T03:46:38.114 INFO:tasks.rados.rados.0.smithi001.stdout:16: seq_num 15 ranges {0=262144} 2024-09-15T03:46:38.116 INFO:tasks.rados.rados.0.smithi001.stdout:16: writing smithi00132172-16 from 0 to 262144 tid 1 2024-09-15T03:46:38.116 INFO:tasks.rados.rados.0.smithi001.stdout: waiting on 16 2024-09-15T03:46:38.116 INFO:tasks.rados.rados.0.smithi001.stdout:13: finishing write tid 1 to smithi00132172-13 2024-09-15T03:46:38.116 INFO:tasks.rados.rados.0.smithi001.stdout:13: finishing write tid 2 to smithi00132172-13 2024-09-15T03:46:38.116 INFO:tasks.rados.rados.0.smithi001.stdout:4: finishing write tid 1 to smithi00132172-4 2024-09-15T03:46:38.117 INFO:tasks.rados.rados.0.smithi001.stdout:4: finishing write tid 2 to smithi00132172-4 2024-09-15T03:46:38.118 INFO:tasks.rados.rados.0.smithi001.stdout:14: finishing write tid 1 to smithi00132172-14 2024-09-15T03:46:38.118 INFO:tasks.rados.rados.0.smithi001.stdout:14: finishing write tid 2 to smithi00132172-14 2024-09-15T03:46:38.118 INFO:tasks.rados.rados.0.smithi001.stdout:13: finishing write tid 3 to smithi00132172-13 2024-09-15T03:46:38.118 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 13 v 2 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-15T03:46:38.118 INFO:tasks.rados.rados.0.smithi001.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-09-15T03:46:38.118 INFO:tasks.rados.rados.0.smithi001.stdout:13: done (15 left) 2024-09-15T03:46:38.118 INFO:tasks.rados.rados.0.smithi001.stdout:17: write initial oid 17 2024-09-15T03:46:38.118 INFO:tasks.rados.rados.0.smithi001.stdout:17: seq_num 16 ranges {0=278528} 2024-09-15T03:46:38.118 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T03:46:38.120 INFO:tasks.rados.rados.0.smithi001.stdout:17: writing smithi00132172-17 from 0 to 278528 tid 1 2024-09-15T03:46:38.120 INFO:tasks.rados.rados.0.smithi001.stdout: waiting on 16 2024-09-15T03:46:38.120 INFO:tasks.rados.rados.0.smithi001.stdout:4: finishing write tid 3 to smithi00132172-4 2024-09-15T03:46:38.121 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 4 v 2 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-15T03:46:38.122 INFO:tasks.rados.rados.0.smithi001.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-09-15T03:46:38.122 INFO:tasks.rados.rados.0.smithi001.stdout:14: finishing write tid 3 to smithi00132172-14 2024-09-15T03:46:38.122 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 14 v 4 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-15T03:46:38.122 INFO:tasks.rados.rados.0.smithi001.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-09-15T03:46:38.122 INFO:tasks.rados.rados.0.smithi001.stdout:4: done (15 left) 2024-09-15T03:46:38.122 INFO:tasks.rados.rados.0.smithi001.stdout:14: done (14 left) 2024-09-15T03:46:38.122 INFO:tasks.rados.rados.0.smithi001.stdout:18: write initial oid 18 2024-09-15T03:46:38.122 INFO:tasks.rados.rados.0.smithi001.stdout:18: seq_num 17 ranges {0=294912} 2024-09-15T03:46:38.123 INFO:tasks.rados.rados.0.smithi001.stdout:18: writing smithi00132172-18 from 0 to 294912 tid 1 2024-09-15T03:46:38.123 INFO:tasks.rados.rados.0.smithi001.stdout:19: write initial oid 19 2024-09-15T03:46:38.123 INFO:tasks.rados.rados.0.smithi001.stdout:19: seq_num 18 ranges {0=311296} 2024-09-15T03:46:38.125 INFO:tasks.rados.rados.0.smithi001.stdout:19: writing smithi00132172-19 from 0 to 311296 tid 1 2024-09-15T03:46:38.125 INFO:tasks.rados.rados.0.smithi001.stdout: waiting on 16 2024-09-15T03:46:38.128 INFO:tasks.rados.rados.0.smithi001.stdout:19: finishing write tid 1 to smithi00132172-19 2024-09-15T03:46:38.128 INFO:tasks.rados.rados.0.smithi001.stdout:19: finishing write tid 2 to smithi00132172-19 2024-09-15T03:46:38.130 INFO:tasks.rados.rados.0.smithi001.stdout:19: finishing write tid 3 to smithi00132172-19 2024-09-15T03:46:38.130 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-15T03:46:38.130 INFO:tasks.rados.rados.0.smithi001.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-09-15T03:46:38.130 INFO:tasks.rados.rados.0.smithi001.stdout:19: done (15 left) 2024-09-15T03:46:38.130 INFO:tasks.rados.rados.0.smithi001.stdout:20: write initial oid 20 2024-09-15T03:46:38.130 INFO:tasks.rados.rados.0.smithi001.stdout:20: seq_num 19 ranges {0=327680} 2024-09-15T03:46:38.131 INFO:tasks.rados.rados.0.smithi001.stdout:20: writing smithi00132172-20 from 0 to 327680 tid 1 2024-09-15T03:46:38.131 INFO:tasks.rados.rados.0.smithi001.stdout: waiting on 16 2024-09-15T03:46:38.173 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:38.173 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:38.182 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:46:38.195 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:38.194+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:46:38.295 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:38.295+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:38.324 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:38.324 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:38.333 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:46:38.396 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:38.396+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:46:38.434 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:38.434 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:38.443 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:46:38.497 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:38.497+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:46:38.557 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:38.557 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:38.566 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:46:38.599 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:38.598+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:38.668 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:38.668 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:38.676 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:46:38.699 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:38.698+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:46:38.777 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:38.777 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:38.785 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:46:38.799 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:38.798+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:38.887 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:38.887 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:38.895 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:46:38.939 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:38.938+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:46:38.996 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:38.996 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:39.000 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:39.000+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:39.004 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:46:39.101 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:39.100+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:46:39.103 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-15T03:46:39.122 INFO:tasks.rados.rados.0.smithi001.stdout:8: finishing write tid 1 to smithi00132172-8 2024-09-15T03:46:39.122 INFO:tasks.rados.rados.0.smithi001.stdout:8: finishing write tid 2 to smithi00132172-8 2024-09-15T03:46:39.124 INFO:tasks.rados.rados.0.smithi001.stdout:8: finishing write tid 3 to smithi00132172-8 2024-09-15T03:46:39.124 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 8 v 2 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-15T03:46:39.124 INFO:tasks.rados.rados.0.smithi001.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-09-15T03:46:39.124 INFO:tasks.rados.rados.0.smithi001.stdout:8: done (15 left) 2024-09-15T03:46:39.124 INFO:tasks.rados.rados.0.smithi001.stdout:21: write initial oid 21 2024-09-15T03:46:39.124 INFO:tasks.rados.rados.0.smithi001.stdout:21: seq_num 20 ranges {0=344064} 2024-09-15T03:46:39.126 INFO:tasks.rados.rados.0.smithi001.stdout:21: writing smithi00132172-21 from 0 to 344064 tid 1 2024-09-15T03:46:39.126 INFO:tasks.rados.rados.0.smithi001.stdout: waiting on 16 2024-09-15T03:46:39.153 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:39.153 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:39.162 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:46:39.202 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:39.201+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:46:39.218 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:46:39.230 INFO:tasks.rados.rados.0.smithi001.stdout:21: finishing write tid 1 to smithi00132172-21 2024-09-15T03:46:39.232 INFO:tasks.rados.rados.0.smithi001.stdout:21: finishing write tid 2 to smithi00132172-21 2024-09-15T03:46:39.234 INFO:tasks.rados.rados.0.smithi001.stdout:21: finishing write tid 3 to smithi00132172-21 2024-09-15T03:46:39.234 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 21 v 2 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-15T03:46:39.234 INFO:tasks.rados.rados.0.smithi001.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-09-15T03:46:39.234 INFO:tasks.rados.rados.0.smithi001.stdout:21: done (15 left) 2024-09-15T03:46:39.234 INFO:tasks.rados.rados.0.smithi001.stdout:22: write initial oid 22 2024-09-15T03:46:39.234 INFO:tasks.rados.rados.0.smithi001.stdout:22: seq_num 21 ranges {0=360448} 2024-09-15T03:46:39.235 INFO:tasks.rados.rados.0.smithi001.stdout:22: writing smithi00132172-22 from 0 to 360448 tid 1 2024-09-15T03:46:39.235 INFO:tasks.rados.rados.0.smithi001.stdout: waiting on 16 2024-09-15T03:46:39.264 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:39.264 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:39.272 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:46:39.316 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:39.316+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:46:39.347 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:46:39.347 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:46:39.361 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:46:39.361 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:46:39.373 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:39.373 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:39.376 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:46:39.376 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:46:39.384 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:46:39.392 INFO:tasks.rados.rados.0.smithi001.stdout:3: finishing write tid 1 to smithi00132172-3 2024-09-15T03:46:39.392 INFO:tasks.rados.rados.0.smithi001.stdout:3: finishing write tid 2 to smithi00132172-3 2024-09-15T03:46:39.394 INFO:tasks.rados.rados.0.smithi001.stdout:11: finishing write tid 1 to smithi00132172-11 2024-09-15T03:46:39.394 INFO:tasks.rados.rados.0.smithi001.stdout:11: finishing write tid 2 to smithi00132172-11 2024-09-15T03:46:39.394 INFO:tasks.rados.rados.0.smithi001.stdout:20: finishing write tid 1 to smithi00132172-20 2024-09-15T03:46:39.394 INFO:tasks.rados.rados.0.smithi001.stdout:20: finishing write tid 2 to smithi00132172-20 2024-09-15T03:46:39.394 INFO:tasks.rados.rados.0.smithi001.stdout:15: finishing write tid 1 to smithi00132172-15 2024-09-15T03:46:39.395 INFO:tasks.rados.rados.0.smithi001.stdout:15: finishing write tid 2 to smithi00132172-15 2024-09-15T03:46:39.395 INFO:tasks.rados.rados.0.smithi001.stdout:17: finishing write tid 1 to smithi00132172-17 2024-09-15T03:46:39.396 INFO:tasks.rados.rados.0.smithi001.stdout:20: finishing write tid 3 to smithi00132172-20 2024-09-15T03:46:39.396 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 20 v 2 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-15T03:46:39.396 INFO:tasks.rados.rados.0.smithi001.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-09-15T03:46:39.396 INFO:tasks.rados.rados.0.smithi001.stdout:20: done (15 left) 2024-09-15T03:46:39.396 INFO:tasks.rados.rados.0.smithi001.stdout:23: write initial oid 23 2024-09-15T03:46:39.396 INFO:tasks.rados.rados.0.smithi001.stdout:23: seq_num 22 ranges {0=376832} 2024-09-15T03:46:39.400 INFO:tasks.rados.rados.0.smithi001.stdout:23: writing smithi00132172-23 from 0 to 376832 tid 1 2024-09-15T03:46:39.400 INFO:tasks.rados.rados.0.smithi001.stdout:1: finishing write tid 1 to smithi00132172-1 2024-09-15T03:46:39.400 INFO:tasks.rados.rados.0.smithi001.stdout:1: finishing write tid 2 to smithi00132172-1 2024-09-15T03:46:39.400 INFO:tasks.rados.rados.0.smithi001.stdout:5: finishing write tid 1 to smithi00132172-5 2024-09-15T03:46:39.400 INFO:tasks.rados.rados.0.smithi001.stdout:17: finishing write tid 2 to smithi00132172-17 2024-09-15T03:46:39.400 INFO:tasks.rados.rados.0.smithi001.stdout:5: finishing write tid 2 to smithi00132172-5 2024-09-15T03:46:39.400 INFO:tasks.rados.rados.0.smithi001.stdout:6: finishing write tid 1 to smithi00132172-6 2024-09-15T03:46:39.400 INFO:tasks.rados.rados.0.smithi001.stdout:6: finishing write tid 2 to smithi00132172-6 2024-09-15T03:46:39.400 INFO:tasks.rados.rados.0.smithi001.stdout:22: finishing write tid 1 to smithi00132172-22 2024-09-15T03:46:39.401 INFO:tasks.rados.rados.0.smithi001.stdout:22: finishing write tid 2 to smithi00132172-22 2024-09-15T03:46:39.401 INFO:tasks.rados.rados.0.smithi001.stdout:12: finishing write tid 1 to smithi00132172-12 2024-09-15T03:46:39.401 INFO:tasks.rados.rados.0.smithi001.stdout:12: finishing write tid 2 to smithi00132172-12 2024-09-15T03:46:39.401 INFO:tasks.rados.rados.0.smithi001.stdout:1: finishing write tid 3 to smithi00132172-1 2024-09-15T03:46:39.401 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 1 v 2 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-15T03:46:39.401 INFO:tasks.rados.rados.0.smithi001.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-09-15T03:46:39.401 INFO:tasks.rados.rados.0.smithi001.stdout:11: finishing write tid 3 to smithi00132172-11 2024-09-15T03:46:39.401 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 11 v 2 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-15T03:46:39.401 INFO:tasks.rados.rados.0.smithi001.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-09-15T03:46:39.401 INFO:tasks.rados.rados.0.smithi001.stdout:16: finishing write tid 1 to smithi00132172-16 2024-09-15T03:46:39.401 INFO:tasks.rados.rados.0.smithi001.stdout:16: finishing write tid 2 to smithi00132172-16 2024-09-15T03:46:39.401 INFO:tasks.rados.rados.0.smithi001.stdout:18: finishing write tid 1 to smithi00132172-18 2024-09-15T03:46:39.402 INFO:tasks.rados.rados.0.smithi001.stdout:3: finishing write tid 3 to smithi00132172-3 2024-09-15T03:46:39.402 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-15T03:46:39.402 INFO:tasks.rados.rados.0.smithi001.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-09-15T03:46:39.402 INFO:tasks.rados.rados.0.smithi001.stdout:18: finishing write tid 2 to smithi00132172-18 2024-09-15T03:46:39.402 INFO:tasks.rados.rados.0.smithi001.stdout:5: finishing write tid 3 to smithi00132172-5 2024-09-15T03:46:39.402 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 5 v 4 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-15T03:46:39.402 INFO:tasks.rados.rados.0.smithi001.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-09-15T03:46:39.402 INFO:tasks.rados.rados.0.smithi001.stdout:15: finishing write tid 3 to smithi00132172-15 2024-09-15T03:46:39.402 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 15 v 5 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-15T03:46:39.402 INFO:tasks.rados.rados.0.smithi001.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-09-15T03:46:39.402 INFO:tasks.rados.rados.0.smithi001.stdout:2: finishing write tid 1 to smithi00132172-2 2024-09-15T03:46:39.402 INFO:tasks.rados.rados.0.smithi001.stdout:2: finishing write tid 2 to smithi00132172-2 2024-09-15T03:46:39.402 INFO:tasks.rados.rados.0.smithi001.stdout:1: done (15 left) 2024-09-15T03:46:39.403 INFO:tasks.rados.rados.0.smithi001.stdout:3: done (14 left) 2024-09-15T03:46:39.403 INFO:tasks.rados.rados.0.smithi001.stdout:5: done (13 left) 2024-09-15T03:46:39.403 INFO:tasks.rados.rados.0.smithi001.stdout:11: done (12 left) 2024-09-15T03:46:39.403 INFO:tasks.rados.rados.0.smithi001.stdout:15: done (11 left) 2024-09-15T03:46:39.403 INFO:tasks.rados.rados.0.smithi001.stdout:24: write initial oid 24 2024-09-15T03:46:39.403 INFO:tasks.rados.rados.0.smithi001.stdout:24: seq_num 23 ranges {0=393216} 2024-09-15T03:46:39.404 INFO:tasks.rados.rados.0.smithi001.stdout:24: writing smithi00132172-24 from 0 to 393216 tid 1 2024-09-15T03:46:39.404 INFO:tasks.rados.rados.0.smithi001.stdout:25: write initial oid 25 2024-09-15T03:46:39.404 INFO:tasks.rados.rados.0.smithi001.stdout:25: seq_num 24 ranges {0=409600} 2024-09-15T03:46:39.404 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:39.403+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:46:39.408 INFO:tasks.rados.rados.0.smithi001.stdout:25: writing smithi00132172-25 from 0 to 409600 tid 1 2024-09-15T03:46:39.408 INFO:tasks.rados.rados.0.smithi001.stdout:26: write initial oid 26 2024-09-15T03:46:39.408 INFO:tasks.rados.rados.0.smithi001.stdout:7: finishing write tid 1 to smithi00132172-7 2024-09-15T03:46:39.408 INFO:tasks.rados.rados.0.smithi001.stdout:7: finishing write tid 2 to smithi00132172-7 2024-09-15T03:46:39.408 INFO:tasks.rados.rados.0.smithi001.stdout:16: finishing write tid 3 to smithi00132172-16 2024-09-15T03:46:39.409 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 16 v 2 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-15T03:46:39.409 INFO:tasks.rados.rados.0.smithi001.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-09-15T03:46:39.409 INFO:tasks.rados.rados.0.smithi001.stdout:9: finishing write tid 1 to smithi00132172-9 2024-09-15T03:46:39.409 INFO:tasks.rados.rados.0.smithi001.stdout:9: finishing write tid 2 to smithi00132172-9 2024-09-15T03:46:39.409 INFO:tasks.rados.rados.0.smithi001.stdout:18: finishing write tid 3 to smithi00132172-18 2024-09-15T03:46:39.409 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 18 v 4 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-15T03:46:39.409 INFO:tasks.rados.rados.0.smithi001.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-09-15T03:46:39.409 INFO:tasks.rados.rados.0.smithi001.stdout:6: finishing write tid 3 to smithi00132172-6 2024-09-15T03:46:39.409 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 6 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-15T03:46:39.409 INFO:tasks.rados.rados.0.smithi001.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-09-15T03:46:39.409 INFO:tasks.rados.rados.0.smithi001.stdout:10: finishing write tid 1 to smithi00132172-10 2024-09-15T03:46:39.409 INFO:tasks.rados.rados.0.smithi001.stdout:10: finishing write tid 2 to smithi00132172-10 2024-09-15T03:46:39.409 INFO:tasks.rados.rados.0.smithi001.stdout:22: finishing write tid 3 to smithi00132172-22 2024-09-15T03:46:39.409 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 22 v 9 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-15T03:46:39.410 INFO:tasks.rados.rados.0.smithi001.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-09-15T03:46:39.410 INFO:tasks.rados.rados.0.smithi001.stdout:23: finishing write tid 1 to smithi00132172-23 2024-09-15T03:46:39.410 INFO:tasks.rados.rados.0.smithi001.stdout:12: finishing write tid 3 to smithi00132172-12 2024-09-15T03:46:39.410 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 12 v 4 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-15T03:46:39.410 INFO:tasks.rados.rados.0.smithi001.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-09-15T03:46:39.410 INFO:tasks.rados.rados.0.smithi001.stdout:23: finishing write tid 2 to smithi00132172-23 2024-09-15T03:46:39.410 INFO:tasks.rados.rados.0.smithi001.stdout:17: finishing write tid 3 to smithi00132172-17 2024-09-15T03:46:39.410 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 17 v 7 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-15T03:46:39.410 INFO:tasks.rados.rados.0.smithi001.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-09-15T03:46:39.410 INFO:tasks.rados.rados.0.smithi001.stdout:2: finishing write tid 3 to smithi00132172-2 2024-09-15T03:46:39.410 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 2 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-15T03:46:39.410 INFO:tasks.rados.rados.0.smithi001.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-09-15T03:46:39.410 INFO:tasks.rados.rados.0.smithi001.stdout:23: finishing write tid 3 to smithi00132172-23 2024-09-15T03:46:39.410 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-15T03:46:39.410 INFO:tasks.rados.rados.0.smithi001.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-09-15T03:46:39.411 INFO:tasks.rados.rados.0.smithi001.stdout:7: finishing write tid 3 to smithi00132172-7 2024-09-15T03:46:39.411 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 7 v 4 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-15T03:46:39.411 INFO:tasks.rados.rados.0.smithi001.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-09-15T03:46:39.411 INFO:tasks.rados.rados.0.smithi001.stdout:9: finishing write tid 3 to smithi00132172-9 2024-09-15T03:46:39.411 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 9 v 6 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-15T03:46:39.411 INFO:tasks.rados.rados.0.smithi001.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-09-15T03:46:39.411 INFO:tasks.rados.rados.0.smithi001.stdout:10: finishing write tid 3 to smithi00132172-10 2024-09-15T03:46:39.411 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 10 v 8 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-15T03:46:39.411 INFO:tasks.rados.rados.0.smithi001.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-09-15T03:46:39.411 INFO:tasks.rados.rados.0.smithi001.stdout:24: finishing write tid 1 to smithi00132172-24 2024-09-15T03:46:39.411 INFO:tasks.rados.rados.0.smithi001.stdout:24: finishing write tid 2 to smithi00132172-24 2024-09-15T03:46:39.411 INFO:tasks.rados.rados.0.smithi001.stdout:26: seq_num 25 ranges {0=425984} 2024-09-15T03:46:39.412 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:46:39.412 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:46:39.413 INFO:tasks.rados.rados.0.smithi001.stdout:26: writing smithi00132172-26 from 0 to 425984 tid 1 2024-09-15T03:46:39.413 INFO:tasks.rados.rados.0.smithi001.stdout:2: done (13 left) 2024-09-15T03:46:39.413 INFO:tasks.rados.rados.0.smithi001.stdout:6: done (12 left) 2024-09-15T03:46:39.413 INFO:tasks.rados.rados.0.smithi001.stdout:7: done (11 left) 2024-09-15T03:46:39.413 INFO:tasks.rados.rados.0.smithi001.stdout:9: done (10 left) 2024-09-15T03:46:39.413 INFO:tasks.rados.rados.0.smithi001.stdout:10: done (9 left) 2024-09-15T03:46:39.413 INFO:tasks.rados.rados.0.smithi001.stdout:12: done (8 left) 2024-09-15T03:46:39.414 INFO:tasks.rados.rados.0.smithi001.stdout:16: done (7 left) 2024-09-15T03:46:39.414 INFO:tasks.rados.rados.0.smithi001.stdout:17: done (6 left) 2024-09-15T03:46:39.414 INFO:tasks.rados.rados.0.smithi001.stdout:18: done (5 left) 2024-09-15T03:46:39.414 INFO:tasks.rados.rados.0.smithi001.stdout:22: done (4 left) 2024-09-15T03:46:39.414 INFO:tasks.rados.rados.0.smithi001.stdout:23: done (3 left) 2024-09-15T03:46:39.414 INFO:tasks.rados.rados.0.smithi001.stdout:27: write initial oid 27 2024-09-15T03:46:39.414 INFO:tasks.rados.rados.0.smithi001.stdout:27: seq_num 26 ranges {0=442368} 2024-09-15T03:46:39.414 INFO:tasks.rados.rados.0.smithi001.stdout:27: writing smithi00132172-27 from 0 to 442368 tid 1 2024-09-15T03:46:39.414 INFO:tasks.rados.rados.0.smithi001.stdout:28: write initial oid 28 2024-09-15T03:46:39.414 INFO:tasks.rados.rados.0.smithi001.stdout:28: seq_num 27 ranges {0=458752} 2024-09-15T03:46:39.416 INFO:tasks.rados.rados.0.smithi001.stdout:28: writing smithi00132172-28 from 0 to 458752 tid 1 2024-09-15T03:46:39.417 INFO:tasks.rados.rados.0.smithi001.stdout:29: write initial oid 29 2024-09-15T03:46:39.417 INFO:tasks.rados.rados.0.smithi001.stdout:29: seq_num 28 ranges {0=475136} 2024-09-15T03:46:39.420 INFO:tasks.rados.rados.0.smithi001.stdout:29: writing smithi00132172-29 from 0 to 475136 tid 1 2024-09-15T03:46:39.421 INFO:tasks.rados.rados.0.smithi001.stdout:30: write initial oid 30 2024-09-15T03:46:39.421 INFO:tasks.rados.rados.0.smithi001.stdout:30: seq_num 29 ranges {0=491520} 2024-09-15T03:46:39.423 INFO:tasks.rados.rados.0.smithi001.stdout:30: writing smithi00132172-30 from 0 to 491520 tid 1 2024-09-15T03:46:39.423 INFO:tasks.rados.rados.0.smithi001.stdout:31: write initial oid 31 2024-09-15T03:46:39.423 INFO:tasks.rados.rados.0.smithi001.stdout:31: seq_num 30 ranges {0=507904} 2024-09-15T03:46:39.427 INFO:tasks.rados.rados.0.smithi001.stdout:31: writing smithi00132172-31 from 0 to 507904 tid 1 2024-09-15T03:46:39.427 INFO:tasks.rados.rados.0.smithi001.stdout:32: write initial oid 32 2024-09-15T03:46:39.427 INFO:tasks.rados.rados.0.smithi001.stdout:32: seq_num 31 ranges {0=524288} 2024-09-15T03:46:39.430 INFO:tasks.rados.rados.0.smithi001.stdout:32: writing smithi00132172-32 from 0 to 524288 tid 1 2024-09-15T03:46:39.431 INFO:tasks.rados.rados.0.smithi001.stdout:33: write initial oid 33 2024-09-15T03:46:39.431 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:46:39.431 INFO:tasks.rados.rados.0.smithi001.stdout:33: seq_num 32 ranges {0=540672} 2024-09-15T03:46:39.431 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:46:39.434 INFO:tasks.rados.rados.0.smithi001.stdout:33: writing smithi00132172-33 from 0 to 540672 tid 1 2024-09-15T03:46:39.434 INFO:tasks.rados.rados.0.smithi001.stdout:34: write initial oid 34 2024-09-15T03:46:39.434 INFO:tasks.rados.rados.0.smithi001.stdout:34: seq_num 33 ranges {0=557056} 2024-09-15T03:46:39.438 INFO:tasks.rados.rados.0.smithi001.stdout:34: writing smithi00132172-34 from 0 to 557056 tid 1 2024-09-15T03:46:39.438 INFO:tasks.rados.rados.0.smithi001.stdout:35: write initial oid 35 2024-09-15T03:46:39.439 INFO:tasks.rados.rados.0.smithi001.stdout:24: finishing write tid 3 to smithi00132172-24 2024-09-15T03:46:39.439 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 24 v 8 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-15T03:46:39.439 INFO:tasks.rados.rados.0.smithi001.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-09-15T03:46:39.439 INFO:tasks.rados.rados.0.smithi001.stdout:25: finishing write tid 1 to smithi00132172-25 2024-09-15T03:46:39.439 INFO:tasks.rados.rados.0.smithi001.stdout:25: finishing write tid 2 to smithi00132172-25 2024-09-15T03:46:39.439 INFO:tasks.rados.rados.0.smithi001.stdout:25: finishing write tid 3 to smithi00132172-25 2024-09-15T03:46:39.439 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 25 v 10 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-15T03:46:39.439 INFO:tasks.rados.rados.0.smithi001.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-09-15T03:46:39.439 INFO:tasks.rados.rados.0.smithi001.stdout:26: finishing write tid 1 to smithi00132172-26 2024-09-15T03:46:39.439 INFO:tasks.rados.rados.0.smithi001.stdout:26: finishing write tid 2 to smithi00132172-26 2024-09-15T03:46:39.439 INFO:tasks.rados.rados.0.smithi001.stdout:26: finishing write tid 3 to smithi00132172-26 2024-09-15T03:46:39.439 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 26 v 4 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-15T03:46:39.440 INFO:tasks.rados.rados.0.smithi001.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-09-15T03:46:39.440 INFO:tasks.rados.rados.0.smithi001.stdout:27: finishing write tid 1 to smithi00132172-27 2024-09-15T03:46:39.440 INFO:tasks.rados.rados.0.smithi001.stdout:27: finishing write tid 2 to smithi00132172-27 2024-09-15T03:46:39.440 INFO:tasks.rados.rados.0.smithi001.stdout:27: finishing write tid 3 to smithi00132172-27 2024-09-15T03:46:39.440 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 27 v 2 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-15T03:46:39.440 INFO:tasks.rados.rados.0.smithi001.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-09-15T03:46:39.440 INFO:tasks.rados.rados.0.smithi001.stdout:28: finishing write tid 1 to smithi00132172-28 2024-09-15T03:46:39.440 INFO:tasks.rados.rados.0.smithi001.stdout:28: finishing write tid 2 to smithi00132172-28 2024-09-15T03:46:39.440 INFO:tasks.rados.rados.0.smithi001.stdout:28: finishing write tid 3 to smithi00132172-28 2024-09-15T03:46:39.440 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 28 v 10 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-15T03:46:39.440 INFO:tasks.rados.rados.0.smithi001.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-09-15T03:46:39.440 INFO:tasks.rados.rados.0.smithi001.stdout:29: finishing write tid 1 to smithi00132172-29 2024-09-15T03:46:39.441 INFO:tasks.rados.rados.0.smithi001.stdout:29: finishing write tid 2 to smithi00132172-29 2024-09-15T03:46:39.441 INFO:tasks.rados.rados.0.smithi001.stdout:29: finishing write tid 3 to smithi00132172-29 2024-09-15T03:46:39.441 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 29 v 11 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-15T03:46:39.441 INFO:tasks.rados.rados.0.smithi001.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-09-15T03:46:39.441 INFO:tasks.rados.rados.0.smithi001.stdout:30: finishing write tid 1 to smithi00132172-30 2024-09-15T03:46:39.441 INFO:tasks.rados.rados.0.smithi001.stdout:30: finishing write tid 2 to smithi00132172-30 2024-09-15T03:46:39.441 INFO:tasks.rados.rados.0.smithi001.stdout:30: finishing write tid 3 to smithi00132172-30 2024-09-15T03:46:39.441 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-15T03:46:39.441 INFO:tasks.rados.rados.0.smithi001.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-09-15T03:46:39.441 INFO:tasks.rados.rados.0.smithi001.stdout:31: finishing write tid 1 to smithi00132172-31 2024-09-15T03:46:39.441 INFO:tasks.rados.rados.0.smithi001.stdout:35: seq_num 34 ranges {0=573440} 2024-09-15T03:46:39.442 INFO:tasks.rados.rados.0.smithi001.stdout:35: writing smithi00132172-35 from 0 to 573440 tid 1 2024-09-15T03:46:39.442 INFO:tasks.rados.rados.0.smithi001.stdout:24: done (11 left) 2024-09-15T03:46:39.442 INFO:tasks.rados.rados.0.smithi001.stdout:25: done (10 left) 2024-09-15T03:46:39.442 INFO:tasks.rados.rados.0.smithi001.stdout:26: done (9 left) 2024-09-15T03:46:39.442 INFO:tasks.rados.rados.0.smithi001.stdout:27: done (8 left) 2024-09-15T03:46:39.443 INFO:tasks.rados.rados.0.smithi001.stdout:28: done (7 left) 2024-09-15T03:46:39.443 INFO:tasks.rados.rados.0.smithi001.stdout:29: done (6 left) 2024-09-15T03:46:39.443 INFO:tasks.rados.rados.0.smithi001.stdout:30: done (5 left) 2024-09-15T03:46:39.443 INFO:tasks.rados.rados.0.smithi001.stdout:36: write initial oid 36 2024-09-15T03:46:39.443 INFO:tasks.rados.rados.0.smithi001.stdout:36: seq_num 35 ranges {0=589824} 2024-09-15T03:46:39.446 INFO:tasks.rados.rados.0.smithi001.stdout:36: writing smithi00132172-36 from 0 to 589824 tid 1 2024-09-15T03:46:39.446 INFO:tasks.rados.rados.0.smithi001.stdout:37: write initial oid 37 2024-09-15T03:46:39.446 INFO:tasks.rados.rados.0.smithi001.stdout:37: seq_num 36 ranges {0=606208} 2024-09-15T03:46:39.451 INFO:tasks.rados.rados.0.smithi001.stdout:37: writing smithi00132172-37 from 0 to 606208 tid 1 2024-09-15T03:46:39.451 INFO:tasks.rados.rados.0.smithi001.stdout:38: write initial oid 38 2024-09-15T03:46:39.451 INFO:tasks.rados.rados.0.smithi001.stdout:38: seq_num 37 ranges {0=622592} 2024-09-15T03:46:39.452 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:46:39.452 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:46:39.455 INFO:tasks.rados.rados.0.smithi001.stdout:38: writing smithi00132172-38 from 0 to 622592 tid 1 2024-09-15T03:46:39.455 INFO:tasks.rados.rados.0.smithi001.stdout:39: write initial oid 39 2024-09-15T03:46:39.455 INFO:tasks.rados.rados.0.smithi001.stdout:39: seq_num 38 ranges {0=638976} 2024-09-15T03:46:39.459 INFO:tasks.rados.rados.0.smithi001.stdout:39: writing smithi00132172-39 from 0 to 638976 tid 1 2024-09-15T03:46:39.459 INFO:tasks.rados.rados.0.smithi001.stdout:40: write initial oid 40 2024-09-15T03:46:39.459 INFO:tasks.rados.rados.0.smithi001.stdout:40: seq_num 39 ranges {0=671744} 2024-09-15T03:46:39.463 INFO:tasks.rados.rados.0.smithi001.stdout:40: writing smithi00132172-40 from 0 to 671744 tid 1 2024-09-15T03:46:39.463 INFO:tasks.rados.rados.0.smithi001.stdout:41: write initial oid 41 2024-09-15T03:46:39.464 INFO:tasks.rados.rados.0.smithi001.stdout:41: seq_num 40 ranges {0=688128} 2024-09-15T03:46:39.468 INFO:tasks.rados.rados.0.smithi001.stdout:41: writing smithi00132172-41 from 0 to 688128 tid 1 2024-09-15T03:46:39.468 INFO:tasks.rados.rados.0.smithi001.stdout:42: write initial oid 42 2024-09-15T03:46:39.468 INFO:tasks.rados.rados.0.smithi001.stdout:42: seq_num 41 ranges {0=704512} 2024-09-15T03:46:39.471 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:46:39.471 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:46:39.472 INFO:tasks.rados.rados.0.smithi001.stdout:42: writing smithi00132172-42 from 0 to 704512 tid 1 2024-09-15T03:46:39.472 INFO:tasks.rados.rados.0.smithi001.stdout:43: write initial oid 43 2024-09-15T03:46:39.473 INFO:tasks.rados.rados.0.smithi001.stdout:43: seq_num 42 ranges {0=720896} 2024-09-15T03:46:39.478 INFO:tasks.rados.rados.0.smithi001.stdout:43: writing smithi00132172-43 from 0 to 720896 tid 1 2024-09-15T03:46:39.478 INFO:tasks.rados.rados.0.smithi001.stdout:44: write initial oid 44 2024-09-15T03:46:39.478 INFO:tasks.rados.rados.0.smithi001.stdout:44: seq_num 43 ranges {0=737280} 2024-09-15T03:46:39.482 INFO:tasks.rados.rados.0.smithi001.stdout:44: writing smithi00132172-44 from 0 to 737280 tid 1 2024-09-15T03:46:39.482 INFO:tasks.rados.rados.0.smithi001.stdout:45: write initial oid 45 2024-09-15T03:46:39.482 INFO:tasks.rados.rados.0.smithi001.stdout:45: seq_num 44 ranges {0=753664} 2024-09-15T03:46:39.487 INFO:tasks.rados.rados.0.smithi001.stdout:45: writing smithi00132172-45 from 0 to 753664 tid 1 2024-09-15T03:46:39.487 INFO:tasks.rados.rados.0.smithi001.stdout:46: write initial oid 46 2024-09-15T03:46:39.488 INFO:tasks.rados.rados.0.smithi001.stdout:46: seq_num 45 ranges {0=770048} 2024-09-15T03:46:39.490 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:46:39.490 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:46:39.495 INFO:tasks.rados.rados.0.smithi001.stdout:46: writing smithi00132172-46 from 0 to 770048 tid 1 2024-09-15T03:46:39.495 INFO:tasks.rados.rados.0.smithi001.stdout: waiting on 16 2024-09-15T03:46:39.495 INFO:tasks.rados.rados.0.smithi001.stdout:31: finishing write tid 2 to smithi00132172-31 2024-09-15T03:46:39.495 INFO:tasks.rados.rados.0.smithi001.stdout:31: finishing write tid 3 to smithi00132172-31 2024-09-15T03:46:39.495 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 31 v 12 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-15T03:46:39.495 INFO:tasks.rados.rados.0.smithi001.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-09-15T03:46:39.495 INFO:tasks.rados.rados.0.smithi001.stdout:32: finishing write tid 1 to smithi00132172-32 2024-09-15T03:46:39.495 INFO:tasks.rados.rados.0.smithi001.stdout:32: finishing write tid 2 to smithi00132172-32 2024-09-15T03:46:39.495 INFO:tasks.rados.rados.0.smithi001.stdout:32: finishing write tid 3 to smithi00132172-32 2024-09-15T03:46:39.495 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 32 v 2 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-15T03:46:39.495 INFO:tasks.rados.rados.0.smithi001.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-09-15T03:46:39.495 INFO:tasks.rados.rados.0.smithi001.stdout:33: finishing write tid 1 to smithi00132172-33 2024-09-15T03:46:39.496 INFO:tasks.rados.rados.0.smithi001.stdout:33: finishing write tid 2 to smithi00132172-33 2024-09-15T03:46:39.496 INFO:tasks.rados.rados.0.smithi001.stdout:33: finishing write tid 3 to smithi00132172-33 2024-09-15T03:46:39.496 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 33 v 2 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-15T03:46:39.496 INFO:tasks.rados.rados.0.smithi001.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-09-15T03:46:39.496 INFO:tasks.rados.rados.0.smithi001.stdout:34: finishing write tid 1 to smithi00132172-34 2024-09-15T03:46:39.496 INFO:tasks.rados.rados.0.smithi001.stdout:34: finishing write tid 2 to smithi00132172-34 2024-09-15T03:46:39.496 INFO:tasks.rados.rados.0.smithi001.stdout:34: finishing write tid 3 to smithi00132172-34 2024-09-15T03:46:39.496 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 34 v 4 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-15T03:46:39.496 INFO:tasks.rados.rados.0.smithi001.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-09-15T03:46:39.496 INFO:tasks.rados.rados.0.smithi001.stdout:35: finishing write tid 1 to smithi00132172-35 2024-09-15T03:46:39.496 INFO:tasks.rados.rados.0.smithi001.stdout:35: finishing write tid 2 to smithi00132172-35 2024-09-15T03:46:39.496 INFO:tasks.rados.rados.0.smithi001.stdout:35: finishing write tid 3 to smithi00132172-35 2024-09-15T03:46:39.497 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 35 v 2 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-15T03:46:39.497 INFO:tasks.rados.rados.0.smithi001.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-09-15T03:46:39.497 INFO:tasks.rados.rados.0.smithi001.stdout:36: finishing write tid 1 to smithi00132172-36 2024-09-15T03:46:39.497 INFO:tasks.rados.rados.0.smithi001.stdout:36: finishing write tid 2 to smithi00132172-36 2024-09-15T03:46:39.497 INFO:tasks.rados.rados.0.smithi001.stdout:36: finishing write tid 3 to smithi00132172-36 2024-09-15T03:46:39.497 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 36 v 4 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-15T03:46:39.497 INFO:tasks.rados.rados.0.smithi001.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-09-15T03:46:39.497 INFO:tasks.rados.rados.0.smithi001.stdout:37: finishing write tid 1 to smithi00132172-37 2024-09-15T03:46:39.497 INFO:tasks.rados.rados.0.smithi001.stdout:37: finishing write tid 2 to smithi00132172-37 2024-09-15T03:46:39.497 INFO:tasks.rados.rados.0.smithi001.stdout:37: finishing write tid 3 to smithi00132172-37 2024-09-15T03:46:39.497 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 37 v 4 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-15T03:46:39.497 INFO:tasks.rados.rados.0.smithi001.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-09-15T03:46:39.497 INFO:tasks.rados.rados.0.smithi001.stdout:38: finishing write tid 1 to smithi00132172-38 2024-09-15T03:46:39.498 INFO:tasks.rados.rados.0.smithi001.stdout:38: finishing write tid 2 to smithi00132172-38 2024-09-15T03:46:39.498 INFO:tasks.rados.rados.0.smithi001.stdout:38: finishing write tid 3 to smithi00132172-38 2024-09-15T03:46:39.498 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 38 v 4 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-15T03:46:39.498 INFO:tasks.rados.rados.0.smithi001.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-09-15T03:46:39.498 INFO:tasks.rados.rados.0.smithi001.stdout:39: finishing write tid 1 to smithi00132172-39 2024-09-15T03:46:39.498 INFO:tasks.rados.rados.0.smithi001.stdout:39: finishing write tid 2 to smithi00132172-39 2024-09-15T03:46:39.498 INFO:tasks.rados.rados.0.smithi001.stdout:31: done (15 left) 2024-09-15T03:46:39.498 INFO:tasks.rados.rados.0.smithi001.stdout:32: done (14 left) 2024-09-15T03:46:39.498 INFO:tasks.rados.rados.0.smithi001.stdout:33: done (13 left) 2024-09-15T03:46:39.498 INFO:tasks.rados.rados.0.smithi001.stdout:34: done (12 left) 2024-09-15T03:46:39.498 INFO:tasks.rados.rados.0.smithi001.stdout:35: done (11 left) 2024-09-15T03:46:39.498 INFO:tasks.rados.rados.0.smithi001.stdout:36: done (10 left) 2024-09-15T03:46:39.499 INFO:tasks.rados.rados.0.smithi001.stdout:37: done (9 left) 2024-09-15T03:46:39.499 INFO:tasks.rados.rados.0.smithi001.stdout:38: done (8 left) 2024-09-15T03:46:39.499 INFO:tasks.rados.rados.0.smithi001.stdout:47: write initial oid 47 2024-09-15T03:46:39.499 INFO:tasks.rados.rados.0.smithi001.stdout:47: seq_num 46 ranges {0=786432} 2024-09-15T03:46:39.499 INFO:tasks.rados.rados.0.smithi001.stdout:47: writing smithi00132172-47 from 0 to 786432 tid 1 2024-09-15T03:46:39.500 INFO:tasks.rados.rados.0.smithi001.stdout:48: write initial oid 48 2024-09-15T03:46:39.500 INFO:tasks.rados.rados.0.smithi001.stdout:48: seq_num 47 ranges {0=425984,425984=376832} 2024-09-15T03:46:39.500 INFO:tasks.rados.rados.0.smithi001.stdout:48: writing smithi00132172-48 from 0 to 425984 tid 1 2024-09-15T03:46:39.503 INFO:tasks.rados.rados.0.smithi001.stdout:48: writing smithi00132172-48 from 425984 to 802816 tid 2 2024-09-15T03:46:39.503 INFO:tasks.rados.rados.0.smithi001.stdout:49: write initial oid 49 2024-09-15T03:46:39.504 INFO:tasks.rados.rados.0.smithi001.stdout:49: seq_num 48 ranges {0=442368,442368=376832} 2024-09-15T03:46:39.505 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:39.504+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:46:39.507 INFO:tasks.rados.rados.0.smithi001.stdout:49: writing smithi00132172-49 from 0 to 442368 tid 1 2024-09-15T03:46:39.509 INFO:tasks.rados.rados.0.smithi001.stdout:49: writing smithi00132172-49 from 442368 to 819200 tid 2 2024-09-15T03:46:39.509 INFO:tasks.rados.rados.0.smithi001.stdout:50: write initial oid 50 2024-09-15T03:46:39.509 INFO:tasks.rados.rados.0.smithi001.stdout:50: seq_num 49 ranges {0=458752,458752=376832} 2024-09-15T03:46:39.509 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:46:39.510 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:46:39.511 INFO:tasks.rados.rados.0.smithi001.stdout:50: writing smithi00132172-50 from 0 to 458752 tid 1 2024-09-15T03:46:39.514 INFO:tasks.rados.rados.0.smithi001.stdout:50: writing smithi00132172-50 from 458752 to 835584 tid 2 2024-09-15T03:46:39.514 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 6 current snap is 0 2024-09-15T03:46:39.514 INFO:tasks.rados.rados.0.smithi001.stdout:39: finishing write tid 3 to smithi00132172-39 2024-09-15T03:46:39.515 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 39 v 14 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-15T03:46:39.515 INFO:tasks.rados.rados.0.smithi001.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-09-15T03:46:39.515 INFO:tasks.rados.rados.0.smithi001.stdout:51: seq_num 50 ranges {98304=475136,573440=376832} 2024-09-15T03:46:39.518 INFO:tasks.rados.rados.0.smithi001.stdout:51: writing smithi00132172-6 from 98304 to 573440 tid 1 2024-09-15T03:46:39.521 INFO:tasks.rados.rados.0.smithi001.stdout:51: writing smithi00132172-6 from 573440 to 950272 tid 2 2024-09-15T03:46:39.521 INFO:tasks.rados.rados.0.smithi001.stdout:39: done (12 left) 2024-09-15T03:46:39.521 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 25 current snap is 0 2024-09-15T03:46:39.521 INFO:tasks.rados.rados.0.smithi001.stdout:40: finishing write tid 1 to smithi00132172-40 2024-09-15T03:46:39.521 INFO:tasks.rados.rados.0.smithi001.stdout:52: seq_num 51 ranges {409600=491520,901120=376832} 2024-09-15T03:46:39.526 INFO:tasks.rados.rados.0.smithi001.stdout:52: writing smithi00132172-25 from 409600 to 901120 tid 1 2024-09-15T03:46:39.530 INFO:tasks.rados.rados.0.smithi001.stdout:52: writing smithi00132172-25 from 901120 to 1277952 tid 2 2024-09-15T03:46:39.530 INFO:tasks.rados.rados.0.smithi001.stdout:53: read oid 19 snap -1 2024-09-15T03:46:39.530 INFO:tasks.rados.rados.0.smithi001.stdout:53: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-15T03:46:39.530 INFO:tasks.rados.rados.0.smithi001.stdout:54: setattr oid 29 current snap is 0 2024-09-15T03:46:39.531 INFO:tasks.rados.rados.0.smithi001.stdout:40: finishing write tid 2 to smithi00132172-40 2024-09-15T03:46:39.531 INFO:tasks.rados.rados.0.smithi001.stdout:40: finishing write tid 3 to smithi00132172-40 2024-09-15T03:46:39.531 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 40 v 13 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-15T03:46:39.531 INFO:tasks.rados.rados.0.smithi001.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-09-15T03:46:39.531 INFO:tasks.rados.rados.0.smithi001.stdout:41: finishing write tid 1 to smithi00132172-41 2024-09-15T03:46:39.531 INFO:tasks.rados.rados.0.smithi001.stdout:41: finishing write tid 2 to smithi00132172-41 2024-09-15T03:46:39.531 INFO:tasks.rados.rados.0.smithi001.stdout:41: finishing write tid 3 to smithi00132172-41 2024-09-15T03:46:39.531 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 41 v 4 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-15T03:46:39.531 INFO:tasks.rados.rados.0.smithi001.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-09-15T03:46:39.531 INFO:tasks.rados.rados.0.smithi001.stdout:42: finishing write tid 1 to smithi00132172-42 2024-09-15T03:46:39.531 INFO:tasks.rados.rados.0.smithi001.stdout:42: finishing write tid 2 to smithi00132172-42 2024-09-15T03:46:39.531 INFO:tasks.rados.rados.0.smithi001.stdout:42: finishing write tid 3 to smithi00132172-42 2024-09-15T03:46:39.531 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 42 v 6 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-15T03:46:39.532 INFO:tasks.rados.rados.0.smithi001.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-09-15T03:46:39.532 INFO:tasks.rados.rados.0.smithi001.stdout:43: finishing write tid 1 to smithi00132172-43 2024-09-15T03:46:39.532 INFO:tasks.rados.rados.0.smithi001.stdout:43: finishing write tid 2 to smithi00132172-43 2024-09-15T03:46:39.532 INFO:tasks.rados.rados.0.smithi001.stdout:43: finishing write tid 3 to smithi00132172-43 2024-09-15T03:46:39.532 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 43 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-15T03:46:39.532 INFO:tasks.rados.rados.0.smithi001.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-09-15T03:46:39.532 INFO:tasks.rados.rados.0.smithi001.stdout:44: finishing write tid 1 to smithi00132172-44 2024-09-15T03:46:39.532 INFO:tasks.rados.rados.0.smithi001.stdout:44: finishing write tid 2 to smithi00132172-44 2024-09-15T03:46:39.532 INFO:tasks.rados.rados.0.smithi001.stdout:44: finishing write tid 3 to smithi00132172-44 2024-09-15T03:46:39.532 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 44 v 15 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-15T03:46:39.532 INFO:tasks.rados.rados.0.smithi001.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-09-15T03:46:39.532 INFO:tasks.rados.rados.0.smithi001.stdout:45: finishing write tid 1 to smithi00132172-45 2024-09-15T03:46:39.532 INFO:tasks.rados.rados.0.smithi001.stdout:45: finishing write tid 2 to smithi00132172-45 2024-09-15T03:46:39.532 INFO:tasks.rados.rados.0.smithi001.stdout:45: finishing write tid 3 to smithi00132172-45 2024-09-15T03:46:39.533 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 45 v 2 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-15T03:46:39.533 INFO:tasks.rados.rados.0.smithi001.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-09-15T03:46:39.533 INFO:tasks.rados.rados.0.smithi001.stdout:46: finishing write tid 1 to smithi00132172-46 2024-09-15T03:46:39.533 INFO:tasks.rados.rados.0.smithi001.stdout:46: finishing write tid 2 to smithi00132172-46 2024-09-15T03:46:39.533 INFO:tasks.rados.rados.0.smithi001.stdout:46: finishing write tid 3 to smithi00132172-46 2024-09-15T03:46:39.533 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 46 v 12 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-15T03:46:39.533 INFO:tasks.rados.rados.0.smithi001.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-09-15T03:46:39.533 INFO:tasks.rados.rados.0.smithi001.stdout:47: finishing write tid 1 to smithi00132172-47 2024-09-15T03:46:39.533 INFO:tasks.rados.rados.0.smithi001.stdout:47: finishing write tid 2 to smithi00132172-47 2024-09-15T03:46:39.533 INFO:tasks.rados.rados.0.smithi001.stdout:47: finishing write tid 3 to smithi00132172-47 2024-09-15T03:46:39.533 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 47 v 6 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-15T03:46:39.533 INFO:tasks.rados.rados.0.smithi001.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-09-15T03:46:39.533 INFO:tasks.rados.rados.0.smithi001.stdout:48: finishing write tid 1 to smithi00132172-48 2024-09-15T03:46:39.533 INFO:tasks.rados.rados.0.smithi001.stdout:48: finishing write tid 2 to smithi00132172-48 2024-09-15T03:46:39.534 INFO:tasks.rados.rados.0.smithi001.stdout:48: finishing write tid 3 to smithi00132172-48 2024-09-15T03:46:39.534 INFO:tasks.rados.rados.0.smithi001.stdout:48: finishing write tid 4 to smithi00132172-48 2024-09-15T03:46:39.534 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 48 v 9 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-15T03:46:39.534 INFO:tasks.rados.rados.0.smithi001.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-09-15T03:46:39.534 INFO:tasks.rados.rados.0.smithi001.stdout:49: finishing write tid 1 to smithi00132172-49 2024-09-15T03:46:39.534 INFO:tasks.rados.rados.0.smithi001.stdout:49: finishing write tid 2 to smithi00132172-49 2024-09-15T03:46:39.534 INFO:tasks.rados.rados.0.smithi001.stdout:49: finishing write tid 3 to smithi00132172-49 2024-09-15T03:46:39.534 INFO:tasks.rados.rados.0.smithi001.stdout:49: finishing write tid 4 to smithi00132172-49 2024-09-15T03:46:39.534 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 49 v 5 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-15T03:46:39.534 INFO:tasks.rados.rados.0.smithi001.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-09-15T03:46:39.534 INFO:tasks.rados.rados.0.smithi001.stdout:50: finishing write tid 1 to smithi00132172-50 2024-09-15T03:46:39.534 INFO:tasks.rados.rados.0.smithi001.stdout:50: finishing write tid 2 to smithi00132172-50 2024-09-15T03:46:39.534 INFO:tasks.rados.rados.0.smithi001.stdout:50: finishing write tid 3 to smithi00132172-50 2024-09-15T03:46:39.534 INFO:tasks.rados.rados.0.smithi001.stdout:50: finishing write tid 4 to smithi00132172-50 2024-09-15T03:46:39.534 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 50 v 12 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-15T03:46:39.535 INFO:tasks.rados.rados.0.smithi001.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-09-15T03:46:39.535 INFO:tasks.rados.rados.0.smithi001.stdout:51: finishing write tid 1 to smithi00132172-6 2024-09-15T03:46:39.535 INFO:tasks.rados.rados.0.smithi001.stdout:51: finishing write tid 2 to smithi00132172-6 2024-09-15T03:46:39.535 INFO:tasks.rados.rados.0.smithi001.stdout:51: finishing write tid 3 to smithi00132172-6 2024-09-15T03:46:39.535 INFO:tasks.rados.rados.0.smithi001.stdout:51: finishing write tid 4 to smithi00132172-6 2024-09-15T03:46:39.535 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 6 v 9 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-15T03:46:39.535 INFO:tasks.rados.rados.0.smithi001.stdout:51: left oid 6 (ObjNum 50 snap 0 seq_num 50) 2024-09-15T03:46:39.535 INFO:tasks.rados.rados.0.smithi001.stdout:52: finishing write tid 1 to smithi00132172-25 2024-09-15T03:46:39.535 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:46:39.536 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:46:39.536 INFO:tasks.rados.rados.0.smithi001.stdout:40: done (14 left) 2024-09-15T03:46:39.536 INFO:tasks.rados.rados.0.smithi001.stdout:41: done (13 left) 2024-09-15T03:46:39.537 INFO:tasks.rados.rados.0.smithi001.stdout:42: done (12 left) 2024-09-15T03:46:39.537 INFO:tasks.rados.rados.0.smithi001.stdout:43: done (11 left) 2024-09-15T03:46:39.537 INFO:tasks.rados.rados.0.smithi001.stdout:44: done (10 left) 2024-09-15T03:46:39.537 INFO:tasks.rados.rados.0.smithi001.stdout:45: done (9 left) 2024-09-15T03:46:39.537 INFO:tasks.rados.rados.0.smithi001.stdout:46: done (8 left) 2024-09-15T03:46:39.537 INFO:tasks.rados.rados.0.smithi001.stdout:47: done (7 left) 2024-09-15T03:46:39.537 INFO:tasks.rados.rados.0.smithi001.stdout:48: done (6 left) 2024-09-15T03:46:39.537 INFO:tasks.rados.rados.0.smithi001.stdout:49: done (5 left) 2024-09-15T03:46:39.537 INFO:tasks.rados.rados.0.smithi001.stdout:50: done (4 left) 2024-09-15T03:46:39.537 INFO:tasks.rados.rados.0.smithi001.stdout:51: done (3 left) 2024-09-15T03:46:39.537 INFO:tasks.rados.rados.0.smithi001.stdout:55: delete oid 41 current snap is 0 2024-09-15T03:46:39.537 INFO:tasks.rados.rados.0.smithi001.stdout:52: finishing write tid 2 to smithi00132172-25 2024-09-15T03:46:39.537 INFO:tasks.rados.rados.0.smithi001.stdout:52: finishing write tid 3 to smithi00132172-25 2024-09-15T03:46:39.537 INFO:tasks.rados.rados.0.smithi001.stdout:53: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-15T03:46:39.542 INFO:tasks.rados.rados.0.smithi001.stdout:52: finishing write tid 4 to smithi00132172-25 2024-09-15T03:46:39.542 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 25 v 15 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-15T03:46:39.542 INFO:tasks.rados.rados.0.smithi001.stdout:52: left oid 25 (ObjNum 51 snap 0 seq_num 51) 2024-09-15T03:46:39.543 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 29 v 16 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-15T03:46:39.543 INFO:tasks.rados.rados.0.smithi001.stdout:52: done (3 left) 2024-09-15T03:46:39.543 INFO:tasks.rados.rados.0.smithi001.stdout:53: done (2 left) 2024-09-15T03:46:39.543 INFO:tasks.rados.rados.0.smithi001.stdout:54: done (1 left) 2024-09-15T03:46:39.543 INFO:tasks.rados.rados.0.smithi001.stdout:55: done (0 left) 2024-09-15T03:46:39.543 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 6 current snap is 0 2024-09-15T03:46:39.543 INFO:tasks.rados.rados.0.smithi001.stdout:56: seq_num 53 ranges {950272=524288,1474560=376832} 2024-09-15T03:46:39.546 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:46:39.546 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:46:39.550 INFO:tasks.rados.rados.0.smithi001.stdout:56: writing smithi00132172-6 from 950272 to 1474560 tid 1 2024-09-15T03:46:39.551 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:46:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:46:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:46:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4263.0:5 2.2 e23/21 2:5cea7035:::rbd_info:head v 23'1, mlcod=0'0)", 2024-09-15T03:46:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:25.042195+0000", 2024-09-15T03:46:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.508763614999999, 2024-09-15T03:46:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00060975899999999997, 2024-09-15T03:46:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:39.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:39.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042195+0000", 2024-09-15T03:46:39.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:39.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042195+0000", 2024-09-15T03:46:39.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042195+0000", 2024-09-15T03:46:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.91e-07 2024-09-15T03:46:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042200+0000", 2024-09-15T03:46:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4e-06 2024-09-15T03:46:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:39.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042216+0000", 2024-09-15T03:46:39.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5204999999999999e-05 2024-09-15T03:46:39.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:39.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042224+0000", 2024-09-15T03:46:39.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7860000000000002e-06 2024-09-15T03:46:39.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042287+0000", 2024-09-15T03:46:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2348999999999995e-05 2024-09-15T03:46:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042326+0000", 2024-09-15T03:46:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9218000000000003e-05 2024-09-15T03:46:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042480+0000", 2024-09-15T03:46:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015405799999999999 2024-09-15T03:46:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042489+0000", 2024-09-15T03:46:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5879999999999998e-06 2024-09-15T03:46:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042547+0000", 2024-09-15T03:46:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8689e-05 2024-09-15T03:46:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042574+0000", 2024-09-15T03:46:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6478999999999998e-05 2024-09-15T03:46:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:25.042804+0000", 2024-09-15T03:46:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023079599999999999 2024-09-15T03:46:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4263.0:7 2.2 e24/21 2:5cea7035:::rbd_info:head v 24'2, mlcod=0'0)", 2024-09-15T03:46:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:26.044758+0000", 2024-09-15T03:46:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.506200624, 2024-09-15T03:46:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040876500000000001, 2024-09-15T03:46:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.044758+0000", 2024-09-15T03:46:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.044758+0000", 2024-09-15T03:46:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.044758+0000", 2024-09-15T03:46:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.04e-07 2024-09-15T03:46:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.044763+0000", 2024-09-15T03:46:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8269999999999997e-06 2024-09-15T03:46:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.044774+0000", 2024-09-15T03:46:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1207e-05 2024-09-15T03:46:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.044782+0000", 2024-09-15T03:46:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9219999999999997e-06 2024-09-15T03:46:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.044843+0000", 2024-09-15T03:46:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1327000000000004e-05 2024-09-15T03:46:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.044861+0000", 2024-09-15T03:46:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7368999999999999e-05 2024-09-15T03:46:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.044943+0000", 2024-09-15T03:46:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2583000000000005e-05 2024-09-15T03:46:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.044955+0000", 2024-09-15T03:46:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1708e-05 2024-09-15T03:46:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.045004+0000", 2024-09-15T03:46:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9218999999999998e-05 2024-09-15T03:46:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.045026+0000", 2024-09-15T03:46:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2289999999999998e-05 2024-09-15T03:46:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:26.045166+0000", 2024-09-15T03:46:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000140109 2024-09-15T03:46:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:39.563 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:39.565 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:46:39.566 INFO:tasks.rados.rados.0.smithi001.stdout:56: writing smithi00132172-6 from 1474560 to 1851392 tid 2 2024-09-15T03:46:39.566 INFO:tasks.rados.rados.0.smithi001.stdout:57: copy_from oid 45 from oid 22 current snap is 0 2024-09-15T03:46:39.566 INFO:tasks.rados.rados.0.smithi001.stdout:58: copy_from oid 5 from oid 42 current snap is 0 2024-09-15T03:46:39.566 INFO:tasks.rados.rados.0.smithi001.stdout:59: delete oid 34 current snap is 0 2024-09-15T03:46:39.566 INFO:tasks.rados.rados.0.smithi001.stdout:56: finishing write tid 1 to smithi00132172-6 2024-09-15T03:46:39.566 INFO:tasks.rados.rados.0.smithi001.stdout:59: done (3 left) 2024-09-15T03:46:39.566 INFO:tasks.rados.rados.0.smithi001.stdout:60: read oid 19 snap -1 2024-09-15T03:46:39.566 INFO:tasks.rados.rados.0.smithi001.stdout:60: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-15T03:46:39.566 INFO:tasks.rados.rados.0.smithi001.stdout:61: rmattr oid 15 current snap is 0 2024-09-15T03:46:39.566 INFO:tasks.rados.rados.0.smithi001.stdout:62: snap_create 2024-09-15T03:46:39.566 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-15T03:46:39.566 INFO:tasks.rados.rados.0.smithi001.stdout:60: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-15T03:46:39.567 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:46:39.567 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:46:39.568 INFO:tasks.rados.rados.0.smithi001.stdout:56: finishing write tid 2 to smithi00132172-6 2024-09-15T03:46:39.568 INFO:tasks.rados.rados.0.smithi001.stdout:56: finishing write tid 3 to smithi00132172-6 2024-09-15T03:46:39.568 INFO:tasks.rados.rados.0.smithi001.stdout:57: finishing copy_from racing read to smithi00132172-45 2024-09-15T03:46:39.568 INFO:tasks.rados.rados.0.smithi001.stdout:57: finishing copy_from to smithi00132172-45 2024-09-15T03:46:39.568 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 45 v 9 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-15T03:46:39.569 INFO:tasks.rados.rados.0.smithi001.stdout:56: finishing write tid 4 to smithi00132172-6 2024-09-15T03:46:39.569 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 6 v 12 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-09-15T03:46:39.569 INFO:tasks.rados.rados.0.smithi001.stdout:56: left oid 6 (ObjNum 53 snap 0 seq_num 53) 2024-09-15T03:46:39.569 INFO:tasks.rados.rados.0.smithi001.stdout:58: finishing copy_from racing read to smithi00132172-5 2024-09-15T03:46:39.569 INFO:tasks.rados.rados.0.smithi001.stdout:58: finishing copy_from to smithi00132172-5 2024-09-15T03:46:39.569 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 5 v 6 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-15T03:46:39.619 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:39.618+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:39.668 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:39.668 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:46:39.668 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:46:39.668 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:39.676 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:46:39.707 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:39.705+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:39.776 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:39.777 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:46:39.777 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:46:39.777 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:46:39.777 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:39.785 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:46:39.808 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:39.807+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:46:39.888 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:46:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:46:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:46:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.3s3 32/31 ECSubWrite(tid=25, reqid=client.4521.0:136, at_version=32'11, trim_to=0'0, roll_forward_to=32'10))", 2024-09-15T03:46:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.494657+0000", 2024-09-15T03:46:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.39109011799999999, 2024-09-15T03:46:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00088601200000000004, 2024-09-15T03:46:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.494657+0000", 2024-09-15T03:46:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.494657+0000", 2024-09-15T03:46:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.494658+0000", 2024-09-15T03:46:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6799999999999999e-07 2024-09-15T03:46:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.494808+0000", 2024-09-15T03:46:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015017599999999999 2024-09-15T03:46:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.494825+0000", 2024-09-15T03:46:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6611000000000001e-05 2024-09-15T03:46:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.494832+0000", 2024-09-15T03:46:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2590000000000003e-06 2024-09-15T03:46:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.494876+0000", 2024-09-15T03:46:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4563e-05 2024-09-15T03:46:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.494885+0000", 2024-09-15T03:46:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8189999999999998e-06 2024-09-15T03:46:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.494985+0000", 2024-09-15T03:46:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9378999999999998e-05 2024-09-15T03:46:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:39.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.494994+0000", 2024-09-15T03:46:39.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2280000000000004e-06 2024-09-15T03:46:39.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:39.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.495116+0000", 2024-09-15T03:46:39.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122059 2024-09-15T03:46:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.495137+0000", 2024-09-15T03:46:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1477000000000001e-05 2024-09-15T03:46:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.495543+0000", 2024-09-15T03:46:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000405673 2024-09-15T03:46:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs3 32/31 ECSubWrite(tid=17, reqid=client.4521.0:139, at_version=32'5, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.500582+0000", 2024-09-15T03:46:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.38516547200000001, 2024-09-15T03:46:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015438559999999999, 2024-09-15T03:46:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:39.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:39.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.500582+0000", 2024-09-15T03:46:39.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:39.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.500582+0000", 2024-09-15T03:46:39.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.500582+0000", 2024-09-15T03:46:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1999999999999999e-07 2024-09-15T03:46:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.501373+0000", 2024-09-15T03:46:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079045500000000002 2024-09-15T03:46:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.501386+0000", 2024-09-15T03:46:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2721000000000001e-05 2024-09-15T03:46:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.501390+0000", 2024-09-15T03:46:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9300000000000002e-06 2024-09-15T03:46:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.501432+0000", 2024-09-15T03:46:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1390000000000002e-05 2024-09-15T03:46:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.501440+0000", 2024-09-15T03:46:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0940000000000008e-06 2024-09-15T03:46:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.501557+0000", 2024-09-15T03:46:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011739999999999999 2024-09-15T03:46:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.501567+0000", 2024-09-15T03:46:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2869999999999993e-06 2024-09-15T03:46:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.501690+0000", 2024-09-15T03:46:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012290000000000001 2024-09-15T03:46:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.501772+0000", 2024-09-15T03:46:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2758999999999998e-05 2024-09-15T03:46:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.502126+0000", 2024-09-15T03:46:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035330000000000002 2024-09-15T03:46:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs3 32/31 ECSubWrite(tid=18, reqid=client.4521.0:140, at_version=32'6, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.501399+0000", 2024-09-15T03:46:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.384348052, 2024-09-15T03:46:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082877500000000002, 2024-09-15T03:46:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.501399+0000", 2024-09-15T03:46:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.501399+0000", 2024-09-15T03:46:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.501400+0000", 2024-09-15T03:46:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6300000000000001e-07 2024-09-15T03:46:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.501411+0000", 2024-09-15T03:46:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.186e-05 2024-09-15T03:46:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.501419+0000", 2024-09-15T03:46:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2779999999999996e-06 2024-09-15T03:46:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.501423+0000", 2024-09-15T03:46:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6140000000000002e-06 2024-09-15T03:46:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.501607+0000", 2024-09-15T03:46:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018346700000000001 2024-09-15T03:46:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.501615+0000", 2024-09-15T03:46:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0439999999999996e-06 2024-09-15T03:46:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.501697+0000", 2024-09-15T03:46:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1989000000000004e-05 2024-09-15T03:46:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.501703+0000", 2024-09-15T03:46:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8250000000000001e-06 2024-09-15T03:46:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.501729+0000", 2024-09-15T03:46:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6407e-05 2024-09-15T03:46:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.501805+0000", 2024-09-15T03:46:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6254999999999994e-05 2024-09-15T03:46:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.502228+0000", 2024-09-15T03:46:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042267299999999998 2024-09-15T03:46:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.7s3 32/31 ECSubWrite(tid=31, reqid=client.4521.0:142, at_version=32'7, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.502818+0000", 2024-09-15T03:46:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.38292917100000001, 2024-09-15T03:46:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00060139799999999995, 2024-09-15T03:46:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.502818+0000", 2024-09-15T03:46:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.502818+0000", 2024-09-15T03:46:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.502819+0000", 2024-09-15T03:46:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0999999999999997e-07 2024-09-15T03:46:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.502881+0000", 2024-09-15T03:46:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2408000000000003e-05 2024-09-15T03:46:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.502894+0000", 2024-09-15T03:46:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2533e-05 2024-09-15T03:46:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.502900+0000", 2024-09-15T03:46:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2899999999999999e-06 2024-09-15T03:46:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.502944+0000", 2024-09-15T03:46:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3423000000000001e-05 2024-09-15T03:46:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.502952+0000", 2024-09-15T03:46:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3240000000000003e-06 2024-09-15T03:46:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.503060+0000", 2024-09-15T03:46:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010828900000000001 2024-09-15T03:46:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.503069+0000", 2024-09-15T03:46:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.003e-06 2024-09-15T03:46:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.503144+0000", 2024-09-15T03:46:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5228000000000005e-05 2024-09-15T03:46:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.503158+0000", 2024-09-15T03:46:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3886000000000001e-05 2024-09-15T03:46:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.503419+0000", 2024-09-15T03:46:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026120400000000002 2024-09-15T03:46:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.ds5 32/31 ECSubWrite(tid=99, reqid=client.4521.0:146, at_version=32'3, trim_to=0'0, roll_forward_to=32'2))", 2024-09-15T03:46:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.507819+0000", 2024-09-15T03:46:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.37792839900000003, 2024-09-15T03:46:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079862100000000001, 2024-09-15T03:46:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.507819+0000", 2024-09-15T03:46:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.507819+0000", 2024-09-15T03:46:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.507820+0000", 2024-09-15T03:46:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0999999999999998e-07 2024-09-15T03:46:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.508145+0000", 2024-09-15T03:46:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032496100000000002 2024-09-15T03:46:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.508154+0000", 2024-09-15T03:46:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9899999999999992e-06 2024-09-15T03:46:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.508159+0000", 2024-09-15T03:46:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9189999999999998e-06 2024-09-15T03:46:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.508206+0000", 2024-09-15T03:46:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6925999999999998e-05 2024-09-15T03:46:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.508215+0000", 2024-09-15T03:46:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7010000000000002e-06 2024-09-15T03:46:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.508324+0000", 2024-09-15T03:46:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108776 2024-09-15T03:46:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.508331+0000", 2024-09-15T03:46:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1230000000000002e-06 2024-09-15T03:46:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.508408+0000", 2024-09-15T03:46:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6705000000000005e-05 2024-09-15T03:46:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.508421+0000", 2024-09-15T03:46:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3813e-05 2024-09-15T03:46:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.508617+0000", 2024-09-15T03:46:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000195997 2024-09-15T03:46:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.ds5 32/31 ECSubWrite(tid=101, reqid=client.4521.0:147, at_version=32'4, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:46:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.510697+0000", 2024-09-15T03:46:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.37505055500000001, 2024-09-15T03:46:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00088994400000000004, 2024-09-15T03:46:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.510697+0000", 2024-09-15T03:46:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.510697+0000", 2024-09-15T03:46:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.510697+0000", 2024-09-15T03:46:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1699999999999999e-07 2024-09-15T03:46:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.511132+0000", 2024-09-15T03:46:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043454700000000003 2024-09-15T03:46:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.511141+0000", 2024-09-15T03:46:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7629999999999995e-06 2024-09-15T03:46:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.511147+0000", 2024-09-15T03:46:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5339999999999999e-06 2024-09-15T03:46:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.511190+0000", 2024-09-15T03:46:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2753999999999998e-05 2024-09-15T03:46:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.511199+0000", 2024-09-15T03:46:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7450000000000007e-06 2024-09-15T03:46:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.511300+0000", 2024-09-15T03:46:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010037 2024-09-15T03:46:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.511307+0000", 2024-09-15T03:46:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3119999999999996e-06 2024-09-15T03:46:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.511372+0000", 2024-09-15T03:46:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4739000000000005e-05 2024-09-15T03:46:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.511384+0000", 2024-09-15T03:46:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2065999999999999e-05 2024-09-15T03:46:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.511587+0000", 2024-09-15T03:46:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000202697 2024-09-15T03:46:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.7s3 32/31 ECSubWrite(tid=36, reqid=client.4521.0:150, at_version=32'10, trim_to=0'0, roll_forward_to=32'9))", 2024-09-15T03:46:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.513452+0000", 2024-09-15T03:46:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.37229475200000001, 2024-09-15T03:46:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054416100000000004, 2024-09-15T03:46:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.513452+0000", 2024-09-15T03:46:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.513452+0000", 2024-09-15T03:46:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.513453+0000", 2024-09-15T03:46:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3200000000000004e-07 2024-09-15T03:46:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.513502+0000", 2024-09-15T03:46:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8387999999999998e-05 2024-09-15T03:46:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.513515+0000", 2024-09-15T03:46:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2921e-05 2024-09-15T03:46:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.513520+0000", 2024-09-15T03:46:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0629999999999996e-06 2024-09-15T03:46:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.513566+0000", 2024-09-15T03:46:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6745000000000001e-05 2024-09-15T03:46:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.513575+0000", 2024-09-15T03:46:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7800000000000006e-06 2024-09-15T03:46:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.513653+0000", 2024-09-15T03:46:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7806999999999996e-05 2024-09-15T03:46:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.513666+0000", 2024-09-15T03:46:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2669e-05 2024-09-15T03:46:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.513735+0000", 2024-09-15T03:46:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9214999999999994e-05 2024-09-15T03:46:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.513754+0000", 2024-09-15T03:46:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9589000000000002e-05 2024-09-15T03:46:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.513997+0000", 2024-09-15T03:46:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024225200000000001 2024-09-15T03:46:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.3s3 32/31 ECSubWrite(tid=29, reqid=client.4521.0:158, at_version=32'13, trim_to=0'0, roll_forward_to=32'12))", 2024-09-15T03:46:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.527974+0000", 2024-09-15T03:46:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.35777351899999998, 2024-09-15T03:46:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00062969499999999999, 2024-09-15T03:46:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.527974+0000", 2024-09-15T03:46:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.527974+0000", 2024-09-15T03:46:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.527975+0000", 2024-09-15T03:46:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.322e-06 2024-09-15T03:46:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.528033+0000", 2024-09-15T03:46:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8309e-05 2024-09-15T03:46:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.528058+0000", 2024-09-15T03:46:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5103e-05 2024-09-15T03:46:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.528067+0000", 2024-09-15T03:46:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2330000000000006e-06 2024-09-15T03:46:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.528128+0000", 2024-09-15T03:46:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0999000000000003e-05 2024-09-15T03:46:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.528142+0000", 2024-09-15T03:46:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4497e-05 2024-09-15T03:46:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.528253+0000", 2024-09-15T03:46:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011087299999999999 2024-09-15T03:46:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.528262+0000", 2024-09-15T03:46:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2639999999999997e-06 2024-09-15T03:46:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.528348+0000", 2024-09-15T03:46:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5607999999999998e-05 2024-09-15T03:46:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.528369+0000", 2024-09-15T03:46:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0706000000000001e-05 2024-09-15T03:46:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.528603+0000", 2024-09-15T03:46:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023478099999999999 2024-09-15T03:46:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.3s3 32/31 ECSubWrite(tid=31, reqid=client.4521.0:159, at_version=32'14, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:46:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.530133+0000", 2024-09-15T03:46:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.35561398700000002, 2024-09-15T03:46:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069238999999999998, 2024-09-15T03:46:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.530133+0000", 2024-09-15T03:46:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.530133+0000", 2024-09-15T03:46:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.530134+0000", 2024-09-15T03:46:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0800000000000004e-07 2024-09-15T03:46:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.530231+0000", 2024-09-15T03:46:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7004999999999997e-05 2024-09-15T03:46:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.530255+0000", 2024-09-15T03:46:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3879e-05 2024-09-15T03:46:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.530261+0000", 2024-09-15T03:46:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1190000000000002e-06 2024-09-15T03:46:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.530359+0000", 2024-09-15T03:46:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8088e-05 2024-09-15T03:46:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.530385+0000", 2024-09-15T03:46:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6358000000000001e-05 2024-09-15T03:46:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.530508+0000", 2024-09-15T03:46:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122934 2024-09-15T03:46:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.530518+0000", 2024-09-15T03:46:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6039999999999999e-06 2024-09-15T03:46:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.530585+0000", 2024-09-15T03:46:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7194999999999996e-05 2024-09-15T03:46:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.530606+0000", 2024-09-15T03:46:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1172e-05 2024-09-15T03:46:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.530826+0000", 2024-09-15T03:46:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000219328 2024-09-15T03:46:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:165 3.bs0 3:d6f79b7b:::smithi00132172-29:head [create,setxattr aSrcO`NnS[qadSjpQT[PSmfh\\PfYagpctmbQ_kXi (1753) in=40b,setxattr hratdsb^VNONN^^acUkPeXNZohcdVkmXXhdTeofs (2805) in=40b,setxattr abOo``aaqamjUn`hc`Qde]XU\\lqYVrmVdcimNs_a (35) in=40b,setxattr ZheopalZpQPnebTbena`gaWXtmaSTdQkji`aZYOi (34) in=40b,setxattr katXYockjtYtieO]`jgpSrZZ\\mq`gOYplPOVal`Y (842) in=40b,setxattr [^]lbsa\\UX[na]akS\\hYcdqU^]aWe_[U[OlaQXUY (1894) in=40b,setxattr We][_Xheab\\Ylg`q^hsi`XTX_tQS`Ng\\]aqZTnV[ (1473) in=40b,setxattr TTRSo^W`XoqVWpSjRgc[adm`ROSSVYUNlsilh\\kS (911) in=40b,setxattr bqbUQROpngcT`ndhrZQP`iraoUgs[ZZatacnZoc_ (2876) in=40b,setxattr qPPSibdg_e\\Y_V`kTQO^\\`_jr^jt\\aiadkepqYUj (3576) in=40b,setxattr q`N\\bPOqejbpq`QZVQfWgjScjfahlgXWodnRReWS (1017) in=40b,setxattr `pdT]PSaeeZ^jhWYc]iO`^tRXWcjedgQdke^adpf (7928) in=40b,setxattr matSmfe^casVRfWbNqkt_mZNWbhhh`\\Waf\\djeca (7157) in=40b,setxattr ]SWX\\hrhracj[TeSRYP[bZ[[lSSRghmfarTYpp]U (735) in=40b,setxattr WrnQrVa (13964) in=7b,setxattr PcP_]XU]rhapTs\\X`ojNmm_Qsjkkjecf`VgrdTrR (1542) in=40b,setxattr o^RO^oeZlji]\\\\UhVsnb]t_WXomaRr^higjSUShQ (1016) in=40b,setxattr Z_ofVVcfZdUV[UaRnTV_Ni^_rRNl`\\kt`]aaaoed (701) in=40b,setxattr X[m^[`bVlbgi``^mWZ]cl[S_nn[Ok\\ldS\\_h]]qk (1999) in=40b,setxattr fpmQVVnVPSmZmt`OeaT`adhoNjhpOb[`\\aPWQ^Rd (700) in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-15T03:46:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.530978+0000", 2024-09-15T03:46:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.354769214, 2024-09-15T03:46:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033826339999999998, 2024-09-15T03:46:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 165 2024-09-15T03:46:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.530978+0000", 2024-09-15T03:46:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.530978+0000", 2024-09-15T03:46:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.530980+0000", 2024-09-15T03:46:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6440000000000001e-06 2024-09-15T03:46:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.531028+0000", 2024-09-15T03:46:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8198000000000002e-05 2024-09-15T03:46:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.531041+0000", 2024-09-15T03:46:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2847000000000001e-05 2024-09-15T03:46:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.531047+0000", 2024-09-15T03:46:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4069999999999998e-06 2024-09-15T03:46:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.531170+0000", 2024-09-15T03:46:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012244999999999999 2024-09-15T03:46:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.531299+0000", 2024-09-15T03:46:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012957499999999999 2024-09-15T03:46:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532344+0000", 2024-09-15T03:46:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010449140000000001 2024-09-15T03:46:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532456+0000", 2024-09-15T03:46:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011162699999999999 2024-09-15T03:46:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532694+0000", 2024-09-15T03:46:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023811500000000001 2024-09-15T03:46:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.533445+0000", 2024-09-15T03:46:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00075162999999999998 2024-09-15T03:46:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.533507+0000", 2024-09-15T03:46:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1630000000000005e-05 2024-09-15T03:46:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.534019+0000", 2024-09-15T03:46:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051244400000000003 2024-09-15T03:46:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.534361+0000", 2024-09-15T03:46:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034115300000000002 2024-09-15T03:46:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.7s3 32/31 ECSubWrite(tid=42, reqid=client.4521.0:166, at_version=32'13, trim_to=0'0, roll_forward_to=32'12))", 2024-09-15T03:46:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.532065+0000", 2024-09-15T03:46:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.35368214799999997, 2024-09-15T03:46:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028783820000000001, 2024-09-15T03:46:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532065+0000", 2024-09-15T03:46:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532065+0000", 2024-09-15T03:46:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532066+0000", 2024-09-15T03:46:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.82e-07 2024-09-15T03:46:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532078+0000", 2024-09-15T03:46:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2417e-05 2024-09-15T03:46:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532086+0000", 2024-09-15T03:46:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9529999999999999e-06 2024-09-15T03:46:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532090+0000", 2024-09-15T03:46:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0090000000000001e-06 2024-09-15T03:46:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532150+0000", 2024-09-15T03:46:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0451000000000001e-05 2024-09-15T03:46:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532172+0000", 2024-09-15T03:46:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1708e-05 2024-09-15T03:46:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532279+0000", 2024-09-15T03:46:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106307 2024-09-15T03:46:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532318+0000", 2024-09-15T03:46:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9854000000000002e-05 2024-09-15T03:46:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532691+0000", 2024-09-15T03:46:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037270100000000002 2024-09-15T03:46:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532711+0000", 2024-09-15T03:46:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9619999999999998e-05 2024-09-15T03:46:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.534943+0000", 2024-09-15T03:46:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022327800000000002 2024-09-15T03:46:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.3s3 32/31 ECSubWrite(tid=33, reqid=client.4521.0:160, at_version=0'0, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:46:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.532453+0000", 2024-09-15T03:46:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.35329376200000001, 2024-09-15T03:46:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025066350000000001, 2024-09-15T03:46:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532453+0000", 2024-09-15T03:46:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532453+0000", 2024-09-15T03:46:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532454+0000", 2024-09-15T03:46:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5899999999999996e-07 2024-09-15T03:46:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532464+0000", 2024-09-15T03:46:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0359e-05 2024-09-15T03:46:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532476+0000", 2024-09-15T03:46:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1541e-05 2024-09-15T03:46:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532483+0000", 2024-09-15T03:46:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8020000000000003e-06 2024-09-15T03:46:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532535+0000", 2024-09-15T03:46:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1894000000000002e-05 2024-09-15T03:46:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532543+0000", 2024-09-15T03:46:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2430000000000005e-06 2024-09-15T03:46:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532627+0000", 2024-09-15T03:46:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3813999999999994e-05 2024-09-15T03:46:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532694+0000", 2024-09-15T03:46:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7638999999999994e-05 2024-09-15T03:46:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.533449+0000", 2024-09-15T03:46:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00075452600000000005 2024-09-15T03:46:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.533511+0000", 2024-09-15T03:46:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2452000000000004e-05 2024-09-15T03:46:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.534960+0000", 2024-09-15T03:46:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014487059999999999 2024-09-15T03:46:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.7s3 32/31 ECSubWrite(tid=43, reqid=client.4521.0:166, at_version=0'0, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:46:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.534156+0000", 2024-09-15T03:46:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.35159100500000001, 2024-09-15T03:46:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00091828899999999998, 2024-09-15T03:46:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.534156+0000", 2024-09-15T03:46:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.534156+0000", 2024-09-15T03:46:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.534157+0000", 2024-09-15T03:46:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0399999999999996e-07 2024-09-15T03:46:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.534172+0000", 2024-09-15T03:46:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5189e-05 2024-09-15T03:46:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.534184+0000", 2024-09-15T03:46:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2135e-05 2024-09-15T03:46:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.534191+0000", 2024-09-15T03:46:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6290000000000004e-06 2024-09-15T03:46:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.534240+0000", 2024-09-15T03:46:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8936e-05 2024-09-15T03:46:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.534249+0000", 2024-09-15T03:46:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0920000000000004e-06 2024-09-15T03:46:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.534320+0000", 2024-09-15T03:46:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1364000000000003e-05 2024-09-15T03:46:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.534330+0000", 2024-09-15T03:46:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7459999999999995e-06 2024-09-15T03:46:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.534363+0000", 2024-09-15T03:46:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2749000000000001e-05 2024-09-15T03:46:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.534394+0000", 2024-09-15T03:46:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1739000000000002e-05 2024-09-15T03:46:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.535075+0000", 2024-09-15T03:46:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068010599999999998 2024-09-15T03:46:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.cs1 32/28 ECSubWrite(tid=8, reqid=client.4521.0:175, at_version=32'5, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.553784+0000", 2024-09-15T03:46:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.33196290699999997, 2024-09-15T03:46:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042737419999999996, 2024-09-15T03:46:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553784+0000", 2024-09-15T03:46:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553784+0000", 2024-09-15T03:46:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553793+0000", 2024-09-15T03:46:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2779999999999993e-06 2024-09-15T03:46:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553815+0000", 2024-09-15T03:46:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2062999999999999e-05 2024-09-15T03:46:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553831+0000", 2024-09-15T03:46:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6589e-05 2024-09-15T03:46:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553840+0000", 2024-09-15T03:46:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8789999999999992e-06 2024-09-15T03:46:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553909+0000", 2024-09-15T03:46:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8397000000000003e-05 2024-09-15T03:46:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553926+0000", 2024-09-15T03:46:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7662000000000001e-05 2024-09-15T03:46:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554044+0000", 2024-09-15T03:46:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118165 2024-09-15T03:46:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554061+0000", 2024-09-15T03:46:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.683e-05 2024-09-15T03:46:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554102+0000", 2024-09-15T03:46:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0451999999999998e-05 2024-09-15T03:46:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554119+0000", 2024-09-15T03:46:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7428999999999999e-05 2024-09-15T03:46:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.558058+0000", 2024-09-15T03:46:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003938998 2024-09-15T03:46:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(osd.6.17:1 3.bs0 3:dda01a53:::smithi00132172-22:head [assert-version v9,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-09-15T03:46:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.554245+0000", 2024-09-15T03:46:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.33150219399999997, 2024-09-15T03:46:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0034389799999999999, 2024-09-15T03:46:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:46:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554245+0000", 2024-09-15T03:46:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554245+0000", 2024-09-15T03:46:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554247+0000", 2024-09-15T03:46:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7859999999999999e-06 2024-09-15T03:46:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554252+0000", 2024-09-15T03:46:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7049999999999998e-06 2024-09-15T03:46:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554262+0000", 2024-09-15T03:46:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0954e-05 2024-09-15T03:46:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554267+0000", 2024-09-15T03:46:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4109999999999998e-06 2024-09-15T03:46:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554311+0000", 2024-09-15T03:46:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4440999999999997e-05 2024-09-15T03:46:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554383+0000", 2024-09-15T03:46:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1791000000000004e-05 2024-09-15T03:46:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.557684+0000", 2024-09-15T03:46:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033008920000000001 2024-09-15T03:46:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.cs1 32/28 ECSubWrite(tid=9, reqid=client.4521.0:175, at_version=0'0, trim_to=0'0, roll_forward_to=32'5))", 2024-09-15T03:46:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.554474+0000", 2024-09-15T03:46:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.33127274299999998, 2024-09-15T03:46:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003720681, 2024-09-15T03:46:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554474+0000", 2024-09-15T03:46:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554474+0000", 2024-09-15T03:46:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554475+0000", 2024-09-15T03:46:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1900000000000001e-07 2024-09-15T03:46:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554487+0000", 2024-09-15T03:46:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1691999999999999e-05 2024-09-15T03:46:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554500+0000", 2024-09-15T03:46:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3261999999999999e-05 2024-09-15T03:46:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554507+0000", 2024-09-15T03:46:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5969999999999997e-06 2024-09-15T03:46:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554555+0000", 2024-09-15T03:46:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8338999999999999e-05 2024-09-15T03:46:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554564+0000", 2024-09-15T03:46:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6619999999999994e-06 2024-09-15T03:46:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554715+0000", 2024-09-15T03:46:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015073699999999999 2024-09-15T03:46:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554734+0000", 2024-09-15T03:46:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9159000000000001e-05 2024-09-15T03:46:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.555298+0000", 2024-09-15T03:46:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056395800000000004 2024-09-15T03:46:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.555313+0000", 2024-09-15T03:46:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5000999999999999e-05 2024-09-15T03:46:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.558195+0000", 2024-09-15T03:46:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028824549999999999 2024-09-15T03:46:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s6 32/31 ECSubWrite(tid=35, reqid=client.4521.0:171, at_version=32'3, trim_to=0'0, roll_forward_to=32'2))", 2024-09-15T03:46:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.559205+0000", 2024-09-15T03:46:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.32654175699999999, 2024-09-15T03:46:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029444279999999998, 2024-09-15T03:46:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559205+0000", 2024-09-15T03:46:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559205+0000", 2024-09-15T03:46:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559207+0000", 2024-09-15T03:46:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0610000000000001e-06 2024-09-15T03:46:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559263+0000", 2024-09-15T03:46:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6174000000000001e-05 2024-09-15T03:46:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559285+0000", 2024-09-15T03:46:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1824e-05 2024-09-15T03:46:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559293+0000", 2024-09-15T03:46:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9899999999999997e-06 2024-09-15T03:46:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559388+0000", 2024-09-15T03:46:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5815000000000003e-05 2024-09-15T03:46:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559399+0000", 2024-09-15T03:46:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1099e-05 2024-09-15T03:46:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559509+0000", 2024-09-15T03:46:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110058 2024-09-15T03:46:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559534+0000", 2024-09-15T03:46:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4167000000000001e-05 2024-09-15T03:46:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559619+0000", 2024-09-15T03:46:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4870999999999994e-05 2024-09-15T03:46:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559639+0000", 2024-09-15T03:46:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0721999999999999e-05 2024-09-15T03:46:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.562150+0000", 2024-09-15T03:46:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002510647 2024-09-15T03:46:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s6 32/31 ECSubWrite(tid=36, reqid=client.4521.0:171, at_version=0'0, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:46:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.560599+0000", 2024-09-15T03:46:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.32514834199999998, 2024-09-15T03:46:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017314870000000001, 2024-09-15T03:46:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.560599+0000", 2024-09-15T03:46:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.560599+0000", 2024-09-15T03:46:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.560600+0000", 2024-09-15T03:46:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1939999999999999e-06 2024-09-15T03:46:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.560611+0000", 2024-09-15T03:46:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.11e-05 2024-09-15T03:46:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.560628+0000", 2024-09-15T03:46:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6509000000000001e-05 2024-09-15T03:46:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.560635+0000", 2024-09-15T03:46:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1010000000000003e-06 2024-09-15T03:46:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.560677+0000", 2024-09-15T03:46:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2221000000000001e-05 2024-09-15T03:46:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.560686+0000", 2024-09-15T03:46:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8410000000000006e-06 2024-09-15T03:46:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.560757+0000", 2024-09-15T03:46:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1635000000000001e-05 2024-09-15T03:46:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.560773+0000", 2024-09-15T03:46:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5645999999999999e-05 2024-09-15T03:46:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.560819+0000", 2024-09-15T03:46:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5704000000000002e-05 2024-09-15T03:46:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.560840+0000", 2024-09-15T03:46:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1396e-05 2024-09-15T03:46:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.562330+0000", 2024-09-15T03:46:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00149014 2024-09-15T03:46:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.2s3 32/31 ECSubWrite(tid=44, reqid=client.4521.0:173, at_version=32'7, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.562064+0000", 2024-09-15T03:46:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.32368350600000001, 2024-09-15T03:46:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041627890000000001, 2024-09-15T03:46:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.562064+0000", 2024-09-15T03:46:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.562064+0000", 2024-09-15T03:46:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.562065+0000", 2024-09-15T03:46:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5999999999999991e-07 2024-09-15T03:46:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.562158+0000", 2024-09-15T03:46:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3455999999999994e-05 2024-09-15T03:46:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.562176+0000", 2024-09-15T03:46:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7872000000000001e-05 2024-09-15T03:46:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.562183+0000", 2024-09-15T03:46:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.934e-06 2024-09-15T03:46:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.562255+0000", 2024-09-15T03:46:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2032000000000004e-05 2024-09-15T03:46:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.562274+0000", 2024-09-15T03:46:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9165999999999999e-05 2024-09-15T03:46:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.562397+0000", 2024-09-15T03:46:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122774 2024-09-15T03:46:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.562426+0000", 2024-09-15T03:46:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9408e-05 2024-09-15T03:46:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.562529+0000", 2024-09-15T03:46:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010228700000000001 2024-09-15T03:46:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.562572+0000", 2024-09-15T03:46:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3010999999999999e-05 2024-09-15T03:46:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.566227+0000", 2024-09-15T03:46:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036548890000000001 2024-09-15T03:46:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.2s3 32/31 ECSubWrite(tid=45, reqid=client.4521.0:173, at_version=0'0, trim_to=0'0, roll_forward_to=32'7))", 2024-09-15T03:46:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.563405+0000", 2024-09-15T03:46:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.322342723, 2024-09-15T03:46:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002934332, 2024-09-15T03:46:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.563405+0000", 2024-09-15T03:46:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.563405+0000", 2024-09-15T03:46:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.563405+0000", 2024-09-15T03:46:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8999999999999996e-07 2024-09-15T03:46:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.563415+0000", 2024-09-15T03:46:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0023e-05 2024-09-15T03:46:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.563430+0000", 2024-09-15T03:46:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5165e-05 2024-09-15T03:46:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.563438+0000", 2024-09-15T03:46:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4250000000000001e-06 2024-09-15T03:46:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.563485+0000", 2024-09-15T03:46:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6792000000000002e-05 2024-09-15T03:46:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.563494+0000", 2024-09-15T03:46:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2869999999999993e-06 2024-09-15T03:46:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.563565+0000", 2024-09-15T03:46:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1513000000000005e-05 2024-09-15T03:46:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.563576+0000", 2024-09-15T03:46:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0291e-05 2024-09-15T03:46:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.563612+0000", 2024-09-15T03:46:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6365000000000002e-05 2024-09-15T03:46:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.563637+0000", 2024-09-15T03:46:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5041e-05 2024-09-15T03:46:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.566339+0000", 2024-09-15T03:46:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027017399999999998 2024-09-15T03:46:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:40.005 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:40.005 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:40.005 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:40.005 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:40.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:46:40.007 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:40.005+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:46:40.105 INFO:tasks.rados.rados.0.smithi001.stdout:56: done (5 left) 2024-09-15T03:46:40.105 INFO:tasks.rados.rados.0.smithi001.stdout:57: done (4 left) 2024-09-15T03:46:40.105 INFO:tasks.rados.rados.0.smithi001.stdout:58: done (3 left) 2024-09-15T03:46:40.105 INFO:tasks.rados.rados.0.smithi001.stdout:60: done (2 left) 2024-09-15T03:46:40.105 INFO:tasks.rados.rados.0.smithi001.stdout:61: done (1 left) 2024-09-15T03:46:40.105 INFO:tasks.rados.rados.0.smithi001.stdout:62: done (0 left) 2024-09-15T03:46:40.105 INFO:tasks.rados.rados.0.smithi001.stdout:63: read oid 42 snap -1 2024-09-15T03:46:40.105 INFO:tasks.rados.rados.0.smithi001.stdout:63: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-15T03:46:40.105 INFO:tasks.rados.rados.0.smithi001.stdout:64: rollback oid 13 current snap is 1 2024-09-15T03:46:40.105 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 13 to 0 2024-09-15T03:46:40.105 INFO:tasks.rados.rados.0.smithi001.stdout:65: rollback oid 47 current snap is 1 2024-09-15T03:46:40.106 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 47 to 0 2024-09-15T03:46:40.106 INFO:tasks.rados.rados.0.smithi001.stdout:66: snap_create 2024-09-15T03:46:40.106 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:40.106 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:40.106 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:46:40.107 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:40.107 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:40.106+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:46:40.109 INFO:tasks.rados.rados.0.smithi001.stdout:64: finishing rollback tid 0 to smithi00132172-13 2024-09-15T03:46:40.111 INFO:tasks.rados.rados.0.smithi001.stdout:65: finishing rollback tid 0 to smithi00132172-47 2024-09-15T03:46:40.111 INFO:tasks.rados.rados.0.smithi001.stdout:65: finishing rollback tid 1 to smithi00132172-47 2024-09-15T03:46:40.113 INFO:tasks.rados.rados.0.smithi001.stdout:63: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-15T03:46:40.118 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:46:40.132 INFO:tasks.rados.rados.0.smithi001.stdout:64: finishing rollback tid 1 to smithi00132172-13 2024-09-15T03:46:40.132 INFO:tasks.rados.rados.0.smithi001.stdout:65: finishing rollback tid 2 to smithi00132172-47 2024-09-15T03:46:40.132 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 47 v 10 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-15T03:46:40.132 INFO:tasks.rados.rados.0.smithi001.stdout:64: finishing rollback tid 2 to smithi00132172-13 2024-09-15T03:46:40.133 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 13 v 8 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-15T03:46:40.208 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:40.207+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:46:40.213 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:40.213 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:40.213 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:46:40.213 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:46:40.213 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:40.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:46:40.309 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:40.308+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:46:40.321 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:40.322 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:46:40.322 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:46:40.322 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:46:40.322 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.322 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:40.322 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:46:40.322 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 22.417134865000001, 2024-09-15T03:46:40.322 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10988437500000001, 2024-09-15T03:46:40.322 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.322 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:40.322 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:40.322 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:40.323 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:40.323 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1 2024-09-15T03:46:40.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.323 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.323 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.323 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.323 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:46:40.323 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.323 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.323 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.323 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:46:40.323 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.324 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.324 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.324 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902355+0000", 2024-09-15T03:46:40.324 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.429e-06 2024-09-15T03:46:40.324 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.324 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.324 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.324 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902362+0000", 2024-09-15T03:46:40.324 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7290000000000003e-06 2024-09-15T03:46:40.324 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.324 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.324 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.324 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902373+0000", 2024-09-15T03:46:40.325 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0533999999999999e-05 2024-09-15T03:46:40.325 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.325 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.325 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.325 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902379+0000", 2024-09-15T03:46:40.325 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4350000000000002e-06 2024-09-15T03:46:40.325 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.325 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.325 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.325 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902428+0000", 2024-09-15T03:46:40.325 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8571e-05 2024-09-15T03:46:40.325 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.325 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.326 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:46:40.326 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902434+0000", 2024-09-15T03:46:40.326 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5610000000000004e-06 2024-09-15T03:46:40.326 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.326 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.326 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.326 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011307+0000", 2024-09-15T03:46:40.326 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.108872306 2024-09-15T03:46:40.326 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.326 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.326 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:40.326 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011448+0000", 2024-09-15T03:46:40.326 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014137199999999999 2024-09-15T03:46:40.326 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.327 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.327 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:40.327 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011569+0000", 2024-09-15T03:46:40.327 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120925 2024-09-15T03:46:40.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.327 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.327 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.327 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011695+0000", 2024-09-15T03:46:40.327 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012586800000000001 2024-09-15T03:46:40.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.327 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.327 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.327 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011703+0000", 2024-09-15T03:46:40.328 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4190000000000002e-06 2024-09-15T03:46:40.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.328 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.328 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011739+0000", 2024-09-15T03:46:40.328 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5474e-05 2024-09-15T03:46:40.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.328 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.328 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:40.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011940+0000", 2024-09-15T03:46:40.328 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020108699999999999 2024-09-15T03:46:40.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.329 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.329 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:40.329 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012179+0000", 2024-09-15T03:46:40.329 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023950299999999999 2024-09-15T03:46:40.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.329 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.329 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:40.329 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012215+0000", 2024-09-15T03:46:40.329 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5332e-05 2024-09-15T03:46:40.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.329 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.329 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.329 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012238+0000", 2024-09-15T03:46:40.329 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3830000000000001e-05 2024-09-15T03:46:40.330 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.330 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.330 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.330 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.330 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:40.330 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.170563+0000", 2024-09-15T03:46:40.330 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 22.148925498000001, 2024-09-15T03:46:40.330 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0033790360000000002, 2024-09-15T03:46:40.330 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.330 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:40.330 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:40.330 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:40.331 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:40.331 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 47 2024-09-15T03:46:40.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.331 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.331 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170563+0000", 2024-09-15T03:46:40.331 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.331 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170563+0000", 2024-09-15T03:46:40.331 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.332 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.332 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.332 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170564+0000", 2024-09-15T03:46:40.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7099999999999998e-07 2024-09-15T03:46:40.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.332 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.332 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.332 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170658+0000", 2024-09-15T03:46:40.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4112999999999998e-05 2024-09-15T03:46:40.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.332 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.332 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.332 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170666+0000", 2024-09-15T03:46:40.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0800000000000006e-06 2024-09-15T03:46:40.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.333 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.333 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.333 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170669+0000", 2024-09-15T03:46:40.333 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5920000000000001e-06 2024-09-15T03:46:40.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.333 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.333 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.333 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.172869+0000", 2024-09-15T03:46:40.333 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021995640000000002 2024-09-15T03:46:40.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.333 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.333 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:40.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.172919+0000", 2024-09-15T03:46:40.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9907999999999997e-05 2024-09-15T03:46:40.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.334 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.334 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:40.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.172997+0000", 2024-09-15T03:46:40.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7896000000000001e-05 2024-09-15T03:46:40.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.334 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.334 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173102+0000", 2024-09-15T03:46:40.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000105269 2024-09-15T03:46:40.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.334 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173109+0000", 2024-09-15T03:46:40.335 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2089999999999999e-06 2024-09-15T03:46:40.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173167+0000", 2024-09-15T03:46:40.335 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7627999999999998e-05 2024-09-15T03:46:40.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:40.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173229+0000", 2024-09-15T03:46:40.335 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2039999999999996e-05 2024-09-15T03:46:40.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.336 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.336 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:40.336 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173897+0000", 2024-09-15T03:46:40.336 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00066788899999999996 2024-09-15T03:46:40.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.336 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.336 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:40.336 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173923+0000", 2024-09-15T03:46:40.336 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5871e-05 2024-09-15T03:46:40.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.336 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.336 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.336 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173942+0000", 2024-09-15T03:46:40.337 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9706000000000001e-05 2024-09-15T03:46:40.337 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.337 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.337 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.337 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.337 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.337 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:40.337 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.170687+0000", 2024-09-15T03:46:40.337 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 22.148801690999999, 2024-09-15T03:46:40.337 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0039080060000000003, 2024-09-15T03:46:40.337 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.337 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:40.337 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:40.338 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:40.338 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:40.338 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 49 2024-09-15T03:46:40.338 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.338 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.338 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.338 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.338 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170687+0000", 2024-09-15T03:46:40.338 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.338 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.338 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.338 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.338 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170687+0000", 2024-09-15T03:46:40.338 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.339 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.339 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.339 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170687+0000", 2024-09-15T03:46:40.339 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5899999999999998e-07 2024-09-15T03:46:40.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.339 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.339 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.339 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170689+0000", 2024-09-15T03:46:40.339 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9120000000000001e-06 2024-09-15T03:46:40.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.339 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.339 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170695+0000", 2024-09-15T03:46:40.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1550000000000004e-06 2024-09-15T03:46:40.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170698+0000", 2024-09-15T03:46:40.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2089999999999999e-06 2024-09-15T03:46:40.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173634+0000", 2024-09-15T03:46:40.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002936204 2024-09-15T03:46:40.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.341 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.341 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:40.341 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173684+0000", 2024-09-15T03:46:40.341 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9798e-05 2024-09-15T03:46:40.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.341 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.341 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:40.341 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173755+0000", 2024-09-15T03:46:40.341 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0957999999999994e-05 2024-09-15T03:46:40.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.341 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.341 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.341 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173850+0000", 2024-09-15T03:46:40.342 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4936e-05 2024-09-15T03:46:40.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.342 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.342 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173857+0000", 2024-09-15T03:46:40.342 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2830000000000004e-06 2024-09-15T03:46:40.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.342 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.342 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173883+0000", 2024-09-15T03:46:40.342 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5910000000000001e-05 2024-09-15T03:46:40.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.342 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:40.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.173969+0000", 2024-09-15T03:46:40.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5894000000000001e-05 2024-09-15T03:46:40.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:40.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174550+0000", 2024-09-15T03:46:40.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00058144000000000002 2024-09-15T03:46:40.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:40.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174576+0000", 2024-09-15T03:46:40.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5624999999999999e-05 2024-09-15T03:46:40.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174595+0000", 2024-09-15T03:46:40.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9423e-05 2024-09-15T03:46:40.344 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.344 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.344 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.344 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:40.344 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.170804+0000", 2024-09-15T03:46:40.344 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 22.148685017999998, 2024-09-15T03:46:40.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0041461739999999999, 2024-09-15T03:46:40.344 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.345 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:40.345 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:40.345 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:40.345 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:40.345 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 50 2024-09-15T03:46:40.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.345 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170804+0000", 2024-09-15T03:46:40.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170804+0000", 2024-09-15T03:46:40.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170804+0000", 2024-09-15T03:46:40.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7199999999999999e-07 2024-09-15T03:46:40.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170856+0000", 2024-09-15T03:46:40.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1647000000000002e-05 2024-09-15T03:46:40.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170865+0000", 2024-09-15T03:46:40.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0829999999999993e-06 2024-09-15T03:46:40.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170869+0000", 2024-09-15T03:46:40.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8369999999999999e-06 2024-09-15T03:46:40.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174053+0000", 2024-09-15T03:46:40.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031840639999999999 2024-09-15T03:46:40.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:40.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174099+0000", 2024-09-15T03:46:40.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5578000000000003e-05 2024-09-15T03:46:40.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:40.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174178+0000", 2024-09-15T03:46:40.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9268000000000001e-05 2024-09-15T03:46:40.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174261+0000", 2024-09-15T03:46:40.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3023999999999997e-05 2024-09-15T03:46:40.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174269+0000", 2024-09-15T03:46:40.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2460000000000003e-06 2024-09-15T03:46:40.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174322+0000", 2024-09-15T03:46:40.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2957999999999997e-05 2024-09-15T03:46:40.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.350 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:40.350 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174614+0000", 2024-09-15T03:46:40.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029206300000000001 2024-09-15T03:46:40.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.350 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:40.350 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174906+0000", 2024-09-15T03:46:40.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029138299999999998 2024-09-15T03:46:40.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:40.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174931+0000", 2024-09-15T03:46:40.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5809999999999999e-05 2024-09-15T03:46:40.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174950+0000", 2024-09-15T03:46:40.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8641e-05 2024-09-15T03:46:40.351 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.351 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.351 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.352 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:40.352 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.170873+0000", 2024-09-15T03:46:40.352 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 22.148615809999999, 2024-09-15T03:46:40.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0041437130000000003, 2024-09-15T03:46:40.352 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.352 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:40.352 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:40.352 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:40.352 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:40.352 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 51 2024-09-15T03:46:40.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.352 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170873+0000", 2024-09-15T03:46:40.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170873+0000", 2024-09-15T03:46:40.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170873+0000", 2024-09-15T03:46:40.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:46:40.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.354 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170876+0000", 2024-09-15T03:46:40.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3010000000000001e-06 2024-09-15T03:46:40.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.354 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170879+0000", 2024-09-15T03:46:40.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2710000000000001e-06 2024-09-15T03:46:40.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.354 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170881+0000", 2024-09-15T03:46:40.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3630000000000001e-06 2024-09-15T03:46:40.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174304+0000", 2024-09-15T03:46:40.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0034224139999999999 2024-09-15T03:46:40.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:40.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174351+0000", 2024-09-15T03:46:40.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7638e-05 2024-09-15T03:46:40.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:40.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174421+0000", 2024-09-15T03:46:40.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9283999999999996e-05 2024-09-15T03:46:40.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174501+0000", 2024-09-15T03:46:40.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0652e-05 2024-09-15T03:46:40.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174508+0000", 2024-09-15T03:46:40.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6540000000000002e-06 2024-09-15T03:46:40.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174530+0000", 2024-09-15T03:46:40.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2473999999999999e-05 2024-09-15T03:46:40.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:40.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174621+0000", 2024-09-15T03:46:40.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0803000000000005e-05 2024-09-15T03:46:40.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:40.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174975+0000", 2024-09-15T03:46:40.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00035398800000000001 2024-09-15T03:46:40.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:40.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174999+0000", 2024-09-15T03:46:40.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3241999999999999e-05 2024-09-15T03:46:40.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.175017+0000", 2024-09-15T03:46:40.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8219e-05 2024-09-15T03:46:40.358 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.358 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.359 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.359 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:40.359 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.170884+0000", 2024-09-15T03:46:40.359 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 22.148604451000001, 2024-09-15T03:46:40.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042576849999999998, 2024-09-15T03:46:40.359 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.359 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:40.359 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:40.359 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:40.359 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:40.359 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 52 2024-09-15T03:46:40.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.360 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170884+0000", 2024-09-15T03:46:40.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170884+0000", 2024-09-15T03:46:40.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170885+0000", 2024-09-15T03:46:40.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0599999999999999e-07 2024-09-15T03:46:40.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170887+0000", 2024-09-15T03:46:40.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.17e-06 2024-09-15T03:46:40.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170890+0000", 2024-09-15T03:46:40.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9210000000000001e-06 2024-09-15T03:46:40.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170892+0000", 2024-09-15T03:46:40.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.779e-06 2024-09-15T03:46:40.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174638+0000", 2024-09-15T03:46:40.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0037452269999999998 2024-09-15T03:46:40.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:40.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174694+0000", 2024-09-15T03:46:40.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6378000000000001e-05 2024-09-15T03:46:40.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.363 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:40.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174770+0000", 2024-09-15T03:46:40.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5654000000000004e-05 2024-09-15T03:46:40.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.363 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174859+0000", 2024-09-15T03:46:40.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9074999999999997e-05 2024-09-15T03:46:40.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.363 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174865+0000", 2024-09-15T03:46:40.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.178e-06 2024-09-15T03:46:40.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.364 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.364 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174890+0000", 2024-09-15T03:46:40.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4956e-05 2024-09-15T03:46:40.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.364 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:40.364 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.175029+0000", 2024-09-15T03:46:40.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013951999999999999 2024-09-15T03:46:40.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:40.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.175102+0000", 2024-09-15T03:46:40.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2713000000000007e-05 2024-09-15T03:46:40.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:40.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.175126+0000", 2024-09-15T03:46:40.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3541e-05 2024-09-15T03:46:40.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.175142+0000", 2024-09-15T03:46:40.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6367000000000001e-05 2024-09-15T03:46:40.366 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.366 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.366 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.366 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:40.366 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.600852+0000", 2024-09-15T03:46:40.366 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.718637008999998, 2024-09-15T03:46:40.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003621132, 2024-09-15T03:46:40.366 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.366 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:40.366 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:40.366 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:40.366 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:40.367 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 74 2024-09-15T03:46:40.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.367 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600852+0000", 2024-09-15T03:46:40.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600852+0000", 2024-09-15T03:46:40.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600852+0000", 2024-09-15T03:46:40.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1399999999999997e-07 2024-09-15T03:46:40.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600856+0000", 2024-09-15T03:46:40.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7009999999999998e-06 2024-09-15T03:46:40.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600863+0000", 2024-09-15T03:46:40.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9689999999999997e-06 2024-09-15T03:46:40.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600867+0000", 2024-09-15T03:46:40.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7129999999999999e-06 2024-09-15T03:46:40.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603471+0000", 2024-09-15T03:46:40.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002604711 2024-09-15T03:46:40.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:40.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603521+0000", 2024-09-15T03:46:40.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0148999999999997e-05 2024-09-15T03:46:40.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:40.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603596+0000", 2024-09-15T03:46:40.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4144e-05 2024-09-15T03:46:40.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603690+0000", 2024-09-15T03:46:40.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4098999999999995e-05 2024-09-15T03:46:40.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603698+0000", 2024-09-15T03:46:40.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6309999999999992e-06 2024-09-15T03:46:40.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603726+0000", 2024-09-15T03:46:40.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7810999999999999e-05 2024-09-15T03:46:40.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:40.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603825+0000", 2024-09-15T03:46:40.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9321999999999995e-05 2024-09-15T03:46:40.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.372 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:40.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604424+0000", 2024-09-15T03:46:40.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00059867700000000004 2024-09-15T03:46:40.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.372 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:40.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604451+0000", 2024-09-15T03:46:40.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7277e-05 2024-09-15T03:46:40.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604473+0000", 2024-09-15T03:46:40.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1514000000000001e-05 2024-09-15T03:46:40.373 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.373 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.373 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.373 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:40.373 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.600872+0000", 2024-09-15T03:46:40.373 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.718617251000001, 2024-09-15T03:46:40.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0037887039999999999, 2024-09-15T03:46:40.374 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.374 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:40.374 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:40.374 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:40.374 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:40.374 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 75 2024-09-15T03:46:40.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.374 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600872+0000", 2024-09-15T03:46:40.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600872+0000", 2024-09-15T03:46:40.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600872+0000", 2024-09-15T03:46:40.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4700000000000002e-07 2024-09-15T03:46:40.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600933+0000", 2024-09-15T03:46:40.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0671999999999998e-05 2024-09-15T03:46:40.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600941+0000", 2024-09-15T03:46:40.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5779999999999999e-06 2024-09-15T03:46:40.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600946+0000", 2024-09-15T03:46:40.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8810000000000004e-06 2024-09-15T03:46:40.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603916+0000", 2024-09-15T03:46:40.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0029702359999999998 2024-09-15T03:46:40.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:40.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.603962+0000", 2024-09-15T03:46:40.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5148999999999998e-05 2024-09-15T03:46:40.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:40.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604037+0000", 2024-09-15T03:46:40.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5352999999999995e-05 2024-09-15T03:46:40.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.378 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604136+0000", 2024-09-15T03:46:40.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8655999999999999e-05 2024-09-15T03:46:40.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.378 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604142+0000", 2024-09-15T03:46:40.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4060000000000002e-06 2024-09-15T03:46:40.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.378 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604199+0000", 2024-09-15T03:46:40.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6820999999999997e-05 2024-09-15T03:46:40.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:40.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604497+0000", 2024-09-15T03:46:40.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029777299999999998 2024-09-15T03:46:40.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:40.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604617+0000", 2024-09-15T03:46:40.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012054599999999999 2024-09-15T03:46:40.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:40.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604642+0000", 2024-09-15T03:46:40.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4513000000000001e-05 2024-09-15T03:46:40.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604660+0000", 2024-09-15T03:46:40.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8672999999999999e-05 2024-09-15T03:46:40.380 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.380 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.380 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.380 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:40.380 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.600953+0000", 2024-09-15T03:46:40.380 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.718536223000001, 2024-09-15T03:46:40.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003775086, 2024-09-15T03:46:40.381 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.381 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:40.381 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:40.381 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:40.381 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:40.381 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 76 2024-09-15T03:46:40.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.381 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600953+0000", 2024-09-15T03:46:40.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600953+0000", 2024-09-15T03:46:40.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600953+0000", 2024-09-15T03:46:40.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8100000000000002e-07 2024-09-15T03:46:40.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600957+0000", 2024-09-15T03:46:40.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1579999999999998e-06 2024-09-15T03:46:40.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600964+0000", 2024-09-15T03:46:40.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3219999999999998e-06 2024-09-15T03:46:40.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.600967+0000", 2024-09-15T03:46:40.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6339999999999999e-06 2024-09-15T03:46:40.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604177+0000", 2024-09-15T03:46:40.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00320967 2024-09-15T03:46:40.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:40.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604226+0000", 2024-09-15T03:46:40.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8646000000000001e-05 2024-09-15T03:46:40.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:40.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604297+0000", 2024-09-15T03:46:40.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1390000000000006e-05 2024-09-15T03:46:40.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604383+0000", 2024-09-15T03:46:40.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5927999999999995e-05 2024-09-15T03:46:40.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604392+0000", 2024-09-15T03:46:40.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.799e-06 2024-09-15T03:46:40.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604418+0000", 2024-09-15T03:46:40.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6585999999999999e-05 2024-09-15T03:46:40.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:40.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604506+0000", 2024-09-15T03:46:40.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7806000000000006e-05 2024-09-15T03:46:40.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:40.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604683+0000", 2024-09-15T03:46:40.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000176971 2024-09-15T03:46:40.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:40.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604709+0000", 2024-09-15T03:46:40.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6044e-05 2024-09-15T03:46:40.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.604728+0000", 2024-09-15T03:46:40.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8451e-05 2024-09-15T03:46:40.387 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.387 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.387 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.387 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:96 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-15T03:46:40.387 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:19.008531+0000", 2024-09-15T03:46:40.388 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.310957807000001, 2024-09-15T03:46:40.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0034252969999999999, 2024-09-15T03:46:40.388 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.388 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:40.388 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:40.388 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:40.388 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:40.388 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 96 2024-09-15T03:46:40.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.388 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008531+0000", 2024-09-15T03:46:40.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008531+0000", 2024-09-15T03:46:40.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008531+0000", 2024-09-15T03:46:40.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3900000000000001e-07 2024-09-15T03:46:40.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008596+0000", 2024-09-15T03:46:40.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4475999999999997e-05 2024-09-15T03:46:40.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008603+0000", 2024-09-15T03:46:40.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3749999999999997e-06 2024-09-15T03:46:40.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008608+0000", 2024-09-15T03:46:40.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8600000000000001e-06 2024-09-15T03:46:40.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.010839+0000", 2024-09-15T03:46:40.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022307889999999999 2024-09-15T03:46:40.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:40.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.010881+0000", 2024-09-15T03:46:40.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1766e-05 2024-09-15T03:46:40.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:40.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.010945+0000", 2024-09-15T03:46:40.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4196000000000007e-05 2024-09-15T03:46:40.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011025+0000", 2024-09-15T03:46:40.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9818e-05 2024-09-15T03:46:40.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011032+0000", 2024-09-15T03:46:40.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1019999999999999e-06 2024-09-15T03:46:40.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011083+0000", 2024-09-15T03:46:40.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1687999999999997e-05 2024-09-15T03:46:40.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:40.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011402+0000", 2024-09-15T03:46:40.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00031831600000000002 2024-09-15T03:46:40.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:40.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011899+0000", 2024-09-15T03:46:40.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00049779100000000003 2024-09-15T03:46:40.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:40.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011924+0000", 2024-09-15T03:46:40.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4595000000000001e-05 2024-09-15T03:46:40.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011956+0000", 2024-09-15T03:46:40.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2286000000000002e-05 2024-09-15T03:46:40.394 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.394 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.394 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.395 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-15T03:46:40.395 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:19.008611+0000", 2024-09-15T03:46:40.395 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.310877462000001, 2024-09-15T03:46:40.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00344955, 2024-09-15T03:46:40.395 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.395 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:40.395 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:40.395 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:40.395 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:40.395 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 97 2024-09-15T03:46:40.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.395 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008611+0000", 2024-09-15T03:46:40.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008611+0000", 2024-09-15T03:46:40.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008612+0000", 2024-09-15T03:46:40.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.39e-07 2024-09-15T03:46:40.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008616+0000", 2024-09-15T03:46:40.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0350000000000003e-06 2024-09-15T03:46:40.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008619+0000", 2024-09-15T03:46:40.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4359999999999998e-06 2024-09-15T03:46:40.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008626+0000", 2024-09-15T03:46:40.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1609999999999997e-06 2024-09-15T03:46:40.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011071+0000", 2024-09-15T03:46:40.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024443569999999999 2024-09-15T03:46:40.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:40.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011126+0000", 2024-09-15T03:46:40.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5300000000000002e-05 2024-09-15T03:46:40.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:40.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011205+0000", 2024-09-15T03:46:40.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9221000000000006e-05 2024-09-15T03:46:40.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011307+0000", 2024-09-15T03:46:40.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010136000000000001 2024-09-15T03:46:40.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011313+0000", 2024-09-15T03:46:40.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.844e-06 2024-09-15T03:46:40.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011336+0000", 2024-09-15T03:46:40.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2993000000000002e-05 2024-09-15T03:46:40.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:40.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011407+0000", 2024-09-15T03:46:40.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1002999999999998e-05 2024-09-15T03:46:40.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:40.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012007+0000", 2024-09-15T03:46:40.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000599652 2024-09-15T03:46:40.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:40.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012029+0000", 2024-09-15T03:46:40.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2067999999999999e-05 2024-09-15T03:46:40.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012061+0000", 2024-09-15T03:46:40.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1780999999999999e-05 2024-09-15T03:46:40.401 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.402 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.402 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.402 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-15T03:46:40.402 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:19.008629+0000", 2024-09-15T03:46:40.402 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.310859490999999, 2024-09-15T03:46:40.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0035108140000000001, 2024-09-15T03:46:40.402 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.402 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:40.402 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:40.402 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:40.402 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:40.402 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 98 2024-09-15T03:46:40.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.403 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008629+0000", 2024-09-15T03:46:40.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008629+0000", 2024-09-15T03:46:40.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008630+0000", 2024-09-15T03:46:40.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2100000000000001e-07 2024-09-15T03:46:40.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008633+0000", 2024-09-15T03:46:40.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.366e-06 2024-09-15T03:46:40.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008636+0000", 2024-09-15T03:46:40.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3330000000000001e-06 2024-09-15T03:46:40.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008638+0000", 2024-09-15T03:46:40.405 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.024e-06 2024-09-15T03:46:40.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011420+0000", 2024-09-15T03:46:40.405 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027817599999999999 2024-09-15T03:46:40.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:40.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011461+0000", 2024-09-15T03:46:40.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1180000000000002e-05 2024-09-15T03:46:40.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.406 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.406 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:40.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011525+0000", 2024-09-15T03:46:40.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3917000000000005e-05 2024-09-15T03:46:40.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.406 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.406 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011605+0000", 2024-09-15T03:46:40.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9813000000000003e-05 2024-09-15T03:46:40.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.407 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.407 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011612+0000", 2024-09-15T03:46:40.407 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5880000000000003e-06 2024-09-15T03:46:40.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.407 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.407 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011632+0000", 2024-09-15T03:46:40.407 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0882000000000001e-05 2024-09-15T03:46:40.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.407 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:40.407 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011977+0000", 2024-09-15T03:46:40.407 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034493000000000002 2024-09-15T03:46:40.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.408 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:40.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012095+0000", 2024-09-15T03:46:40.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000117791 2024-09-15T03:46:40.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.408 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:40.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012121+0000", 2024-09-15T03:46:40.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5729e-05 2024-09-15T03:46:40.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012140+0000", 2024-09-15T03:46:40.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9279999999999998e-05 2024-09-15T03:46:40.409 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.409 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.409 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.409 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-15T03:46:40.409 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:19.008643+0000", 2024-09-15T03:46:40.409 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.310846248000001, 2024-09-15T03:46:40.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003782948, 2024-09-15T03:46:40.409 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.409 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:40.410 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:40.410 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:40.410 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:40.410 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 99 2024-09-15T03:46:40.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.410 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008643+0000", 2024-09-15T03:46:40.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008643+0000", 2024-09-15T03:46:40.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008643+0000", 2024-09-15T03:46:40.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2499999999999999e-07 2024-09-15T03:46:40.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008717+0000", 2024-09-15T03:46:40.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4688e-05 2024-09-15T03:46:40.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008726+0000", 2024-09-15T03:46:40.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2109999999999998e-06 2024-09-15T03:46:40.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008730+0000", 2024-09-15T03:46:40.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7169999999999999e-06 2024-09-15T03:46:40.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011646+0000", 2024-09-15T03:46:40.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0029156759999999999 2024-09-15T03:46:40.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.413 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:40.413 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011698+0000", 2024-09-15T03:46:40.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1918e-05 2024-09-15T03:46:40.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.413 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:40.413 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011768+0000", 2024-09-15T03:46:40.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9741999999999997e-05 2024-09-15T03:46:40.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.413 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.413 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011858+0000", 2024-09-15T03:46:40.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0117999999999993e-05 2024-09-15T03:46:40.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011864+0000", 2024-09-15T03:46:40.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5980000000000002e-06 2024-09-15T03:46:40.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011914+0000", 2024-09-15T03:46:40.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9428000000000002e-05 2024-09-15T03:46:40.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:40.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.011987+0000", 2024-09-15T03:46:40.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2873999999999994e-05 2024-09-15T03:46:40.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:40.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012383+0000", 2024-09-15T03:46:40.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039653399999999998 2024-09-15T03:46:40.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:40.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012408+0000", 2024-09-15T03:46:40.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4890000000000001e-05 2024-09-15T03:46:40.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012426+0000", 2024-09-15T03:46:40.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7329e-05 2024-09-15T03:46:40.416 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.416 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.416 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.416 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-15T03:46:40.416 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:19.008737+0000", 2024-09-15T03:46:40.416 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.310752263000001, 2024-09-15T03:46:40.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003912993, 2024-09-15T03:46:40.417 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.417 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:40.417 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:40.417 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:40.417 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:40.417 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 100 2024-09-15T03:46:40.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.417 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008737+0000", 2024-09-15T03:46:40.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.418 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008737+0000", 2024-09-15T03:46:40.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.418 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008737+0000", 2024-09-15T03:46:40.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4700000000000001e-07 2024-09-15T03:46:40.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.418 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008746+0000", 2024-09-15T03:46:40.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7229999999999993e-06 2024-09-15T03:46:40.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.419 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008756+0000", 2024-09-15T03:46:40.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6800000000000005e-06 2024-09-15T03:46:40.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.419 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.008760+0000", 2024-09-15T03:46:40.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9210000000000002e-06 2024-09-15T03:46:40.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012160+0000", 2024-09-15T03:46:40.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0034000469999999998 2024-09-15T03:46:40.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:40.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012205+0000", 2024-09-15T03:46:40.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5405000000000003e-05 2024-09-15T03:46:40.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:40.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012269+0000", 2024-09-15T03:46:40.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4068999999999999e-05 2024-09-15T03:46:40.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012346+0000", 2024-09-15T03:46:40.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6903999999999994e-05 2024-09-15T03:46:40.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012353+0000", 2024-09-15T03:46:40.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3100000000000003e-06 2024-09-15T03:46:40.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012377+0000", 2024-09-15T03:46:40.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3730000000000001e-05 2024-09-15T03:46:40.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:40.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012442+0000", 2024-09-15T03:46:40.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4827999999999997e-05 2024-09-15T03:46:40.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:40.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012595+0000", 2024-09-15T03:46:40.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015312 2024-09-15T03:46:40.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:40.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012618+0000", 2024-09-15T03:46:40.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3045e-05 2024-09-15T03:46:40.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:19.012650+0000", 2024-09-15T03:46:40.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1563999999999998e-05 2024-09-15T03:46:40.423 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.423 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.423 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.424 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.424 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4263.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2024-09-15T03:46:40.424 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:46:40.424 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.275738179000001, 2024-09-15T03:46:40.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019495788999999999, 2024-09-15T03:46:40.424 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.424 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.424 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:40.424 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4263", 2024-09-15T03:46:40.424 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/646873808", 2024-09-15T03:46:40.424 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2 2024-09-15T03:46:40.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.425 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.425 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.425 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.425 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:46:40.425 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.425 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.425 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.425 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.425 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:46:40.425 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.425 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.425 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.425 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.426 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043752+0000", 2024-09-15T03:46:40.426 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.815e-06 2024-09-15T03:46:40.426 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.426 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.426 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.426 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043757+0000", 2024-09-15T03:46:40.426 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4240000000000003e-06 2024-09-15T03:46:40.426 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.426 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.426 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.426 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043766+0000", 2024-09-15T03:46:40.426 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8629999999999997e-06 2024-09-15T03:46:40.426 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.426 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043773+0000", 2024-09-15T03:46:40.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0099999999999998e-06 2024-09-15T03:46:40.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.427 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043830+0000", 2024-09-15T03:46:40.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7089999999999997e-05 2024-09-15T03:46:40.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.427 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:46:40.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043837+0000", 2024-09-15T03:46:40.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.08e-06 2024-09-15T03:46:40.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063103+0000", 2024-09-15T03:46:40.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019265619000000001 2024-09-15T03:46:40.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063246+0000", 2024-09-15T03:46:40.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000143888 2024-09-15T03:46:40.428 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.428 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.428 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.429 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s4 32/31 ECSubWrite(tid=44, reqid=client.4521.0:173, at_version=32'7, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:40.429 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.562108+0000", 2024-09-15T03:46:40.429 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.75738063499999997, 2024-09-15T03:46:40.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0039284979999999999, 2024-09-15T03:46:40.429 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.429 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.429 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562108+0000", 2024-09-15T03:46:40.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562108+0000", 2024-09-15T03:46:40.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562109+0000", 2024-09-15T03:46:40.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.85e-07 2024-09-15T03:46:40.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562489+0000", 2024-09-15T03:46:40.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00038030300000000002 2024-09-15T03:46:40.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562500+0000", 2024-09-15T03:46:40.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0607e-05 2024-09-15T03:46:40.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562505+0000", 2024-09-15T03:46:40.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8849999999999998e-06 2024-09-15T03:46:40.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.432 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.432 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562548+0000", 2024-09-15T03:46:40.432 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3541999999999997e-05 2024-09-15T03:46:40.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.432 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.432 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562558+0000", 2024-09-15T03:46:40.432 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2830000000000007e-06 2024-09-15T03:46:40.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.432 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.432 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562667+0000", 2024-09-15T03:46:40.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000109016 2024-09-15T03:46:40.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562676+0000", 2024-09-15T03:46:40.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3689999999999996e-06 2024-09-15T03:46:40.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562771+0000", 2024-09-15T03:46:40.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4659000000000003e-05 2024-09-15T03:46:40.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562781+0000", 2024-09-15T03:46:40.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0233e-05 2024-09-15T03:46:40.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.566037+0000", 2024-09-15T03:46:40.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032557160000000001 2024-09-15T03:46:40.434 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.434 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.434 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.435 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.fs5 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:40.435 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:46:40.435 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.21189027299999999, 2024-09-15T03:46:40.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098369180000000001, 2024-09-15T03:46:40.435 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.435 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.435 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:46:40.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:46:40.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:46:40.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4499999999999998e-07 2024-09-15T03:46:40.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107630+0000", 2024-09-15T03:46:40.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0877999999999998e-05 2024-09-15T03:46:40.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107642+0000", 2024-09-15T03:46:40.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2567e-05 2024-09-15T03:46:40.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107648+0000", 2024-09-15T03:46:40.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8690000000000001e-06 2024-09-15T03:46:40.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.438 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107894+0000", 2024-09-15T03:46:40.438 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024540400000000001 2024-09-15T03:46:40.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.438 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.438 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.438 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107900+0000", 2024-09-15T03:46:40.438 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5710000000000003e-06 2024-09-15T03:46:40.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.438 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.438 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.438 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107968+0000", 2024-09-15T03:46:40.438 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7957e-05 2024-09-15T03:46:40.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108274+0000", 2024-09-15T03:46:40.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030591700000000001 2024-09-15T03:46:40.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108504+0000", 2024-09-15T03:46:40.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022952199999999999 2024-09-15T03:46:40.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109438+0000", 2024-09-15T03:46:40.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00093439100000000002 2024-09-15T03:46:40.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.440 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117435+0000", 2024-09-15T03:46:40.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0079974969999999992 2024-09-15T03:46:40.440 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.440 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.440 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.440 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:40.440 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:46:40.440 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.21183302100000001, 2024-09-15T03:46:40.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084894540000000004, 2024-09-15T03:46:40.441 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.441 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.441 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:46:40.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:46:40.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.442 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.442 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.442 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:46:40.442 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4700000000000002e-07 2024-09-15T03:46:40.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107670+0000", 2024-09-15T03:46:40.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3892e-05 2024-09-15T03:46:40.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107678+0000", 2024-09-15T03:46:40.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9589999999999995e-06 2024-09-15T03:46:40.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107683+0000", 2024-09-15T03:46:40.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7199999999999997e-06 2024-09-15T03:46:40.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107768+0000", 2024-09-15T03:46:40.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4968000000000004e-05 2024-09-15T03:46:40.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.446 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107776+0000", 2024-09-15T03:46:40.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9109999999999993e-06 2024-09-15T03:46:40.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.446 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107858+0000", 2024-09-15T03:46:40.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2349999999999996e-05 2024-09-15T03:46:40.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.446 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108273+0000", 2024-09-15T03:46:40.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00041525600000000002 2024-09-15T03:46:40.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108503+0000", 2024-09-15T03:46:40.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022990599999999999 2024-09-15T03:46:40.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108705+0000", 2024-09-15T03:46:40.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020173000000000001 2024-09-15T03:46:40.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.116145+0000", 2024-09-15T03:46:40.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0074404149999999997 2024-09-15T03:46:40.448 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.448 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.448 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.448 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.fs5 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:40.448 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.109087+0000", 2024-09-15T03:46:40.448 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.21040202699999999, 2024-09-15T03:46:40.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084235809999999994, 2024-09-15T03:46:40.448 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.448 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.448 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109087+0000", 2024-09-15T03:46:40.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109087+0000", 2024-09-15T03:46:40.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109087+0000", 2024-09-15T03:46:40.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7599999999999997e-07 2024-09-15T03:46:40.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109107+0000", 2024-09-15T03:46:40.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9511999999999999e-05 2024-09-15T03:46:40.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109127+0000", 2024-09-15T03:46:40.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0268e-05 2024-09-15T03:46:40.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109136+0000", 2024-09-15T03:46:40.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1330000000000005e-06 2024-09-15T03:46:40.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109594+0000", 2024-09-15T03:46:40.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045802100000000002 2024-09-15T03:46:40.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109602+0000", 2024-09-15T03:46:40.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7020000000000001e-06 2024-09-15T03:46:40.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109678+0000", 2024-09-15T03:46:40.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5428999999999999e-05 2024-09-15T03:46:40.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109699+0000", 2024-09-15T03:46:40.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1759999999999998e-05 2024-09-15T03:46:40.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109728+0000", 2024-09-15T03:46:40.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8886000000000001e-05 2024-09-15T03:46:40.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109736+0000", 2024-09-15T03:46:40.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6310000000000003e-06 2024-09-15T03:46:40.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117510+0000", 2024-09-15T03:46:40.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.007774563 2024-09-15T03:46:40.453 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.453 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.453 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.454 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:40.454 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.109145+0000", 2024-09-15T03:46:40.454 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.210344119, 2024-09-15T03:46:40.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075088480000000003, 2024-09-15T03:46:40.454 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.454 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.454 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109145+0000", 2024-09-15T03:46:40.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109145+0000", 2024-09-15T03:46:40.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109145+0000", 2024-09-15T03:46:40.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7399999999999999e-07 2024-09-15T03:46:40.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109160+0000", 2024-09-15T03:46:40.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4544e-05 2024-09-15T03:46:40.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109168+0000", 2024-09-15T03:46:40.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9200000000000004e-06 2024-09-15T03:46:40.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109172+0000", 2024-09-15T03:46:40.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.498e-06 2024-09-15T03:46:40.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109221+0000", 2024-09-15T03:46:40.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9348000000000003e-05 2024-09-15T03:46:40.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109230+0000", 2024-09-15T03:46:40.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2020000000000004e-06 2024-09-15T03:46:40.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109314+0000", 2024-09-15T03:46:40.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4445999999999998e-05 2024-09-15T03:46:40.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109570+0000", 2024-09-15T03:46:40.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00025616300000000001 2024-09-15T03:46:40.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109642+0000", 2024-09-15T03:46:40.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1600999999999994e-05 2024-09-15T03:46:40.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109653+0000", 2024-09-15T03:46:40.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1048e-05 2024-09-15T03:46:40.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.459 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.116654+0000", 2024-09-15T03:46:40.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.007000704 2024-09-15T03:46:40.459 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.459 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.459 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.459 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.459 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.459 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:40.461 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:46:40.461 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:40.460+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:46:40.558 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:40.558 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:40.558 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:46:40.558 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:40.561 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:40.561+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:46:40.566 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:46:40.662 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:40.662 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:40.662 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:46:40.662 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:46:40.662 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:40.663 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:40.660+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:40.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:46:40.763 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:40.763+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:46:40.808 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:40.808 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:46:40.808 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:46:40.808 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:46:40.808 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.808 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4263.0:1 2.3 e22/21 2:c4c92e5a:::rbd_trash:head v 22'1, mlcod=0'0)", 2024-09-15T03:46:40.809 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:24.039192+0000", 2024-09-15T03:46:40.809 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.767211361000001, 2024-09-15T03:46:40.809 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001176968, 2024-09-15T03:46:40.809 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.809 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:40.809 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039192+0000", 2024-09-15T03:46:40.809 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.810 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039192+0000", 2024-09-15T03:46:40.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.810 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.810 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039192+0000", 2024-09-15T03:46:40.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3999999999999997e-07 2024-09-15T03:46:40.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.810 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.810 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039199+0000", 2024-09-15T03:46:40.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2580000000000001e-06 2024-09-15T03:46:40.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.811 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.811 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039217+0000", 2024-09-15T03:46:40.811 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8895000000000001e-05 2024-09-15T03:46:40.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.811 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.811 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039226+0000", 2024-09-15T03:46:40.811 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4840000000000004e-06 2024-09-15T03:46:40.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.811 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.811 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039268+0000", 2024-09-15T03:46:40.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2104000000000002e-05 2024-09-15T03:46:40.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for flush", 2024-09-15T03:46:40.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039283+0000", 2024-09-15T03:46:40.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5009e-05 2024-09-15T03:46:40.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040042+0000", 2024-09-15T03:46:40.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00075874199999999997 2024-09-15T03:46:40.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:40.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040067+0000", 2024-09-15T03:46:40.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4888999999999998e-05 2024-09-15T03:46:40.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040169+0000", 2024-09-15T03:46:40.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010252299999999999 2024-09-15T03:46:40.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040178+0000", 2024-09-15T03:46:40.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7709999999999995e-06 2024-09-15T03:46:40.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.814 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040211+0000", 2024-09-15T03:46:40.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3552e-05 2024-09-15T03:46:40.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.814 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:40.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040226+0000", 2024-09-15T03:46:40.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4695000000000001e-05 2024-09-15T03:46:40.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.814 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040369+0000", 2024-09-15T03:46:40.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014270600000000001 2024-09-15T03:46:40.815 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.815 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.815 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.815 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.815 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 32/31 ECSubWrite(tid=31, reqid=client.4521.0:159, at_version=32'14, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:46:40.815 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.530266+0000", 2024-09-15T03:46:40.815 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.27613738, 2024-09-15T03:46:40.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00059373300000000004, 2024-09-15T03:46:40.815 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.815 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.815 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.816 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.530266+0000", 2024-09-15T03:46:40.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.816 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.530266+0000", 2024-09-15T03:46:40.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.816 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.530267+0000", 2024-09-15T03:46:40.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6899999999999997e-07 2024-09-15T03:46:40.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.530442+0000", 2024-09-15T03:46:40.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017512800000000001 2024-09-15T03:46:40.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.530451+0000", 2024-09-15T03:46:40.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4299999999999995e-06 2024-09-15T03:46:40.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.530457+0000", 2024-09-15T03:46:40.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6470000000000003e-06 2024-09-15T03:46:40.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.818 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.530499+0000", 2024-09-15T03:46:40.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2172999999999998e-05 2024-09-15T03:46:40.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.818 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.530508+0000", 2024-09-15T03:46:40.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9500000000000007e-06 2024-09-15T03:46:40.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.530615+0000", 2024-09-15T03:46:40.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000106648 2024-09-15T03:46:40.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.530621+0000", 2024-09-15T03:46:40.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4189999999999999e-06 2024-09-15T03:46:40.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.530687+0000", 2024-09-15T03:46:40.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6217999999999995e-05 2024-09-15T03:46:40.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.530700+0000", 2024-09-15T03:46:40.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2962000000000001e-05 2024-09-15T03:46:40.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.530860+0000", 2024-09-15T03:46:40.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015948899999999999 2024-09-15T03:46:40.820 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.820 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.820 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.821 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 32/31 ECSubWrite(tid=32, reqid=client.4521.0:160, at_version=32'15, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:46:40.821 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.530979+0000", 2024-09-15T03:46:40.821 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.2754242149999999, 2024-09-15T03:46:40.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00025472800000000002, 2024-09-15T03:46:40.821 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.821 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.821 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.821 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.530979+0000", 2024-09-15T03:46:40.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.822 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.530979+0000", 2024-09-15T03:46:40.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.822 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.530980+0000", 2024-09-15T03:46:40.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.06e-07 2024-09-15T03:46:40.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.822 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.530992+0000", 2024-09-15T03:46:40.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2133999999999999e-05 2024-09-15T03:46:40.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.531002+0000", 2024-09-15T03:46:40.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8779999999999993e-06 2024-09-15T03:46:40.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.531006+0000", 2024-09-15T03:46:40.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3490000000000002e-06 2024-09-15T03:46:40.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.531038+0000", 2024-09-15T03:46:40.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2332000000000001e-05 2024-09-15T03:46:40.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.531044+0000", 2024-09-15T03:46:40.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4260000000000002e-06 2024-09-15T03:46:40.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.531102+0000", 2024-09-15T03:46:40.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8017999999999999e-05 2024-09-15T03:46:40.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.531108+0000", 2024-09-15T03:46:40.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1789999999999996e-06 2024-09-15T03:46:40.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.531134+0000", 2024-09-15T03:46:40.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5799000000000002e-05 2024-09-15T03:46:40.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.531142+0000", 2024-09-15T03:46:40.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4889999999999995e-06 2024-09-15T03:46:40.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.531234+0000", 2024-09-15T03:46:40.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1618000000000003e-05 2024-09-15T03:46:40.826 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.826 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.826 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.827 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 32/31 ECSubWrite(tid=42, reqid=client.4521.0:166, at_version=32'13, trim_to=0'0, roll_forward_to=32'12))", 2024-09-15T03:46:40.827 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.532201+0000", 2024-09-15T03:46:40.827 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.27420226, 2024-09-15T03:46:40.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0036328060000000001, 2024-09-15T03:46:40.827 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.827 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.827 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532201+0000", 2024-09-15T03:46:40.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532201+0000", 2024-09-15T03:46:40.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532202+0000", 2024-09-15T03:46:40.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.44e-07 2024-09-15T03:46:40.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532214+0000", 2024-09-15T03:46:40.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2689e-05 2024-09-15T03:46:40.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.829 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.829 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532224+0000", 2024-09-15T03:46:40.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3549999999999994e-06 2024-09-15T03:46:40.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.829 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.829 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532228+0000", 2024-09-15T03:46:40.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8619999999999996e-06 2024-09-15T03:46:40.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.829 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.830 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532268+0000", 2024-09-15T03:46:40.830 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0899000000000003e-05 2024-09-15T03:46:40.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.830 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.830 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.830 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532277+0000", 2024-09-15T03:46:40.830 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6410000000000008e-06 2024-09-15T03:46:40.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.830 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.830 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.830 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532354+0000", 2024-09-15T03:46:40.830 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6904999999999996e-05 2024-09-15T03:46:40.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532361+0000", 2024-09-15T03:46:40.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.798e-06 2024-09-15T03:46:40.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532387+0000", 2024-09-15T03:46:40.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6346000000000002e-05 2024-09-15T03:46:40.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532400+0000", 2024-09-15T03:46:40.832 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2655e-05 2024-09-15T03:46:40.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.832 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.535834+0000", 2024-09-15T03:46:40.832 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0034340120000000002 2024-09-15T03:46:40.832 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.832 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.832 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.832 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 32/31 ECSubWrite(tid=33, reqid=client.4521.0:160, at_version=0'0, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:46:40.832 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.532332+0000", 2024-09-15T03:46:40.832 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.2740708199999999, 2024-09-15T03:46:40.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023696700000000001, 2024-09-15T03:46:40.833 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.833 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.833 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.833 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532332+0000", 2024-09-15T03:46:40.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.833 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532332+0000", 2024-09-15T03:46:40.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.834 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.834 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.834 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532333+0000", 2024-09-15T03:46:40.834 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4700000000000002e-07 2024-09-15T03:46:40.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.834 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.834 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.834 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532338+0000", 2024-09-15T03:46:40.834 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5600000000000001e-06 2024-09-15T03:46:40.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.834 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.834 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.835 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532344+0000", 2024-09-15T03:46:40.835 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6779999999999997e-06 2024-09-15T03:46:40.835 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.835 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.835 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.835 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532347+0000", 2024-09-15T03:46:40.835 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7650000000000002e-06 2024-09-15T03:46:40.835 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.835 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.835 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.835 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532379+0000", 2024-09-15T03:46:40.835 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2605999999999999e-05 2024-09-15T03:46:40.835 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.835 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.835 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532388+0000", 2024-09-15T03:46:40.836 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8659999999999995e-06 2024-09-15T03:46:40.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.836 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.836 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532462+0000", 2024-09-15T03:46:40.836 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3512999999999999e-05 2024-09-15T03:46:40.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.836 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.836 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532472+0000", 2024-09-15T03:46:40.836 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0434999999999999e-05 2024-09-15T03:46:40.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.837 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.837 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.837 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532506+0000", 2024-09-15T03:46:40.837 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3491000000000002e-05 2024-09-15T03:46:40.837 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.837 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.837 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.837 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532515+0000", 2024-09-15T03:46:40.837 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4080000000000004e-06 2024-09-15T03:46:40.837 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.837 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.837 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.837 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532569+0000", 2024-09-15T03:46:40.837 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4298000000000001e-05 2024-09-15T03:46:40.838 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.838 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.838 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.838 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.838 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.838 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 32/31 ECSubWrite(tid=43, reqid=client.4521.0:166, at_version=0'0, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:46:40.838 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.534156+0000", 2024-09-15T03:46:40.838 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.272247471, 2024-09-15T03:46:40.838 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001796693, 2024-09-15T03:46:40.838 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.838 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.838 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.838 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.838 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.534156+0000", 2024-09-15T03:46:40.839 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.839 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.839 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.534156+0000", 2024-09-15T03:46:40.839 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.839 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.839 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.534156+0000", 2024-09-15T03:46:40.839 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8699999999999995e-07 2024-09-15T03:46:40.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.839 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.840 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.840 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.534168+0000", 2024-09-15T03:46:40.840 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1698e-05 2024-09-15T03:46:40.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.840 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.840 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.840 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.534178+0000", 2024-09-15T03:46:40.840 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0086e-05 2024-09-15T03:46:40.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.840 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.840 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.840 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.534182+0000", 2024-09-15T03:46:40.840 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.194e-06 2024-09-15T03:46:40.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.841 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.841 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.841 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.534216+0000", 2024-09-15T03:46:40.841 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3762e-05 2024-09-15T03:46:40.841 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.841 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.841 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.841 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.534221+0000", 2024-09-15T03:46:40.841 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3430000000000003e-06 2024-09-15T03:46:40.841 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.841 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.841 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.841 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.534269+0000", 2024-09-15T03:46:40.842 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7256999999999998e-05 2024-09-15T03:46:40.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.842 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.842 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.842 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.534275+0000", 2024-09-15T03:46:40.842 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4219999999999997e-06 2024-09-15T03:46:40.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.842 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.842 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.842 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.535553+0000", 2024-09-15T03:46:40.842 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001277423 2024-09-15T03:46:40.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.842 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.842 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.843 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.535586+0000", 2024-09-15T03:46:40.843 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3781000000000001e-05 2024-09-15T03:46:40.843 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.843 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.843 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.843 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.535953+0000", 2024-09-15T03:46:40.843 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036613999999999999 2024-09-15T03:46:40.843 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.843 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.843 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.843 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.843 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.844 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.0s2 32/31 ECSubWrite(tid=21, reqid=client.4521.0:167, at_version=32'10, trim_to=0'0, roll_forward_to=32'9))", 2024-09-15T03:46:40.844 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.552275+0000", 2024-09-15T03:46:40.844 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.254127958, 2024-09-15T03:46:40.844 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00070308299999999996, 2024-09-15T03:46:40.844 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.844 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.844 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.844 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.844 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.844 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.552275+0000", 2024-09-15T03:46:40.844 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.844 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.845 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.845 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.845 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.552275+0000", 2024-09-15T03:46:40.845 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.845 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.845 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.845 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.845 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.552277+0000", 2024-09-15T03:46:40.845 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3969999999999999e-06 2024-09-15T03:46:40.845 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.845 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.845 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.845 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.552550+0000", 2024-09-15T03:46:40.845 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027286500000000001 2024-09-15T03:46:40.846 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.846 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.846 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.846 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.552561+0000", 2024-09-15T03:46:40.846 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1058e-05 2024-09-15T03:46:40.846 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.846 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.846 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.846 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.552565+0000", 2024-09-15T03:46:40.846 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2989999999999998e-06 2024-09-15T03:46:40.846 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.846 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.846 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.847 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.552602+0000", 2024-09-15T03:46:40.847 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6588999999999998e-05 2024-09-15T03:46:40.847 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.847 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.847 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.847 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.552608+0000", 2024-09-15T03:46:40.847 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5440000000000004e-06 2024-09-15T03:46:40.847 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.847 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.847 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.847 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.552724+0000", 2024-09-15T03:46:40.847 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011629399999999999 2024-09-15T03:46:40.847 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.847 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.848 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.848 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.552729+0000", 2024-09-15T03:46:40.848 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0130000000000001e-06 2024-09-15T03:46:40.848 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.848 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.848 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.848 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.552805+0000", 2024-09-15T03:46:40.848 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5585000000000002e-05 2024-09-15T03:46:40.848 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.848 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.848 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.848 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.552818+0000", 2024-09-15T03:46:40.848 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3155e-05 2024-09-15T03:46:40.848 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.849 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.849 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.849 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.552978+0000", 2024-09-15T03:46:40.849 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016028400000000001 2024-09-15T03:46:40.849 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.849 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.849 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.849 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.849 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.849 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:175 3.cs0 3:3164c049:::smithi00132172-34:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e32)", 2024-09-15T03:46:40.849 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.553220+0000", 2024-09-15T03:46:40.849 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.2531830429999999, 2024-09-15T03:46:40.849 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0052725100000000002, 2024-09-15T03:46:40.849 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.850 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:40.850 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:40.850 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:40.850 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:40.850 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 175 2024-09-15T03:46:40.850 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.850 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.850 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.850 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.850 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553220+0000", 2024-09-15T03:46:40.850 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.850 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.851 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.851 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.851 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553220+0000", 2024-09-15T03:46:40.851 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.851 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.851 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.851 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553221+0000", 2024-09-15T03:46:40.851 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0860000000000001e-06 2024-09-15T03:46:40.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.851 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.851 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.852 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553226+0000", 2024-09-15T03:46:40.852 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1630000000000004e-06 2024-09-15T03:46:40.852 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.852 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.852 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.852 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553239+0000", 2024-09-15T03:46:40.852 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2187e-05 2024-09-15T03:46:40.852 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.852 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.852 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.852 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553244+0000", 2024-09-15T03:46:40.852 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8960000000000002e-06 2024-09-15T03:46:40.852 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.853 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.853 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.853 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553276+0000", 2024-09-15T03:46:40.853 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2237000000000003e-05 2024-09-15T03:46:40.853 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.853 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.853 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:40.853 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553345+0000", 2024-09-15T03:46:40.853 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9618999999999999e-05 2024-09-15T03:46:40.853 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.853 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.853 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.854 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553695+0000", 2024-09-15T03:46:40.854 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034979 2024-09-15T03:46:40.854 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.854 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.854 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.854 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553780+0000", 2024-09-15T03:46:40.854 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4633000000000001e-05 2024-09-15T03:46:40.854 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.854 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.854 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.854 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553787+0000", 2024-09-15T03:46:40.854 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2409999999999998e-06 2024-09-15T03:46:40.854 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.854 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.855 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.855 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553828+0000", 2024-09-15T03:46:40.855 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0564e-05 2024-09-15T03:46:40.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.855 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.855 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553841+0000", 2024-09-15T03:46:40.855 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3249e-05 2024-09-15T03:46:40.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.855 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:40.855 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.554266+0000", 2024-09-15T03:46:40.855 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00042530299999999998 2024-09-15T03:46:40.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.856 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.856 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.558493+0000", 2024-09-15T03:46:40.856 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042265419999999998 2024-09-15T03:46:40.856 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.856 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.856 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.856 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.856 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.0s2 32/31 ECSubWrite(tid=22, reqid=client.4521.0:167, at_version=0'0, trim_to=0'0, roll_forward_to=32'10))", 2024-09-15T03:46:40.856 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.553350+0000", 2024-09-15T03:46:40.856 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.2530534260000001, 2024-09-15T03:46:40.856 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020506100000000001, 2024-09-15T03:46:40.857 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.857 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.857 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.857 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.857 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.857 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553350+0000", 2024-09-15T03:46:40.857 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.857 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.857 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.857 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.857 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553350+0000", 2024-09-15T03:46:40.857 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.857 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.857 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.858 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.858 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553350+0000", 2024-09-15T03:46:40.858 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7100000000000002e-07 2024-09-15T03:46:40.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.858 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.858 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.858 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553359+0000", 2024-09-15T03:46:40.858 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5399999999999996e-06 2024-09-15T03:46:40.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.858 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.858 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.858 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553369+0000", 2024-09-15T03:46:40.858 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.859e-06 2024-09-15T03:46:40.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.859 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.859 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.859 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553374+0000", 2024-09-15T03:46:40.859 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0969999999999996e-06 2024-09-15T03:46:40.859 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.859 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.859 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.859 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553405+0000", 2024-09-15T03:46:40.859 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1343999999999997e-05 2024-09-15T03:46:40.859 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.859 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.859 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.859 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553410+0000", 2024-09-15T03:46:40.859 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0359999999999997e-06 2024-09-15T03:46:40.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.860 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.860 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.860 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553461+0000", 2024-09-15T03:46:40.860 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0260999999999999e-05 2024-09-15T03:46:40.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.860 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.860 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.860 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553469+0000", 2024-09-15T03:46:40.860 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6510000000000007e-06 2024-09-15T03:46:40.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.860 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.860 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.861 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553502+0000", 2024-09-15T03:46:40.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2972999999999998e-05 2024-09-15T03:46:40.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.861 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.861 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.861 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553513+0000", 2024-09-15T03:46:40.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.096e-05 2024-09-15T03:46:40.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.861 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.861 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.861 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553555+0000", 2024-09-15T03:46:40.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1768999999999999e-05 2024-09-15T03:46:40.861 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.861 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.862 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.862 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.862 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.862 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 32/28 ECSubWriteReply(tid=8, last_complete=32'5, committed=1, applied=1))", 2024-09-15T03:46:40.862 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.554153+0000", 2024-09-15T03:46:40.862 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.2522506250000001, 2024-09-15T03:46:40.862 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029763800000000003, 2024-09-15T03:46:40.862 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.862 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.862 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.862 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.862 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.862 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.554153+0000", 2024-09-15T03:46:40.862 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.863 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.863 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.863 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.554153+0000", 2024-09-15T03:46:40.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.863 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.863 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.863 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.554153+0000", 2024-09-15T03:46:40.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8299999999999998e-07 2024-09-15T03:46:40.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.863 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.863 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.863 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.554155+0000", 2024-09-15T03:46:40.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2529999999999999e-06 2024-09-15T03:46:40.864 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.864 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.864 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.554161+0000", 2024-09-15T03:46:40.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9920000000000004e-06 2024-09-15T03:46:40.864 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.864 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.864 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.554165+0000", 2024-09-15T03:46:40.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5860000000000001e-06 2024-09-15T03:46:40.864 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.864 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.864 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.554229+0000", 2024-09-15T03:46:40.865 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3732999999999994e-05 2024-09-15T03:46:40.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.865 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.865 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.554450+0000", 2024-09-15T03:46:40.865 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022169099999999999 2024-09-15T03:46:40.865 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.865 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.865 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.865 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 32/28 ECSubWriteReply(tid=9, last_complete=32'5, committed=1, applied=1))", 2024-09-15T03:46:40.865 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.557135+0000", 2024-09-15T03:46:40.865 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.249268278, 2024-09-15T03:46:40.866 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5160000000000005e-05, 2024-09-15T03:46:40.866 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.866 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.866 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.866 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.866 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.866 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557135+0000", 2024-09-15T03:46:40.866 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.866 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.866 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.866 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.866 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557135+0000", 2024-09-15T03:46:40.867 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.867 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.867 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.867 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557136+0000", 2024-09-15T03:46:40.867 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6300000000000005e-07 2024-09-15T03:46:40.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.867 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.867 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.867 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557139+0000", 2024-09-15T03:46:40.867 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3639999999999999e-06 2024-09-15T03:46:40.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.867 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.867 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.868 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557148+0000", 2024-09-15T03:46:40.868 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2119999999999992e-06 2024-09-15T03:46:40.868 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.868 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.868 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.868 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557152+0000", 2024-09-15T03:46:40.868 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6320000000000001e-06 2024-09-15T03:46:40.868 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.868 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.868 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.868 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557187+0000", 2024-09-15T03:46:40.868 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5086999999999998e-05 2024-09-15T03:46:40.868 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.869 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.869 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.869 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557220+0000", 2024-09-15T03:46:40.869 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3302000000000001e-05 2024-09-15T03:46:40.869 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.869 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.869 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.869 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.869 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.0s2 32/31 ECSubWrite(tid=23, reqid=client.4521.0:168, at_version=32'11, trim_to=0'0, roll_forward_to=32'10))", 2024-09-15T03:46:40.869 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.557157+0000", 2024-09-15T03:46:40.869 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.2492466, 2024-09-15T03:46:40.869 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001476724, 2024-09-15T03:46:40.870 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.870 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.870 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.870 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.870 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.870 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557157+0000", 2024-09-15T03:46:40.870 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.870 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.870 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.870 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.870 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557157+0000", 2024-09-15T03:46:40.870 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.870 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.871 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.871 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557157+0000", 2024-09-15T03:46:40.871 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8900000000000001e-07 2024-09-15T03:46:40.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.871 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.871 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557254+0000", 2024-09-15T03:46:40.871 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7207999999999995e-05 2024-09-15T03:46:40.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.871 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.871 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557261+0000", 2024-09-15T03:46:40.871 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7290000000000003e-06 2024-09-15T03:46:40.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.872 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.872 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.872 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557264+0000", 2024-09-15T03:46:40.872 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.145e-06 2024-09-15T03:46:40.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.872 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.872 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.872 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557291+0000", 2024-09-15T03:46:40.872 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7336000000000001e-05 2024-09-15T03:46:40.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.872 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.872 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.873 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557297+0000", 2024-09-15T03:46:40.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3680000000000001e-06 2024-09-15T03:46:40.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.873 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.873 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557366+0000", 2024-09-15T03:46:40.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9548000000000006e-05 2024-09-15T03:46:40.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.873 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.873 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557371+0000", 2024-09-15T03:46:40.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8840000000000002e-06 2024-09-15T03:46:40.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.874 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.558453+0000", 2024-09-15T03:46:40.874 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010819790000000001 2024-09-15T03:46:40.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.874 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.558462+0000", 2024-09-15T03:46:40.874 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7290000000000006e-06 2024-09-15T03:46:40.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.874 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.558633+0000", 2024-09-15T03:46:40.874 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000171609 2024-09-15T03:46:40.874 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.875 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.875 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.875 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.0s2 32/31 ECSubWrite(tid=24, reqid=client.4521.0:169, at_version=32'12, trim_to=0'0, roll_forward_to=32'10))", 2024-09-15T03:46:40.875 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.557269+0000", 2024-09-15T03:46:40.875 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.249134033, 2024-09-15T03:46:40.875 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001442306, 2024-09-15T03:46:40.875 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.875 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.875 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.875 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.875 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557269+0000", 2024-09-15T03:46:40.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.876 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.876 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557269+0000", 2024-09-15T03:46:40.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.876 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.876 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557269+0000", 2024-09-15T03:46:40.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.28e-07 2024-09-15T03:46:40.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.876 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.877 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557276+0000", 2024-09-15T03:46:40.877 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1410000000000002e-06 2024-09-15T03:46:40.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.877 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.877 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.877 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557279+0000", 2024-09-15T03:46:40.877 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.703e-06 2024-09-15T03:46:40.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.877 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.877 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.877 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557286+0000", 2024-09-15T03:46:40.877 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.618e-06 2024-09-15T03:46:40.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.878 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557396+0000", 2024-09-15T03:46:40.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000110441 2024-09-15T03:46:40.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.878 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557402+0000", 2024-09-15T03:46:40.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8590000000000001e-06 2024-09-15T03:46:40.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.878 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557453+0000", 2024-09-15T03:46:40.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0507000000000003e-05 2024-09-15T03:46:40.879 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.879 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.879 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.879 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.558455+0000", 2024-09-15T03:46:40.879 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001002492 2024-09-15T03:46:40.879 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.879 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.879 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.879 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.558483+0000", 2024-09-15T03:46:40.879 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8087999999999999e-05 2024-09-15T03:46:40.879 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.879 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.880 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.880 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.558489+0000", 2024-09-15T03:46:40.881 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6710000000000004e-06 2024-09-15T03:46:40.881 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.881 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.881 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.881 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.558712+0000", 2024-09-15T03:46:40.881 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022255800000000001 2024-09-15T03:46:40.881 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.881 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.881 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.881 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.881 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.881 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.0s2 32/31 ECSubWrite(tid=26, reqid=client.4521.0:169, at_version=0'0, trim_to=0'0, roll_forward_to=32'12))", 2024-09-15T03:46:40.882 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.559135+0000", 2024-09-15T03:46:40.882 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.2472683039999999, 2024-09-15T03:46:40.882 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022680899999999999, 2024-09-15T03:46:40.882 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.882 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.882 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.882 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.882 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.882 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.559135+0000", 2024-09-15T03:46:40.882 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.882 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.882 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.882 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.882 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.559135+0000", 2024-09-15T03:46:40.883 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.883 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.883 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.883 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.559136+0000", 2024-09-15T03:46:40.883 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8100000000000003e-07 2024-09-15T03:46:40.883 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.883 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.883 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.559143+0000", 2024-09-15T03:46:40.883 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6899999999999992e-06 2024-09-15T03:46:40.883 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.883 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.884 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.559154+0000", 2024-09-15T03:46:40.884 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0468e-05 2024-09-15T03:46:40.884 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.884 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.884 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.884 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.559161+0000", 2024-09-15T03:46:40.884 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3599999999999998e-06 2024-09-15T03:46:40.884 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.884 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.884 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.884 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.559204+0000", 2024-09-15T03:46:40.884 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2902999999999999e-05 2024-09-15T03:46:40.884 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.885 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.885 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.885 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.559210+0000", 2024-09-15T03:46:40.885 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1659999999999999e-06 2024-09-15T03:46:40.885 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.885 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.885 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.885 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.559268+0000", 2024-09-15T03:46:40.885 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8026999999999999e-05 2024-09-15T03:46:40.885 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.885 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.885 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.885 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.559275+0000", 2024-09-15T03:46:40.886 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3559999999999998e-06 2024-09-15T03:46:40.886 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.886 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.886 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.886 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.559305+0000", 2024-09-15T03:46:40.886 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9986000000000001e-05 2024-09-15T03:46:40.886 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.886 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.886 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.886 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.559317+0000", 2024-09-15T03:46:40.886 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2112e-05 2024-09-15T03:46:40.886 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.886 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.886 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.887 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.559362+0000", 2024-09-15T03:46:40.887 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5160000000000001e-05 2024-09-15T03:46:40.887 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.887 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.887 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.887 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.887 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.887 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s7 32/31 ECSubWrite(tid=44, reqid=client.4521.0:173, at_version=32'7, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:40.887 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.562211+0000", 2024-09-15T03:46:40.887 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.244192336, 2024-09-15T03:46:40.887 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0038125260000000001, 2024-09-15T03:46:40.887 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.887 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.887 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.888 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.888 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.888 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562211+0000", 2024-09-15T03:46:40.888 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.888 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.888 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.888 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.888 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562211+0000", 2024-09-15T03:46:40.888 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.888 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.888 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.888 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.888 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562212+0000", 2024-09-15T03:46:40.889 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2099999999999996e-07 2024-09-15T03:46:40.889 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.889 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.889 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.889 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562643+0000", 2024-09-15T03:46:40.889 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043174199999999999 2024-09-15T03:46:40.889 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.889 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.889 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.889 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562654+0000", 2024-09-15T03:46:40.889 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0927999999999999e-05 2024-09-15T03:46:40.889 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.889 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.889 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.890 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562666+0000", 2024-09-15T03:46:40.890 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2261e-05 2024-09-15T03:46:40.890 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.890 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.890 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.890 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562709+0000", 2024-09-15T03:46:40.890 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2885e-05 2024-09-15T03:46:40.890 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.890 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.890 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.890 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562716+0000", 2024-09-15T03:46:40.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6490000000000002e-06 2024-09-15T03:46:40.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562882+0000", 2024-09-15T03:46:40.892 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016565500000000001 2024-09-15T03:46:40.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562888+0000", 2024-09-15T03:46:40.892 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.703e-06 2024-09-15T03:46:40.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.893 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.893 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562972+0000", 2024-09-15T03:46:40.893 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3724e-05 2024-09-15T03:46:40.893 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.893 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.893 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.893 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562984+0000", 2024-09-15T03:46:40.893 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2361000000000001e-05 2024-09-15T03:46:40.893 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.893 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.893 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.893 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.566023+0000", 2024-09-15T03:46:40.893 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030389969999999999 2024-09-15T03:46:40.893 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.894 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.894 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.894 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.894 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.894 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s7 32/31 ECSubWrite(tid=45, reqid=client.4521.0:173, at_version=0'0, trim_to=0'0, roll_forward_to=32'7))", 2024-09-15T03:46:40.894 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.563503+0000", 2024-09-15T03:46:40.894 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.2429001390000001, 2024-09-15T03:46:40.894 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002634688, 2024-09-15T03:46:40.894 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.894 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.894 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.894 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.894 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.894 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.563503+0000", 2024-09-15T03:46:40.895 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.895 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.895 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.895 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.895 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.563503+0000", 2024-09-15T03:46:40.895 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.895 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.895 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.895 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.895 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.563504+0000", 2024-09-15T03:46:40.895 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9699999999999996e-07 2024-09-15T03:46:40.895 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.895 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.896 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.896 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.563512+0000", 2024-09-15T03:46:40.896 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4319999999999999e-06 2024-09-15T03:46:40.896 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.896 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.896 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.896 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.563526+0000", 2024-09-15T03:46:40.896 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3891e-05 2024-09-15T03:46:40.896 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.896 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.896 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.896 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.563531+0000", 2024-09-15T03:46:40.896 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7509999999999999e-06 2024-09-15T03:46:40.897 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.897 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.897 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.897 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.563572+0000", 2024-09-15T03:46:40.897 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0963000000000001e-05 2024-09-15T03:46:40.897 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.897 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.897 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.897 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.563581+0000", 2024-09-15T03:46:40.897 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1230000000000006e-06 2024-09-15T03:46:40.897 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.897 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.897 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.898 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.563650+0000", 2024-09-15T03:46:40.898 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9206000000000001e-05 2024-09-15T03:46:40.898 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.898 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.898 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.898 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.563656+0000", 2024-09-15T03:46:40.898 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1639999999999998e-06 2024-09-15T03:46:40.898 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.898 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.898 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.898 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.565765+0000", 2024-09-15T03:46:40.898 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021088600000000002 2024-09-15T03:46:40.898 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.898 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.899 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.899 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.565790+0000", 2024-09-15T03:46:40.899 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.51e-05 2024-09-15T03:46:40.899 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.899 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.899 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.566138+0000", 2024-09-15T03:46:40.899 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034760100000000001 2024-09-15T03:46:40.899 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.899 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.899 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.899 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.900 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 33/28 ECSubWrite(tid=21, reqid=client.4521.0:184, at_version=33'6, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:40.900 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.106912+0000", 2024-09-15T03:46:40.900 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.69949084699999997, 2024-09-15T03:46:40.900 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026439319999999999, 2024-09-15T03:46:40.900 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.900 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.900 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.900 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.900 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.900 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106912+0000", 2024-09-15T03:46:40.900 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.900 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.900 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.901 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.901 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106912+0000", 2024-09-15T03:46:40.901 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.901 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.901 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.901 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.901 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106913+0000", 2024-09-15T03:46:40.901 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.39e-07 2024-09-15T03:46:40.901 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.901 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.901 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.901 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106938+0000", 2024-09-15T03:46:40.901 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5063999999999999e-05 2024-09-15T03:46:40.901 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.902 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.902 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.902 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106949+0000", 2024-09-15T03:46:40.902 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1137e-05 2024-09-15T03:46:40.902 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.902 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.902 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.902 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106956+0000", 2024-09-15T03:46:40.902 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7719999999999997e-06 2024-09-15T03:46:40.902 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.902 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.902 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.902 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107325+0000", 2024-09-15T03:46:40.903 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036909699999999998 2024-09-15T03:46:40.903 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.903 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.903 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.903 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107333+0000", 2024-09-15T03:46:40.903 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9489999999999996e-06 2024-09-15T03:46:40.903 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.903 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.903 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.903 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107497+0000", 2024-09-15T03:46:40.903 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016431900000000001 2024-09-15T03:46:40.903 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.903 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.903 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.904 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107523+0000", 2024-09-15T03:46:40.904 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5678e-05 2024-09-15T03:46:40.904 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.904 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.904 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.904 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107897+0000", 2024-09-15T03:46:40.904 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00037388999999999999 2024-09-15T03:46:40.904 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.904 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.904 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.904 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108093+0000", 2024-09-15T03:46:40.904 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019591199999999999 2024-09-15T03:46:40.904 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.905 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.905 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.905 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109556+0000", 2024-09-15T03:46:40.905 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001463675 2024-09-15T03:46:40.905 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.905 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.905 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.905 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.905 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.905 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:40.905 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.108773+0000", 2024-09-15T03:46:40.905 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.69763042399999997, 2024-09-15T03:46:40.905 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0047381410000000004, 2024-09-15T03:46:40.905 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.906 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.906 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.906 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.906 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.906 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108773+0000", 2024-09-15T03:46:40.906 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.906 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.906 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.906 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.906 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108773+0000", 2024-09-15T03:46:40.906 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.906 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.906 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.906 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.907 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108774+0000", 2024-09-15T03:46:40.907 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8800000000000002e-07 2024-09-15T03:46:40.907 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.907 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.907 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.907 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108801+0000", 2024-09-15T03:46:40.907 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7339999999999999e-05 2024-09-15T03:46:40.907 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.907 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.907 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.907 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108816+0000", 2024-09-15T03:46:40.907 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5359999999999999e-05 2024-09-15T03:46:40.907 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.907 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.908 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.908 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108823+0000", 2024-09-15T03:46:40.908 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0060000000000003e-06 2024-09-15T03:46:40.908 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.908 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.908 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.908 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108930+0000", 2024-09-15T03:46:40.908 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000106569 2024-09-15T03:46:40.908 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.908 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.908 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.908 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108940+0000", 2024-09-15T03:46:40.908 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.679e-06 2024-09-15T03:46:40.909 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.909 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.909 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.909 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109030+0000", 2024-09-15T03:46:40.909 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0618999999999994e-05 2024-09-15T03:46:40.909 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.909 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.909 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.909 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109041+0000", 2024-09-15T03:46:40.909 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1011e-05 2024-09-15T03:46:40.909 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.909 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.909 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.909 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109073+0000", 2024-09-15T03:46:40.910 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1427999999999998e-05 2024-09-15T03:46:40.910 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.910 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.910 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.910 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109088+0000", 2024-09-15T03:46:40.910 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4956e-05 2024-09-15T03:46:40.910 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.910 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.910 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.910 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.113511+0000", 2024-09-15T03:46:40.910 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044233850000000002 2024-09-15T03:46:40.910 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.910 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.911 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.911 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.911 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.911 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:40.911 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.109181+0000", 2024-09-15T03:46:40.911 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.69722252600000001, 2024-09-15T03:46:40.911 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044348879999999997, 2024-09-15T03:46:40.911 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.911 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.911 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.911 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.911 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.911 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109181+0000", 2024-09-15T03:46:40.911 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.912 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.912 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.912 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.912 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109181+0000", 2024-09-15T03:46:40.912 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.912 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.912 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.912 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.912 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109181+0000", 2024-09-15T03:46:40.912 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0900000000000001e-07 2024-09-15T03:46:40.912 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.912 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.912 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.912 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109197+0000", 2024-09-15T03:46:40.913 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5150999999999999e-05 2024-09-15T03:46:40.913 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.913 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.913 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.913 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109211+0000", 2024-09-15T03:46:40.913 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3998999999999999e-05 2024-09-15T03:46:40.913 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.913 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.913 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.913 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109217+0000", 2024-09-15T03:46:40.913 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.686e-06 2024-09-15T03:46:40.913 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.913 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.913 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.914 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109254+0000", 2024-09-15T03:46:40.914 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6424999999999998e-05 2024-09-15T03:46:40.914 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.914 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.914 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.914 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109262+0000", 2024-09-15T03:46:40.914 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8230000000000001e-06 2024-09-15T03:46:40.914 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.914 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.914 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.914 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109350+0000", 2024-09-15T03:46:40.914 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7120999999999995e-05 2024-09-15T03:46:40.914 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.914 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.915 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.915 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109358+0000", 2024-09-15T03:46:40.915 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3399999999999998e-06 2024-09-15T03:46:40.915 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.915 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.915 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.915 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109388+0000", 2024-09-15T03:46:40.915 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9807000000000001e-05 2024-09-15T03:46:40.915 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.915 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.915 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.915 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109398+0000", 2024-09-15T03:46:40.915 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0744000000000001e-05 2024-09-15T03:46:40.916 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.916 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.916 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.916 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.113616+0000", 2024-09-15T03:46:40.916 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042171830000000002 2024-09-15T03:46:40.916 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.916 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.916 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.916 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.916 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.916 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 33/28 ECSubWrite(tid=28, reqid=client.4521.0:186, at_version=0'0, trim_to=0'0, roll_forward_to=33'8))", 2024-09-15T03:46:40.916 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.114810+0000", 2024-09-15T03:46:40.916 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.69159287400000002, 2024-09-15T03:46:40.916 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0011592569999999999, 2024-09-15T03:46:40.917 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:40.917 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:40.917 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:40.917 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.917 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:40.917 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114810+0000", 2024-09-15T03:46:40.917 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.917 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.917 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.917 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:40.917 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114810+0000", 2024-09-15T03:46:40.917 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:40.917 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.917 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.918 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:40.918 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114812+0000", 2024-09-15T03:46:40.918 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.144e-06 2024-09-15T03:46:40.918 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.918 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.918 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:40.918 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114825+0000", 2024-09-15T03:46:40.918 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3893000000000001e-05 2024-09-15T03:46:40.918 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.918 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.918 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:40.918 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114844+0000", 2024-09-15T03:46:40.918 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8111e-05 2024-09-15T03:46:40.918 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.919 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.919 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:40.919 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114851+0000", 2024-09-15T03:46:40.919 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2869999999999999e-06 2024-09-15T03:46:40.919 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.919 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.919 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:40.919 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114905+0000", 2024-09-15T03:46:40.919 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3736999999999997e-05 2024-09-15T03:46:40.919 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.919 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.919 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:40.919 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114914+0000", 2024-09-15T03:46:40.919 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3999999999999998e-06 2024-09-15T03:46:40.920 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.920 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.920 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:40.920 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.115031+0000", 2024-09-15T03:46:40.920 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000116712 2024-09-15T03:46:40.920 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.920 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.920 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:40.920 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.115048+0000", 2024-09-15T03:46:40.920 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7637000000000001e-05 2024-09-15T03:46:40.920 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.920 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.920 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:40.921 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.115772+0000", 2024-09-15T03:46:40.921 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000723681 2024-09-15T03:46:40.921 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.921 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.921 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:40.921 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.115790+0000", 2024-09-15T03:46:40.921 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7958e-05 2024-09-15T03:46:40.921 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:40.921 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:40.921 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:40.921 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.115970+0000", 2024-09-15T03:46:40.921 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017969700000000001 2024-09-15T03:46:40.921 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.921 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.922 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.922 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:40.922 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:40.922 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:40.923 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:46:40.924 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:40.923+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:46:41.020 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:41.020 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:41.020 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:46:41.020 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:41.024 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:41.024+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:41.028 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:46:41.107 INFO:tasks.rados.rados.0.smithi001.stdout:63: done (3 left) 2024-09-15T03:46:41.107 INFO:tasks.rados.rados.0.smithi001.stdout:64: done (2 left) 2024-09-15T03:46:41.107 INFO:tasks.rados.rados.0.smithi001.stdout:65: done (1 left) 2024-09-15T03:46:41.107 INFO:tasks.rados.rados.0.smithi001.stdout:66: done (0 left) 2024-09-15T03:46:41.107 INFO:tasks.rados.rados.0.smithi001.stdout:67: delete oid 1 current snap is 2 2024-09-15T03:46:41.111 INFO:tasks.rados.rados.0.smithi001.stdout:67: done (0 left) 2024-09-15T03:46:41.112 INFO:tasks.rados.rados.0.smithi001.stdout:68: rollback oid 17 current snap is 2 2024-09-15T03:46:41.112 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 17 to 1 2024-09-15T03:46:41.112 INFO:tasks.rados.rados.0.smithi001.stdout:69: copy_from oid 31 from oid 18 current snap is 2 2024-09-15T03:46:41.112 INFO:tasks.rados.rados.0.smithi001.stdout:70: delete oid 3 current snap is 2 2024-09-15T03:46:41.115 INFO:tasks.rados.rados.0.smithi001.stdout:68: finishing rollback tid 0 to smithi00132172-17 2024-09-15T03:46:41.115 INFO:tasks.rados.rados.0.smithi001.stdout:68: finishing rollback tid 1 to smithi00132172-17 2024-09-15T03:46:41.119 INFO:tasks.rados.rados.0.smithi001.stdout:69: finishing copy_from racing read to smithi00132172-31 2024-09-15T03:46:41.124 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T03:46:41.125 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:41.124+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:46:41.126 INFO:tasks.rados.rados.0.smithi001.stdout:69: finishing copy_from to smithi00132172-31 2024-09-15T03:46:41.126 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 31 v 4 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-15T03:46:41.126 INFO:tasks.rados.rados.0.smithi001.stdout:68: finishing rollback tid 2 to smithi00132172-17 2024-09-15T03:46:41.126 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 17 v 19 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-15T03:46:41.126 INFO:tasks.rados.rados.0.smithi001.stdout:68: done (2 left) 2024-09-15T03:46:41.126 INFO:tasks.rados.rados.0.smithi001.stdout:69: done (1 left) 2024-09-15T03:46:41.126 INFO:tasks.rados.rados.0.smithi001.stdout:70: done (0 left) 2024-09-15T03:46:41.126 INFO:tasks.rados.rados.0.smithi001.stdout:71: snap_create 2024-09-15T03:46:41.127 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:41.127 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:41.127 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:46:41.127 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:46:41.128 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:41.136 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:46:41.226 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:41.226+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:46:41.233 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:41.234 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:46:41.234 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:46:41.234 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [] 2024-09-15T03:46:41.234 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:41.242 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:46:41.327 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:41.325+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:41.340 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:41.340 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:41.340 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:46:41.340 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:41.349 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:46:41.428 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:41.427+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:46:41.485 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:41.485 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:41.485 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:46:41.485 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:46:41.485 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:41.493 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:46:41.529 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:41.528+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:41.578 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:46:41.591 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:41.591 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:46:41.591 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:46:41.591 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:46:41.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.591 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4263.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-15T03:46:41.591 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:24.037919+0000", 2024-09-15T03:46:41.591 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 17.551343654, 2024-09-15T03:46:41.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025392869999999999, 2024-09-15T03:46:41.592 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:41.592 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:41.592 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:41.592 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4263", 2024-09-15T03:46:41.592 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/646873808", 2024-09-15T03:46:41.592 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1 2024-09-15T03:46:41.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.592 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:41.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.593 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:41.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.037919+0000", 2024-09-15T03:46:41.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.593 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.593 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:41.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.037919+0000", 2024-09-15T03:46:41.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.593 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.593 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:41.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.037920+0000", 2024-09-15T03:46:41.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.863e-06 2024-09-15T03:46:41.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:41.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.037930+0000", 2024-09-15T03:46:41.594 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1530000000000003e-06 2024-09-15T03:46:41.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:41.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.037948+0000", 2024-09-15T03:46:41.594 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7969e-05 2024-09-15T03:46:41.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:41.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.037958+0000", 2024-09-15T03:46:41.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0288e-05 2024-09-15T03:46:41.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.038029+0000", 2024-09-15T03:46:41.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0778999999999994e-05 2024-09-15T03:46:41.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:46:41.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.038037+0000", 2024-09-15T03:46:41.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6179999999999995e-06 2024-09-15T03:46:41.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.038835+0000", 2024-09-15T03:46:41.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00079816700000000002 2024-09-15T03:46:41.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.596 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.596 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:41.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.038974+0000", 2024-09-15T03:46:41.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013816499999999999 2024-09-15T03:46:41.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.596 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.596 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 9", 2024-09-15T03:46:41.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039080+0000", 2024-09-15T03:46:41.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000105969 2024-09-15T03:46:41.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.597 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.597 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:41.597 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039223+0000", 2024-09-15T03:46:41.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014346399999999999 2024-09-15T03:46:41.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.597 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.597 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:41.597 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039232+0000", 2024-09-15T03:46:41.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.935e-06 2024-09-15T03:46:41.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.597 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.597 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:41.598 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039283+0000", 2024-09-15T03:46:41.598 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0807999999999999e-05 2024-09-15T03:46:41.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.598 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.598 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:41.598 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039475+0000", 2024-09-15T03:46:41.598 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019226000000000001 2024-09-15T03:46:41.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.598 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.598 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:41.598 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040394+0000", 2024-09-15T03:46:41.598 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00091921199999999998 2024-09-15T03:46:41.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.599 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.599 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:41.599 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040430+0000", 2024-09-15T03:46:41.599 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5954999999999997e-05 2024-09-15T03:46:41.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.599 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.599 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:41.599 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040458+0000", 2024-09-15T03:46:41.599 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7682e-05 2024-09-15T03:46:41.599 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.599 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:41.599 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.599 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.600 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:184 3.as0 3:5515acd6:::smithi00132172-13:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:41.600 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104658+0000", 2024-09-15T03:46:41.600 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.4846038909999999, 2024-09-15T03:46:41.600 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006212525, 2024-09-15T03:46:41.600 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:41.600 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:41.600 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:41.600 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:41.600 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:41.600 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 184 2024-09-15T03:46:41.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.600 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:41.600 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.601 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:41.601 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104658+0000", 2024-09-15T03:46:41.601 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.601 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.601 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:41.601 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104658+0000", 2024-09-15T03:46:41.601 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.601 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.601 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:41.601 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104660+0000", 2024-09-15T03:46:41.601 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.17e-06 2024-09-15T03:46:41.602 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.602 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.602 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:41.602 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104672+0000", 2024-09-15T03:46:41.602 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1161999999999999e-05 2024-09-15T03:46:41.602 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.602 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.602 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:41.602 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104693+0000", 2024-09-15T03:46:41.602 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0995000000000001e-05 2024-09-15T03:46:41.602 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.602 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.602 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:41.602 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104700+0000", 2024-09-15T03:46:41.603 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4189999999999996e-06 2024-09-15T03:46:41.603 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.603 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.603 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.603 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104752+0000", 2024-09-15T03:46:41.603 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2413000000000001e-05 2024-09-15T03:46:41.603 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.603 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.603 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-09-15T03:46:41.603 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104757+0000", 2024-09-15T03:46:41.603 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9599999999999999e-06 2024-09-15T03:46:41.603 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.603 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.603 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.604 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.105683+0000", 2024-09-15T03:46:41.604 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00092545599999999996 2024-09-15T03:46:41.604 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.604 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.604 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:41.604 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.105757+0000", 2024-09-15T03:46:41.604 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4567999999999995e-05 2024-09-15T03:46:41.604 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.604 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.604 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:41.604 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106531+0000", 2024-09-15T03:46:41.604 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00077397999999999996 2024-09-15T03:46:41.604 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.605 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.605 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:41.605 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106775+0000", 2024-09-15T03:46:41.605 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024402899999999999 2024-09-15T03:46:41.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.605 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.605 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:41.605 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106786+0000", 2024-09-15T03:46:41.605 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0825000000000001e-05 2024-09-15T03:46:41.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.605 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.605 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:41.605 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106845+0000", 2024-09-15T03:46:41.605 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8733999999999997e-05 2024-09-15T03:46:41.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.606 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.606 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:41.606 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109399+0000", 2024-09-15T03:46:41.606 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002553573 2024-09-15T03:46:41.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.606 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.606 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:41.606 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109844+0000", 2024-09-15T03:46:41.606 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00044492700000000002 2024-09-15T03:46:41.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.606 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.606 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:41.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.110871+0000", 2024-09-15T03:46:41.607 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010273140000000001 2024-09-15T03:46:41.607 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.607 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:41.607 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.607 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.607 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.607 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:185 3.as0 3:5515acd6:::smithi00132172-13:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:41.607 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:46:41.607 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.484553061, 2024-09-15T03:46:41.607 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01268643, 2024-09-15T03:46:41.607 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:41.607 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:41.607 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:41.608 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:41.608 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:41.608 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 185 2024-09-15T03:46:41.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.608 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:41.608 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.608 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:41.608 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:46:41.608 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.608 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.608 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:41.609 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:46:41.609 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.609 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.609 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:41.609 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104710+0000", 2024-09-15T03:46:41.609 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0200000000000003e-07 2024-09-15T03:46:41.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.609 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.609 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:41.609 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104713+0000", 2024-09-15T03:46:41.609 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2339999999999999e-06 2024-09-15T03:46:41.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.610 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.610 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:41.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104718+0000", 2024-09-15T03:46:41.610 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3449999999999996e-06 2024-09-15T03:46:41.610 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.610 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.610 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:41.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104722+0000", 2024-09-15T03:46:41.610 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8399999999999997e-06 2024-09-15T03:46:41.610 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.610 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.610 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104849+0000", 2024-09-15T03:46:41.610 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012709599999999999 2024-09-15T03:46:41.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.611 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.611 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:41.611 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104856+0000", 2024-09-15T03:46:41.611 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.652e-06 2024-09-15T03:46:41.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.611 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.611 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.611 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106851+0000", 2024-09-15T03:46:41.611 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019954840000000001 2024-09-15T03:46:41.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.611 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.611 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:41.611 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106935+0000", 2024-09-15T03:46:41.612 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4095e-05 2024-09-15T03:46:41.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.612 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.612 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:41.612 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107453+0000", 2024-09-15T03:46:41.612 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00051729499999999997 2024-09-15T03:46:41.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.612 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.612 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:41.612 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107554+0000", 2024-09-15T03:46:41.612 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000101251 2024-09-15T03:46:41.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.612 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.613 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:41.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108104+0000", 2024-09-15T03:46:41.613 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00054995100000000002 2024-09-15T03:46:41.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.613 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.613 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:41.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108503+0000", 2024-09-15T03:46:41.613 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039953500000000002 2024-09-15T03:46:41.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.613 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.613 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:41.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109405+0000", 2024-09-15T03:46:41.613 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00090170099999999998 2024-09-15T03:46:41.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.614 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.614 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:41.614 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114365+0000", 2024-09-15T03:46:41.614 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049602120000000003 2024-09-15T03:46:41.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.614 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.614 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:41.614 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117396+0000", 2024-09-15T03:46:41.614 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030303370000000001 2024-09-15T03:46:41.614 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.614 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:41.614 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.615 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.615 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:186 3.as0 3:5515acd6:::smithi00132172-13:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:41.615 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:46:41.615 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.484513271, 2024-09-15T03:46:41.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012817485999999999, 2024-09-15T03:46:41.615 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:41.615 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:41.615 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:41.615 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:41.615 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:41.615 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 186 2024-09-15T03:46:41.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.615 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:41.615 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:41.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:46:41.616 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:41.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:46:41.616 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:41.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104750+0000", 2024-09-15T03:46:41.616 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1500000000000004e-07 2024-09-15T03:46:41.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:41.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104754+0000", 2024-09-15T03:46:41.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.946e-06 2024-09-15T03:46:41.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:41.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104769+0000", 2024-09-15T03:46:41.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.507e-05 2024-09-15T03:46:41.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:41.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104776+0000", 2024-09-15T03:46:41.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7849999999999992e-06 2024-09-15T03:46:41.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.618 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104888+0000", 2024-09-15T03:46:41.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011164500000000001 2024-09-15T03:46:41.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.618 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:41.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104894+0000", 2024-09-15T03:46:41.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.942e-06 2024-09-15T03:46:41.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.618 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.619 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.619 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108172+0000", 2024-09-15T03:46:41.619 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032781939999999999 2024-09-15T03:46:41.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.619 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:41.619 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108257+0000", 2024-09-15T03:46:41.619 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4488999999999997e-05 2024-09-15T03:46:41.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.619 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:41.619 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108732+0000", 2024-09-15T03:46:41.619 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047547399999999998 2024-09-15T03:46:41.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:41.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108823+0000", 2024-09-15T03:46:41.620 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1106000000000005e-05 2024-09-15T03:46:41.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:41.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108961+0000", 2024-09-15T03:46:41.620 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013735099999999999 2024-09-15T03:46:41.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:41.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109065+0000", 2024-09-15T03:46:41.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104715 2024-09-15T03:46:41.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.621 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:41.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109406+0000", 2024-09-15T03:46:41.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034062300000000002 2024-09-15T03:46:41.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.621 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:41.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114529+0000", 2024-09-15T03:46:41.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.005123219 2024-09-15T03:46:41.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.621 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:41.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117566+0000", 2024-09-15T03:46:41.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030372120000000001 2024-09-15T03:46:41.622 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.622 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:41.622 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.622 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:187 3.fs0 3:fe9c5513:::smithi00132172-47:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:41.622 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104782+0000", 2024-09-15T03:46:41.622 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.484480252, 2024-09-15T03:46:41.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069218609999999996, 2024-09-15T03:46:41.622 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:41.622 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:41.623 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:41.623 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:41.623 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:41.623 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 187 2024-09-15T03:46:41.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.623 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:41.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:41.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104782+0000", 2024-09-15T03:46:41.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:41.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104782+0000", 2024-09-15T03:46:41.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:41.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104782+0000", 2024-09-15T03:46:41.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8599999999999998e-07 2024-09-15T03:46:41.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:41.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104785+0000", 2024-09-15T03:46:41.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.649e-06 2024-09-15T03:46:41.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:41.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104800+0000", 2024-09-15T03:46:41.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5305e-05 2024-09-15T03:46:41.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:41.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104805+0000", 2024-09-15T03:46:41.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5569999999999997e-06 2024-09-15T03:46:41.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104854+0000", 2024-09-15T03:46:41.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9039999999999998e-05 2024-09-15T03:46:41.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-09-15T03:46:41.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104860+0000", 2024-09-15T03:46:41.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2979999999999997e-06 2024-09-15T03:46:41.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.627 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.105869+0000", 2024-09-15T03:46:41.627 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001009036 2024-09-15T03:46:41.627 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.627 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.627 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:41.627 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.105954+0000", 2024-09-15T03:46:41.627 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4357000000000006e-05 2024-09-15T03:46:41.627 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.627 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.627 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:41.627 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106626+0000", 2024-09-15T03:46:41.627 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00067191099999999997 2024-09-15T03:46:41.627 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.627 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.628 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:41.628 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106786+0000", 2024-09-15T03:46:41.628 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000160469 2024-09-15T03:46:41.628 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.628 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.628 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:41.628 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106788+0000", 2024-09-15T03:46:41.628 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9630000000000001e-06 2024-09-15T03:46:41.628 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.628 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.628 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:41.628 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106846+0000", 2024-09-15T03:46:41.628 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7719000000000001e-05 2024-09-15T03:46:41.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.629 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:41.629 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108372+0000", 2024-09-15T03:46:41.629 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0015264499999999999 2024-09-15T03:46:41.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.629 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:41.629 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.111475+0000", 2024-09-15T03:46:41.629 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031028459999999998 2024-09-15T03:46:41.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:41.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.111704+0000", 2024-09-15T03:46:41.630 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022877499999999999 2024-09-15T03:46:41.630 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.630 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:41.630 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.630 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:188 3.fs0 3:fe9c5513:::smithi00132172-47:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:41.630 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:46:41.630 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.4844305740000001, 2024-09-15T03:46:41.630 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012637061, 2024-09-15T03:46:41.631 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:41.631 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:41.631 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:41.631 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:41.631 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:41.631 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 188 2024-09-15T03:46:41.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.631 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:41.631 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.631 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:41.631 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:46:41.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.632 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:41.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:46:41.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.632 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:41.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:46:41.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2799999999999996e-07 2024-09-15T03:46:41.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.632 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:41.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104835+0000", 2024-09-15T03:46:41.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7190000000000001e-06 2024-09-15T03:46:41.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.633 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:41.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104842+0000", 2024-09-15T03:46:41.633 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4580000000000004e-06 2024-09-15T03:46:41.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.633 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:41.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104846+0000", 2024-09-15T03:46:41.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1250000000000003e-06 2024-09-15T03:46:41.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104884+0000", 2024-09-15T03:46:41.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7075e-05 2024-09-15T03:46:41.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:41.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104888+0000", 2024-09-15T03:46:41.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8520000000000003e-06 2024-09-15T03:46:41.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.635 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.635 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106860+0000", 2024-09-15T03:46:41.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019719440000000002 2024-09-15T03:46:41.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.635 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.635 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:41.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106938+0000", 2024-09-15T03:46:41.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7421000000000003e-05 2024-09-15T03:46:41.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.635 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.635 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:41.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107368+0000", 2024-09-15T03:46:41.636 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043042999999999998 2024-09-15T03:46:41.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.636 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.636 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:41.636 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107460+0000", 2024-09-15T03:46:41.636 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1392999999999997e-05 2024-09-15T03:46:41.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.636 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.637 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:41.637 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107473+0000", 2024-09-15T03:46:41.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3702e-05 2024-09-15T03:46:41.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.637 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.637 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:41.637 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107504+0000", 2024-09-15T03:46:41.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1000000000000001e-05 2024-09-15T03:46:41.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.637 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.637 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:41.637 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108376+0000", 2024-09-15T03:46:41.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00087162000000000003 2024-09-15T03:46:41.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.638 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:41.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.111657+0000", 2024-09-15T03:46:41.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032815589999999999 2024-09-15T03:46:41.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.638 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:41.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117469+0000", 2024-09-15T03:46:41.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058112349999999997 2024-09-15T03:46:41.639 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.639 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:41.639 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.639 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:189 3.fs0 3:fe9c5513:::smithi00132172-47:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:41.639 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:46:41.639 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.4844113880000001, 2024-09-15T03:46:41.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012731009999999999, 2024-09-15T03:46:41.640 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:41.640 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:41.640 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:41.640 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:41.640 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:41.640 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 189 2024-09-15T03:46:41.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.640 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:41.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:41.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:46:41.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:41.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:46:41.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:41.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:46:41.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7899999999999999e-07 2024-09-15T03:46:41.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:41.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104853+0000", 2024-09-15T03:46:41.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.933e-06 2024-09-15T03:46:41.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:41.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104859+0000", 2024-09-15T03:46:41.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.592e-06 2024-09-15T03:46:41.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:41.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104863+0000", 2024-09-15T03:46:41.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4000000000000002e-06 2024-09-15T03:46:41.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104910+0000", 2024-09-15T03:46:41.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6678000000000003e-05 2024-09-15T03:46:41.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:41.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104916+0000", 2024-09-15T03:46:41.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1079999999999998e-06 2024-09-15T03:46:41.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107550+0000", 2024-09-15T03:46:41.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026344110000000001 2024-09-15T03:46:41.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:41.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107624+0000", 2024-09-15T03:46:41.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4134000000000005e-05 2024-09-15T03:46:41.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:41.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107988+0000", 2024-09-15T03:46:41.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036353600000000002 2024-09-15T03:46:41.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:41.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108066+0000", 2024-09-15T03:46:41.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8047999999999995e-05 2024-09-15T03:46:41.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:41.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108105+0000", 2024-09-15T03:46:41.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8868000000000001e-05 2024-09-15T03:46:41.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:41.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108504+0000", 2024-09-15T03:46:41.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039916600000000001 2024-09-15T03:46:41.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:41.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108955+0000", 2024-09-15T03:46:41.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045126200000000002 2024-09-15T03:46:41.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:41.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114394+0000", 2024-09-15T03:46:41.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0054382750000000002 2024-09-15T03:46:41.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:41.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117582+0000", 2024-09-15T03:46:41.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031882199999999999 2024-09-15T03:46:41.647 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.647 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:41.647 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.647 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.as0 33/28 ECSubWriteReply(tid=21, last_complete=33'6, committed=1, applied=1))", 2024-09-15T03:46:41.647 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.107372+0000", 2024-09-15T03:46:41.647 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.4818904210000001, 2024-09-15T03:46:41.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0015093699999999999, 2024-09-15T03:46:41.647 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:41.647 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:41.648 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:41.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:41.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107372+0000", 2024-09-15T03:46:41.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:41.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107372+0000", 2024-09-15T03:46:41.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:41.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107373+0000", 2024-09-15T03:46:41.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7100000000000001e-07 2024-09-15T03:46:41.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:41.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107377+0000", 2024-09-15T03:46:41.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5900000000000001e-06 2024-09-15T03:46:41.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:41.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107386+0000", 2024-09-15T03:46:41.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8650000000000007e-06 2024-09-15T03:46:41.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:41.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107394+0000", 2024-09-15T03:46:41.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9529999999999999e-06 2024-09-15T03:46:41.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108872+0000", 2024-09-15T03:46:41.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001478247 2024-09-15T03:46:41.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:41.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108881+0000", 2024-09-15T03:46:41.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9439999999999994e-06 2024-09-15T03:46:41.651 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.651 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:41.651 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.651 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.as0 33/28 ECSubWriteReply(tid=21, last_complete=33'6, committed=1, applied=1))", 2024-09-15T03:46:41.651 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.107487+0000", 2024-09-15T03:46:41.651 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.4817752989999999, 2024-09-15T03:46:41.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0014182450000000001, 2024-09-15T03:46:41.651 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:41.651 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:41.651 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:41.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.651 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:41.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107487+0000", 2024-09-15T03:46:41.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:41.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107487+0000", 2024-09-15T03:46:41.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:41.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107488+0000", 2024-09-15T03:46:41.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.92e-07 2024-09-15T03:46:41.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:41.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107499+0000", 2024-09-15T03:46:41.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1201e-05 2024-09-15T03:46:41.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:41.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108217+0000", 2024-09-15T03:46:41.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00071768899999999998 2024-09-15T03:46:41.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:41.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108225+0000", 2024-09-15T03:46:41.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0020000000000006e-06 2024-09-15T03:46:41.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108898+0000", 2024-09-15T03:46:41.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00067329200000000003 2024-09-15T03:46:41.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:41.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108905+0000", 2024-09-15T03:46:41.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2690000000000002e-06 2024-09-15T03:46:41.655 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.655 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:41.655 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.655 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.as0 33/28 ECSubWriteReply(tid=21, last_complete=33'6, committed=1, applied=1))", 2024-09-15T03:46:41.655 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.108283+0000", 2024-09-15T03:46:41.655 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.4809790139999999, 2024-09-15T03:46:41.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00064378199999999997, 2024-09-15T03:46:41.655 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:41.655 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:41.655 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:41.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:41.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108283+0000", 2024-09-15T03:46:41.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:41.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108283+0000", 2024-09-15T03:46:41.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:41.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108284+0000", 2024-09-15T03:46:41.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.34e-07 2024-09-15T03:46:41.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:41.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108286+0000", 2024-09-15T03:46:41.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3819999999999998e-06 2024-09-15T03:46:41.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:41.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108292+0000", 2024-09-15T03:46:41.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4910000000000002e-06 2024-09-15T03:46:41.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:41.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108297+0000", 2024-09-15T03:46:41.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3869999999999997e-06 2024-09-15T03:46:41.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108921+0000", 2024-09-15T03:46:41.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00062375799999999997 2024-09-15T03:46:41.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:41.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108927+0000", 2024-09-15T03:46:41.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4300000000000003e-06 2024-09-15T03:46:41.658 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.658 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:41.659 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.659 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s6 34/31 ECSubWrite(tid=46, reqid=client.4521.0:191, at_version=34'9, trim_to=0'0, roll_forward_to=32'7))", 2024-09-15T03:46:41.659 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.109352+0000", 2024-09-15T03:46:41.659 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.479910063, 2024-09-15T03:46:41.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0035566389999999999, 2024-09-15T03:46:41.659 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:41.659 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:41.659 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:41.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.659 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:41.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109352+0000", 2024-09-15T03:46:41.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.660 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:41.660 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109352+0000", 2024-09-15T03:46:41.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.660 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:41.660 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109352+0000", 2024-09-15T03:46:41.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3099999999999999e-07 2024-09-15T03:46:41.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:41.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109378+0000", 2024-09-15T03:46:41.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5148e-05 2024-09-15T03:46:41.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:41.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109389+0000", 2024-09-15T03:46:41.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1063999999999999e-05 2024-09-15T03:46:41.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:41.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109396+0000", 2024-09-15T03:46:41.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4889999999999998e-06 2024-09-15T03:46:41.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109454+0000", 2024-09-15T03:46:41.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7969e-05 2024-09-15T03:46:41.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:41.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109464+0000", 2024-09-15T03:46:41.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6900000000000004e-06 2024-09-15T03:46:41.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:41.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109668+0000", 2024-09-15T03:46:41.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000204093 2024-09-15T03:46:41.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:41.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109678+0000", 2024-09-15T03:46:41.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0182e-05 2024-09-15T03:46:41.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:41.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109847+0000", 2024-09-15T03:46:41.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016929 2024-09-15T03:46:41.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.664 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:41.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109954+0000", 2024-09-15T03:46:41.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010638500000000001 2024-09-15T03:46:41.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.664 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:41.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.112909+0000", 2024-09-15T03:46:41.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0029550980000000002 2024-09-15T03:46:41.664 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.664 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:41.664 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.665 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s6 34/31 ECSubWrite(tid=47, reqid=client.4521.0:191, at_version=0'0, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:46:41.665 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.111978+0000", 2024-09-15T03:46:41.665 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.47728369900000001, 2024-09-15T03:46:41.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010393069999999999, 2024-09-15T03:46:41.665 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:41.665 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:41.665 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:41.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.665 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:41.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.111978+0000", 2024-09-15T03:46:41.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:41.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.111978+0000", 2024-09-15T03:46:41.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:41.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.111979+0000", 2024-09-15T03:46:41.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9500000000000001e-07 2024-09-15T03:46:41.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:41.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.111998+0000", 2024-09-15T03:46:41.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8357999999999999e-05 2024-09-15T03:46:41.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:41.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.112015+0000", 2024-09-15T03:46:41.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7283999999999999e-05 2024-09-15T03:46:41.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:41.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.112022+0000", 2024-09-15T03:46:41.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3560000000000004e-06 2024-09-15T03:46:41.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.112072+0000", 2024-09-15T03:46:41.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0102000000000002e-05 2024-09-15T03:46:41.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:41.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.112081+0000", 2024-09-15T03:46:41.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5609999999999999e-06 2024-09-15T03:46:41.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:41.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.112171+0000", 2024-09-15T03:46:41.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0031999999999995e-05 2024-09-15T03:46:41.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:41.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.112178+0000", 2024-09-15T03:46:41.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8850000000000001e-06 2024-09-15T03:46:41.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:41.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.112212+0000", 2024-09-15T03:46:41.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4207e-05 2024-09-15T03:46:41.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:41.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.112233+0000", 2024-09-15T03:46:41.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0491000000000001e-05 2024-09-15T03:46:41.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.673 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:41.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.113018+0000", 2024-09-15T03:46:41.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00078513599999999997 2024-09-15T03:46:41.673 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.673 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:41.673 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.673 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 34/31 ECSubWrite(tid=30, reqid=client.4521.0:192, at_version=34'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:46:41.673 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.113372+0000", 2024-09-15T03:46:41.673 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.47589001600000003, 2024-09-15T03:46:41.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000954882, 2024-09-15T03:46:41.673 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:41.673 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:41.674 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:41.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:41.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.113372+0000", 2024-09-15T03:46:41.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:41.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.113372+0000", 2024-09-15T03:46:41.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:41.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.113372+0000", 2024-09-15T03:46:41.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5599999999999999e-07 2024-09-15T03:46:41.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.675 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:41.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.113386+0000", 2024-09-15T03:46:41.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4052e-05 2024-09-15T03:46:41.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.675 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:41.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.113396+0000", 2024-09-15T03:46:41.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2669999999999995e-06 2024-09-15T03:46:41.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.676 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:41.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.113400+0000", 2024-09-15T03:46:41.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.493e-06 2024-09-15T03:46:41.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.676 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.113440+0000", 2024-09-15T03:46:41.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9685999999999999e-05 2024-09-15T03:46:41.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.676 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:41.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.113445+0000", 2024-09-15T03:46:41.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.429e-06 2024-09-15T03:46:41.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:41.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.113586+0000", 2024-09-15T03:46:41.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014060799999999999 2024-09-15T03:46:41.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:41.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.113594+0000", 2024-09-15T03:46:41.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9370000000000004e-06 2024-09-15T03:46:41.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:41.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.113629+0000", 2024-09-15T03:46:41.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5392000000000003e-05 2024-09-15T03:46:41.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:41.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.113645+0000", 2024-09-15T03:46:41.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5537999999999999e-05 2024-09-15T03:46:41.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:41.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.114327+0000", 2024-09-15T03:46:41.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00068232399999999999 2024-09-15T03:46:41.678 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.678 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:41.679 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.679 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 34/31 ECSubWrite(tid=31, reqid=client.4521.0:193, at_version=34'18, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:46:41.679 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.114599+0000", 2024-09-15T03:46:41.679 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.47466352299999998, 2024-09-15T03:46:41.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019551899999999999, 2024-09-15T03:46:41.679 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:41.679 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:41.679 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:41.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:41.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.114599+0000", 2024-09-15T03:46:41.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:41.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.114599+0000", 2024-09-15T03:46:41.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:41.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.114599+0000", 2024-09-15T03:46:41.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7199999999999999e-07 2024-09-15T03:46:41.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:41.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.114612+0000", 2024-09-15T03:46:41.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2839e-05 2024-09-15T03:46:41.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.681 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:41.681 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.114621+0000", 2024-09-15T03:46:41.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0089999999999996e-06 2024-09-15T03:46:41.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.681 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:41.681 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.114625+0000", 2024-09-15T03:46:41.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9110000000000003e-06 2024-09-15T03:46:41.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.681 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.682 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.114662+0000", 2024-09-15T03:46:41.682 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7308999999999998e-05 2024-09-15T03:46:41.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.682 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.682 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:41.682 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.114671+0000", 2024-09-15T03:46:41.682 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7970000000000006e-06 2024-09-15T03:46:41.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.682 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.682 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:41.682 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.114754+0000", 2024-09-15T03:46:41.682 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3026000000000001e-05 2024-09-15T03:46:41.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:41.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.114760+0000", 2024-09-15T03:46:41.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.592e-06 2024-09-15T03:46:41.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:41.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.114810+0000", 2024-09-15T03:46:41.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0448999999999998e-05 2024-09-15T03:46:41.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:41.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.114824+0000", 2024-09-15T03:46:41.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3879999999999999e-05 2024-09-15T03:46:41.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.684 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.684 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:41.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.116554+0000", 2024-09-15T03:46:41.684 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001729906 2024-09-15T03:46:41.684 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.684 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:41.684 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.684 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.684 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 34/31 ECSubWrite(tid=32, reqid=client.4521.0:193, at_version=0'0, trim_to=0'0, roll_forward_to=34'18))", 2024-09-15T03:46:41.684 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.115948+0000", 2024-09-15T03:46:41.684 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.47331391, 2024-09-15T03:46:41.684 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00074094500000000002, 2024-09-15T03:46:41.685 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:41.685 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:41.685 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:41.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.685 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:41.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.115948+0000", 2024-09-15T03:46:41.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.685 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:41.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.115948+0000", 2024-09-15T03:46:41.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.686 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:41.686 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.115949+0000", 2024-09-15T03:46:41.686 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0399999999999996e-07 2024-09-15T03:46:41.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.686 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:41.686 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.115958+0000", 2024-09-15T03:46:41.686 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7290000000000006e-06 2024-09-15T03:46:41.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.686 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:41.686 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.115968+0000", 2024-09-15T03:46:41.687 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0029e-05 2024-09-15T03:46:41.687 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.687 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.687 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:41.687 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.115974+0000", 2024-09-15T03:46:41.687 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1879999999999999e-06 2024-09-15T03:46:41.687 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.687 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.687 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.687 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.116014+0000", 2024-09-15T03:46:41.687 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0500999999999997e-05 2024-09-15T03:46:41.687 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.687 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.688 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:41.688 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.116027+0000", 2024-09-15T03:46:41.688 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2251e-05 2024-09-15T03:46:41.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.688 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.688 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:41.688 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.116104+0000", 2024-09-15T03:46:41.688 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7564999999999993e-05 2024-09-15T03:46:41.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.688 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.688 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:41.688 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.116111+0000", 2024-09-15T03:46:41.688 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2309999999999999e-06 2024-09-15T03:46:41.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.689 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:41.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.116169+0000", 2024-09-15T03:46:41.689 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7351e-05 2024-09-15T03:46:41.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.689 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:41.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.116193+0000", 2024-09-15T03:46:41.689 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4315999999999999e-05 2024-09-15T03:46:41.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.689 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:41.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.116689+0000", 2024-09-15T03:46:41.689 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00049618000000000004 2024-09-15T03:46:41.690 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.690 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:41.690 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.690 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.690 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es3 34/28 ECSubWrite(tid=33, reqid=client.4521.0:195, at_version=34'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-15T03:46:41.690 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.118213+0000", 2024-09-15T03:46:41.690 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.47104916299999999, 2024-09-15T03:46:41.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006662701, 2024-09-15T03:46:41.691 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:41.691 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:41.691 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:41.691 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.691 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:41.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118213+0000", 2024-09-15T03:46:41.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:41.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118213+0000", 2024-09-15T03:46:41.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:41.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118213+0000", 2024-09-15T03:46:41.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6300000000000001e-07 2024-09-15T03:46:41.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:41.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118447+0000", 2024-09-15T03:46:41.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000234016 2024-09-15T03:46:41.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:41.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118465+0000", 2024-09-15T03:46:41.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7978e-05 2024-09-15T03:46:41.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:41.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118475+0000", 2024-09-15T03:46:41.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3179999999999996e-06 2024-09-15T03:46:41.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.694 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118524+0000", 2024-09-15T03:46:41.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9069000000000001e-05 2024-09-15T03:46:41.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.694 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:41.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118534+0000", 2024-09-15T03:46:41.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0222e-05 2024-09-15T03:46:41.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.694 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:41.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118727+0000", 2024-09-15T03:46:41.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000193124 2024-09-15T03:46:41.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:41.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118734+0000", 2024-09-15T03:46:41.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2080000000000003e-06 2024-09-15T03:46:41.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:41.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118804+0000", 2024-09-15T03:46:41.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9720000000000003e-05 2024-09-15T03:46:41.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:41.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118820+0000", 2024-09-15T03:46:41.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5705999999999998e-05 2024-09-15T03:46:41.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:41.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124876+0000", 2024-09-15T03:46:41.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006056077 2024-09-15T03:46:41.696 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.696 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:41.696 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.697 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 34/31 ECSubWrite(tid=33, reqid=client.4521.0:194, at_version=34'19, trim_to=0'0, roll_forward_to=34'18))", 2024-09-15T03:46:41.697 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.123014+0000", 2024-09-15T03:46:41.697 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.46624771799999998, 2024-09-15T03:46:41.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001554539, 2024-09-15T03:46:41.697 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:41.697 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:41.697 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:41.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:41.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123014+0000", 2024-09-15T03:46:41.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:41.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123014+0000", 2024-09-15T03:46:41.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:41.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123015+0000", 2024-09-15T03:46:41.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.23e-07 2024-09-15T03:46:41.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:41.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123033+0000", 2024-09-15T03:46:41.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7723999999999998e-05 2024-09-15T03:46:41.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:41.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123047+0000", 2024-09-15T03:46:41.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4015e-05 2024-09-15T03:46:41.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:41.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123054+0000", 2024-09-15T03:46:41.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6880000000000002e-06 2024-09-15T03:46:41.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123092+0000", 2024-09-15T03:46:41.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8275999999999997e-05 2024-09-15T03:46:41.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:41.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123098+0000", 2024-09-15T03:46:41.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5420000000000002e-06 2024-09-15T03:46:41.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:41.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123160+0000", 2024-09-15T03:46:41.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1795e-05 2024-09-15T03:46:41.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:41.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123168+0000", 2024-09-15T03:46:41.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4359999999999996e-06 2024-09-15T03:46:41.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:41.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123197+0000", 2024-09-15T03:46:41.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9359000000000001e-05 2024-09-15T03:46:41.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:41.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123210+0000", 2024-09-15T03:46:41.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2498e-05 2024-09-15T03:46:41.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:41.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124569+0000", 2024-09-15T03:46:41.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013594830000000001 2024-09-15T03:46:41.702 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.702 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:41.702 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.702 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 34/31 ECSubWrite(tid=34, reqid=client.4521.0:197, at_version=34'21, trim_to=0'0, roll_forward_to=34'18))", 2024-09-15T03:46:41.702 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.123980+0000", 2024-09-15T03:46:41.702 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.46528233400000002, 2024-09-15T03:46:41.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0066547489999999997, 2024-09-15T03:46:41.703 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:41.703 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:41.703 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:41.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:41.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123980+0000", 2024-09-15T03:46:41.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:41.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123980+0000", 2024-09-15T03:46:41.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:41.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123980+0000", 2024-09-15T03:46:41.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8800000000000003e-07 2024-09-15T03:46:41.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:41.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123996+0000", 2024-09-15T03:46:41.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5838e-05 2024-09-15T03:46:41.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:41.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124005+0000", 2024-09-15T03:46:41.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1249999999999999e-06 2024-09-15T03:46:41.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:41.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124010+0000", 2024-09-15T03:46:41.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2640000000000002e-06 2024-09-15T03:46:41.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124045+0000", 2024-09-15T03:46:41.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5145000000000003e-05 2024-09-15T03:46:41.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:41.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124054+0000", 2024-09-15T03:46:41.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8389999999999996e-06 2024-09-15T03:46:41.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:41.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124696+0000", 2024-09-15T03:46:41.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00064247499999999999 2024-09-15T03:46:41.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:41.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124703+0000", 2024-09-15T03:46:41.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.905e-06 2024-09-15T03:46:41.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:41.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124736+0000", 2024-09-15T03:46:41.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2966999999999998e-05 2024-09-15T03:46:41.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:41.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124758+0000", 2024-09-15T03:46:41.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2569999999999999e-05 2024-09-15T03:46:41.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:41.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.130635+0000", 2024-09-15T03:46:41.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058761330000000004 2024-09-15T03:46:41.707 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.708 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:41.708 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.708 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es3 34/28 ECSubWrite(tid=34, reqid=client.4521.0:195, at_version=0'0, trim_to=0'0, roll_forward_to=34'16))", 2024-09-15T03:46:41.708 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.124831+0000", 2024-09-15T03:46:41.708 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.464431129, 2024-09-15T03:46:41.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0028221240000000001, 2024-09-15T03:46:41.708 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:41.708 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:41.708 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:41.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.708 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:41.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124831+0000", 2024-09-15T03:46:41.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:41.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124831+0000", 2024-09-15T03:46:41.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:41.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124832+0000", 2024-09-15T03:46:41.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5000000000000003e-07 2024-09-15T03:46:41.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:41.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124851+0000", 2024-09-15T03:46:41.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9831e-05 2024-09-15T03:46:41.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:41.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124866+0000", 2024-09-15T03:46:41.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4333e-05 2024-09-15T03:46:41.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:41.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124872+0000", 2024-09-15T03:46:41.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3559999999999998e-06 2024-09-15T03:46:41.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124928+0000", 2024-09-15T03:46:41.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.63e-05 2024-09-15T03:46:41.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:41.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124938+0000", 2024-09-15T03:46:41.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6600000000000007e-06 2024-09-15T03:46:41.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:41.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125028+0000", 2024-09-15T03:46:41.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9687999999999999e-05 2024-09-15T03:46:41.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.712 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:41.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125046+0000", 2024-09-15T03:46:41.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8338999999999998e-05 2024-09-15T03:46:41.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.712 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:41.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125092+0000", 2024-09-15T03:46:41.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5544000000000003e-05 2024-09-15T03:46:41.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.712 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:41.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125104+0000", 2024-09-15T03:46:41.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2442e-05 2024-09-15T03:46:41.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.713 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:41.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.127653+0000", 2024-09-15T03:46:41.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025490809999999999 2024-09-15T03:46:41.713 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.713 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:41.713 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.713 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 34/31 ECSubWrite(tid=35, reqid=client.4521.0:197, at_version=0'0, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:46:41.714 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.125913+0000", 2024-09-15T03:46:41.714 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.46334941800000001, 2024-09-15T03:46:41.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0048403999999999999, 2024-09-15T03:46:41.714 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:41.714 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:41.714 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:41.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.714 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:41.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125913+0000", 2024-09-15T03:46:41.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.714 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:41.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125913+0000", 2024-09-15T03:46:41.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:41.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:41.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125913+0000", 2024-09-15T03:46:41.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3499999999999996e-07 2024-09-15T03:46:41.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:41.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125924+0000", 2024-09-15T03:46:41.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0669000000000001e-05 2024-09-15T03:46:41.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:41.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125935+0000", 2024-09-15T03:46:41.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1225e-05 2024-09-15T03:46:41.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:41.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125941+0000", 2024-09-15T03:46:41.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2519999999999996e-06 2024-09-15T03:46:41.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:41.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125979+0000", 2024-09-15T03:46:41.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7765999999999997e-05 2024-09-15T03:46:41.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:41.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125989+0000", 2024-09-15T03:46:41.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5950000000000005e-06 2024-09-15T03:46:41.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:41.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.126092+0000", 2024-09-15T03:46:41.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000103132 2024-09-15T03:46:41.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:41.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.126097+0000", 2024-09-15T03:46:41.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0220000000000004e-06 2024-09-15T03:46:41.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:41.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.126129+0000", 2024-09-15T03:46:41.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1668999999999998e-05 2024-09-15T03:46:41.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:41.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.126136+0000", 2024-09-15T03:46:41.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6179999999999997e-06 2024-09-15T03:46:41.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:41.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:41.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:41.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.130753+0000", 2024-09-15T03:46:41.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0046169169999999999 2024-09-15T03:46:41.719 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.719 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:41.719 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.719 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:41.719 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:41.719 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:41.720 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:46:41.720 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:46:41.721 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:46:41.722 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:46:41.722 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:41.718+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:41.722 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:46:41.725 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:46:41.725 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:46:41.740 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:46:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:46:41.755 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:46:41.756 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:46:41.770 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:46:41.771 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:46:41.786 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:46:41.786 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:46:41.801 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:46:41.802 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:46:41.815 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:46:41.815 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:46:41.820 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:41.819+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:41.825 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:41.825 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:41.830 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:46:41.830 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:46:41.834 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:46:41.845 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:46:41.845 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:46:41.859 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:46:41.859 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:46:41.922 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:41.921+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:41.946 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:41.946 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:41.954 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:46:42.022 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:42.022+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:46:42.095 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:42.095 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:42.104 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:46:42.116 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-15T03:46:42.116 INFO:tasks.rados.rados.0.smithi001.stdout:71: done (0 left) 2024-09-15T03:46:42.116 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 27 current snap is 3 2024-09-15T03:46:42.116 INFO:tasks.rados.rados.0.smithi001.stdout:72: seq_num 57 ranges {442368=589824,1032192=376832} 2024-09-15T03:46:42.121 INFO:tasks.rados.rados.0.smithi001.stdout:72: writing smithi00132172-27 from 442368 to 1032192 tid 1 2024-09-15T03:46:42.124 INFO:tasks.rados.rados.0.smithi001.stdout:72: writing smithi00132172-27 from 1032192 to 1409024 tid 2 2024-09-15T03:46:42.124 INFO:tasks.rados.rados.0.smithi001.stdout:73: delete oid 28 current snap is 3 2024-09-15T03:46:42.124 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:42.123+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:46:42.129 INFO:tasks.rados.rados.0.smithi001.stdout:72: finishing write tid 1 to smithi00132172-27 2024-09-15T03:46:42.130 INFO:tasks.rados.rados.0.smithi001.stdout:72: finishing write tid 2 to smithi00132172-27 2024-09-15T03:46:42.130 INFO:tasks.rados.rados.0.smithi001.stdout:72: finishing write tid 3 to smithi00132172-27 2024-09-15T03:46:42.132 INFO:tasks.rados.rados.0.smithi001.stdout:73: done (1 left) 2024-09-15T03:46:42.132 INFO:tasks.rados.rados.0.smithi001.stdout:74: delete oid 30 current snap is 3 2024-09-15T03:46:42.135 INFO:tasks.rados.rados.0.smithi001.stdout:74: done (1 left) 2024-09-15T03:46:42.135 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 14 current snap is 3 2024-09-15T03:46:42.135 INFO:tasks.rados.rados.0.smithi001.stdout:75: seq_num 60 ranges {229376=638976,868352=376832} 2024-09-15T03:46:42.140 INFO:tasks.rados.rados.0.smithi001.stdout:75: writing smithi00132172-14 from 229376 to 868352 tid 1 2024-09-15T03:46:42.143 INFO:tasks.rados.rados.0.smithi001.stdout:75: writing smithi00132172-14 from 868352 to 1245184 tid 2 2024-09-15T03:46:42.143 INFO:tasks.rados.rados.0.smithi001.stdout:76: delete oid 42 current snap is 3 2024-09-15T03:46:42.143 INFO:tasks.rados.rados.0.smithi001.stdout:72: finishing write tid 4 to smithi00132172-27 2024-09-15T03:46:42.143 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 27 v 6 (ObjNum 57 snap 3 seq_num 57) dirty exists 2024-09-15T03:46:42.143 INFO:tasks.rados.rados.0.smithi001.stdout:72: left oid 27 (ObjNum 57 snap 3 seq_num 57) 2024-09-15T03:46:42.145 INFO:tasks.rados.rados.0.smithi001.stdout:75: finishing write tid 1 to smithi00132172-14 2024-09-15T03:46:42.146 INFO:tasks.rados.rados.0.smithi001.stdout:75: finishing write tid 2 to smithi00132172-14 2024-09-15T03:46:42.146 INFO:tasks.rados.rados.0.smithi001.stdout:75: finishing write tid 3 to smithi00132172-14 2024-09-15T03:46:42.146 INFO:tasks.rados.rados.0.smithi001.stdout:72: done (2 left) 2024-09-15T03:46:42.147 INFO:tasks.rados.rados.0.smithi001.stdout:76: done (1 left) 2024-09-15T03:46:42.147 INFO:tasks.rados.rados.0.smithi001.stdout:77: read oid 5 snap -1 2024-09-15T03:46:42.147 INFO:tasks.rados.rados.0.smithi001.stdout:77: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-15T03:46:42.147 INFO:tasks.rados.rados.0.smithi001.stdout:78: rollback oid 13 current snap is 3 2024-09-15T03:46:42.147 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 13 to 0 2024-09-15T03:46:42.147 INFO:tasks.rados.rados.0.smithi001.stdout:79: snap_create 2024-09-15T03:46:42.148 INFO:tasks.rados.rados.0.smithi001.stdout:75: finishing write tid 4 to smithi00132172-14 2024-09-15T03:46:42.148 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 14 v 22 (ObjNum 60 snap 3 seq_num 60) dirty exists 2024-09-15T03:46:42.148 INFO:tasks.rados.rados.0.smithi001.stdout:75: left oid 14 (ObjNum 60 snap 3 seq_num 60) 2024-09-15T03:46:42.151 INFO:tasks.rados.rados.0.smithi001.stdout:78: finishing rollback tid 0 to smithi00132172-13 2024-09-15T03:46:42.151 INFO:tasks.rados.rados.0.smithi001.stdout:78: finishing rollback tid 1 to smithi00132172-13 2024-09-15T03:46:42.151 INFO:tasks.rados.rados.0.smithi001.stdout:78: finishing rollback tid 2 to smithi00132172-13 2024-09-15T03:46:42.151 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 13 v 12 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-15T03:46:42.153 INFO:tasks.rados.rados.0.smithi001.stdout:77: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-15T03:46:42.216 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:42.216 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:42.225 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:46:42.225 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:42.224+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:46:42.326 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:42.324+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:46:42.365 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:42.419 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:42.420 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:46:42.427 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:42.426+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:42.520 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:42.521 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:42.528 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:42.527+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:46:42.529 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:46:42.629 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:42.627+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:42.671 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:42.671 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:42.679 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:46:42.729 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:46:42.729+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:46:42.779 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:42.780 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:42.788 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:46:42.831 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:42.831+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:46:42.888 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:42.888 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:42.897 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:46:42.932 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:42.931+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:46:43.033 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:43.032+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:43.037 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:43.037 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:43.046 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:46:43.123 INFO:tasks.rados.rados.0.smithi001.stdout:75: done (3 left) 2024-09-15T03:46:43.123 INFO:tasks.rados.rados.0.smithi001.stdout:77: done (2 left) 2024-09-15T03:46:43.123 INFO:tasks.rados.rados.0.smithi001.stdout:78: done (1 left) 2024-09-15T03:46:43.123 INFO:tasks.rados.rados.0.smithi001.stdout:79: done (0 left) 2024-09-15T03:46:43.123 INFO:tasks.rados.rados.0.smithi001.stdout:80: snap_create 2024-09-15T03:46:43.127 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-15T03:46:43.127 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T03:46:43.127 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-15T03:46:43.127 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-15T03:46:43.127 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-15T03:46:43.127 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-09-15T03:46:43.133 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:43.131+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:43.148 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:43.148 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:43.157 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:46:43.234 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:43.233+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:46:43.240 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:43.240 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-15T03:46:43.240 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:43.258 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:43.259 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:43.267 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:46:43.335 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:43.334+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:43.392 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:43.392 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:43.400 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:46:43.436 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:43.435+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:43.500 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:43.501 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:43.509 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:46:43.553 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:43.552+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:46:43.609 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:43.609 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:43.618 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:46:43.660 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:43.658+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:46:43.739 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:43.739+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:46:43.759 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:43.759 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:43.767 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:46:43.839 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:43.839+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:46:43.867 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:43.867 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:43.871 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:46:43.875 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:46:43.941 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:43.940+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:46:43.976 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:43.976 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:43.985 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:46:43.994 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:46:43.994 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:46:44.008 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:46:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:46:44.023 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:46:44.023 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:46:44.037 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:46:44.037 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:46:44.041 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:44.040+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:44.052 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:46:44.053 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:46:44.067 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:46:44.067 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:46:44.081 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:46:44.081 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:46:44.087 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:44.087 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:46:44.088 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:46:44.088 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:44.095 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:46:44.096 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:46:44.097 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:46:44.110 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:46:44.111 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:46:44.125 INFO:tasks.rados.rados.0.smithi001.stdout:80: done (0 left) 2024-09-15T03:46:44.125 INFO:tasks.rados.rados.0.smithi001.stdout:81: snap_create 2024-09-15T03:46:44.126 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:46:44.127 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:46:44.133 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T03:46:44.141 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:46:44.142 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:46:44.142 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:44.140+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:44.156 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:46:44.157 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:46:44.199 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:44.199 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:46:44.199 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:46:44.199 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:46:44.199 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:44.208 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:46:44.252 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:44.251+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:46:44.311 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:44.311 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:46:44.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:46:44.312 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:46:44.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.312 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:135 3.4s0 3:2d1c787c:::smithi00132172-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-15T03:46:44.312 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.488612+0000", 2024-09-15T03:46:44.312 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.8209674659999999, 2024-09-15T03:46:44.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004050606, 2024-09-15T03:46:44.312 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.312 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:46:44.312 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:44.312 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:44.312 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:44.312 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 135 2024-09-15T03:46:44.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.313 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.488612+0000", 2024-09-15T03:46:44.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.488612+0000", 2024-09-15T03:46:44.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.488612+0000", 2024-09-15T03:46:44.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.08e-07 2024-09-15T03:46:44.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.488614+0000", 2024-09-15T03:46:44.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1160000000000002e-06 2024-09-15T03:46:44.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.488624+0000", 2024-09-15T03:46:44.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5729999999999997e-06 2024-09-15T03:46:44.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.488628+0000", 2024-09-15T03:46:44.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8739999999999997e-06 2024-09-15T03:46:44.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.490440+0000", 2024-09-15T03:46:44.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001812632 2024-09-15T03:46:44.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:46:44.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.490504+0000", 2024-09-15T03:46:44.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3848000000000003e-05 2024-09-15T03:46:44.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.491725+0000", 2024-09-15T03:46:44.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001221188 2024-09-15T03:46:44.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:44.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.491807+0000", 2024-09-15T03:46:44.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1845e-05 2024-09-15T03:46:44.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.492662+0000", 2024-09-15T03:46:44.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085512200000000004 2024-09-15T03:46:44.317 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.317 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.317 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.317 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:161 3.3s0 3:c7c3ef16:::smithi00132172-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-15T03:46:44.317 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.529329+0000", 2024-09-15T03:46:44.317 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7802505010000003, 2024-09-15T03:46:44.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042940230000000001, 2024-09-15T03:46:44.317 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.317 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:46:44.317 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:44.318 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:44.318 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:44.318 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 161 2024-09-15T03:46:44.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.318 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.529329+0000", 2024-09-15T03:46:44.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.529329+0000", 2024-09-15T03:46:44.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.529329+0000", 2024-09-15T03:46:44.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2200000000000004e-07 2024-09-15T03:46:44.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.529344+0000", 2024-09-15T03:46:44.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4447000000000001e-05 2024-09-15T03:46:44.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.529353+0000", 2024-09-15T03:46:44.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2730000000000008e-06 2024-09-15T03:46:44.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.529359+0000", 2024-09-15T03:46:44.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7490000000000004e-06 2024-09-15T03:46:44.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.531410+0000", 2024-09-15T03:46:44.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020513760000000002 2024-09-15T03:46:44.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:46:44.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.531489+0000", 2024-09-15T03:46:44.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9102000000000003e-05 2024-09-15T03:46:44.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532453+0000", 2024-09-15T03:46:44.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096348400000000002 2024-09-15T03:46:44.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:44.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.532525+0000", 2024-09-15T03:46:44.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2105000000000002e-05 2024-09-15T03:46:44.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.533623+0000", 2024-09-15T03:46:44.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001097965 2024-09-15T03:46:44.322 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.323 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.323 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.323 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:171 3.4s0 3:2d1c787c:::smithi00132172-45:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-15T03:46:44.323 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.553202+0000", 2024-09-15T03:46:44.323 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7563770979999997, 2024-09-15T03:46:44.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0083718339999999999, 2024-09-15T03:46:44.323 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.323 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:44.323 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:44.324 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:44.324 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:44.324 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 171 2024-09-15T03:46:44.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.324 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553202+0000", 2024-09-15T03:46:44.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553202+0000", 2024-09-15T03:46:44.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553203+0000", 2024-09-15T03:46:44.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.122e-06 2024-09-15T03:46:44.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553207+0000", 2024-09-15T03:46:44.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2200000000000003e-06 2024-09-15T03:46:44.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553219+0000", 2024-09-15T03:46:44.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1316e-05 2024-09-15T03:46:44.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553224+0000", 2024-09-15T03:46:44.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6710000000000004e-06 2024-09-15T03:46:44.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553265+0000", 2024-09-15T03:46:44.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0939000000000002e-05 2024-09-15T03:46:44.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:44.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553341+0000", 2024-09-15T03:46:44.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5939000000000006e-05 2024-09-15T03:46:44.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559019+0000", 2024-09-15T03:46:44.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056774299999999998 2024-09-15T03:46:44.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559137+0000", 2024-09-15T03:46:44.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117751 2024-09-15T03:46:44.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559213+0000", 2024-09-15T03:46:44.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6882e-05 2024-09-15T03:46:44.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559551+0000", 2024-09-15T03:46:44.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033777900000000002 2024-09-15T03:46:44.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559571+0000", 2024-09-15T03:46:44.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9728000000000002e-05 2024-09-15T03:46:44.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:44.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.560343+0000", 2024-09-15T03:46:44.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00077172899999999997 2024-09-15T03:46:44.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.561574+0000", 2024-09-15T03:46:44.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012313280000000001 2024-09-15T03:46:44.330 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.330 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.330 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.330 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:172 3.4s0 3:2d1c787c:::smithi00132172-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-15T03:46:44.330 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.553234+0000", 2024-09-15T03:46:44.330 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7563452509999999, 2024-09-15T03:46:44.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062080829999999997, 2024-09-15T03:46:44.330 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.330 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:46:44.330 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:44.330 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:44.330 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:44.331 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 172 2024-09-15T03:46:44.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.331 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553234+0000", 2024-09-15T03:46:44.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553234+0000", 2024-09-15T03:46:44.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553235+0000", 2024-09-15T03:46:44.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2600000000000002e-07 2024-09-15T03:46:44.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553237+0000", 2024-09-15T03:46:44.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4200000000000001e-06 2024-09-15T03:46:44.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553245+0000", 2024-09-15T03:46:44.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5870000000000004e-06 2024-09-15T03:46:44.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553249+0000", 2024-09-15T03:46:44.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2309999999999998e-06 2024-09-15T03:46:44.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553479+0000", 2024-09-15T03:46:44.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023039299999999999 2024-09-15T03:46:44.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:46:44.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553549+0000", 2024-09-15T03:46:44.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9381999999999994e-05 2024-09-15T03:46:44.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559254+0000", 2024-09-15T03:46:44.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057053210000000002 2024-09-15T03:46:44.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:44.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559348+0000", 2024-09-15T03:46:44.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3715000000000007e-05 2024-09-15T03:46:44.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559442+0000", 2024-09-15T03:46:44.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4507999999999997e-05 2024-09-15T03:46:44.336 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.336 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.336 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.336 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.cs5 32/28 ECSubWrite(tid=8, reqid=client.4521.0:175, at_version=32'5, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:44.337 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.553774+0000", 2024-09-15T03:46:44.337 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7558048050000004, 2024-09-15T03:46:44.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042934490000000004, 2024-09-15T03:46:44.337 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.337 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.337 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553774+0000", 2024-09-15T03:46:44.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553774+0000", 2024-09-15T03:46:44.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553775+0000", 2024-09-15T03:46:44.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9299999999999997e-07 2024-09-15T03:46:44.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553797+0000", 2024-09-15T03:46:44.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1699999999999999e-05 2024-09-15T03:46:44.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553811+0000", 2024-09-15T03:46:44.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4566000000000001e-05 2024-09-15T03:46:44.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553818+0000", 2024-09-15T03:46:44.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7050000000000002e-06 2024-09-15T03:46:44.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553907+0000", 2024-09-15T03:46:44.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8900999999999995e-05 2024-09-15T03:46:44.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553918+0000", 2024-09-15T03:46:44.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0542e-05 2024-09-15T03:46:44.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554010+0000", 2024-09-15T03:46:44.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2868000000000006e-05 2024-09-15T03:46:44.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554032+0000", 2024-09-15T03:46:44.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1826000000000001e-05 2024-09-15T03:46:44.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554081+0000", 2024-09-15T03:46:44.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9067999999999998e-05 2024-09-15T03:46:44.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554100+0000", 2024-09-15T03:46:44.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9123e-05 2024-09-15T03:46:44.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.558068+0000", 2024-09-15T03:46:44.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039674569999999998 2024-09-15T03:46:44.342 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.342 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.342 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.343 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.cs5 32/28 ECSubWrite(tid=9, reqid=client.4521.0:175, at_version=0'0, trim_to=0'0, roll_forward_to=32'5))", 2024-09-15T03:46:44.343 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.554461+0000", 2024-09-15T03:46:44.343 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7551179450000003, 2024-09-15T03:46:44.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037930450000000001, 2024-09-15T03:46:44.343 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.343 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.343 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554461+0000", 2024-09-15T03:46:44.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554461+0000", 2024-09-15T03:46:44.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554462+0000", 2024-09-15T03:46:44.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.61e-07 2024-09-15T03:46:44.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554472+0000", 2024-09-15T03:46:44.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6709999999999994e-06 2024-09-15T03:46:44.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554485+0000", 2024-09-15T03:46:44.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2869e-05 2024-09-15T03:46:44.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554490+0000", 2024-09-15T03:46:44.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.451e-06 2024-09-15T03:46:44.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554533+0000", 2024-09-15T03:46:44.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3420000000000001e-05 2024-09-15T03:46:44.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554542+0000", 2024-09-15T03:46:44.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1499999999999999e-06 2024-09-15T03:46:44.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554610+0000", 2024-09-15T03:46:44.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7886999999999996e-05 2024-09-15T03:46:44.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.554626+0000", 2024-09-15T03:46:44.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6300999999999999e-05 2024-09-15T03:46:44.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.555276+0000", 2024-09-15T03:46:44.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00065056499999999998 2024-09-15T03:46:44.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.555372+0000", 2024-09-15T03:46:44.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5408000000000005e-05 2024-09-15T03:46:44.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.558254+0000", 2024-09-15T03:46:44.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002882562 2024-09-15T03:46:44.348 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.349 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.349 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.349 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=21, reqid=client.4521.0:184, at_version=33'6, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:44.349 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:46:44.349 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.2030701280000002, 2024-09-15T03:46:44.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010901256999999999, 2024-09-15T03:46:44.349 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.350 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.350 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:46:44.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:46:44.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:46:44.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0200000000000001e-07 2024-09-15T03:46:44.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106529+0000", 2024-09-15T03:46:44.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9539e-05 2024-09-15T03:46:44.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106542+0000", 2024-09-15T03:46:44.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2731999999999999e-05 2024-09-15T03:46:44.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106549+0000", 2024-09-15T03:46:44.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3339999999999996e-06 2024-09-15T03:46:44.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106613+0000", 2024-09-15T03:46:44.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4527000000000001e-05 2024-09-15T03:46:44.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106622+0000", 2024-09-15T03:46:44.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.755e-06 2024-09-15T03:46:44.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106841+0000", 2024-09-15T03:46:44.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021906200000000001 2024-09-15T03:46:44.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106869+0000", 2024-09-15T03:46:44.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.739e-05 2024-09-15T03:46:44.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106948+0000", 2024-09-15T03:46:44.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9422e-05 2024-09-15T03:46:44.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106965+0000", 2024-09-15T03:46:44.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7010000000000001e-05 2024-09-15T03:46:44.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.117410+0000", 2024-09-15T03:46:44.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010445284000000001 2024-09-15T03:46:44.355 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.355 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.355 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.355 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=22, reqid=client.4521.0:187, at_version=33'8, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:44.355 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.106795+0000", 2024-09-15T03:46:44.355 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.202784533, 2024-09-15T03:46:44.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0075031530000000003, 2024-09-15T03:46:44.355 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.355 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.355 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106795+0000", 2024-09-15T03:46:44.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106795+0000", 2024-09-15T03:46:44.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106795+0000", 2024-09-15T03:46:44.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1900000000000002e-07 2024-09-15T03:46:44.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106814+0000", 2024-09-15T03:46:44.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8376000000000002e-05 2024-09-15T03:46:44.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106825+0000", 2024-09-15T03:46:44.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1096e-05 2024-09-15T03:46:44.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106832+0000", 2024-09-15T03:46:44.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4939999999999997e-06 2024-09-15T03:46:44.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106877+0000", 2024-09-15T03:46:44.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4799999999999998e-05 2024-09-15T03:46:44.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106888+0000", 2024-09-15T03:46:44.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0621e-05 2024-09-15T03:46:44.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.107075+0000", 2024-09-15T03:46:44.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000187104 2024-09-15T03:46:44.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.107099+0000", 2024-09-15T03:46:44.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4369e-05 2024-09-15T03:46:44.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.107135+0000", 2024-09-15T03:46:44.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.553e-05 2024-09-15T03:46:44.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.107453+0000", 2024-09-15T03:46:44.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031841299999999998 2024-09-15T03:46:44.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114298+0000", 2024-09-15T03:46:44.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068447309999999997 2024-09-15T03:46:44.360 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.360 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.360 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.361 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:44.361 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:46:44.361 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.2011566409999999, 2024-09-15T03:46:44.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015296944, 2024-09-15T03:46:44.361 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.361 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.361 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:46:44.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:46:44.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:46:44.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5499999999999998e-07 2024-09-15T03:46:44.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108447+0000", 2024-09-15T03:46:44.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3314000000000001e-05 2024-09-15T03:46:44.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108460+0000", 2024-09-15T03:46:44.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3444e-05 2024-09-15T03:46:44.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108464+0000", 2024-09-15T03:46:44.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3610000000000003e-06 2024-09-15T03:46:44.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108494+0000", 2024-09-15T03:46:44.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9688000000000001e-05 2024-09-15T03:46:44.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108502+0000", 2024-09-15T03:46:44.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.678e-06 2024-09-15T03:46:44.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108601+0000", 2024-09-15T03:46:44.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9748999999999997e-05 2024-09-15T03:46:44.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108610+0000", 2024-09-15T03:46:44.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2490000000000001e-06 2024-09-15T03:46:44.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109977+0000", 2024-09-15T03:46:44.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013669450000000001 2024-09-15T03:46:44.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109993+0000", 2024-09-15T03:46:44.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6688e-05 2024-09-15T03:46:44.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123720+0000", 2024-09-15T03:46:44.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013726172999999999 2024-09-15T03:46:44.366 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.366 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.366 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.366 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:44.366 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:46:44.366 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.2011069799999996, 2024-09-15T03:46:44.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015105210000000001, 2024-09-15T03:46:44.367 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.367 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.367 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:46:44.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:46:44.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108473+0000", 2024-09-15T03:46:44.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.84e-07 2024-09-15T03:46:44.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108486+0000", 2024-09-15T03:46:44.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3845999999999999e-05 2024-09-15T03:46:44.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108491+0000", 2024-09-15T03:46:44.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4719999999999997e-06 2024-09-15T03:46:44.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108495+0000", 2024-09-15T03:46:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.822e-06 2024-09-15T03:46:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108546+0000", 2024-09-15T03:46:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0853000000000003e-05 2024-09-15T03:46:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108555+0000", 2024-09-15T03:46:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3319999999999998e-06 2024-09-15T03:46:44.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108638+0000", 2024-09-15T03:46:44.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3580999999999998e-05 2024-09-15T03:46:44.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109983+0000", 2024-09-15T03:46:44.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001344392 2024-09-15T03:46:44.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110023+0000", 2024-09-15T03:46:44.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9895999999999999e-05 2024-09-15T03:46:44.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110040+0000", 2024-09-15T03:46:44.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.747e-05 2024-09-15T03:46:44.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123577+0000", 2024-09-15T03:46:44.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013537262 2024-09-15T03:46:44.371 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.371 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.372 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.372 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:44.372 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:46:44.372 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.2010731010000004, 2024-09-15T03:46:44.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015311313999999999, 2024-09-15T03:46:44.372 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.372 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.372 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:46:44.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:46:44.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:46:44.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8500000000000002e-07 2024-09-15T03:46:44.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108517+0000", 2024-09-15T03:46:44.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0543000000000001e-05 2024-09-15T03:46:44.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108524+0000", 2024-09-15T03:46:44.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7170000000000002e-06 2024-09-15T03:46:44.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108528+0000", 2024-09-15T03:46:44.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3970000000000004e-06 2024-09-15T03:46:44.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108632+0000", 2024-09-15T03:46:44.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103687 2024-09-15T03:46:44.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108637+0000", 2024-09-15T03:46:44.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4389999999999999e-06 2024-09-15T03:46:44.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108713+0000", 2024-09-15T03:46:44.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6223000000000005e-05 2024-09-15T03:46:44.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109984+0000", 2024-09-15T03:46:44.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012700719999999999 2024-09-15T03:46:44.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110023+0000", 2024-09-15T03:46:44.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9864999999999999e-05 2024-09-15T03:46:44.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110041+0000", 2024-09-15T03:46:44.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8040999999999999e-05 2024-09-15T03:46:44.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123817+0000", 2024-09-15T03:46:44.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013776045000000001 2024-09-15T03:46:44.377 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.377 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.377 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.378 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:44.378 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:46:44.378 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.2005794930000002, 2024-09-15T03:46:44.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014864502, 2024-09-15T03:46:44.378 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.378 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.378 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:46:44.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:46:44.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:46:44.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:46:44.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109016+0000", 2024-09-15T03:46:44.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5889999999999999e-05 2024-09-15T03:46:44.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109029+0000", 2024-09-15T03:46:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2812e-05 2024-09-15T03:46:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109035+0000", 2024-09-15T03:46:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0109999999999997e-06 2024-09-15T03:46:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109073+0000", 2024-09-15T03:46:44.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8260000000000003e-05 2024-09-15T03:46:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109080+0000", 2024-09-15T03:46:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0190000000000001e-06 2024-09-15T03:46:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109165+0000", 2024-09-15T03:46:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4957000000000007e-05 2024-09-15T03:46:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109984+0000", 2024-09-15T03:46:44.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081866399999999998 2024-09-15T03:46:44.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110024+0000", 2024-09-15T03:46:44.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9644999999999997e-05 2024-09-15T03:46:44.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110072+0000", 2024-09-15T03:46:44.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8418999999999998e-05 2024-09-15T03:46:44.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123864+0000", 2024-09-15T03:46:44.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013792234 2024-09-15T03:46:44.383 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.383 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.383 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.383 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=27, reqid=client.4521.0:189, at_version=0'0, trim_to=0'0, roll_forward_to=33'10))", 2024-09-15T03:46:44.383 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.114624+0000", 2024-09-15T03:46:44.383 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.1949555040000002, 2024-09-15T03:46:44.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092946850000000004, 2024-09-15T03:46:44.384 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.384 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.384 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114624+0000", 2024-09-15T03:46:44.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114624+0000", 2024-09-15T03:46:44.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114624+0000", 2024-09-15T03:46:44.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3600000000000004e-07 2024-09-15T03:46:44.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114633+0000", 2024-09-15T03:46:44.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1579999999999994e-06 2024-09-15T03:46:44.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114644+0000", 2024-09-15T03:46:44.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.04e-05 2024-09-15T03:46:44.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114649+0000", 2024-09-15T03:46:44.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8760000000000004e-06 2024-09-15T03:46:44.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114687+0000", 2024-09-15T03:46:44.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8438e-05 2024-09-15T03:46:44.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114697+0000", 2024-09-15T03:46:44.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6479999999999998e-06 2024-09-15T03:46:44.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114818+0000", 2024-09-15T03:46:44.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121116 2024-09-15T03:46:44.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114830+0000", 2024-09-15T03:46:44.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1844e-05 2024-09-15T03:46:44.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114921+0000", 2024-09-15T03:46:44.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1461999999999999e-05 2024-09-15T03:46:44.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114934+0000", 2024-09-15T03:46:44.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2857000000000001e-05 2024-09-15T03:46:44.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123918+0000", 2024-09-15T03:46:44.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089843500000000003 2024-09-15T03:46:44.389 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.389 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.389 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.389 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=28, reqid=client.4521.0:186, at_version=0'0, trim_to=0'0, roll_forward_to=33'8))", 2024-09-15T03:46:44.389 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.114815+0000", 2024-09-15T03:46:44.389 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.1947646470000004, 2024-09-15T03:46:44.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.009158846, 2024-09-15T03:46:44.389 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.389 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.390 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114815+0000", 2024-09-15T03:46:44.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114815+0000", 2024-09-15T03:46:44.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114815+0000", 2024-09-15T03:46:44.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.75e-07 2024-09-15T03:46:44.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114824+0000", 2024-09-15T03:46:44.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9819999999999997e-06 2024-09-15T03:46:44.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114839+0000", 2024-09-15T03:46:44.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4521999999999999e-05 2024-09-15T03:46:44.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114845+0000", 2024-09-15T03:46:44.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7399999999999998e-06 2024-09-15T03:46:44.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114910+0000", 2024-09-15T03:46:44.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4492999999999994e-05 2024-09-15T03:46:44.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114919+0000", 2024-09-15T03:46:44.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9299999999999992e-06 2024-09-15T03:46:44.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.115022+0000", 2024-09-15T03:46:44.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103019 2024-09-15T03:46:44.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.115030+0000", 2024-09-15T03:46:44.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6690000000000006e-06 2024-09-15T03:46:44.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.115056+0000", 2024-09-15T03:46:44.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6001000000000001e-05 2024-09-15T03:46:44.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.115070+0000", 2024-09-15T03:46:44.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4552999999999999e-05 2024-09-15T03:46:44.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123973+0000", 2024-09-15T03:46:44.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089033619999999997 2024-09-15T03:46:44.395 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.395 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.395 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.395 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.2s5 34/31 ECSubWrite(tid=46, reqid=client.4521.0:191, at_version=34'9, trim_to=0'0, roll_forward_to=32'7))", 2024-09-15T03:46:44.395 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.109201+0000", 2024-09-15T03:46:44.395 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.2003783609999998, 2024-09-15T03:46:44.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052504969999999998, 2024-09-15T03:46:44.395 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.395 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.395 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.109201+0000", 2024-09-15T03:46:44.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.109201+0000", 2024-09-15T03:46:44.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.109201+0000", 2024-09-15T03:46:44.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3699999999999999e-07 2024-09-15T03:46:44.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.109242+0000", 2024-09-15T03:46:44.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0692999999999998e-05 2024-09-15T03:46:44.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.109257+0000", 2024-09-15T03:46:44.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5656000000000001e-05 2024-09-15T03:46:44.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.109267+0000", 2024-09-15T03:46:44.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3300000000000005e-06 2024-09-15T03:46:44.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.109440+0000", 2024-09-15T03:46:44.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017318600000000001 2024-09-15T03:46:44.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.109452+0000", 2024-09-15T03:46:44.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1678000000000001e-05 2024-09-15T03:46:44.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.109741+0000", 2024-09-15T03:46:44.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028973699999999998 2024-09-15T03:46:44.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.109757+0000", 2024-09-15T03:46:44.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5647000000000001e-05 2024-09-15T03:46:44.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.109915+0000", 2024-09-15T03:46:44.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015837800000000001 2024-09-15T03:46:44.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.109982+0000", 2024-09-15T03:46:44.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6241999999999994e-05 2024-09-15T03:46:44.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114451+0000", 2024-09-15T03:46:44.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0044697130000000002 2024-09-15T03:46:44.401 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.401 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.401 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.401 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=30, reqid=client.4521.0:192, at_version=34'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:46:44.401 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.113395+0000", 2024-09-15T03:46:44.401 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.196184642, 2024-09-15T03:46:44.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003810875, 2024-09-15T03:46:44.401 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.401 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.401 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.113395+0000", 2024-09-15T03:46:44.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.113395+0000", 2024-09-15T03:46:44.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.113395+0000", 2024-09-15T03:46:44.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0900000000000001e-07 2024-09-15T03:46:44.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.113411+0000", 2024-09-15T03:46:44.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6311999999999999e-05 2024-09-15T03:46:44.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.113425+0000", 2024-09-15T03:46:44.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.396e-05 2024-09-15T03:46:44.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.113434+0000", 2024-09-15T03:46:44.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7280000000000001e-06 2024-09-15T03:46:44.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.113503+0000", 2024-09-15T03:46:44.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8897000000000001e-05 2024-09-15T03:46:44.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.113516+0000", 2024-09-15T03:46:44.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2951e-05 2024-09-15T03:46:44.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.113829+0000", 2024-09-15T03:46:44.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031321300000000001 2024-09-15T03:46:44.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.113849+0000", 2024-09-15T03:46:44.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0109999999999999e-05 2024-09-15T03:46:44.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.113906+0000", 2024-09-15T03:46:44.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7509000000000001e-05 2024-09-15T03:46:44.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.113950+0000", 2024-09-15T03:46:44.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3563999999999998e-05 2024-09-15T03:46:44.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.117205+0000", 2024-09-15T03:46:44.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003255422 2024-09-15T03:46:44.406 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.406 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.406 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.407 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=31, reqid=client.4521.0:193, at_version=34'18, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:46:44.407 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.114621+0000", 2024-09-15T03:46:44.407 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.194958186, 2024-09-15T03:46:44.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049155020000000004, 2024-09-15T03:46:44.407 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.407 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.407 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114621+0000", 2024-09-15T03:46:44.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114621+0000", 2024-09-15T03:46:44.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114622+0000", 2024-09-15T03:46:44.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.341e-06 2024-09-15T03:46:44.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114645+0000", 2024-09-15T03:46:44.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3016000000000001e-05 2024-09-15T03:46:44.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114664+0000", 2024-09-15T03:46:44.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8681e-05 2024-09-15T03:46:44.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114672+0000", 2024-09-15T03:46:44.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7319999999999998e-06 2024-09-15T03:46:44.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114801+0000", 2024-09-15T03:46:44.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000129319 2024-09-15T03:46:44.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114811+0000", 2024-09-15T03:46:44.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0179e-05 2024-09-15T03:46:44.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114915+0000", 2024-09-15T03:46:44.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103317 2024-09-15T03:46:44.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114939+0000", 2024-09-15T03:46:44.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4539e-05 2024-09-15T03:46:44.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.115043+0000", 2024-09-15T03:46:44.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103865 2024-09-15T03:46:44.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.115068+0000", 2024-09-15T03:46:44.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5381e-05 2024-09-15T03:46:44.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.119537+0000", 2024-09-15T03:46:44.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0044681319999999997 2024-09-15T03:46:44.412 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.412 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.412 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.412 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=32, reqid=client.4521.0:193, at_version=0'0, trim_to=0'0, roll_forward_to=34'18))", 2024-09-15T03:46:44.412 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.115983+0000", 2024-09-15T03:46:44.412 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.1935957460000002, 2024-09-15T03:46:44.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037144729999999998, 2024-09-15T03:46:44.413 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.413 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.413 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.115983+0000", 2024-09-15T03:46:44.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.115983+0000", 2024-09-15T03:46:44.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.115984+0000", 2024-09-15T03:46:44.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5300000000000002e-07 2024-09-15T03:46:44.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116002+0000", 2024-09-15T03:46:44.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.772e-05 2024-09-15T03:46:44.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116017+0000", 2024-09-15T03:46:44.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5130000000000001e-05 2024-09-15T03:46:44.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116025+0000", 2024-09-15T03:46:44.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3459999999999996e-06 2024-09-15T03:46:44.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116100+0000", 2024-09-15T03:46:44.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5378999999999998e-05 2024-09-15T03:46:44.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116109+0000", 2024-09-15T03:46:44.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0799999999999995e-06 2024-09-15T03:46:44.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116218+0000", 2024-09-15T03:46:44.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010895099999999999 2024-09-15T03:46:44.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116231+0000", 2024-09-15T03:46:44.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2965999999999999e-05 2024-09-15T03:46:44.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116881+0000", 2024-09-15T03:46:44.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00065018900000000002 2024-09-15T03:46:44.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116897+0000", 2024-09-15T03:46:44.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5811000000000002e-05 2024-09-15T03:46:44.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.119698+0000", 2024-09-15T03:46:44.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028009480000000002 2024-09-15T03:46:44.418 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.418 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.418 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.418 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=34, reqid=client.4521.0:197, at_version=34'21, trim_to=0'0, roll_forward_to=34'18))", 2024-09-15T03:46:44.418 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:46:44.418 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.1855676129999999, 2024-09-15T03:46:44.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012992399, 2024-09-15T03:46:44.418 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.418 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.418 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:46:44.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:46:44.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124013+0000", 2024-09-15T03:46:44.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6299999999999993e-07 2024-09-15T03:46:44.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124044+0000", 2024-09-15T03:46:44.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0995999999999999e-05 2024-09-15T03:46:44.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124062+0000", 2024-09-15T03:46:44.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8413000000000001e-05 2024-09-15T03:46:44.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124069+0000", 2024-09-15T03:46:44.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3930000000000002e-06 2024-09-15T03:46:44.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124150+0000", 2024-09-15T03:46:44.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1097e-05 2024-09-15T03:46:44.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124161+0000", 2024-09-15T03:46:44.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0720000000000001e-05 2024-09-15T03:46:44.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124402+0000", 2024-09-15T03:46:44.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000240442 2024-09-15T03:46:44.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125141+0000", 2024-09-15T03:46:44.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00073915500000000002 2024-09-15T03:46:44.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125191+0000", 2024-09-15T03:46:44.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9951999999999999e-05 2024-09-15T03:46:44.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125251+0000", 2024-09-15T03:46:44.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9938000000000001e-05 2024-09-15T03:46:44.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.137004+0000", 2024-09-15T03:46:44.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011753329999999999 2024-09-15T03:46:44.423 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.423 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.423 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.423 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=35, reqid=client.4521.0:197, at_version=0'0, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:46:44.423 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:46:44.424 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.183627569, 2024-09-15T03:46:44.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011202456, 2024-09-15T03:46:44.424 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.424 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.424 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:46:44.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:46:44.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125953+0000", 2024-09-15T03:46:44.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9100000000000002e-07 2024-09-15T03:46:44.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125964+0000", 2024-09-15T03:46:44.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1399999999999999e-05 2024-09-15T03:46:44.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125979+0000", 2024-09-15T03:46:44.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4810000000000001e-05 2024-09-15T03:46:44.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126004+0000", 2024-09-15T03:46:44.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4913e-05 2024-09-15T03:46:44.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126059+0000", 2024-09-15T03:46:44.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5668999999999998e-05 2024-09-15T03:46:44.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126071+0000", 2024-09-15T03:46:44.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1219999999999999e-05 2024-09-15T03:46:44.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126151+0000", 2024-09-15T03:46:44.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0572e-05 2024-09-15T03:46:44.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126163+0000", 2024-09-15T03:46:44.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2101e-05 2024-09-15T03:46:44.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.127945+0000", 2024-09-15T03:46:44.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001781803 2024-09-15T03:46:44.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.127982+0000", 2024-09-15T03:46:44.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7129999999999999e-05 2024-09-15T03:46:44.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.137154+0000", 2024-09-15T03:46:44.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0091719469999999997 2024-09-15T03:46:44.429 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.429 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.429 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.429 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.429 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.429 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:44.430 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:46:44.431 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:44.430+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:46:44.531 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:44.531+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:46:44.572 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:44.572 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:46:44.572 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:46:44.572 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:44.580 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:46:44.632 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:44.631+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:44.722 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:44.722 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:46:44.722 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:46:44.722 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:46:44.722 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:44.730 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:46:44.733 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:44.731+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:46:44.832 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:44.832 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:46:44.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:46:44.833 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:46:44.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.833 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as2 33/28 ECSubWrite(tid=21, reqid=client.4521.0:184, at_version=33'6, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:44.833 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.106605+0000", 2024-09-15T03:46:44.833 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7239583840000003, 2024-09-15T03:46:44.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0048367940000000002, 2024-09-15T03:46:44.833 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.833 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.833 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106605+0000", 2024-09-15T03:46:44.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106605+0000", 2024-09-15T03:46:44.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106605+0000", 2024-09-15T03:46:44.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3200000000000005e-07 2024-09-15T03:46:44.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106626+0000", 2024-09-15T03:46:44.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0781e-05 2024-09-15T03:46:44.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106636+0000", 2024-09-15T03:46:44.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0137e-05 2024-09-15T03:46:44.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106643+0000", 2024-09-15T03:46:44.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7020000000000004e-06 2024-09-15T03:46:44.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106689+0000", 2024-09-15T03:46:44.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6640000000000001e-05 2024-09-15T03:46:44.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106698+0000", 2024-09-15T03:46:44.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8310000000000007e-06 2024-09-15T03:46:44.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106906+0000", 2024-09-15T03:46:44.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000207216 2024-09-15T03:46:44.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106925+0000", 2024-09-15T03:46:44.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9814e-05 2024-09-15T03:46:44.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106989+0000", 2024-09-15T03:46:44.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3336000000000005e-05 2024-09-15T03:46:44.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.107185+0000", 2024-09-15T03:46:44.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019641299999999999 2024-09-15T03:46:44.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111441+0000", 2024-09-15T03:46:44.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042563920000000003 2024-09-15T03:46:44.838 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.838 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.838 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.839 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs3 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:44.839 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108456+0000", 2024-09-15T03:46:44.839 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.722106986, 2024-09-15T03:46:44.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015075257, 2024-09-15T03:46:44.839 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.839 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.839 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108456+0000", 2024-09-15T03:46:44.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108456+0000", 2024-09-15T03:46:44.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108457+0000", 2024-09-15T03:46:44.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9999999999999997e-07 2024-09-15T03:46:44.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108477+0000", 2024-09-15T03:46:44.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0344e-05 2024-09-15T03:46:44.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108497+0000", 2024-09-15T03:46:44.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0143999999999999e-05 2024-09-15T03:46:44.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108504+0000", 2024-09-15T03:46:44.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8499999999999996e-06 2024-09-15T03:46:44.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108554+0000", 2024-09-15T03:46:44.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9567999999999997e-05 2024-09-15T03:46:44.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108568+0000", 2024-09-15T03:46:44.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4326e-05 2024-09-15T03:46:44.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108654+0000", 2024-09-15T03:46:44.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6193999999999995e-05 2024-09-15T03:46:44.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108663+0000", 2024-09-15T03:46:44.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7479999999999999e-06 2024-09-15T03:46:44.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111234+0000", 2024-09-15T03:46:44.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025709079999999998 2024-09-15T03:46:44.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111259+0000", 2024-09-15T03:46:44.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4797999999999999e-05 2024-09-15T03:46:44.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123531+0000", 2024-09-15T03:46:44.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012272777 2024-09-15T03:46:44.844 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.844 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.844 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.844 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as2 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:44.844 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:46:44.844 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7218783320000002, 2024-09-15T03:46:44.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0149108, 2024-09-15T03:46:44.845 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.845 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.845 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:46:44.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:46:44.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:46:44.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1699999999999999e-07 2024-09-15T03:46:44.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108710+0000", 2024-09-15T03:46:44.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5032999999999999e-05 2024-09-15T03:46:44.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108725+0000", 2024-09-15T03:46:44.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4788e-05 2024-09-15T03:46:44.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108733+0000", 2024-09-15T03:46:44.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4610000000000008e-06 2024-09-15T03:46:44.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108778+0000", 2024-09-15T03:46:44.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4224999999999998e-05 2024-09-15T03:46:44.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108800+0000", 2024-09-15T03:46:44.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2189999999999999e-05 2024-09-15T03:46:44.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108884+0000", 2024-09-15T03:46:44.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4569999999999998e-05 2024-09-15T03:46:44.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111237+0000", 2024-09-15T03:46:44.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002352313 2024-09-15T03:46:44.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114139+0000", 2024-09-15T03:46:44.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002902057 2024-09-15T03:46:44.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114173+0000", 2024-09-15T03:46:44.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4461000000000001e-05 2024-09-15T03:46:44.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123596+0000", 2024-09-15T03:46:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094222850000000007 2024-09-15T03:46:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs3 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:46:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.721819054, 2024-09-15T03:46:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014886043, 2024-09-15T03:46:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:46:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:46:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:46:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5100000000000001e-07 2024-09-15T03:46:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108759+0000", 2024-09-15T03:46:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4945e-05 2024-09-15T03:46:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108768+0000", 2024-09-15T03:46:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2309999999999996e-06 2024-09-15T03:46:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108795+0000", 2024-09-15T03:46:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7829000000000001e-05 2024-09-15T03:46:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108841+0000", 2024-09-15T03:46:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5330999999999997e-05 2024-09-15T03:46:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108850+0000", 2024-09-15T03:46:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9959999999999999e-06 2024-09-15T03:46:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108940+0000", 2024-09-15T03:46:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9823999999999999e-05 2024-09-15T03:46:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111237+0000", 2024-09-15T03:46:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002297927 2024-09-15T03:46:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114139+0000", 2024-09-15T03:46:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029020029999999998 2024-09-15T03:46:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114212+0000", 2024-09-15T03:46:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2177999999999999e-05 2024-09-15T03:46:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123630+0000", 2024-09-15T03:46:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094184279999999995 2024-09-15T03:46:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as2 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.109397+0000", 2024-09-15T03:46:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7211659959999999, 2024-09-15T03:46:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014322056, 2024-09-15T03:46:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109397+0000", 2024-09-15T03:46:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109397+0000", 2024-09-15T03:46:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109398+0000", 2024-09-15T03:46:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1600000000000001e-07 2024-09-15T03:46:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109410+0000", 2024-09-15T03:46:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2294e-05 2024-09-15T03:46:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109420+0000", 2024-09-15T03:46:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.764e-06 2024-09-15T03:46:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109426+0000", 2024-09-15T03:46:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9309999999999996e-06 2024-09-15T03:46:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109457+0000", 2024-09-15T03:46:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1047000000000003e-05 2024-09-15T03:46:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109462+0000", 2024-09-15T03:46:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1499999999999998e-06 2024-09-15T03:46:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109513+0000", 2024-09-15T03:46:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0722e-05 2024-09-15T03:46:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111238+0000", 2024-09-15T03:46:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017253380000000001 2024-09-15T03:46:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114140+0000", 2024-09-15T03:46:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029017370000000002 2024-09-15T03:46:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114201+0000", 2024-09-15T03:46:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1047e-05 2024-09-15T03:46:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123719+0000", 2024-09-15T03:46:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095184099999999997 2024-09-15T03:46:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs3 33/31 ECSubWrite(tid=27, reqid=client.4521.0:189, at_version=0'0, trim_to=0'0, roll_forward_to=33'10))", 2024-09-15T03:46:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.114638+0000", 2024-09-15T03:46:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7159247989999997, 2024-09-15T03:46:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090975689999999998, 2024-09-15T03:46:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114638+0000", 2024-09-15T03:46:44.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114638+0000", 2024-09-15T03:46:44.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114639+0000", 2024-09-15T03:46:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7899999999999998e-07 2024-09-15T03:46:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114648+0000", 2024-09-15T03:46:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.969e-06 2024-09-15T03:46:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114663+0000", 2024-09-15T03:46:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4813e-05 2024-09-15T03:46:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114669+0000", 2024-09-15T03:46:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9710000000000001e-06 2024-09-15T03:46:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114728+0000", 2024-09-15T03:46:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9794999999999999e-05 2024-09-15T03:46:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114737+0000", 2024-09-15T03:46:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3639999999999999e-06 2024-09-15T03:46:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114862+0000", 2024-09-15T03:46:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124826 2024-09-15T03:46:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114881+0000", 2024-09-15T03:46:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8901000000000001e-05 2024-09-15T03:46:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114920+0000", 2024-09-15T03:46:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9258000000000003e-05 2024-09-15T03:46:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114936+0000", 2024-09-15T03:46:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5665999999999999e-05 2024-09-15T03:46:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123736+0000", 2024-09-15T03:46:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088003270000000001 2024-09-15T03:46:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as2 33/28 ECSubWrite(tid=28, reqid=client.4521.0:186, at_version=0'0, trim_to=0'0, roll_forward_to=33'8))", 2024-09-15T03:46:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.114840+0000", 2024-09-15T03:46:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7157232630000001, 2024-09-15T03:46:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090295170000000008, 2024-09-15T03:46:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114840+0000", 2024-09-15T03:46:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114840+0000", 2024-09-15T03:46:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114840+0000", 2024-09-15T03:46:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6899999999999997e-07 2024-09-15T03:46:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114850+0000", 2024-09-15T03:46:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0023e-05 2024-09-15T03:46:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114862+0000", 2024-09-15T03:46:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1615e-05 2024-09-15T03:46:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114868+0000", 2024-09-15T03:46:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3570000000000003e-06 2024-09-15T03:46:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114917+0000", 2024-09-15T03:46:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8838999999999998e-05 2024-09-15T03:46:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114926+0000", 2024-09-15T03:46:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7390000000000005e-06 2024-09-15T03:46:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.115039+0000", 2024-09-15T03:46:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113114 2024-09-15T03:46:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.115048+0000", 2024-09-15T03:46:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5080000000000005e-06 2024-09-15T03:46:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.115077+0000", 2024-09-15T03:46:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.97e-05 2024-09-15T03:46:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.115104+0000", 2024-09-15T03:46:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6197999999999999e-05 2024-09-15T03:46:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123869+0000", 2024-09-15T03:46:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087657550000000001 2024-09-15T03:46:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:192 3.bs0 3:df8b18a3:::smithi00132172-17:head [append 0~0] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:46:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.112141+0000", 2024-09-15T03:46:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.7184216370000001, 2024-09-15T03:46:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050052550000000001, 2024-09-15T03:46:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 192 2024-09-15T03:46:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112141+0000", 2024-09-15T03:46:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112141+0000", 2024-09-15T03:46:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112143+0000", 2024-09-15T03:46:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6640000000000001e-06 2024-09-15T03:46:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112149+0000", 2024-09-15T03:46:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1800000000000001e-06 2024-09-15T03:46:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112161+0000", 2024-09-15T03:46:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1909000000000001e-05 2024-09-15T03:46:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112167+0000", 2024-09-15T03:46:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5559999999999998e-06 2024-09-15T03:46:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112234+0000", 2024-09-15T03:46:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6822000000000005e-05 2024-09-15T03:46:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112343+0000", 2024-09-15T03:46:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109814 2024-09-15T03:46:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.113245+0000", 2024-09-15T03:46:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00090176799999999995 2024-09-15T03:46:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.113718+0000", 2024-09-15T03:46:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047238500000000002 2024-09-15T03:46:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.113767+0000", 2024-09-15T03:46:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9589000000000002e-05 2024-09-15T03:46:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.113823+0000", 2024-09-15T03:46:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6107000000000003e-05 2024-09-15T03:46:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.115269+0000", 2024-09-15T03:46:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014454750000000001 2024-09-15T03:46:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.115439+0000", 2024-09-15T03:46:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017025300000000001 2024-09-15T03:46:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.117147+0000", 2024-09-15T03:46:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017077329999999999 2024-09-15T03:46:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:193 3.bs0 3:df8b18a3:::smithi00132172-17:head [rollback 3] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:46:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.112181+0000", 2024-09-15T03:46:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.7183819859999998, 2024-09-15T03:46:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010302426999999999, 2024-09-15T03:46:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 193 2024-09-15T03:46:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112181+0000", 2024-09-15T03:46:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112181+0000", 2024-09-15T03:46:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112182+0000", 2024-09-15T03:46:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6000000000000004e-07 2024-09-15T03:46:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112185+0000", 2024-09-15T03:46:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.892e-06 2024-09-15T03:46:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112192+0000", 2024-09-15T03:46:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8110000000000003e-06 2024-09-15T03:46:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112197+0000", 2024-09-15T03:46:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3640000000000001e-06 2024-09-15T03:46:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.113873+0000", 2024-09-15T03:46:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016760760000000001 2024-09-15T03:46:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.113974+0000", 2024-09-15T03:46:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101303 2024-09-15T03:46:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114575+0000", 2024-09-15T03:46:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00060097600000000003 2024-09-15T03:46:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114707+0000", 2024-09-15T03:46:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000131543 2024-09-15T03:46:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114723+0000", 2024-09-15T03:46:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6449000000000001e-05 2024-09-15T03:46:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114771+0000", 2024-09-15T03:46:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8312000000000001e-05 2024-09-15T03:46:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.115309+0000", 2024-09-15T03:46:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053803300000000004 2024-09-15T03:46:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.115740+0000", 2024-09-15T03:46:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043093100000000001 2024-09-15T03:46:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.122484+0000", 2024-09-15T03:46:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0067431770000000004 2024-09-15T03:46:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:194 3.bs0 3:df8b18a3:::smithi00132172-17:head [append 0~0] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:46:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.112221+0000", 2024-09-15T03:46:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.7183419569999998, 2024-09-15T03:46:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013410515, 2024-09-15T03:46:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 194 2024-09-15T03:46:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112221+0000", 2024-09-15T03:46:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112221+0000", 2024-09-15T03:46:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112222+0000", 2024-09-15T03:46:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:46:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112226+0000", 2024-09-15T03:46:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6840000000000004e-06 2024-09-15T03:46:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112232+0000", 2024-09-15T03:46:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9660000000000001e-06 2024-09-15T03:46:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112236+0000", 2024-09-15T03:46:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2740000000000001e-06 2024-09-15T03:46:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116143+0000", 2024-09-15T03:46:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039065549999999999 2024-09-15T03:46:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116257+0000", 2024-09-15T03:46:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113655 2024-09-15T03:46:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.122980+0000", 2024-09-15T03:46:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0067229860000000002 2024-09-15T03:46:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123101+0000", 2024-09-15T03:46:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121143 2024-09-15T03:46:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123117+0000", 2024-09-15T03:46:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6279000000000001e-05 2024-09-15T03:46:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123185+0000", 2024-09-15T03:46:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7613000000000005e-05 2024-09-15T03:46:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124310+0000", 2024-09-15T03:46:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011254679999999999 2024-09-15T03:46:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125591+0000", 2024-09-15T03:46:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012803669999999999 2024-09-15T03:46:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125632+0000", 2024-09-15T03:46:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1115000000000002e-05 2024-09-15T03:46:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:197 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [delete] snapc 3=[3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e34)", 2024-09-15T03:46:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:46:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.7182978200000001, 2024-09-15T03:46:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021934704999999999, 2024-09-15T03:46:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197 2024-09-15T03:46:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:46:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:46:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112266+0000", 2024-09-15T03:46:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5000000000000002e-07 2024-09-15T03:46:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112269+0000", 2024-09-15T03:46:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.066e-06 2024-09-15T03:46:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112276+0000", 2024-09-15T03:46:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-09-15T03:46:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112281+0000", 2024-09-15T03:46:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3970000000000004e-06 2024-09-15T03:46:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123281+0000", 2024-09-15T03:46:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011000575 2024-09-15T03:46:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123395+0000", 2024-09-15T03:46:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113398 2024-09-15T03:46:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123963+0000", 2024-09-15T03:46:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000568101 2024-09-15T03:46:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124220+0000", 2024-09-15T03:46:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025704900000000003 2024-09-15T03:46:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124229+0000", 2024-09-15T03:46:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4860000000000003e-06 2024-09-15T03:46:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124277+0000", 2024-09-15T03:46:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7574000000000002e-05 2024-09-15T03:46:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124312+0000", 2024-09-15T03:46:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5256000000000003e-05 2024-09-15T03:46:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125716+0000", 2024-09-15T03:46:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014043720000000001 2024-09-15T03:46:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.134200+0000", 2024-09-15T03:46:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084835120000000003 2024-09-15T03:46:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 34/31 ECSubWriteReply(tid=32, last_complete=34'18, committed=1, applied=1))", 2024-09-15T03:46:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.116301+0000", 2024-09-15T03:46:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.7142623490000002, 2024-09-15T03:46:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0080085850000000004, 2024-09-15T03:46:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116301+0000", 2024-09-15T03:46:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116301+0000", 2024-09-15T03:46:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116302+0000", 2024-09-15T03:46:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6899999999999996e-07 2024-09-15T03:46:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116307+0000", 2024-09-15T03:46:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1089999999999997e-06 2024-09-15T03:46:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116317+0000", 2024-09-15T03:46:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0689999999999999e-05 2024-09-15T03:46:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116323+0000", 2024-09-15T03:46:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1599999999999997e-06 2024-09-15T03:46:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124294+0000", 2024-09-15T03:46:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0079718770000000005 2024-09-15T03:46:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124309+0000", 2024-09-15T03:46:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.488e-05 2024-09-15T03:46:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 34/31 ECSubWriteReply(tid=32, last_complete=34'18, committed=1, applied=1))", 2024-09-15T03:46:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.116340+0000", 2024-09-15T03:46:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.7142235599999998, 2024-09-15T03:46:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082310500000000002, 2024-09-15T03:46:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116340+0000", 2024-09-15T03:46:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116340+0000", 2024-09-15T03:46:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116340+0000", 2024-09-15T03:46:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8100000000000001e-07 2024-09-15T03:46:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116347+0000", 2024-09-15T03:46:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9369999999999998e-06 2024-09-15T03:46:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116364+0000", 2024-09-15T03:46:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6169000000000001e-05 2024-09-15T03:46:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116371+0000", 2024-09-15T03:46:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9870000000000002e-06 2024-09-15T03:46:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124547+0000", 2024-09-15T03:46:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0081764009999999998 2024-09-15T03:46:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124571+0000", 2024-09-15T03:46:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3575e-05 2024-09-15T03:46:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 34/31 ECSubWriteReply(tid=32, last_complete=34'18, committed=1, applied=1))", 2024-09-15T03:46:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.116401+0000", 2024-09-15T03:46:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.7141617330000001, 2024-09-15T03:46:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0080117220000000006, 2024-09-15T03:46:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116401+0000", 2024-09-15T03:46:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116401+0000", 2024-09-15T03:46:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116402+0000", 2024-09-15T03:46:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5000000000000002e-07 2024-09-15T03:46:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116406+0000", 2024-09-15T03:46:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7060000000000002e-06 2024-09-15T03:46:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116417+0000", 2024-09-15T03:46:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1612e-05 2024-09-15T03:46:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116422+0000", 2024-09-15T03:46:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4089999999999996e-06 2024-09-15T03:46:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124406+0000", 2024-09-15T03:46:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0079847270000000005 2024-09-15T03:46:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124413+0000", 2024-09-15T03:46:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.618e-06 2024-09-15T03:46:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 34/31 ECSubWriteReply(tid=32, last_complete=34'18, committed=1, applied=1))", 2024-09-15T03:46:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.116488+0000", 2024-09-15T03:46:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.7140752080000001, 2024-09-15T03:46:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0078941749999999998, 2024-09-15T03:46:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116488+0000", 2024-09-15T03:46:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116488+0000", 2024-09-15T03:46:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116489+0000", 2024-09-15T03:46:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8400000000000003e-07 2024-09-15T03:46:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116494+0000", 2024-09-15T03:46:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3000000000000001e-06 2024-09-15T03:46:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116512+0000", 2024-09-15T03:46:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7805e-05 2024-09-15T03:46:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116519+0000", 2024-09-15T03:46:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0219999999999999e-06 2024-09-15T03:46:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124374+0000", 2024-09-15T03:46:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0078555910000000003 2024-09-15T03:46:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124382+0000", 2024-09-15T03:46:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6729999999999992e-06 2024-09-15T03:46:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 34/31 ECSubWriteReply(tid=32, last_complete=34'18, committed=1, applied=1))", 2024-09-15T03:46:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.116583+0000", 2024-09-15T03:46:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.7139797159999999, 2024-09-15T03:46:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077626049999999997, 2024-09-15T03:46:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116583+0000", 2024-09-15T03:46:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116583+0000", 2024-09-15T03:46:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116584+0000", 2024-09-15T03:46:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1000000000000001e-06 2024-09-15T03:46:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116589+0000", 2024-09-15T03:46:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9860000000000002e-06 2024-09-15T03:46:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116603+0000", 2024-09-15T03:46:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3312000000000001e-05 2024-09-15T03:46:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116608+0000", 2024-09-15T03:46:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2739999999999998e-06 2024-09-15T03:46:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124332+0000", 2024-09-15T03:46:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077239539999999999 2024-09-15T03:46:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124346+0000", 2024-09-15T03:46:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3978999999999999e-05 2024-09-15T03:46:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 34/31 ECSubWriteReply(tid=32, last_complete=34'18, committed=1, applied=1))", 2024-09-15T03:46:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.116993+0000", 2024-09-15T03:46:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.7135696569999999, 2024-09-15T03:46:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074568159999999998, 2024-09-15T03:46:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116993+0000", 2024-09-15T03:46:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116993+0000", 2024-09-15T03:46:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116994+0000", 2024-09-15T03:46:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5199999999999996e-07 2024-09-15T03:46:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116999+0000", 2024-09-15T03:46:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.369e-06 2024-09-15T03:46:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.117012+0000", 2024-09-15T03:46:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2989e-05 2024-09-15T03:46:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.117018+0000", 2024-09-15T03:46:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4529999999999999e-06 2024-09-15T03:46:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124438+0000", 2024-09-15T03:46:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074203669999999998 2024-09-15T03:46:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124450+0000", 2024-09-15T03:46:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1885999999999999e-05 2024-09-15T03:46:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 34/31 ECSubWriteReply(tid=32, last_complete=34'18, committed=1, applied=1))", 2024-09-15T03:46:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.117040+0000", 2024-09-15T03:46:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.7135232770000002, 2024-09-15T03:46:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074689819999999999, 2024-09-15T03:46:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.117040+0000", 2024-09-15T03:46:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.117040+0000", 2024-09-15T03:46:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.117040+0000", 2024-09-15T03:46:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8999999999999996e-07 2024-09-15T03:46:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.117044+0000", 2024-09-15T03:46:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6370000000000001e-06 2024-09-15T03:46:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.117056+0000", 2024-09-15T03:46:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1523e-05 2024-09-15T03:46:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.117061+0000", 2024-09-15T03:46:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0989999999999998e-06 2024-09-15T03:46:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124495+0000", 2024-09-15T03:46:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074343329999999996 2024-09-15T03:46:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124509+0000", 2024-09-15T03:46:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.38e-05 2024-09-15T03:46:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es4 34/28 ECSubWrite(tid=33, reqid=client.4521.0:195, at_version=34'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-15T03:46:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:46:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.7124704610000001, 2024-09-15T03:46:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015907482000000001, 2024-09-15T03:46:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:46:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:46:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:46:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9400000000000004e-07 2024-09-15T03:46:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118299+0000", 2024-09-15T03:46:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000205677 2024-09-15T03:46:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118315+0000", 2024-09-15T03:46:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5429000000000001e-05 2024-09-15T03:46:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118322+0000", 2024-09-15T03:46:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8290000000000008e-06 2024-09-15T03:46:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118416+0000", 2024-09-15T03:46:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3140000000000006e-05 2024-09-15T03:46:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118431+0000", 2024-09-15T03:46:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5648e-05 2024-09-15T03:46:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118678+0000", 2024-09-15T03:46:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024701200000000001 2024-09-15T03:46:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118691+0000", 2024-09-15T03:46:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2653000000000001e-05 2024-09-15T03:46:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.119431+0000", 2024-09-15T03:46:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00074060399999999996 2024-09-15T03:46:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124280+0000", 2024-09-15T03:46:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0048484410000000002 2024-09-15T03:46:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.134000+0000", 2024-09-15T03:46:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0097201549999999994 2024-09-15T03:46:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es4 34/28 ECSubWrite(tid=34, reqid=client.4521.0:195, at_version=0'0, trim_to=0'0, roll_forward_to=34'16))", 2024-09-15T03:46:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.124887+0000", 2024-09-15T03:46:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.7056758090000002, 2024-09-15T03:46:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092852760000000003, 2024-09-15T03:46:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124887+0000", 2024-09-15T03:46:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124887+0000", 2024-09-15T03:46:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124888+0000", 2024-09-15T03:46:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2600000000000001e-07 2024-09-15T03:46:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124899+0000", 2024-09-15T03:46:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1256e-05 2024-09-15T03:46:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124910+0000", 2024-09-15T03:46:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0986e-05 2024-09-15T03:46:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124917+0000", 2024-09-15T03:46:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.297e-06 2024-09-15T03:46:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124985+0000", 2024-09-15T03:46:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7838999999999999e-05 2024-09-15T03:46:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124999+0000", 2024-09-15T03:46:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4201e-05 2024-09-15T03:46:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125095+0000", 2024-09-15T03:46:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5797000000000005e-05 2024-09-15T03:46:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125107+0000", 2024-09-15T03:46:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2194999999999999e-05 2024-09-15T03:46:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125145+0000", 2024-09-15T03:46:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8355000000000001e-05 2024-09-15T03:46:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125189+0000", 2024-09-15T03:46:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3741e-05 2024-09-15T03:46:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:44.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.134173+0000", 2024-09-15T03:46:44.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089836829999999993 2024-09-15T03:46:44.941 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.941 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.941 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.941 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:44.941 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:44.941 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:44.943 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:46:44.943 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:44.942+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:46:45.044 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:45.042+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:45.051 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:45.051 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:45.051 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:46:45.051 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:45.060 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:46:45.127 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-15T03:46:45.127 INFO:tasks.rados.rados.0.smithi001.stdout:81: done (0 left) 2024-09-15T03:46:45.127 INFO:tasks.rados.rados.0.smithi001.stdout:82: rollback oid 1 current snap is 6 2024-09-15T03:46:45.128 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 1 to 4 2024-09-15T03:46:45.128 INFO:tasks.rados.rados.0.smithi001.stdout:83: copy_from oid 44 from oid 14 current snap is 6 2024-09-15T03:46:45.128 INFO:tasks.rados.rados.0.smithi001.stdout:84: delete oid 13 current snap is 6 2024-09-15T03:46:45.130 INFO:tasks.rados.rados.0.smithi001.stdout:82: finishing rollback tid 1 to smithi00132172-1 2024-09-15T03:46:45.130 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 1 v 9 (ObjNum 81 snap 0 seq_num 2582500959) dirty dne 2024-09-15T03:46:45.133 INFO:tasks.rados.rados.0.smithi001.stdout:82: done (2 left) 2024-09-15T03:46:45.133 INFO:tasks.rados.rados.0.smithi001.stdout:84: done (1 left) 2024-09-15T03:46:45.133 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 27 current snap is 6 2024-09-15T03:46:45.133 INFO:tasks.rados.rados.0.smithi001.stdout:85: seq_num 63 ranges {1409024=688128,2097152=376832} 2024-09-15T03:46:45.144 INFO:tasks.rados.rados.0.smithi001.stdout:85: writing smithi00132172-27 from 1409024 to 2097152 tid 1 2024-09-15T03:46:45.144 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:45.144+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:46:45.147 INFO:tasks.rados.rados.0.smithi001.stdout:85: writing smithi00132172-27 from 2097152 to 2473984 tid 2 2024-09-15T03:46:45.147 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 4 current snap is 6 2024-09-15T03:46:45.147 INFO:tasks.rados.rados.0.smithi001.stdout:86: seq_num 64 ranges {65536=704512,770048=376832} 2024-09-15T03:46:45.152 INFO:tasks.rados.rados.0.smithi001.stdout:86: writing smithi00132172-4 from 65536 to 770048 tid 1 2024-09-15T03:46:45.156 INFO:tasks.rados.rados.0.smithi001.stdout:86: writing smithi00132172-4 from 770048 to 1146880 tid 2 2024-09-15T03:46:45.156 INFO:tasks.rados.rados.0.smithi001.stdout:87: copy_from oid 35 from oid 43 current snap is 6 2024-09-15T03:46:45.156 INFO:tasks.rados.rados.0.smithi001.stdout:88: read oid 5 snap -1 2024-09-15T03:46:45.156 INFO:tasks.rados.rados.0.smithi001.stdout:88: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-15T03:46:45.156 INFO:tasks.rados.rados.0.smithi001.stdout:89: rollback oid 45 current snap is 6 2024-09-15T03:46:45.156 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 45 to 1 2024-09-15T03:46:45.156 INFO:tasks.rados.rados.0.smithi001.stdout:83: finishing copy_from racing read to smithi00132172-44 2024-09-15T03:46:45.156 INFO:tasks.rados.rados.0.smithi001.stdout:85: finishing write tid 1 to smithi00132172-27 2024-09-15T03:46:45.156 INFO:tasks.rados.rados.0.smithi001.stdout:83: finishing copy_from to smithi00132172-44 2024-09-15T03:46:45.156 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 44 v 22 (ObjNum 60 snap 3 seq_num 60) dirty exists 2024-09-15T03:46:45.157 INFO:tasks.rados.rados.0.smithi001.stdout:85: finishing write tid 2 to smithi00132172-27 2024-09-15T03:46:45.157 INFO:tasks.rados.rados.0.smithi001.stdout:85: finishing write tid 3 to smithi00132172-27 2024-09-15T03:46:45.157 INFO:tasks.rados.rados.0.smithi001.stdout:83: done (5 left) 2024-09-15T03:46:45.157 INFO:tasks.rados.rados.0.smithi001.stdout:90: snap_remove snap 5 2024-09-15T03:46:45.206 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:45.206 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:45.206 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:46:45.206 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:46:45.206 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:45.214 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:46:45.257 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:45.256+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:45.313 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:45.313 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:46:45.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:46:45.313 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:46:45.313 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.313 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:45.313 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:46:45.314 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 27.408491768000001, 2024-09-15T03:46:45.314 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10988437500000001, 2024-09-15T03:46:45.315 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.315 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:45.315 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:45.315 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:45.315 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:45.315 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1 2024-09-15T03:46:45.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.315 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.315 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.315 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.315 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:46:45.315 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.316 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.316 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:46:45.316 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.316 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.316 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902355+0000", 2024-09-15T03:46:45.316 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.429e-06 2024-09-15T03:46:45.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.316 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.316 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902362+0000", 2024-09-15T03:46:45.317 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7290000000000003e-06 2024-09-15T03:46:45.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.317 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.317 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902373+0000", 2024-09-15T03:46:45.317 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0533999999999999e-05 2024-09-15T03:46:45.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.317 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.317 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902379+0000", 2024-09-15T03:46:45.317 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4350000000000002e-06 2024-09-15T03:46:45.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.317 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.317 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.318 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902428+0000", 2024-09-15T03:46:45.318 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8571e-05 2024-09-15T03:46:45.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.318 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.318 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:46:45.318 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902434+0000", 2024-09-15T03:46:45.318 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5610000000000004e-06 2024-09-15T03:46:45.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.318 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.318 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.318 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011307+0000", 2024-09-15T03:46:45.319 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.108872306 2024-09-15T03:46:45.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.319 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.319 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:45.319 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011448+0000", 2024-09-15T03:46:45.319 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014137199999999999 2024-09-15T03:46:45.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.319 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.319 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:45.319 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011569+0000", 2024-09-15T03:46:45.319 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120925 2024-09-15T03:46:45.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.320 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.320 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.320 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011695+0000", 2024-09-15T03:46:45.320 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012586800000000001 2024-09-15T03:46:45.320 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.320 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.320 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.320 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011703+0000", 2024-09-15T03:46:45.320 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4190000000000002e-06 2024-09-15T03:46:45.320 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.320 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.320 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.320 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011739+0000", 2024-09-15T03:46:45.320 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5474e-05 2024-09-15T03:46:45.321 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.321 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.321 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:45.321 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011940+0000", 2024-09-15T03:46:45.321 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020108699999999999 2024-09-15T03:46:45.321 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.321 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.321 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:45.321 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012179+0000", 2024-09-15T03:46:45.321 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023950299999999999 2024-09-15T03:46:45.321 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.321 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.322 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:45.322 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012215+0000", 2024-09-15T03:46:45.322 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5332e-05 2024-09-15T03:46:45.322 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.322 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.322 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.322 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012238+0000", 2024-09-15T03:46:45.322 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3830000000000001e-05 2024-09-15T03:46:45.322 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.322 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.322 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.322 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.322 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.323 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:45.323 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.170804+0000", 2024-09-15T03:46:45.323 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 27.140041921000002, 2024-09-15T03:46:45.323 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0041461739999999999, 2024-09-15T03:46:45.323 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.323 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:45.323 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:45.323 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:45.323 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:45.323 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 50 2024-09-15T03:46:45.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.323 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.324 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.324 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.324 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170804+0000", 2024-09-15T03:46:45.324 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.324 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.324 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.324 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.324 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170804+0000", 2024-09-15T03:46:45.325 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.325 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.325 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.325 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.325 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170804+0000", 2024-09-15T03:46:45.325 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7199999999999999e-07 2024-09-15T03:46:45.325 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.325 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.325 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.325 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170856+0000", 2024-09-15T03:46:45.325 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1647000000000002e-05 2024-09-15T03:46:45.326 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.326 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.326 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.326 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170865+0000", 2024-09-15T03:46:45.326 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0829999999999993e-06 2024-09-15T03:46:45.326 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.326 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.326 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.326 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170869+0000", 2024-09-15T03:46:45.326 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8369999999999999e-06 2024-09-15T03:46:45.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.327 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.327 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.327 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174053+0000", 2024-09-15T03:46:45.327 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031840639999999999 2024-09-15T03:46:45.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.327 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.327 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:45.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174099+0000", 2024-09-15T03:46:45.328 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5578000000000003e-05 2024-09-15T03:46:45.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.328 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.328 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:45.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174178+0000", 2024-09-15T03:46:45.328 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9268000000000001e-05 2024-09-15T03:46:45.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.328 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.328 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174261+0000", 2024-09-15T03:46:45.328 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3023999999999997e-05 2024-09-15T03:46:45.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.329 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.329 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.329 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174269+0000", 2024-09-15T03:46:45.329 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2460000000000003e-06 2024-09-15T03:46:45.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.330 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.330 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.330 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174322+0000", 2024-09-15T03:46:45.330 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2957999999999997e-05 2024-09-15T03:46:45.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.331 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:45.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174614+0000", 2024-09-15T03:46:45.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029206300000000001 2024-09-15T03:46:45.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.332 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.332 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:45.332 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174906+0000", 2024-09-15T03:46:45.333 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029138299999999998 2024-09-15T03:46:45.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.333 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.333 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:45.333 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174931+0000", 2024-09-15T03:46:45.333 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5809999999999999e-05 2024-09-15T03:46:45.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.334 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.334 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174950+0000", 2024-09-15T03:46:45.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8641e-05 2024-09-15T03:46:45.335 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.335 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.335 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.336 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.336 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:45.336 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.170873+0000", 2024-09-15T03:46:45.336 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 27.139972712999999, 2024-09-15T03:46:45.336 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0041437130000000003, 2024-09-15T03:46:45.337 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.337 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:45.337 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:45.337 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:45.337 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:45.338 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 51 2024-09-15T03:46:45.338 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.338 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.338 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.338 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.338 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170873+0000", 2024-09-15T03:46:45.339 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170873+0000", 2024-09-15T03:46:45.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170873+0000", 2024-09-15T03:46:45.341 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:46:45.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.341 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.341 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.341 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170876+0000", 2024-09-15T03:46:45.341 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3010000000000001e-06 2024-09-15T03:46:45.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.341 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.341 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.341 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170879+0000", 2024-09-15T03:46:45.341 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2710000000000001e-06 2024-09-15T03:46:45.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.342 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.342 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170881+0000", 2024-09-15T03:46:45.342 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3630000000000001e-06 2024-09-15T03:46:45.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.342 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.342 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174304+0000", 2024-09-15T03:46:45.342 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0034224139999999999 2024-09-15T03:46:45.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.342 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:45.342 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174351+0000", 2024-09-15T03:46:45.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7638e-05 2024-09-15T03:46:45.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:45.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174421+0000", 2024-09-15T03:46:45.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9283999999999996e-05 2024-09-15T03:46:45.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174501+0000", 2024-09-15T03:46:45.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0652e-05 2024-09-15T03:46:45.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174508+0000", 2024-09-15T03:46:45.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6540000000000002e-06 2024-09-15T03:46:45.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174530+0000", 2024-09-15T03:46:45.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2473999999999999e-05 2024-09-15T03:46:45.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:45.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174621+0000", 2024-09-15T03:46:45.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0803000000000005e-05 2024-09-15T03:46:45.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:45.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174975+0000", 2024-09-15T03:46:45.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00035398800000000001 2024-09-15T03:46:45.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:45.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174999+0000", 2024-09-15T03:46:45.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3241999999999999e-05 2024-09-15T03:46:45.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.175017+0000", 2024-09-15T03:46:45.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8219e-05 2024-09-15T03:46:45.346 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.346 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.346 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.347 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:45.347 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:18.170884+0000", 2024-09-15T03:46:45.347 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 27.139961354, 2024-09-15T03:46:45.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042576849999999998, 2024-09-15T03:46:45.347 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.347 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:45.347 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:45.347 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:45.347 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:45.347 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 52 2024-09-15T03:46:45.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.348 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170884+0000", 2024-09-15T03:46:45.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170884+0000", 2024-09-15T03:46:45.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170885+0000", 2024-09-15T03:46:45.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0599999999999999e-07 2024-09-15T03:46:45.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.350 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.350 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170887+0000", 2024-09-15T03:46:45.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.17e-06 2024-09-15T03:46:45.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.350 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.350 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170890+0000", 2024-09-15T03:46:45.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9210000000000001e-06 2024-09-15T03:46:45.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.170892+0000", 2024-09-15T03:46:45.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.779e-06 2024-09-15T03:46:45.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174638+0000", 2024-09-15T03:46:45.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0037452269999999998 2024-09-15T03:46:45.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:45.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174694+0000", 2024-09-15T03:46:45.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6378000000000001e-05 2024-09-15T03:46:45.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:45.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174770+0000", 2024-09-15T03:46:45.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5654000000000004e-05 2024-09-15T03:46:45.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174859+0000", 2024-09-15T03:46:45.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9074999999999997e-05 2024-09-15T03:46:45.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174865+0000", 2024-09-15T03:46:45.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.178e-06 2024-09-15T03:46:45.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.174890+0000", 2024-09-15T03:46:45.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4956e-05 2024-09-15T03:46:45.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:45.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.175029+0000", 2024-09-15T03:46:45.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013951999999999999 2024-09-15T03:46:45.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:45.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.175102+0000", 2024-09-15T03:46:45.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2713000000000007e-05 2024-09-15T03:46:45.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:45.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.175126+0000", 2024-09-15T03:46:45.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3541e-05 2024-09-15T03:46:45.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.175142+0000", 2024-09-15T03:46:45.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6367000000000001e-05 2024-09-15T03:46:45.361 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.362 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.363 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.363 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4263.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2024-09-15T03:46:45.363 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:46:45.363 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.267095082000001, 2024-09-15T03:46:45.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019495788999999999, 2024-09-15T03:46:45.364 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.364 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.364 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:45.364 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4263", 2024-09-15T03:46:45.364 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/646873808", 2024-09-15T03:46:45.364 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2 2024-09-15T03:46:45.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.364 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.364 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.364 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:46:45.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:46:45.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043752+0000", 2024-09-15T03:46:45.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.815e-06 2024-09-15T03:46:45.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043757+0000", 2024-09-15T03:46:45.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4240000000000003e-06 2024-09-15T03:46:45.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.366 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043766+0000", 2024-09-15T03:46:45.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8629999999999997e-06 2024-09-15T03:46:45.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.366 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043773+0000", 2024-09-15T03:46:45.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0099999999999998e-06 2024-09-15T03:46:45.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043830+0000", 2024-09-15T03:46:45.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7089999999999997e-05 2024-09-15T03:46:45.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:46:45.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043837+0000", 2024-09-15T03:46:45.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.08e-06 2024-09-15T03:46:45.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063103+0000", 2024-09-15T03:46:45.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019265619000000001 2024-09-15T03:46:45.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063246+0000", 2024-09-15T03:46:45.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000143888 2024-09-15T03:46:45.369 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.369 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.369 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.369 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.fs5 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:45.369 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:46:45.369 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.2032471759999996, 2024-09-15T03:46:45.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098369180000000001, 2024-09-15T03:46:45.370 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.370 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.370 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:46:45.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:46:45.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:46:45.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4499999999999998e-07 2024-09-15T03:46:45.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107630+0000", 2024-09-15T03:46:45.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0877999999999998e-05 2024-09-15T03:46:45.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.372 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107642+0000", 2024-09-15T03:46:45.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2567e-05 2024-09-15T03:46:45.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.372 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107648+0000", 2024-09-15T03:46:45.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8690000000000001e-06 2024-09-15T03:46:45.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107894+0000", 2024-09-15T03:46:45.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024540400000000001 2024-09-15T03:46:45.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107900+0000", 2024-09-15T03:46:45.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5710000000000003e-06 2024-09-15T03:46:45.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107968+0000", 2024-09-15T03:46:45.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7957e-05 2024-09-15T03:46:45.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108274+0000", 2024-09-15T03:46:45.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030591700000000001 2024-09-15T03:46:45.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108504+0000", 2024-09-15T03:46:45.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022952199999999999 2024-09-15T03:46:45.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109438+0000", 2024-09-15T03:46:45.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00093439100000000002 2024-09-15T03:46:45.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117435+0000", 2024-09-15T03:46:45.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0079974969999999992 2024-09-15T03:46:45.376 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.376 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.376 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.376 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:45.376 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:46:45.376 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.2031899240000001, 2024-09-15T03:46:45.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084894540000000004, 2024-09-15T03:46:45.376 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.376 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.376 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:46:45.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:46:45.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:46:45.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4700000000000002e-07 2024-09-15T03:46:45.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.378 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107670+0000", 2024-09-15T03:46:45.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3892e-05 2024-09-15T03:46:45.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.378 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107678+0000", 2024-09-15T03:46:45.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9589999999999995e-06 2024-09-15T03:46:45.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107683+0000", 2024-09-15T03:46:45.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7199999999999997e-06 2024-09-15T03:46:45.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107768+0000", 2024-09-15T03:46:45.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4968000000000004e-05 2024-09-15T03:46:45.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107776+0000", 2024-09-15T03:46:45.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9109999999999993e-06 2024-09-15T03:46:45.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107858+0000", 2024-09-15T03:46:45.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2349999999999996e-05 2024-09-15T03:46:45.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108273+0000", 2024-09-15T03:46:45.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00041525600000000002 2024-09-15T03:46:45.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108503+0000", 2024-09-15T03:46:45.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022990599999999999 2024-09-15T03:46:45.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108705+0000", 2024-09-15T03:46:45.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020173000000000001 2024-09-15T03:46:45.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.116145+0000", 2024-09-15T03:46:45.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0074404149999999997 2024-09-15T03:46:45.383 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.383 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.383 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.383 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.fs5 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:45.383 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.109087+0000", 2024-09-15T03:46:45.383 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.2017589300000004, 2024-09-15T03:46:45.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084235809999999994, 2024-09-15T03:46:45.383 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.383 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.383 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109087+0000", 2024-09-15T03:46:45.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109087+0000", 2024-09-15T03:46:45.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109087+0000", 2024-09-15T03:46:45.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7599999999999997e-07 2024-09-15T03:46:45.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109107+0000", 2024-09-15T03:46:45.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9511999999999999e-05 2024-09-15T03:46:45.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109127+0000", 2024-09-15T03:46:45.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0268e-05 2024-09-15T03:46:45.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109136+0000", 2024-09-15T03:46:45.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1330000000000005e-06 2024-09-15T03:46:45.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109594+0000", 2024-09-15T03:46:45.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045802100000000002 2024-09-15T03:46:45.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109602+0000", 2024-09-15T03:46:45.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7020000000000001e-06 2024-09-15T03:46:45.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109678+0000", 2024-09-15T03:46:45.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5428999999999999e-05 2024-09-15T03:46:45.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109699+0000", 2024-09-15T03:46:45.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1759999999999998e-05 2024-09-15T03:46:45.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109728+0000", 2024-09-15T03:46:45.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8886000000000001e-05 2024-09-15T03:46:45.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109736+0000", 2024-09-15T03:46:45.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6310000000000003e-06 2024-09-15T03:46:45.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117510+0000", 2024-09-15T03:46:45.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.007774563 2024-09-15T03:46:45.388 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.389 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.389 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.389 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:45.389 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.109145+0000", 2024-09-15T03:46:45.389 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.201701022, 2024-09-15T03:46:45.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075088480000000003, 2024-09-15T03:46:45.389 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.389 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.389 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109145+0000", 2024-09-15T03:46:45.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109145+0000", 2024-09-15T03:46:45.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109145+0000", 2024-09-15T03:46:45.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7399999999999999e-07 2024-09-15T03:46:45.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109160+0000", 2024-09-15T03:46:45.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4544e-05 2024-09-15T03:46:45.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109168+0000", 2024-09-15T03:46:45.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9200000000000004e-06 2024-09-15T03:46:45.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109172+0000", 2024-09-15T03:46:45.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.498e-06 2024-09-15T03:46:45.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109221+0000", 2024-09-15T03:46:45.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9348000000000003e-05 2024-09-15T03:46:45.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109230+0000", 2024-09-15T03:46:45.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2020000000000004e-06 2024-09-15T03:46:45.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109314+0000", 2024-09-15T03:46:45.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4445999999999998e-05 2024-09-15T03:46:45.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109570+0000", 2024-09-15T03:46:45.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00025616300000000001 2024-09-15T03:46:45.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109642+0000", 2024-09-15T03:46:45.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1600999999999994e-05 2024-09-15T03:46:45.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109653+0000", 2024-09-15T03:46:45.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1048e-05 2024-09-15T03:46:45.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.116654+0000", 2024-09-15T03:46:45.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.007000704 2024-09-15T03:46:45.394 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.394 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.394 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.395 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es5 34/28 ECSubWrite(tid=33, reqid=client.4521.0:195, at_version=34'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-15T03:46:45.395 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.118140+0000", 2024-09-15T03:46:45.395 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.1927056279999997, 2024-09-15T03:46:45.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069410629999999999, 2024-09-15T03:46:45.395 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.395 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.395 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118140+0000", 2024-09-15T03:46:45.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118140+0000", 2024-09-15T03:46:45.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118140+0000", 2024-09-15T03:46:45.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.29e-07 2024-09-15T03:46:45.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118354+0000", 2024-09-15T03:46:45.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000213291 2024-09-15T03:46:45.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118363+0000", 2024-09-15T03:46:45.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4739999999999994e-06 2024-09-15T03:46:45.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118368+0000", 2024-09-15T03:46:45.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2499999999999997e-06 2024-09-15T03:46:45.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118418+0000", 2024-09-15T03:46:45.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0166e-05 2024-09-15T03:46:45.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118427+0000", 2024-09-15T03:46:45.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0459999999999994e-06 2024-09-15T03:46:45.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118648+0000", 2024-09-15T03:46:45.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022084599999999999 2024-09-15T03:46:45.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118656+0000", 2024-09-15T03:46:45.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9899999999999997e-06 2024-09-15T03:46:45.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118725+0000", 2024-09-15T03:46:45.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8569e-05 2024-09-15T03:46:45.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118742+0000", 2024-09-15T03:46:45.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6776999999999999e-05 2024-09-15T03:46:45.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125081+0000", 2024-09-15T03:46:45.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0063395250000000004 2024-09-15T03:46:45.400 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.401 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.401 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.401 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 34/31 ECSubWrite(tid=34, reqid=client.4521.0:197, at_version=34'21, trim_to=0'0, roll_forward_to=34'18))", 2024-09-15T03:46:45.401 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.124009+0000", 2024-09-15T03:46:45.401 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.1868365790000004, 2024-09-15T03:46:45.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0066902589999999996, 2024-09-15T03:46:45.401 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.401 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.401 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.402 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124009+0000", 2024-09-15T03:46:45.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.402 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.402 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124009+0000", 2024-09-15T03:46:45.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124010+0000", 2024-09-15T03:46:45.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.63e-07 2024-09-15T03:46:45.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124031+0000", 2024-09-15T03:46:45.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1043000000000002e-05 2024-09-15T03:46:45.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124043+0000", 2024-09-15T03:46:45.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2500999999999999e-05 2024-09-15T03:46:45.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124049+0000", 2024-09-15T03:46:45.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.331e-06 2024-09-15T03:46:45.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124090+0000", 2024-09-15T03:46:45.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0692999999999998e-05 2024-09-15T03:46:45.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124097+0000", 2024-09-15T03:46:45.405 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.216e-06 2024-09-15T03:46:45.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124657+0000", 2024-09-15T03:46:45.405 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00055963300000000003 2024-09-15T03:46:45.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124664+0000", 2024-09-15T03:46:45.405 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0310000000000001e-06 2024-09-15T03:46:45.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.406 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124694+0000", 2024-09-15T03:46:45.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0383999999999999e-05 2024-09-15T03:46:45.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.406 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.406 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124707+0000", 2024-09-15T03:46:45.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2098e-05 2024-09-15T03:46:45.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.406 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.406 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.130699+0000", 2024-09-15T03:46:45.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0059928660000000003 2024-09-15T03:46:45.407 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.407 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.407 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.407 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 34/31 ECSubWrite(tid=35, reqid=client.4521.0:197, at_version=0'0, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:46:45.407 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.125940+0000", 2024-09-15T03:46:45.407 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.1849059740000003, 2024-09-15T03:46:45.407 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0048729330000000003, 2024-09-15T03:46:45.407 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.407 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.407 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.407 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125940+0000", 2024-09-15T03:46:45.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.408 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125940+0000", 2024-09-15T03:46:45.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.408 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125940+0000", 2024-09-15T03:46:45.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6499999999999999e-07 2024-09-15T03:46:45.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125950+0000", 2024-09-15T03:46:45.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6050000000000004e-06 2024-09-15T03:46:45.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125961+0000", 2024-09-15T03:46:45.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0692e-05 2024-09-15T03:46:45.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125966+0000", 2024-09-15T03:46:45.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.203e-06 2024-09-15T03:46:45.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.126012+0000", 2024-09-15T03:46:45.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5840000000000002e-05 2024-09-15T03:46:45.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.126021+0000", 2024-09-15T03:46:45.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6059999999999992e-06 2024-09-15T03:46:45.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.126114+0000", 2024-09-15T03:46:45.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2746999999999998e-05 2024-09-15T03:46:45.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.126125+0000", 2024-09-15T03:46:45.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0682e-05 2024-09-15T03:46:45.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.126148+0000", 2024-09-15T03:46:45.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3078000000000002e-05 2024-09-15T03:46:45.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.126161+0000", 2024-09-15T03:46:45.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2917e-05 2024-09-15T03:46:45.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.130813+0000", 2024-09-15T03:46:45.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0046519980000000001 2024-09-15T03:46:45.412 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.412 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.412 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.413 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/31 ECSubWrite(tid=38, reqid=client.4521.0:223, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:46:45.413 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:46:45.413 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.163638745, 2024-09-15T03:46:45.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010504493, 2024-09-15T03:46:45.413 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.413 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.413 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.413 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.413 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:46:45.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:46:45.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:46:45.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4999999999999999e-07 2024-09-15T03:46:45.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147648+0000", 2024-09-15T03:46:45.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00044030399999999998 2024-09-15T03:46:45.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147660+0000", 2024-09-15T03:46:45.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2517000000000001e-05 2024-09-15T03:46:45.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147667+0000", 2024-09-15T03:46:45.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.737e-06 2024-09-15T03:46:45.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147713+0000", 2024-09-15T03:46:45.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6405e-05 2024-09-15T03:46:45.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147719+0000", 2024-09-15T03:46:45.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0580000000000003e-06 2024-09-15T03:46:45.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147905+0000", 2024-09-15T03:46:45.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018574600000000001 2024-09-15T03:46:45.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147917+0000", 2024-09-15T03:46:45.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1749999999999999e-05 2024-09-15T03:46:45.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148022+0000", 2024-09-15T03:46:45.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104863 2024-09-15T03:46:45.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148037+0000", 2024-09-15T03:46:45.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5302999999999999e-05 2024-09-15T03:46:45.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.418 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157711+0000", 2024-09-15T03:46:45.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0096745600000000005 2024-09-15T03:46:45.418 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.418 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.418 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.418 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 38/31 ECSubWrite(tid=36, reqid=client.4521.0:220, at_version=38'23, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:46:45.418 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.148570+0000", 2024-09-15T03:46:45.418 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.16227540800000001, 2024-09-15T03:46:45.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0056594480000000001, 2024-09-15T03:46:45.419 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.419 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.419 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.419 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148570+0000", 2024-09-15T03:46:45.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.419 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148570+0000", 2024-09-15T03:46:45.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148571+0000", 2024-09-15T03:46:45.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9000000000000001e-07 2024-09-15T03:46:45.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149166+0000", 2024-09-15T03:46:45.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00059582600000000004 2024-09-15T03:46:45.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149176+0000", 2024-09-15T03:46:45.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2280000000000004e-06 2024-09-15T03:46:45.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149180+0000", 2024-09-15T03:46:45.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.109e-06 2024-09-15T03:46:45.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149214+0000", 2024-09-15T03:46:45.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4795999999999997e-05 2024-09-15T03:46:45.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149221+0000", 2024-09-15T03:46:45.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.951e-06 2024-09-15T03:46:45.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149582+0000", 2024-09-15T03:46:45.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036058299999999999 2024-09-15T03:46:45.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149589+0000", 2024-09-15T03:46:45.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4059999999999999e-06 2024-09-15T03:46:45.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149717+0000", 2024-09-15T03:46:45.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000127689 2024-09-15T03:46:45.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149730+0000", 2024-09-15T03:46:45.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3040999999999999e-05 2024-09-15T03:46:45.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.154230+0000", 2024-09-15T03:46:45.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044996289999999998 2024-09-15T03:46:45.424 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.424 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.424 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.424 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.424 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/31 ECSubWrite(tid=39, reqid=client.4521.0:224, at_version=38'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:46:45.424 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.149886+0000", 2024-09-15T03:46:45.424 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.16095931799999999, 2024-09-15T03:46:45.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.008019155, 2024-09-15T03:46:45.424 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.424 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.425 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.425 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.425 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.425 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149886+0000", 2024-09-15T03:46:45.425 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.425 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.425 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.425 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.425 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149886+0000", 2024-09-15T03:46:45.426 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.426 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.426 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.426 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.426 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149887+0000", 2024-09-15T03:46:45.426 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3399999999999999e-07 2024-09-15T03:46:45.426 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.426 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.426 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.426 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150075+0000", 2024-09-15T03:46:45.426 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018829200000000001 2024-09-15T03:46:45.426 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.426 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150089+0000", 2024-09-15T03:46:45.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3886000000000001e-05 2024-09-15T03:46:45.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.427 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150096+0000", 2024-09-15T03:46:45.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7429999999999996e-06 2024-09-15T03:46:45.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.427 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150137+0000", 2024-09-15T03:46:45.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1170000000000001e-05 2024-09-15T03:46:45.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150146+0000", 2024-09-15T03:46:45.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8610000000000004e-06 2024-09-15T03:46:45.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150218+0000", 2024-09-15T03:46:45.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2588000000000004e-05 2024-09-15T03:46:45.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150224+0000", 2024-09-15T03:46:45.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9390000000000002e-06 2024-09-15T03:46:45.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150308+0000", 2024-09-15T03:46:45.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3222999999999999e-05 2024-09-15T03:46:45.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150318+0000", 2024-09-15T03:46:45.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0407e-05 2024-09-15T03:46:45.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157906+0000", 2024-09-15T03:46:45.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075875120000000002 2024-09-15T03:46:45.430 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.430 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.430 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.430 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 38/31 ECSubWrite(tid=37, reqid=client.4521.0:220, at_version=0'0, trim_to=0'0, roll_forward_to=38'23))", 2024-09-15T03:46:45.430 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.151298+0000", 2024-09-15T03:46:45.430 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.15954774899999999, 2024-09-15T03:46:45.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0066381979999999997, 2024-09-15T03:46:45.431 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.431 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.431 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151298+0000", 2024-09-15T03:46:45.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151298+0000", 2024-09-15T03:46:45.432 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.432 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.432 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151298+0000", 2024-09-15T03:46:45.432 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5400000000000002e-07 2024-09-15T03:46:45.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.432 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.432 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151306+0000", 2024-09-15T03:46:45.432 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5689999999999999e-06 2024-09-15T03:46:45.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151315+0000", 2024-09-15T03:46:45.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4239999999999999e-06 2024-09-15T03:46:45.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151321+0000", 2024-09-15T03:46:45.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.118e-06 2024-09-15T03:46:45.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151364+0000", 2024-09-15T03:46:45.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3509e-05 2024-09-15T03:46:45.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151372+0000", 2024-09-15T03:46:45.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2239999999999995e-06 2024-09-15T03:46:45.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151461+0000", 2024-09-15T03:46:45.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8721000000000007e-05 2024-09-15T03:46:45.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151468+0000", 2024-09-15T03:46:45.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0330000000000003e-06 2024-09-15T03:46:45.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.153509+0000", 2024-09-15T03:46:45.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020411549999999998 2024-09-15T03:46:45.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.153529+0000", 2024-09-15T03:46:45.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0072e-05 2024-09-15T03:46:45.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157936+0000", 2024-09-15T03:46:45.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044069189999999996 2024-09-15T03:46:45.436 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.436 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.436 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.436 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/31 ECSubWrite(tid=40, reqid=client.4521.0:225, at_version=38'10, trim_to=0'0, roll_forward_to=38'8))", 2024-09-15T03:46:45.436 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.151586+0000", 2024-09-15T03:46:45.436 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.15925946999999999, 2024-09-15T03:46:45.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064316060000000003, 2024-09-15T03:46:45.436 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.437 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.437 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151586+0000", 2024-09-15T03:46:45.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151586+0000", 2024-09-15T03:46:45.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.438 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.438 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151587+0000", 2024-09-15T03:46:45.438 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6400000000000003e-07 2024-09-15T03:46:45.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.438 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.438 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.438 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151600+0000", 2024-09-15T03:46:45.438 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3662999999999999e-05 2024-09-15T03:46:45.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.438 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.438 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.438 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151609+0000", 2024-09-15T03:46:45.438 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9120000000000004e-06 2024-09-15T03:46:45.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151619+0000", 2024-09-15T03:46:45.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0107e-05 2024-09-15T03:46:45.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151654+0000", 2024-09-15T03:46:45.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4295000000000003e-05 2024-09-15T03:46:45.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151662+0000", 2024-09-15T03:46:45.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6959999999999994e-06 2024-09-15T03:46:45.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.440 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151761+0000", 2024-09-15T03:46:45.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8356000000000005e-05 2024-09-15T03:46:45.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.440 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.153516+0000", 2024-09-15T03:46:45.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001754986 2024-09-15T03:46:45.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.153546+0000", 2024-09-15T03:46:45.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0028000000000001e-05 2024-09-15T03:46:45.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.153557+0000", 2024-09-15T03:46:45.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1561e-05 2024-09-15T03:46:45.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158018+0000", 2024-09-15T03:46:45.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.004460538 2024-09-15T03:46:45.442 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.442 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.442 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.442 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.442 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es5 38/28 ECSubWrite(tid=45, reqid=client.4521.0:227, at_version=38'24, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:46:45.442 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.155440+0000", 2024-09-15T03:46:45.442 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.15540541299999999, 2024-09-15T03:46:45.442 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0059398020000000001, 2024-09-15T03:46:45.442 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.442 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.442 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.442 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155440+0000", 2024-09-15T03:46:45.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155440+0000", 2024-09-15T03:46:45.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155441+0000", 2024-09-15T03:46:45.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.077e-06 2024-09-15T03:46:45.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155981+0000", 2024-09-15T03:46:45.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00053971599999999998 2024-09-15T03:46:45.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155994+0000", 2024-09-15T03:46:45.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3339000000000001e-05 2024-09-15T03:46:45.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156000+0000", 2024-09-15T03:46:45.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9529999999999996e-06 2024-09-15T03:46:45.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156048+0000", 2024-09-15T03:46:45.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7981e-05 2024-09-15T03:46:45.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156054+0000", 2024-09-15T03:46:45.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5169999999999999e-06 2024-09-15T03:46:45.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.446 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157320+0000", 2024-09-15T03:46:45.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00126636 2024-09-15T03:46:45.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.446 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157331+0000", 2024-09-15T03:46:45.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0577000000000001e-05 2024-09-15T03:46:45.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.446 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157434+0000", 2024-09-15T03:46:45.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010285000000000001 2024-09-15T03:46:45.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157449+0000", 2024-09-15T03:46:45.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5545000000000001e-05 2024-09-15T03:46:45.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.161380+0000", 2024-09-15T03:46:45.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0039308870000000001 2024-09-15T03:46:45.447 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.447 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.447 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.448 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es5 38/28 ECSubWrite(tid=46, reqid=client.4521.0:228, at_version=38'25, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:46:45.448 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.157517+0000", 2024-09-15T03:46:45.448 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.15332843099999999, 2024-09-15T03:46:45.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0040409620000000004, 2024-09-15T03:46:45.448 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.448 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.448 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.448 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157517+0000", 2024-09-15T03:46:45.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157517+0000", 2024-09-15T03:46:45.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157518+0000", 2024-09-15T03:46:45.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0200000000000002e-07 2024-09-15T03:46:45.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157837+0000", 2024-09-15T03:46:45.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00031912999999999999 2024-09-15T03:46:45.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157849+0000", 2024-09-15T03:46:45.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2075e-05 2024-09-15T03:46:45.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157858+0000", 2024-09-15T03:46:45.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8240000000000006e-06 2024-09-15T03:46:45.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157902+0000", 2024-09-15T03:46:45.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4121e-05 2024-09-15T03:46:45.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157912+0000", 2024-09-15T03:46:45.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.713e-06 2024-09-15T03:46:45.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158022+0000", 2024-09-15T03:46:45.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000110413 2024-09-15T03:46:45.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158031+0000", 2024-09-15T03:46:45.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6300000000000004e-06 2024-09-15T03:46:45.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158297+0000", 2024-09-15T03:46:45.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00026585699999999999 2024-09-15T03:46:45.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158373+0000", 2024-09-15T03:46:45.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6606999999999993e-05 2024-09-15T03:46:45.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.161558+0000", 2024-09-15T03:46:45.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031850899999999998 2024-09-15T03:46:45.453 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.453 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.453 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.453 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.4s3 38/31 ECSubWrite(tid=38, reqid=client.4521.0:237, at_version=38'6, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:46:45.453 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.158265+0000", 2024-09-15T03:46:45.454 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.15258065700000001, 2024-09-15T03:46:45.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042638069999999997, 2024-09-15T03:46:45.454 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.454 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.454 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158265+0000", 2024-09-15T03:46:45.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158265+0000", 2024-09-15T03:46:45.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158266+0000", 2024-09-15T03:46:45.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6100000000000001e-07 2024-09-15T03:46:45.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158279+0000", 2024-09-15T03:46:45.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3918e-05 2024-09-15T03:46:45.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158289+0000", 2024-09-15T03:46:45.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5149999999999995e-06 2024-09-15T03:46:45.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158294+0000", 2024-09-15T03:46:45.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2739999999999998e-06 2024-09-15T03:46:45.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158500+0000", 2024-09-15T03:46:45.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020625099999999999 2024-09-15T03:46:45.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158506+0000", 2024-09-15T03:46:45.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4729999999999999e-06 2024-09-15T03:46:45.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158563+0000", 2024-09-15T03:46:45.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6781999999999999e-05 2024-09-15T03:46:45.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158590+0000", 2024-09-15T03:46:45.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6823000000000001e-05 2024-09-15T03:46:45.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.160867+0000", 2024-09-15T03:46:45.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022772080000000002 2024-09-15T03:46:45.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.160880+0000", 2024-09-15T03:46:45.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3131e-05 2024-09-15T03:46:45.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.162529+0000", 2024-09-15T03:46:45.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0016489709999999999 2024-09-15T03:46:45.460 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.460 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.460 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.460 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.460 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.460 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:45.462 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:46:45.462 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:45.461+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:45.562 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:45.561+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:46:45.599 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:45.599 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:45.599 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:46:45.599 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:45.607 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:46:45.664 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:45.663+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:46:45.745 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:45.746 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:45.746 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:46:45.746 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:46:45.746 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:45.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:46:45.798 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:45.796+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:45.853 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:45.853 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:46:45.853 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:46:45.853 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:46:45.853 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.853 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4263.0:1 2.3 e22/21 2:c4c92e5a:::rbd_trash:head v 22'1, mlcod=0'0)", 2024-09-15T03:46:45.853 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:24.039192+0000", 2024-09-15T03:46:45.854 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.811926884000002, 2024-09-15T03:46:45.854 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001176968, 2024-09-15T03:46:45.854 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.854 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:45.854 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.854 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.854 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.854 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039192+0000", 2024-09-15T03:46:45.854 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.854 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.854 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.854 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.854 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039192+0000", 2024-09-15T03:46:45.855 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.855 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.855 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039192+0000", 2024-09-15T03:46:45.855 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3999999999999997e-07 2024-09-15T03:46:45.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.855 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.855 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039199+0000", 2024-09-15T03:46:45.855 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2580000000000001e-06 2024-09-15T03:46:45.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.856 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039217+0000", 2024-09-15T03:46:45.856 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8895000000000001e-05 2024-09-15T03:46:45.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.856 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.856 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039226+0000", 2024-09-15T03:46:45.856 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4840000000000004e-06 2024-09-15T03:46:45.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.856 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.856 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039268+0000", 2024-09-15T03:46:45.857 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2104000000000002e-05 2024-09-15T03:46:45.857 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.857 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.857 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for flush", 2024-09-15T03:46:45.857 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.039283+0000", 2024-09-15T03:46:45.857 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5009e-05 2024-09-15T03:46:45.857 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.857 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.857 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.857 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040042+0000", 2024-09-15T03:46:45.857 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00075874199999999997 2024-09-15T03:46:45.857 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.857 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.857 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:45.858 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040067+0000", 2024-09-15T03:46:45.858 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4888999999999998e-05 2024-09-15T03:46:45.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.858 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.858 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.858 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040169+0000", 2024-09-15T03:46:45.858 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010252299999999999 2024-09-15T03:46:45.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.858 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.858 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.858 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040178+0000", 2024-09-15T03:46:45.858 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7709999999999995e-06 2024-09-15T03:46:45.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.858 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.859 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.859 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040211+0000", 2024-09-15T03:46:45.859 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3552e-05 2024-09-15T03:46:45.859 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.859 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.859 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:45.859 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040226+0000", 2024-09-15T03:46:45.859 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4695000000000001e-05 2024-09-15T03:46:45.859 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.859 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.859 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.859 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.040369+0000", 2024-09-15T03:46:45.859 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014270600000000001 2024-09-15T03:46:45.860 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.860 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.860 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.860 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.860 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 32/31 ECSubWrite(tid=42, reqid=client.4521.0:166, at_version=32'13, trim_to=0'0, roll_forward_to=32'12))", 2024-09-15T03:46:45.860 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.532201+0000", 2024-09-15T03:46:45.860 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.3189177829999998, 2024-09-15T03:46:45.860 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0036328060000000001, 2024-09-15T03:46:45.860 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.860 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.860 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.861 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.861 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.861 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532201+0000", 2024-09-15T03:46:45.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.861 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.861 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.861 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532201+0000", 2024-09-15T03:46:45.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.861 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.861 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.862 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532202+0000", 2024-09-15T03:46:45.862 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.44e-07 2024-09-15T03:46:45.862 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.862 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.862 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.862 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532214+0000", 2024-09-15T03:46:45.862 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2689e-05 2024-09-15T03:46:45.862 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.862 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.862 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.862 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532224+0000", 2024-09-15T03:46:45.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3549999999999994e-06 2024-09-15T03:46:45.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.863 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.863 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.863 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532228+0000", 2024-09-15T03:46:45.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8619999999999996e-06 2024-09-15T03:46:45.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.863 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.863 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.863 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532268+0000", 2024-09-15T03:46:45.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0899000000000003e-05 2024-09-15T03:46:45.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.863 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.863 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532277+0000", 2024-09-15T03:46:45.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6410000000000008e-06 2024-09-15T03:46:45.864 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.864 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.864 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532354+0000", 2024-09-15T03:46:45.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6904999999999996e-05 2024-09-15T03:46:45.864 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.864 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.864 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532361+0000", 2024-09-15T03:46:45.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.798e-06 2024-09-15T03:46:45.864 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.865 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.865 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532387+0000", 2024-09-15T03:46:45.865 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6346000000000002e-05 2024-09-15T03:46:45.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.865 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.865 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532400+0000", 2024-09-15T03:46:45.865 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2655e-05 2024-09-15T03:46:45.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.866 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.867 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.535834+0000", 2024-09-15T03:46:45.867 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0034340120000000002 2024-09-15T03:46:45.867 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.867 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.867 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.867 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.867 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 32/31 ECSubWrite(tid=43, reqid=client.4521.0:166, at_version=0'0, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:46:45.868 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.534156+0000", 2024-09-15T03:46:45.868 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.3169629939999998, 2024-09-15T03:46:45.868 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001796693, 2024-09-15T03:46:45.868 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.868 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.868 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.868 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.868 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.868 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.534156+0000", 2024-09-15T03:46:45.868 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.868 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.868 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.868 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.868 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.534156+0000", 2024-09-15T03:46:45.869 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.869 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.869 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.869 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.534156+0000", 2024-09-15T03:46:45.869 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8699999999999995e-07 2024-09-15T03:46:45.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.869 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.869 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.869 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.534168+0000", 2024-09-15T03:46:45.869 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1698e-05 2024-09-15T03:46:45.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.870 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.870 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.870 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.534178+0000", 2024-09-15T03:46:45.870 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0086e-05 2024-09-15T03:46:45.870 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.870 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.870 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.870 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.534182+0000", 2024-09-15T03:46:45.870 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.194e-06 2024-09-15T03:46:45.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.871 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.871 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.534216+0000", 2024-09-15T03:46:45.871 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3762e-05 2024-09-15T03:46:45.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.871 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.871 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.534221+0000", 2024-09-15T03:46:45.871 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3430000000000003e-06 2024-09-15T03:46:45.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.871 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.872 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.872 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.534269+0000", 2024-09-15T03:46:45.872 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7256999999999998e-05 2024-09-15T03:46:45.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.872 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.872 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.872 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.534275+0000", 2024-09-15T03:46:45.872 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4219999999999997e-06 2024-09-15T03:46:45.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.872 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.872 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.872 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.535553+0000", 2024-09-15T03:46:45.872 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001277423 2024-09-15T03:46:45.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.873 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.873 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.535586+0000", 2024-09-15T03:46:45.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3781000000000001e-05 2024-09-15T03:46:45.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.873 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.873 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.535953+0000", 2024-09-15T03:46:45.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036613999999999999 2024-09-15T03:46:45.873 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.873 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.873 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.874 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:175 3.cs0 3:3164c049:::smithi00132172-34:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e32)", 2024-09-15T03:46:45.874 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.553220+0000", 2024-09-15T03:46:45.874 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.2978985659999998, 2024-09-15T03:46:45.874 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0052725100000000002, 2024-09-15T03:46:45.874 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.874 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:45.874 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:45.874 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:45.874 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:45.874 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 175 2024-09-15T03:46:45.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.875 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.875 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.875 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553220+0000", 2024-09-15T03:46:45.875 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.875 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.875 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553220+0000", 2024-09-15T03:46:45.875 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.875 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.876 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553221+0000", 2024-09-15T03:46:45.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0860000000000001e-06 2024-09-15T03:46:45.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.876 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.876 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553226+0000", 2024-09-15T03:46:45.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1630000000000004e-06 2024-09-15T03:46:45.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.876 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.876 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553239+0000", 2024-09-15T03:46:45.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2187e-05 2024-09-15T03:46:45.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.877 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.877 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.877 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553244+0000", 2024-09-15T03:46:45.877 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8960000000000002e-06 2024-09-15T03:46:45.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.877 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.877 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.877 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553276+0000", 2024-09-15T03:46:45.877 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2237000000000003e-05 2024-09-15T03:46:45.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.877 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:45.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553345+0000", 2024-09-15T03:46:45.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9618999999999999e-05 2024-09-15T03:46:45.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.878 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553695+0000", 2024-09-15T03:46:45.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034979 2024-09-15T03:46:45.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.878 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553780+0000", 2024-09-15T03:46:45.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4633000000000001e-05 2024-09-15T03:46:45.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.879 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.879 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.879 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553787+0000", 2024-09-15T03:46:45.879 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2409999999999998e-06 2024-09-15T03:46:45.880 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.880 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.880 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.880 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553828+0000", 2024-09-15T03:46:45.880 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0564e-05 2024-09-15T03:46:45.880 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.880 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.880 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.881 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553841+0000", 2024-09-15T03:46:45.881 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3249e-05 2024-09-15T03:46:45.881 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.881 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.881 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:45.881 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.554266+0000", 2024-09-15T03:46:45.881 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00042530299999999998 2024-09-15T03:46:45.881 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.881 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.881 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.881 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.558493+0000", 2024-09-15T03:46:45.881 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042265419999999998 2024-09-15T03:46:45.881 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.882 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.882 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.882 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.882 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.882 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.0s2 32/31 ECSubWrite(tid=23, reqid=client.4521.0:168, at_version=32'11, trim_to=0'0, roll_forward_to=32'10))", 2024-09-15T03:46:45.882 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.557157+0000", 2024-09-15T03:46:45.882 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.293962123, 2024-09-15T03:46:45.882 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001476724, 2024-09-15T03:46:45.882 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.882 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.882 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.882 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.882 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.883 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557157+0000", 2024-09-15T03:46:45.883 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.883 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.883 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.883 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557157+0000", 2024-09-15T03:46:45.883 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.883 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.883 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.883 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557157+0000", 2024-09-15T03:46:45.883 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8900000000000001e-07 2024-09-15T03:46:45.883 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.883 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.884 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557254+0000", 2024-09-15T03:46:45.884 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7207999999999995e-05 2024-09-15T03:46:45.884 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.884 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.884 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.884 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557261+0000", 2024-09-15T03:46:45.884 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7290000000000003e-06 2024-09-15T03:46:45.884 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.884 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.884 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.884 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557264+0000", 2024-09-15T03:46:45.884 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.145e-06 2024-09-15T03:46:45.884 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.885 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.885 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.885 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557291+0000", 2024-09-15T03:46:45.885 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7336000000000001e-05 2024-09-15T03:46:45.885 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.885 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.885 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.885 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557297+0000", 2024-09-15T03:46:45.885 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3680000000000001e-06 2024-09-15T03:46:45.885 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.885 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.885 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.885 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557366+0000", 2024-09-15T03:46:45.886 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9548000000000006e-05 2024-09-15T03:46:45.886 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.886 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.886 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.886 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557371+0000", 2024-09-15T03:46:45.886 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8840000000000002e-06 2024-09-15T03:46:45.886 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.887 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.887 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.887 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.558453+0000", 2024-09-15T03:46:45.888 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010819790000000001 2024-09-15T03:46:45.888 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.888 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.888 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.888 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.558462+0000", 2024-09-15T03:46:45.888 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7290000000000006e-06 2024-09-15T03:46:45.888 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.888 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.888 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.888 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.558633+0000", 2024-09-15T03:46:45.888 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000171609 2024-09-15T03:46:45.889 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.889 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.889 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.889 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.889 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.889 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.0s2 32/31 ECSubWrite(tid=24, reqid=client.4521.0:169, at_version=32'12, trim_to=0'0, roll_forward_to=32'10))", 2024-09-15T03:46:45.889 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.557269+0000", 2024-09-15T03:46:45.889 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.2938495559999996, 2024-09-15T03:46:45.889 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001442306, 2024-09-15T03:46:45.889 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.889 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.889 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.889 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.889 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.890 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557269+0000", 2024-09-15T03:46:45.890 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.890 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.890 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.890 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.890 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557269+0000", 2024-09-15T03:46:45.890 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.890 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.890 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.890 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.890 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557269+0000", 2024-09-15T03:46:45.890 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.28e-07 2024-09-15T03:46:45.890 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.890 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.891 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557276+0000", 2024-09-15T03:46:45.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1410000000000002e-06 2024-09-15T03:46:45.891 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.891 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557279+0000", 2024-09-15T03:46:45.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.703e-06 2024-09-15T03:46:45.891 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.891 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557286+0000", 2024-09-15T03:46:45.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.618e-06 2024-09-15T03:46:45.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557396+0000", 2024-09-15T03:46:45.892 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000110441 2024-09-15T03:46:45.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557402+0000", 2024-09-15T03:46:45.892 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8590000000000001e-06 2024-09-15T03:46:45.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.893 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.557453+0000", 2024-09-15T03:46:45.893 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0507000000000003e-05 2024-09-15T03:46:45.893 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.893 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.893 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.893 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.558455+0000", 2024-09-15T03:46:45.893 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001002492 2024-09-15T03:46:45.893 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.893 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.893 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.893 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.558483+0000", 2024-09-15T03:46:45.894 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8087999999999999e-05 2024-09-15T03:46:45.894 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.894 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.894 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.894 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.558489+0000", 2024-09-15T03:46:45.894 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6710000000000004e-06 2024-09-15T03:46:45.894 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.895 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.896 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.896 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.558712+0000", 2024-09-15T03:46:45.896 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022255800000000001 2024-09-15T03:46:45.896 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.896 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.896 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.896 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.896 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.896 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s7 32/31 ECSubWrite(tid=44, reqid=client.4521.0:173, at_version=32'7, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:45.896 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.562211+0000", 2024-09-15T03:46:45.897 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.288907859, 2024-09-15T03:46:45.897 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0038125260000000001, 2024-09-15T03:46:45.897 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.897 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.897 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.897 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.897 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.897 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562211+0000", 2024-09-15T03:46:45.897 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.897 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.897 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.897 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.898 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562211+0000", 2024-09-15T03:46:45.898 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.898 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.898 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.898 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.898 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562212+0000", 2024-09-15T03:46:45.898 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2099999999999996e-07 2024-09-15T03:46:45.898 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.898 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.898 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.898 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562643+0000", 2024-09-15T03:46:45.898 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043174199999999999 2024-09-15T03:46:45.898 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.899 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.899 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562654+0000", 2024-09-15T03:46:45.899 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0927999999999999e-05 2024-09-15T03:46:45.899 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.899 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.899 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562666+0000", 2024-09-15T03:46:45.899 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2261e-05 2024-09-15T03:46:45.899 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.899 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.899 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562709+0000", 2024-09-15T03:46:45.899 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2885e-05 2024-09-15T03:46:45.900 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.900 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.900 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.900 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562716+0000", 2024-09-15T03:46:45.900 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6490000000000002e-06 2024-09-15T03:46:45.900 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.900 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.900 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.900 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562882+0000", 2024-09-15T03:46:45.900 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016565500000000001 2024-09-15T03:46:45.900 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.900 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.901 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.901 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562888+0000", 2024-09-15T03:46:45.901 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.703e-06 2024-09-15T03:46:45.901 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.901 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.901 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.901 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562972+0000", 2024-09-15T03:46:45.901 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3724e-05 2024-09-15T03:46:45.901 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.901 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.901 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.901 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562984+0000", 2024-09-15T03:46:45.901 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2361000000000001e-05 2024-09-15T03:46:45.902 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.902 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.902 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.902 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.566023+0000", 2024-09-15T03:46:45.902 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030389969999999999 2024-09-15T03:46:45.902 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.902 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.902 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.902 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.902 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.902 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s7 32/31 ECSubWrite(tid=45, reqid=client.4521.0:173, at_version=0'0, trim_to=0'0, roll_forward_to=32'7))", 2024-09-15T03:46:45.902 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.563503+0000", 2024-09-15T03:46:45.902 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.2876156620000003, 2024-09-15T03:46:45.903 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002634688, 2024-09-15T03:46:45.903 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.903 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.903 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.903 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.903 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.903 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.563503+0000", 2024-09-15T03:46:45.903 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.903 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.903 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.904 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.904 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.563503+0000", 2024-09-15T03:46:45.904 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.904 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.904 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.904 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.904 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.563504+0000", 2024-09-15T03:46:45.904 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9699999999999996e-07 2024-09-15T03:46:45.904 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.904 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.904 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.905 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.563512+0000", 2024-09-15T03:46:45.905 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4319999999999999e-06 2024-09-15T03:46:45.905 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.905 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.905 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.905 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.563526+0000", 2024-09-15T03:46:45.905 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3891e-05 2024-09-15T03:46:45.905 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.905 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.905 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.905 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.563531+0000", 2024-09-15T03:46:45.905 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7509999999999999e-06 2024-09-15T03:46:45.905 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.906 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.906 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.906 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.563572+0000", 2024-09-15T03:46:45.906 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0963000000000001e-05 2024-09-15T03:46:45.906 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.906 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.906 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.906 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.563581+0000", 2024-09-15T03:46:45.906 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1230000000000006e-06 2024-09-15T03:46:45.906 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.906 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.906 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.906 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.563650+0000", 2024-09-15T03:46:45.907 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9206000000000001e-05 2024-09-15T03:46:45.907 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.907 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.907 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.907 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.563656+0000", 2024-09-15T03:46:45.907 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1639999999999998e-06 2024-09-15T03:46:45.907 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.907 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.907 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.907 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.565765+0000", 2024-09-15T03:46:45.907 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021088600000000002 2024-09-15T03:46:45.907 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.908 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.908 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.908 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.565790+0000", 2024-09-15T03:46:45.908 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.51e-05 2024-09-15T03:46:45.908 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.908 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.908 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.908 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.566138+0000", 2024-09-15T03:46:45.908 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034760100000000001 2024-09-15T03:46:45.908 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.908 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.908 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.908 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.909 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.909 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 33/28 ECSubWrite(tid=21, reqid=client.4521.0:184, at_version=33'6, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:45.909 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.106912+0000", 2024-09-15T03:46:45.909 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.7442063699999997, 2024-09-15T03:46:45.909 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026439319999999999, 2024-09-15T03:46:45.909 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.909 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.909 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.909 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.909 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.909 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106912+0000", 2024-09-15T03:46:45.909 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.909 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.910 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.910 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.910 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106912+0000", 2024-09-15T03:46:45.910 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.910 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.910 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.910 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.910 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106913+0000", 2024-09-15T03:46:45.910 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.39e-07 2024-09-15T03:46:45.910 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.910 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.910 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.910 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106938+0000", 2024-09-15T03:46:45.910 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5063999999999999e-05 2024-09-15T03:46:45.911 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.911 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.911 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.911 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106949+0000", 2024-09-15T03:46:45.911 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1137e-05 2024-09-15T03:46:45.911 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.911 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.911 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.911 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106956+0000", 2024-09-15T03:46:45.911 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7719999999999997e-06 2024-09-15T03:46:45.911 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.911 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.911 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.912 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107325+0000", 2024-09-15T03:46:45.912 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036909699999999998 2024-09-15T03:46:45.912 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.912 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.912 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.912 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107333+0000", 2024-09-15T03:46:45.912 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9489999999999996e-06 2024-09-15T03:46:45.912 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.912 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.912 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.912 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107497+0000", 2024-09-15T03:46:45.912 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016431900000000001 2024-09-15T03:46:45.912 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.913 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.913 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.913 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107523+0000", 2024-09-15T03:46:45.913 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5678e-05 2024-09-15T03:46:45.913 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.913 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.913 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.913 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107897+0000", 2024-09-15T03:46:45.913 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00037388999999999999 2024-09-15T03:46:45.914 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.914 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.914 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.914 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108093+0000", 2024-09-15T03:46:45.914 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019591199999999999 2024-09-15T03:46:45.914 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.914 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.914 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.914 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109556+0000", 2024-09-15T03:46:45.914 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001463675 2024-09-15T03:46:45.915 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.915 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.915 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.915 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.915 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.915 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:45.915 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.108773+0000", 2024-09-15T03:46:45.915 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.7423459470000005, 2024-09-15T03:46:45.915 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0047381410000000004, 2024-09-15T03:46:45.915 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.915 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.915 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.916 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.916 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.916 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108773+0000", 2024-09-15T03:46:45.916 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.916 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.916 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.916 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.916 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108773+0000", 2024-09-15T03:46:45.916 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.916 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.916 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.916 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.916 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108774+0000", 2024-09-15T03:46:45.917 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8800000000000002e-07 2024-09-15T03:46:45.917 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.917 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.917 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.917 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108801+0000", 2024-09-15T03:46:45.917 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7339999999999999e-05 2024-09-15T03:46:45.917 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.917 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.917 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.917 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108816+0000", 2024-09-15T03:46:45.918 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5359999999999999e-05 2024-09-15T03:46:45.918 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.918 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.918 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.918 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108823+0000", 2024-09-15T03:46:45.918 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0060000000000003e-06 2024-09-15T03:46:45.918 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.918 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.918 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.918 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108930+0000", 2024-09-15T03:46:45.918 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000106569 2024-09-15T03:46:45.918 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.918 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.918 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.919 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108940+0000", 2024-09-15T03:46:45.919 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.679e-06 2024-09-15T03:46:45.919 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.919 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.919 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.919 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109030+0000", 2024-09-15T03:46:45.919 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0618999999999994e-05 2024-09-15T03:46:45.919 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.919 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.919 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.919 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109041+0000", 2024-09-15T03:46:45.919 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1011e-05 2024-09-15T03:46:45.919 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.920 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.920 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.920 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109073+0000", 2024-09-15T03:46:45.920 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1427999999999998e-05 2024-09-15T03:46:45.920 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.920 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.920 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.920 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109088+0000", 2024-09-15T03:46:45.920 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4956e-05 2024-09-15T03:46:45.920 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.920 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.920 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.921 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.113511+0000", 2024-09-15T03:46:45.921 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044233850000000002 2024-09-15T03:46:45.921 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.921 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.921 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.921 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.921 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.921 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:45.921 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.109181+0000", 2024-09-15T03:46:45.922 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.7419380489999998, 2024-09-15T03:46:45.922 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044348879999999997, 2024-09-15T03:46:45.922 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.922 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.922 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.922 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.922 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.922 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109181+0000", 2024-09-15T03:46:45.922 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.922 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.922 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.922 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.922 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109181+0000", 2024-09-15T03:46:45.923 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.923 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.923 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.923 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.923 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109181+0000", 2024-09-15T03:46:45.923 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0900000000000001e-07 2024-09-15T03:46:45.923 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.923 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.923 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.923 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109197+0000", 2024-09-15T03:46:45.924 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5150999999999999e-05 2024-09-15T03:46:45.924 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.924 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.924 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.924 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109211+0000", 2024-09-15T03:46:45.924 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3998999999999999e-05 2024-09-15T03:46:45.924 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.924 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.924 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.924 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109217+0000", 2024-09-15T03:46:45.924 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.686e-06 2024-09-15T03:46:45.924 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.924 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.925 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.925 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109254+0000", 2024-09-15T03:46:45.925 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6424999999999998e-05 2024-09-15T03:46:45.925 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.925 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.925 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.925 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109262+0000", 2024-09-15T03:46:45.925 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8230000000000001e-06 2024-09-15T03:46:45.925 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.925 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.925 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.925 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109350+0000", 2024-09-15T03:46:45.925 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7120999999999995e-05 2024-09-15T03:46:45.926 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.926 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109358+0000", 2024-09-15T03:46:45.926 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3399999999999998e-06 2024-09-15T03:46:45.926 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.926 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109388+0000", 2024-09-15T03:46:45.926 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9807000000000001e-05 2024-09-15T03:46:45.926 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.927 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109398+0000", 2024-09-15T03:46:45.927 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0744000000000001e-05 2024-09-15T03:46:45.927 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.927 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.927 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.927 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.113616+0000", 2024-09-15T03:46:45.927 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042171830000000002 2024-09-15T03:46:45.927 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.927 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.927 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.927 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.927 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.927 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 33/28 ECSubWrite(tid=28, reqid=client.4521.0:186, at_version=0'0, trim_to=0'0, roll_forward_to=33'8))", 2024-09-15T03:46:45.928 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.114810+0000", 2024-09-15T03:46:45.928 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.7363083970000002, 2024-09-15T03:46:45.928 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0011592569999999999, 2024-09-15T03:46:45.928 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.928 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.928 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.928 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.928 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.928 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114810+0000", 2024-09-15T03:46:45.928 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.928 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.928 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.928 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.929 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114810+0000", 2024-09-15T03:46:45.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.929 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.929 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.929 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.929 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114812+0000", 2024-09-15T03:46:45.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.144e-06 2024-09-15T03:46:45.929 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.929 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.929 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.929 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114825+0000", 2024-09-15T03:46:45.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3893000000000001e-05 2024-09-15T03:46:45.930 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.930 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.930 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.930 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114844+0000", 2024-09-15T03:46:45.930 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8111e-05 2024-09-15T03:46:45.930 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.930 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.930 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.930 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114851+0000", 2024-09-15T03:46:45.930 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2869999999999999e-06 2024-09-15T03:46:45.930 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.931 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.931 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.931 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114905+0000", 2024-09-15T03:46:45.931 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3736999999999997e-05 2024-09-15T03:46:45.931 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.931 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.931 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.931 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114914+0000", 2024-09-15T03:46:45.931 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3999999999999998e-06 2024-09-15T03:46:45.931 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.931 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.931 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.932 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.115031+0000", 2024-09-15T03:46:45.932 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000116712 2024-09-15T03:46:45.932 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.932 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.932 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.932 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.115048+0000", 2024-09-15T03:46:45.932 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7637000000000001e-05 2024-09-15T03:46:45.932 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.932 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.932 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.932 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.115772+0000", 2024-09-15T03:46:45.932 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000723681 2024-09-15T03:46:45.932 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.933 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.933 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.933 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.115790+0000", 2024-09-15T03:46:45.933 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7958e-05 2024-09-15T03:46:45.933 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.933 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.933 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.933 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.115970+0000", 2024-09-15T03:46:45.933 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017969700000000001 2024-09-15T03:46:45.933 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.933 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.933 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.934 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s7 34/31 ECSubWrite(tid=46, reqid=client.4521.0:191, at_version=34'9, trim_to=0'0, roll_forward_to=32'7))", 2024-09-15T03:46:45.934 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.109318+0000", 2024-09-15T03:46:45.934 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.7418012659999995, 2024-09-15T03:46:45.934 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024895289999999999, 2024-09-15T03:46:45.934 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.934 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.934 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.934 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.935 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109318+0000", 2024-09-15T03:46:45.935 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.935 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.935 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.935 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.935 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109318+0000", 2024-09-15T03:46:45.935 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.935 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.935 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.935 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.935 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109318+0000", 2024-09-15T03:46:45.935 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5800000000000001e-07 2024-09-15T03:46:45.935 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.936 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.936 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.936 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109344+0000", 2024-09-15T03:46:45.936 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6597999999999998e-05 2024-09-15T03:46:45.936 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.936 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.936 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.936 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109358+0000", 2024-09-15T03:46:45.936 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.394e-05 2024-09-15T03:46:45.936 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.936 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.936 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.937 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109369+0000", 2024-09-15T03:46:45.937 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1068e-05 2024-09-15T03:46:45.937 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.937 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.937 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.937 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109432+0000", 2024-09-15T03:46:45.937 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2360000000000006e-05 2024-09-15T03:46:45.937 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.937 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.937 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.937 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109441+0000", 2024-09-15T03:46:45.937 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1209999999999996e-06 2024-09-15T03:46:45.938 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.938 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.938 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109660+0000", 2024-09-15T03:46:45.938 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000219609 2024-09-15T03:46:45.938 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.938 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.938 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109685+0000", 2024-09-15T03:46:45.938 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4861000000000001e-05 2024-09-15T03:46:45.938 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.938 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.939 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.109722+0000", 2024-09-15T03:46:45.939 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6313999999999998e-05 2024-09-15T03:46:45.939 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.939 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.939 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.939 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.110431+0000", 2024-09-15T03:46:45.939 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00070969800000000003 2024-09-15T03:46:45.939 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.939 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.939 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.940 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.111807+0000", 2024-09-15T03:46:45.940 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013757019999999999 2024-09-15T03:46:45.940 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.940 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.940 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.940 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.940 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.940 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es7 34/28 ECSubWrite(tid=33, reqid=client.4521.0:195, at_version=34'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-15T03:46:45.940 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.118211+0000", 2024-09-15T03:46:45.940 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.7329074559999995, 2024-09-15T03:46:45.940 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0070162009999999997, 2024-09-15T03:46:45.940 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.940 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.941 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.941 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.941 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.941 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118211+0000", 2024-09-15T03:46:45.941 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.941 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.941 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.941 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.941 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118211+0000", 2024-09-15T03:46:45.941 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.941 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.941 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.941 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.941 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118212+0000", 2024-09-15T03:46:45.942 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5599999999999999e-07 2024-09-15T03:46:45.942 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.942 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.942 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.942 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118422+0000", 2024-09-15T03:46:45.942 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021081 2024-09-15T03:46:45.942 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.942 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.942 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.942 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118433+0000", 2024-09-15T03:46:45.942 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0529e-05 2024-09-15T03:46:45.942 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.942 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.943 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.943 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118438+0000", 2024-09-15T03:46:45.943 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8210000000000001e-06 2024-09-15T03:46:45.943 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.943 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.943 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.943 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118479+0000", 2024-09-15T03:46:45.943 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1493999999999999e-05 2024-09-15T03:46:45.943 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.944 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.944 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.944 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118490+0000", 2024-09-15T03:46:45.944 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0550000000000001e-05 2024-09-15T03:46:45.944 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.944 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.944 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.944 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118700+0000", 2024-09-15T03:46:45.944 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021063500000000001 2024-09-15T03:46:45.944 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.945 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.945 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.945 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118711+0000", 2024-09-15T03:46:45.945 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0849000000000001e-05 2024-09-15T03:46:45.945 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.945 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.945 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.945 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118776+0000", 2024-09-15T03:46:45.945 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4702999999999994e-05 2024-09-15T03:46:45.945 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.945 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.945 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.945 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118797+0000", 2024-09-15T03:46:45.946 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1265999999999999e-05 2024-09-15T03:46:45.946 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.946 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.946 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.946 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125228+0000", 2024-09-15T03:46:45.946 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064303880000000004 2024-09-15T03:46:45.946 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.946 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.946 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.946 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.946 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.946 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 38/28 ECSubWrite(tid=33, reqid=client.4521.0:222, at_version=38'14, trim_to=0'0, roll_forward_to=35'12))", 2024-09-15T03:46:45.946 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.131009+0000", 2024-09-15T03:46:45.947 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.72011027500000002, 2024-09-15T03:46:45.947 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0018653039999999999, 2024-09-15T03:46:45.947 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.947 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.947 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.947 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.947 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.947 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.131009+0000", 2024-09-15T03:46:45.947 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.947 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.947 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.947 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.947 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.131009+0000", 2024-09-15T03:46:45.947 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.948 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.948 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.949 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.949 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.131009+0000", 2024-09-15T03:46:45.949 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5499999999999999e-07 2024-09-15T03:46:45.949 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.949 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.949 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.949 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.131028+0000", 2024-09-15T03:46:45.949 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9279e-05 2024-09-15T03:46:45.949 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.949 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.950 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.950 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.131038+0000", 2024-09-15T03:46:45.950 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0156999999999999e-05 2024-09-15T03:46:45.950 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.950 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.950 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.950 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.131045+0000", 2024-09-15T03:46:45.950 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6220000000000003e-06 2024-09-15T03:46:45.950 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.950 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.950 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.950 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.131141+0000", 2024-09-15T03:46:45.951 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6077999999999997e-05 2024-09-15T03:46:45.951 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.951 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.951 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.951 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.131163+0000", 2024-09-15T03:46:45.951 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2107999999999999e-05 2024-09-15T03:46:45.951 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.951 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.951 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.951 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.131387+0000", 2024-09-15T03:46:45.951 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022441299999999999 2024-09-15T03:46:45.952 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.952 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.952 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.952 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.131398+0000", 2024-09-15T03:46:45.952 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0993e-05 2024-09-15T03:46:45.952 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.952 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.952 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.952 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.131445+0000", 2024-09-15T03:46:45.952 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6757e-05 2024-09-15T03:46:45.952 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.953 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.953 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.953 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.131468+0000", 2024-09-15T03:46:45.953 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2725999999999999e-05 2024-09-15T03:46:45.953 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.953 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.953 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.953 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.132874+0000", 2024-09-15T03:46:45.953 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0014059159999999999 2024-09-15T03:46:45.953 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.953 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.953 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.953 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.954 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.954 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 38/28 ECSubWrite(tid=34, reqid=client.4521.0:222, at_version=0'0, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T03:46:45.954 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.132394+0000", 2024-09-15T03:46:45.954 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.71872511800000005, 2024-09-15T03:46:45.954 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001546277, 2024-09-15T03:46:45.954 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.954 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.954 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.954 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.954 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.955 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.132394+0000", 2024-09-15T03:46:45.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.955 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.955 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.955 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.955 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.132394+0000", 2024-09-15T03:46:45.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.955 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.955 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.955 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.955 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.132394+0000", 2024-09-15T03:46:45.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8800000000000002e-07 2024-09-15T03:46:45.955 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.956 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.956 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.956 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.132405+0000", 2024-09-15T03:46:45.956 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0628000000000001e-05 2024-09-15T03:46:45.956 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.956 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.956 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.956 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.132418+0000", 2024-09-15T03:46:45.956 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.306e-05 2024-09-15T03:46:45.956 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.956 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.956 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.956 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.132424+0000", 2024-09-15T03:46:45.957 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0499999999999997e-06 2024-09-15T03:46:45.957 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.957 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.957 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.957 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.132543+0000", 2024-09-15T03:46:45.957 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000119079 2024-09-15T03:46:45.957 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.957 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.957 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.957 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.132552+0000", 2024-09-15T03:46:45.957 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9220000000000003e-06 2024-09-15T03:46:45.957 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.957 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.957 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.958 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.132639+0000", 2024-09-15T03:46:45.958 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6693999999999994e-05 2024-09-15T03:46:45.958 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.958 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.958 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.958 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.132646+0000", 2024-09-15T03:46:45.958 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8619999999999997e-06 2024-09-15T03:46:45.958 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.958 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.959 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.959 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.133756+0000", 2024-09-15T03:46:45.959 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001110649 2024-09-15T03:46:45.959 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.959 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.959 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.959 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.133769+0000", 2024-09-15T03:46:45.959 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3161e-05 2024-09-15T03:46:45.959 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.959 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.959 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.959 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.133940+0000", 2024-09-15T03:46:45.959 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017048399999999999 2024-09-15T03:46:45.960 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.960 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.960 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.960 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.960 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.960 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s3 38/31 ECSubWrite(tid=38, reqid=client.4521.0:223, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:46:45.960 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.146924+0000", 2024-09-15T03:46:45.960 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.70419476800000003, 2024-09-15T03:46:45.960 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002513852, 2024-09-15T03:46:45.961 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.961 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.961 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.961 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.961 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.146924+0000", 2024-09-15T03:46:45.961 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.146924+0000", 2024-09-15T03:46:45.962 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.146924+0000", 2024-09-15T03:46:45.962 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.73e-07 2024-09-15T03:46:45.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147412+0000", 2024-09-15T03:46:45.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00048740699999999998 2024-09-15T03:46:45.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147423+0000", 2024-09-15T03:46:45.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1578e-05 2024-09-15T03:46:45.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147428+0000", 2024-09-15T03:46:45.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2839999999999997e-06 2024-09-15T03:46:45.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.964 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147484+0000", 2024-09-15T03:46:45.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5043000000000001e-05 2024-09-15T03:46:45.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.964 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147502+0000", 2024-09-15T03:46:45.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.802e-05 2024-09-15T03:46:45.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.964 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147845+0000", 2024-09-15T03:46:45.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000343431 2024-09-15T03:46:45.965 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.965 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.965 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147856+0000", 2024-09-15T03:46:45.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0847e-05 2024-09-15T03:46:45.965 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.965 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.965 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147966+0000", 2024-09-15T03:46:45.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000110059 2024-09-15T03:46:45.965 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.966 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.966 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147980+0000", 2024-09-15T03:46:45.966 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3766e-05 2024-09-15T03:46:45.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.966 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.966 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.966 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149438+0000", 2024-09-15T03:46:45.966 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001458244 2024-09-15T03:46:45.966 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.966 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.966 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.966 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.966 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s3 38/31 ECSubWrite(tid=40, reqid=client.4521.0:225, at_version=38'10, trim_to=0'0, roll_forward_to=38'8))", 2024-09-15T03:46:45.967 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.150686+0000", 2024-09-15T03:46:45.967 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.70043241700000003, 2024-09-15T03:46:45.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0029850269999999999, 2024-09-15T03:46:45.967 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.967 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.967 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.967 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.967 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150686+0000", 2024-09-15T03:46:45.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.967 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.968 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150686+0000", 2024-09-15T03:46:45.968 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.968 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150687+0000", 2024-09-15T03:46:45.968 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1399999999999997e-07 2024-09-15T03:46:45.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.968 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150698+0000", 2024-09-15T03:46:45.968 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1294000000000001e-05 2024-09-15T03:46:45.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.969 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.969 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.969 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150707+0000", 2024-09-15T03:46:45.969 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9460000000000004e-06 2024-09-15T03:46:45.969 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.969 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.969 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.969 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150711+0000", 2024-09-15T03:46:45.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7620000000000001e-06 2024-09-15T03:46:45.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.972 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150745+0000", 2024-09-15T03:46:45.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4356000000000001e-05 2024-09-15T03:46:45.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.973 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150752+0000", 2024-09-15T03:46:45.973 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.263e-06 2024-09-15T03:46:45.973 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.973 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.973 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.973 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150833+0000", 2024-09-15T03:46:45.973 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1786000000000006e-05 2024-09-15T03:46:45.973 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.973 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.973 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.973 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150840+0000", 2024-09-15T03:46:45.973 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6200000000000001e-06 2024-09-15T03:46:45.973 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.974 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.974 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.974 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.153509+0000", 2024-09-15T03:46:45.974 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026692840000000001 2024-09-15T03:46:45.974 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.974 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.974 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.974 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.153522+0000", 2024-09-15T03:46:45.974 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3138e-05 2024-09-15T03:46:45.974 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.974 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.974 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.974 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.153671+0000", 2024-09-15T03:46:45.975 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014906399999999999 2024-09-15T03:46:45.975 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.975 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.975 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.975 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.975 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es7 38/28 ECSubWrite(tid=45, reqid=client.4521.0:227, at_version=38'24, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:46:45.975 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.155205+0000", 2024-09-15T03:46:45.975 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.69591425399999995, 2024-09-15T03:46:45.975 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0035096289999999998, 2024-09-15T03:46:45.975 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.975 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.975 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155205+0000", 2024-09-15T03:46:45.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155205+0000", 2024-09-15T03:46:45.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.977 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155205+0000", 2024-09-15T03:46:45.977 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2100000000000001e-07 2024-09-15T03:46:45.977 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.977 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.977 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.977 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155523+0000", 2024-09-15T03:46:45.977 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00031849999999999999 2024-09-15T03:46:45.977 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.977 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.977 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.977 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155533+0000", 2024-09-15T03:46:45.977 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6260000000000007e-06 2024-09-15T03:46:45.977 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.978 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155537+0000", 2024-09-15T03:46:45.978 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2490000000000003e-06 2024-09-15T03:46:45.978 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.978 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155598+0000", 2024-09-15T03:46:45.978 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0915999999999997e-05 2024-09-15T03:46:45.978 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.978 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155607+0000", 2024-09-15T03:46:45.978 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4909999999999994e-06 2024-09-15T03:46:45.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155937+0000", 2024-09-15T03:46:45.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00032933700000000002 2024-09-15T03:46:45.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155943+0000", 2024-09-15T03:46:45.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0599999999999996e-06 2024-09-15T03:46:45.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157049+0000", 2024-09-15T03:46:45.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001105766 2024-09-15T03:46:45.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157069+0000", 2024-09-15T03:46:45.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0477999999999999e-05 2024-09-15T03:46:45.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158714+0000", 2024-09-15T03:46:45.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0016450849999999999 2024-09-15T03:46:45.980 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.981 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.981 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.981 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.981 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.981 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es7 38/28 ECSubWrite(tid=46, reqid=client.4521.0:228, at_version=38'25, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:46:45.981 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.157437+0000", 2024-09-15T03:46:45.981 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.69368131899999996, 2024-09-15T03:46:45.981 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0014321620000000001, 2024-09-15T03:46:45.981 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:45.981 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:45.981 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:45.981 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.981 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:45.981 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157437+0000", 2024-09-15T03:46:45.982 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.982 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:45.982 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157437+0000", 2024-09-15T03:46:45.982 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:45.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.982 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:45.982 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157438+0000", 2024-09-15T03:46:45.982 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.68e-07 2024-09-15T03:46:45.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.982 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:45.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157862+0000", 2024-09-15T03:46:45.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000423786 2024-09-15T03:46:45.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:45.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157897+0000", 2024-09-15T03:46:45.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5484000000000001e-05 2024-09-15T03:46:45.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:45.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157906+0000", 2024-09-15T03:46:45.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6070000000000008e-06 2024-09-15T03:46:45.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.984 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:45.984 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157972+0000", 2024-09-15T03:46:45.984 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5617999999999994e-05 2024-09-15T03:46:45.984 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.984 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:45.984 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157980+0000", 2024-09-15T03:46:45.984 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9980000000000003e-06 2024-09-15T03:46:45.984 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.984 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:45.984 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158094+0000", 2024-09-15T03:46:45.985 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011465599999999999 2024-09-15T03:46:45.985 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.985 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.985 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:45.985 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158114+0000", 2024-09-15T03:46:45.985 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0072999999999999e-05 2024-09-15T03:46:45.985 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.985 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.985 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:45.985 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158188+0000", 2024-09-15T03:46:45.985 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3616999999999997e-05 2024-09-15T03:46:45.985 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.985 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.986 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:45.986 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158428+0000", 2024-09-15T03:46:45.986 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023979399999999999 2024-09-15T03:46:45.986 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:45.986 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:45.986 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:45.986 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158870+0000", 2024-09-15T03:46:45.986 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00044186099999999999 2024-09-15T03:46:45.986 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.986 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.986 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.986 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:45.986 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:45.986 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:45.988 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:46:45.989 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:45.987+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:46:46.088 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:46.087+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:46.126 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:46.126 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:46.127 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:46:46.127 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:46.135 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:46:46.141 INFO:tasks.rados.rados.0.smithi001.stdout:90: done (5 left) 2024-09-15T03:46:46.142 INFO:tasks.rados.rados.0.smithi001.stdout:91: rollback oid 6 current snap is 6 2024-09-15T03:46:46.142 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 6 to 0 2024-09-15T03:46:46.142 INFO:tasks.rados.rados.0.smithi001.stdout:86: finishing write tid 1 to smithi00132172-4 2024-09-15T03:46:46.142 INFO:tasks.rados.rados.0.smithi001.stdout:85: finishing write tid 4 to smithi00132172-27 2024-09-15T03:46:46.142 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 27 v 10 (ObjNum 63 snap 6 seq_num 63) dirty exists 2024-09-15T03:46:46.142 INFO:tasks.rados.rados.0.smithi001.stdout:85: left oid 27 (ObjNum 63 snap 6 seq_num 63) 2024-09-15T03:46:46.142 INFO:tasks.rados.rados.0.smithi001.stdout:89: finishing rollback tid 0 to smithi00132172-45 2024-09-15T03:46:46.142 INFO:tasks.rados.rados.0.smithi001.stdout:86: finishing write tid 2 to smithi00132172-4 2024-09-15T03:46:46.142 INFO:tasks.rados.rados.0.smithi001.stdout:86: finishing write tid 3 to smithi00132172-4 2024-09-15T03:46:46.142 INFO:tasks.rados.rados.0.smithi001.stdout:89: finishing rollback tid 1 to smithi00132172-45 2024-09-15T03:46:46.142 INFO:tasks.rados.rados.0.smithi001.stdout:89: finishing rollback tid 2 to smithi00132172-45 2024-09-15T03:46:46.142 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 45 v 12 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-15T03:46:46.142 INFO:tasks.rados.rados.0.smithi001.stdout:86: finishing write tid 4 to smithi00132172-4 2024-09-15T03:46:46.143 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 4 v 26 (ObjNum 64 snap 6 seq_num 64) dirty exists 2024-09-15T03:46:46.143 INFO:tasks.rados.rados.0.smithi001.stdout:86: left oid 4 (ObjNum 64 snap 6 seq_num 64) 2024-09-15T03:46:46.143 INFO:tasks.rados.rados.0.smithi001.stdout:88: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-15T03:46:46.167 INFO:tasks.rados.rados.0.smithi001.stdout:87: finishing copy_from racing read to smithi00132172-35 2024-09-15T03:46:46.167 INFO:tasks.rados.rados.0.smithi001.stdout:87: finishing copy_from to smithi00132172-35 2024-09-15T03:46:46.167 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 35 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-15T03:46:46.167 INFO:tasks.rados.rados.0.smithi001.stdout:85: done (5 left) 2024-09-15T03:46:46.167 INFO:tasks.rados.rados.0.smithi001.stdout:86: done (4 left) 2024-09-15T03:46:46.167 INFO:tasks.rados.rados.0.smithi001.stdout:87: done (3 left) 2024-09-15T03:46:46.168 INFO:tasks.rados.rados.0.smithi001.stdout:88: done (2 left) 2024-09-15T03:46:46.168 INFO:tasks.rados.rados.0.smithi001.stdout:89: done (1 left) 2024-09-15T03:46:46.168 INFO:tasks.rados.rados.0.smithi001.stdout:92: setattr oid 25 current snap is 6 2024-09-15T03:46:46.168 INFO:tasks.rados.rados.0.smithi001.stdout:91: finishing rollback tid 0 to smithi00132172-6 2024-09-15T03:46:46.168 INFO:tasks.rados.rados.0.smithi001.stdout:91: finishing rollback tid 1 to smithi00132172-6 2024-09-15T03:46:46.168 INFO:tasks.rados.rados.0.smithi001.stdout:91: finishing rollback tid 2 to smithi00132172-6 2024-09-15T03:46:46.168 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 6 v 18 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-09-15T03:46:46.168 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:46:46.170 INFO:tasks.rados.rados.0.smithi001.stdout:91: done (1 left) 2024-09-15T03:46:46.170 INFO:tasks.rados.rados.0.smithi001.stdout:93: read oid 17 snap 1 2024-09-15T03:46:46.170 INFO:tasks.rados.rados.0.smithi001.stdout:93: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-15T03:46:46.170 INFO:tasks.rados.rados.0.smithi001.stdout:94: copy_from oid 40 from oid 42 current snap is 6 2024-09-15T03:46:46.170 INFO:tasks.rados.rados.0.smithi001.stdout:95: read oid 48 snap -1 2024-09-15T03:46:46.171 INFO:tasks.rados.rados.0.smithi001.stdout:95: expect (ObjNum 47 snap 0 seq_num 47) 2024-09-15T03:46:46.171 INFO:tasks.rados.rados.0.smithi001.stdout:96: copy_from oid 29 from oid 23 current snap is 6 2024-09-15T03:46:46.171 INFO:tasks.rados.rados.0.smithi001.stdout:97: rollback oid 38 current snap is 6 2024-09-15T03:46:46.171 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 38 to 0 2024-09-15T03:46:46.171 INFO:tasks.rados.rados.0.smithi001.stdout:98: snap_remove snap 2 2024-09-15T03:46:46.189 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:46:46.189+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:46:46.272 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:46.272 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:46.272 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:46:46.272 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:46:46.272 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:46.280 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:46:46.290 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:46.289+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:46:46.296 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:46:46.297 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:46:46.311 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:46:46.311 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:46:46.325 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:46:46.326 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:46:46.339 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:46:46.339 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:46:46.354 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:46:46.354 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:46:46.377 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:46.378 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:46:46.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:46:46.378 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [] 2024-09-15T03:46:46.378 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:46.386 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:46:46.391 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:46:46.391+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:46:46.482 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:46.483 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:46.483 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:46:46.483 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:46.491 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:46:46.493 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:46.492+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:46.588 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:46.588 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:46.588 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:46:46.588 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:46:46.588 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:46.594 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:46.593+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:46.598 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:46:46.695 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:46.694+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:46.696 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:46:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:46:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:46:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:184 3.as0 3:5515acd6:::smithi00132172-13:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104658+0000", 2024-09-15T03:46:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.5894201480000003, 2024-09-15T03:46:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006212525, 2024-09-15T03:46:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 184 2024-09-15T03:46:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104658+0000", 2024-09-15T03:46:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104658+0000", 2024-09-15T03:46:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:46.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104660+0000", 2024-09-15T03:46:46.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.17e-06 2024-09-15T03:46:46.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:46.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104672+0000", 2024-09-15T03:46:46.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1161999999999999e-05 2024-09-15T03:46:46.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:46.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104693+0000", 2024-09-15T03:46:46.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0995000000000001e-05 2024-09-15T03:46:46.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:46.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104700+0000", 2024-09-15T03:46:46.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4189999999999996e-06 2024-09-15T03:46:46.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104752+0000", 2024-09-15T03:46:46.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2413000000000001e-05 2024-09-15T03:46:46.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-09-15T03:46:46.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104757+0000", 2024-09-15T03:46:46.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9599999999999999e-06 2024-09-15T03:46:46.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.105683+0000", 2024-09-15T03:46:46.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00092545599999999996 2024-09-15T03:46:46.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:46.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.105757+0000", 2024-09-15T03:46:46.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4567999999999995e-05 2024-09-15T03:46:46.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:46.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106531+0000", 2024-09-15T03:46:46.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00077397999999999996 2024-09-15T03:46:46.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:46.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106775+0000", 2024-09-15T03:46:46.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024402899999999999 2024-09-15T03:46:46.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:46.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106786+0000", 2024-09-15T03:46:46.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0825000000000001e-05 2024-09-15T03:46:46.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:46.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106845+0000", 2024-09-15T03:46:46.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8733999999999997e-05 2024-09-15T03:46:46.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:46.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109399+0000", 2024-09-15T03:46:46.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002553573 2024-09-15T03:46:46.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:46.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109844+0000", 2024-09-15T03:46:46.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00044492700000000002 2024-09-15T03:46:46.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:46.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.110871+0000", 2024-09-15T03:46:46.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010273140000000001 2024-09-15T03:46:46.705 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.705 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:46.706 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.706 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:185 3.as0 3:5515acd6:::smithi00132172-13:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:46.706 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:46:46.706 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.5893693180000001, 2024-09-15T03:46:46.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01268643, 2024-09-15T03:46:46.706 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:46.706 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:46.706 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:46.706 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:46.706 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:46.706 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 185 2024-09-15T03:46:46.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.707 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:46.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:46.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:46:46.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:46.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:46:46.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.708 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:46.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104710+0000", 2024-09-15T03:46:46.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0200000000000003e-07 2024-09-15T03:46:46.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.708 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:46.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104713+0000", 2024-09-15T03:46:46.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2339999999999999e-06 2024-09-15T03:46:46.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.708 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:46.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104718+0000", 2024-09-15T03:46:46.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3449999999999996e-06 2024-09-15T03:46:46.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:46.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104722+0000", 2024-09-15T03:46:46.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8399999999999997e-06 2024-09-15T03:46:46.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104849+0000", 2024-09-15T03:46:46.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012709599999999999 2024-09-15T03:46:46.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:46.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104856+0000", 2024-09-15T03:46:46.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.652e-06 2024-09-15T03:46:46.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106851+0000", 2024-09-15T03:46:46.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019954840000000001 2024-09-15T03:46:46.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:46.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106935+0000", 2024-09-15T03:46:46.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4095e-05 2024-09-15T03:46:46.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:46.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107453+0000", 2024-09-15T03:46:46.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00051729499999999997 2024-09-15T03:46:46.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:46.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107554+0000", 2024-09-15T03:46:46.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000101251 2024-09-15T03:46:46.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.712 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:46.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108104+0000", 2024-09-15T03:46:46.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00054995100000000002 2024-09-15T03:46:46.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.712 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:46.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108503+0000", 2024-09-15T03:46:46.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039953500000000002 2024-09-15T03:46:46.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.712 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:46.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109405+0000", 2024-09-15T03:46:46.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00090170099999999998 2024-09-15T03:46:46.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.713 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:46.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114365+0000", 2024-09-15T03:46:46.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049602120000000003 2024-09-15T03:46:46.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.713 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:46.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117396+0000", 2024-09-15T03:46:46.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030303370000000001 2024-09-15T03:46:46.713 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.713 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:46.713 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.714 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:186 3.as0 3:5515acd6:::smithi00132172-13:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:46.714 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:46:46.714 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.5893295280000004, 2024-09-15T03:46:46.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012817485999999999, 2024-09-15T03:46:46.714 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:46.714 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:46.714 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:46.714 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:46.714 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:46.714 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 186 2024-09-15T03:46:46.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.715 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:46.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:46.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:46:46.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:46.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:46:46.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:46.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104750+0000", 2024-09-15T03:46:46.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1500000000000004e-07 2024-09-15T03:46:46.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:46.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104754+0000", 2024-09-15T03:46:46.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.946e-06 2024-09-15T03:46:46.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:46.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104769+0000", 2024-09-15T03:46:46.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.507e-05 2024-09-15T03:46:46.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:46.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104776+0000", 2024-09-15T03:46:46.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7849999999999992e-06 2024-09-15T03:46:46.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104888+0000", 2024-09-15T03:46:46.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011164500000000001 2024-09-15T03:46:46.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:46.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104894+0000", 2024-09-15T03:46:46.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.942e-06 2024-09-15T03:46:46.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108172+0000", 2024-09-15T03:46:46.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032781939999999999 2024-09-15T03:46:46.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:46.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108257+0000", 2024-09-15T03:46:46.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4488999999999997e-05 2024-09-15T03:46:46.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:46.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108732+0000", 2024-09-15T03:46:46.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047547399999999998 2024-09-15T03:46:46.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.719 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.719 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:46.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108823+0000", 2024-09-15T03:46:46.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1106000000000005e-05 2024-09-15T03:46:46.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.719 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.719 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:46.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108961+0000", 2024-09-15T03:46:46.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013735099999999999 2024-09-15T03:46:46.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.720 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:46.720 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109065+0000", 2024-09-15T03:46:46.720 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104715 2024-09-15T03:46:46.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.720 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:46.720 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109406+0000", 2024-09-15T03:46:46.720 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034062300000000002 2024-09-15T03:46:46.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.720 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:46.720 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114529+0000", 2024-09-15T03:46:46.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.005123219 2024-09-15T03:46:46.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:46.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117566+0000", 2024-09-15T03:46:46.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030372120000000001 2024-09-15T03:46:46.721 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.721 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:46.721 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.721 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:187 3.fs0 3:fe9c5513:::smithi00132172-47:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:46.721 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104782+0000", 2024-09-15T03:46:46.722 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.5892965090000004, 2024-09-15T03:46:46.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069218609999999996, 2024-09-15T03:46:46.722 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:46.722 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:46.722 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:46.722 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:46.722 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:46.722 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 187 2024-09-15T03:46:46.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.722 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:46.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:46.722 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104782+0000", 2024-09-15T03:46:46.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:46.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104782+0000", 2024-09-15T03:46:46.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:46.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104782+0000", 2024-09-15T03:46:46.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8599999999999998e-07 2024-09-15T03:46:46.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:46.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104785+0000", 2024-09-15T03:46:46.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.649e-06 2024-09-15T03:46:46.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:46.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104800+0000", 2024-09-15T03:46:46.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5305e-05 2024-09-15T03:46:46.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:46.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104805+0000", 2024-09-15T03:46:46.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5569999999999997e-06 2024-09-15T03:46:46.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104854+0000", 2024-09-15T03:46:46.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9039999999999998e-05 2024-09-15T03:46:46.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-09-15T03:46:46.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104860+0000", 2024-09-15T03:46:46.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2979999999999997e-06 2024-09-15T03:46:46.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.105869+0000", 2024-09-15T03:46:46.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001009036 2024-09-15T03:46:46.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.726 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:46.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.105954+0000", 2024-09-15T03:46:46.726 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4357000000000006e-05 2024-09-15T03:46:46.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.726 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:46.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106626+0000", 2024-09-15T03:46:46.726 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00067191099999999997 2024-09-15T03:46:46.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.726 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:46.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106786+0000", 2024-09-15T03:46:46.726 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000160469 2024-09-15T03:46:46.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:46.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106788+0000", 2024-09-15T03:46:46.727 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9630000000000001e-06 2024-09-15T03:46:46.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:46.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106846+0000", 2024-09-15T03:46:46.727 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7719000000000001e-05 2024-09-15T03:46:46.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:46.728 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108372+0000", 2024-09-15T03:46:46.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0015264499999999999 2024-09-15T03:46:46.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.728 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.728 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:46.728 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.111475+0000", 2024-09-15T03:46:46.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031028459999999998 2024-09-15T03:46:46.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.728 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.728 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:46.728 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.111704+0000", 2024-09-15T03:46:46.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022877499999999999 2024-09-15T03:46:46.728 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.729 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:46.729 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.729 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:188 3.fs0 3:fe9c5513:::smithi00132172-47:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:46.729 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:46:46.729 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.5892468309999996, 2024-09-15T03:46:46.729 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012637061, 2024-09-15T03:46:46.729 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:46.729 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:46.729 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:46.729 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:46.729 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:46.730 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 188 2024-09-15T03:46:46.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.730 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:46.730 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.730 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:46.730 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:46:46.730 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.730 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.730 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:46.730 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:46:46.730 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:46.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:46:46.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2799999999999996e-07 2024-09-15T03:46:46.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:46.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104835+0000", 2024-09-15T03:46:46.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7190000000000001e-06 2024-09-15T03:46:46.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:46.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104842+0000", 2024-09-15T03:46:46.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4580000000000004e-06 2024-09-15T03:46:46.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.732 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:46.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104846+0000", 2024-09-15T03:46:46.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1250000000000003e-06 2024-09-15T03:46:46.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.732 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104884+0000", 2024-09-15T03:46:46.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7075e-05 2024-09-15T03:46:46.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.732 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:46.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104888+0000", 2024-09-15T03:46:46.733 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8520000000000003e-06 2024-09-15T03:46:46.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106860+0000", 2024-09-15T03:46:46.733 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019719440000000002 2024-09-15T03:46:46.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:46.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106938+0000", 2024-09-15T03:46:46.733 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7421000000000003e-05 2024-09-15T03:46:46.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.734 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.734 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:46.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107368+0000", 2024-09-15T03:46:46.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043042999999999998 2024-09-15T03:46:46.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.734 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.734 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:46.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107460+0000", 2024-09-15T03:46:46.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1392999999999997e-05 2024-09-15T03:46:46.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.734 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.734 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:46.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107473+0000", 2024-09-15T03:46:46.735 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3702e-05 2024-09-15T03:46:46.735 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.735 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.735 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:46.735 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107504+0000", 2024-09-15T03:46:46.735 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1000000000000001e-05 2024-09-15T03:46:46.735 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.735 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.735 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:46.735 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108376+0000", 2024-09-15T03:46:46.735 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00087162000000000003 2024-09-15T03:46:46.735 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.735 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.735 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:46.736 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.111657+0000", 2024-09-15T03:46:46.736 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032815589999999999 2024-09-15T03:46:46.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.736 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.736 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:46.736 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117469+0000", 2024-09-15T03:46:46.736 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058112349999999997 2024-09-15T03:46:46.736 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.736 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:46.736 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.736 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.736 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:189 3.fs0 3:fe9c5513:::smithi00132172-47:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:46.737 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:46:46.737 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.5892276450000002, 2024-09-15T03:46:46.737 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012731009999999999, 2024-09-15T03:46:46.737 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:46.737 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:46.737 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:46.737 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:46.737 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:46.737 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 189 2024-09-15T03:46:46.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.737 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:46.737 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.737 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:46.738 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:46:46.738 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.738 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:46.738 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:46:46.738 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.738 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:46.738 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:46:46.738 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7899999999999999e-07 2024-09-15T03:46:46.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.739 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:46.739 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104853+0000", 2024-09-15T03:46:46.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.933e-06 2024-09-15T03:46:46.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.739 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.739 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:46.739 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104859+0000", 2024-09-15T03:46:46.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.592e-06 2024-09-15T03:46:46.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.739 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.739 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:46.739 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104863+0000", 2024-09-15T03:46:46.740 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4000000000000002e-06 2024-09-15T03:46:46.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.740 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104910+0000", 2024-09-15T03:46:46.740 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6678000000000003e-05 2024-09-15T03:46:46.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.740 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:46.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104916+0000", 2024-09-15T03:46:46.740 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1079999999999998e-06 2024-09-15T03:46:46.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.740 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.741 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107550+0000", 2024-09-15T03:46:46.741 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026344110000000001 2024-09-15T03:46:46.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.741 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.741 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:46.741 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107624+0000", 2024-09-15T03:46:46.741 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4134000000000005e-05 2024-09-15T03:46:46.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.741 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.741 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:46.741 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107988+0000", 2024-09-15T03:46:46.741 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036353600000000002 2024-09-15T03:46:46.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.742 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.742 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:46.742 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108066+0000", 2024-09-15T03:46:46.742 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8047999999999995e-05 2024-09-15T03:46:46.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.742 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.742 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:46.742 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108105+0000", 2024-09-15T03:46:46.742 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8868000000000001e-05 2024-09-15T03:46:46.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.742 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.742 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:46.742 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108504+0000", 2024-09-15T03:46:46.743 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039916600000000001 2024-09-15T03:46:46.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.743 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:46.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108955+0000", 2024-09-15T03:46:46.743 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045126200000000002 2024-09-15T03:46:46.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.743 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:46.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114394+0000", 2024-09-15T03:46:46.743 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0054382750000000002 2024-09-15T03:46:46.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.743 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:46.744 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117582+0000", 2024-09-15T03:46:46.744 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031882199999999999 2024-09-15T03:46:46.744 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.744 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:46.744 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.744 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.744 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es3 34/28 ECSubWrite(tid=33, reqid=client.4521.0:195, at_version=34'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-15T03:46:46.744 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.118213+0000", 2024-09-15T03:46:46.744 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.5758654199999995, 2024-09-15T03:46:46.744 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006662701, 2024-09-15T03:46:46.744 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:46.744 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:46.745 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:46.745 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.745 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:46.745 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118213+0000", 2024-09-15T03:46:46.745 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.745 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.745 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:46.745 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118213+0000", 2024-09-15T03:46:46.745 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.745 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.745 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:46.746 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118213+0000", 2024-09-15T03:46:46.746 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6300000000000001e-07 2024-09-15T03:46:46.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.746 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.746 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:46.746 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118447+0000", 2024-09-15T03:46:46.746 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000234016 2024-09-15T03:46:46.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.746 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.746 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:46.746 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118465+0000", 2024-09-15T03:46:46.746 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7978e-05 2024-09-15T03:46:46.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.746 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.747 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:46.747 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118475+0000", 2024-09-15T03:46:46.747 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3179999999999996e-06 2024-09-15T03:46:46.747 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.747 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.747 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.747 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118524+0000", 2024-09-15T03:46:46.747 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9069000000000001e-05 2024-09-15T03:46:46.747 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.747 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.747 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:46.747 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118534+0000", 2024-09-15T03:46:46.747 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0222e-05 2024-09-15T03:46:46.747 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.748 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.748 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:46.748 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118727+0000", 2024-09-15T03:46:46.748 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000193124 2024-09-15T03:46:46.748 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.748 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.748 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:46.748 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118734+0000", 2024-09-15T03:46:46.748 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2080000000000003e-06 2024-09-15T03:46:46.748 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.748 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.748 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:46.749 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118804+0000", 2024-09-15T03:46:46.749 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9720000000000003e-05 2024-09-15T03:46:46.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.749 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.749 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:46.749 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118820+0000", 2024-09-15T03:46:46.749 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5705999999999998e-05 2024-09-15T03:46:46.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.749 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.749 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:46.749 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124876+0000", 2024-09-15T03:46:46.749 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006056077 2024-09-15T03:46:46.749 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.749 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:46.750 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.750 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.750 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.750 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 34/31 ECSubWrite(tid=34, reqid=client.4521.0:197, at_version=34'21, trim_to=0'0, roll_forward_to=34'18))", 2024-09-15T03:46:46.750 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.123980+0000", 2024-09-15T03:46:46.750 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.5700985909999998, 2024-09-15T03:46:46.750 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0066547489999999997, 2024-09-15T03:46:46.750 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:46.750 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:46.750 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:46.750 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.750 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:46.750 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123980+0000", 2024-09-15T03:46:46.750 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.751 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.751 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.751 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:46.751 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123980+0000", 2024-09-15T03:46:46.751 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.751 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.751 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.751 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:46.751 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123980+0000", 2024-09-15T03:46:46.751 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8800000000000003e-07 2024-09-15T03:46:46.751 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.751 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.751 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:46.752 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123996+0000", 2024-09-15T03:46:46.752 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5838e-05 2024-09-15T03:46:46.752 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.752 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.752 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:46.752 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124005+0000", 2024-09-15T03:46:46.752 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1249999999999999e-06 2024-09-15T03:46:46.752 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.752 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.752 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:46.752 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124010+0000", 2024-09-15T03:46:46.752 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2640000000000002e-06 2024-09-15T03:46:46.752 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.753 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.753 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.753 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124045+0000", 2024-09-15T03:46:46.753 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5145000000000003e-05 2024-09-15T03:46:46.754 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.754 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.754 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:46.754 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124054+0000", 2024-09-15T03:46:46.754 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8389999999999996e-06 2024-09-15T03:46:46.754 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.754 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.754 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:46.754 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124696+0000", 2024-09-15T03:46:46.754 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00064247499999999999 2024-09-15T03:46:46.754 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.754 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.754 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:46.754 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124703+0000", 2024-09-15T03:46:46.755 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.905e-06 2024-09-15T03:46:46.755 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.755 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.755 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:46.755 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124736+0000", 2024-09-15T03:46:46.755 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2966999999999998e-05 2024-09-15T03:46:46.755 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.755 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.755 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:46.755 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124758+0000", 2024-09-15T03:46:46.755 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2569999999999999e-05 2024-09-15T03:46:46.755 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.755 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.755 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:46.756 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.130635+0000", 2024-09-15T03:46:46.756 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058761330000000004 2024-09-15T03:46:46.756 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.756 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:46.756 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.756 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.756 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.756 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 34/31 ECSubWrite(tid=35, reqid=client.4521.0:197, at_version=0'0, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:46:46.756 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.125913+0000", 2024-09-15T03:46:46.756 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.5681656750000004, 2024-09-15T03:46:46.757 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0048403999999999999, 2024-09-15T03:46:46.757 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:46.757 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:46.757 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:46.757 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.757 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:46.757 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125913+0000", 2024-09-15T03:46:46.757 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.757 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.757 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.757 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:46.758 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125913+0000", 2024-09-15T03:46:46.758 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.758 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.758 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.758 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:46.758 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125913+0000", 2024-09-15T03:46:46.758 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3499999999999996e-07 2024-09-15T03:46:46.758 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.758 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.758 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:46.758 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125924+0000", 2024-09-15T03:46:46.758 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0669000000000001e-05 2024-09-15T03:46:46.758 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.759 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.759 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:46.759 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125935+0000", 2024-09-15T03:46:46.759 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1225e-05 2024-09-15T03:46:46.759 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.759 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.759 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:46.759 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125941+0000", 2024-09-15T03:46:46.759 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2519999999999996e-06 2024-09-15T03:46:46.759 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.759 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.759 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.759 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125979+0000", 2024-09-15T03:46:46.760 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7765999999999997e-05 2024-09-15T03:46:46.760 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.760 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.760 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:46.760 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125989+0000", 2024-09-15T03:46:46.760 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5950000000000005e-06 2024-09-15T03:46:46.760 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.760 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.760 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:46.760 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.126092+0000", 2024-09-15T03:46:46.760 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000103132 2024-09-15T03:46:46.760 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.761 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.761 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:46.761 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.126097+0000", 2024-09-15T03:46:46.761 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0220000000000004e-06 2024-09-15T03:46:46.761 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.761 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.761 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:46.761 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.126129+0000", 2024-09-15T03:46:46.761 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1668999999999998e-05 2024-09-15T03:46:46.761 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.761 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.761 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:46.762 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.126136+0000", 2024-09-15T03:46:46.762 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6179999999999997e-06 2024-09-15T03:46:46.762 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.762 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.762 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:46.762 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.130753+0000", 2024-09-15T03:46:46.762 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0046169169999999999 2024-09-15T03:46:46.762 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.762 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:46.762 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.762 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.762 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.762 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:222 3.as0 3:5515acd6:::smithi00132172-13:head [delete] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)", 2024-09-15T03:46:46.763 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.127651+0000", 2024-09-15T03:46:46.763 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.5664274599999999, 2024-09-15T03:46:46.763 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0063992839999999999, 2024-09-15T03:46:46.763 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:46.763 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:46.763 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:46.763 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:46.763 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:46.763 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 222 2024-09-15T03:46:46.763 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.763 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:46.764 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.764 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:46.764 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.127651+0000", 2024-09-15T03:46:46.764 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.764 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.764 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.764 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:46.764 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.127651+0000", 2024-09-15T03:46:46.764 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.764 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.764 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.764 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:46.765 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.127653+0000", 2024-09-15T03:46:46.765 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8190000000000001e-06 2024-09-15T03:46:46.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.765 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.765 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:46.765 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.127660+0000", 2024-09-15T03:46:46.765 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9500000000000004e-06 2024-09-15T03:46:46.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.765 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.765 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:46.765 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.127677+0000", 2024-09-15T03:46:46.765 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7255999999999999e-05 2024-09-15T03:46:46.766 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.766 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.766 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:46.766 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.127688+0000", 2024-09-15T03:46:46.766 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0762e-05 2024-09-15T03:46:46.766 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.766 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.766 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.766 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.127732+0000", 2024-09-15T03:46:46.766 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4308000000000003e-05 2024-09-15T03:46:46.766 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.766 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.766 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-09-15T03:46:46.767 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.127737+0000", 2024-09-15T03:46:46.767 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8099999999999997e-06 2024-09-15T03:46:46.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.767 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.767 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.767 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.130409+0000", 2024-09-15T03:46:46.767 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026722809999999999 2024-09-15T03:46:46.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.767 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.767 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:46.767 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.130471+0000", 2024-09-15T03:46:46.767 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1709000000000002e-05 2024-09-15T03:46:46.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.768 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.768 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:46.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.130960+0000", 2024-09-15T03:46:46.768 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00048949300000000002 2024-09-15T03:46:46.768 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.768 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.768 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:46.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.131233+0000", 2024-09-15T03:46:46.768 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000272627 2024-09-15T03:46:46.768 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.768 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.768 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:46.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.131242+0000", 2024-09-15T03:46:46.769 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.115e-06 2024-09-15T03:46:46.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.769 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:46.769 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.131280+0000", 2024-09-15T03:46:46.769 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8324000000000001e-05 2024-09-15T03:46:46.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.769 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:46.769 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.131294+0000", 2024-09-15T03:46:46.769 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3896999999999999e-05 2024-09-15T03:46:46.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.770 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:46.770 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.132132+0000", 2024-09-15T03:46:46.770 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00083733500000000003 2024-09-15T03:46:46.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.770 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.770 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:46.770 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.134050+0000", 2024-09-15T03:46:46.770 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019185980000000001 2024-09-15T03:46:46.770 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.770 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:46.770 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.770 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.770 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 38/31 ECSubWrite(tid=36, reqid=client.4521.0:220, at_version=38'23, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:46:46.771 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.148035+0000", 2024-09-15T03:46:46.771 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.546043539, 2024-09-15T03:46:46.771 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010074534, 2024-09-15T03:46:46.771 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:46.771 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:46.771 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:46.771 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.771 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:46.771 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148035+0000", 2024-09-15T03:46:46.771 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.771 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.771 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.772 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:46.772 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148035+0000", 2024-09-15T03:46:46.772 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.772 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.772 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.772 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:46.772 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148036+0000", 2024-09-15T03:46:46.772 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3939999999999999e-06 2024-09-15T03:46:46.772 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.772 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.772 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:46.772 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148681+0000", 2024-09-15T03:46:46.772 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00064493899999999997 2024-09-15T03:46:46.773 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.773 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.773 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:46.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148695+0000", 2024-09-15T03:46:46.773 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4078e-05 2024-09-15T03:46:46.773 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.773 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.773 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:46.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148701+0000", 2024-09-15T03:46:46.773 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7230000000000001e-06 2024-09-15T03:46:46.773 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.773 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.773 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148742+0000", 2024-09-15T03:46:46.774 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0781000000000001e-05 2024-09-15T03:46:46.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.774 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.774 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:46.774 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148749+0000", 2024-09-15T03:46:46.774 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6109999999999996e-06 2024-09-15T03:46:46.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.774 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.774 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:46.774 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149077+0000", 2024-09-15T03:46:46.774 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000327597 2024-09-15T03:46:46.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.774 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.775 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:46.775 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149087+0000", 2024-09-15T03:46:46.775 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4739999999999994e-06 2024-09-15T03:46:46.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.775 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.775 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:46.775 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149230+0000", 2024-09-15T03:46:46.775 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014398300000000001 2024-09-15T03:46:46.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.775 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.775 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:46.775 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149247+0000", 2024-09-15T03:46:46.775 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6201e-05 2024-09-15T03:46:46.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.776 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.776 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:46.776 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158109+0000", 2024-09-15T03:46:46.776 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0088627529999999993 2024-09-15T03:46:46.776 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.776 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:46.776 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.776 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.776 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.776 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 38/31 ECSubWrite(tid=37, reqid=client.4521.0:220, at_version=0'0, trim_to=0'0, roll_forward_to=38'23))", 2024-09-15T03:46:46.776 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.151308+0000", 2024-09-15T03:46:46.776 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.5427702939999999, 2024-09-15T03:46:46.776 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069978310000000004, 2024-09-15T03:46:46.777 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:46.777 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:46.777 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:46.777 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.777 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:46.777 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151308+0000", 2024-09-15T03:46:46.777 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.777 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.777 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.777 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:46.777 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151308+0000", 2024-09-15T03:46:46.777 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.777 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.777 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.778 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:46.778 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151309+0000", 2024-09-15T03:46:46.778 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2200000000000003e-07 2024-09-15T03:46:46.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.778 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.778 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:46.778 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151318+0000", 2024-09-15T03:46:46.778 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7250000000000003e-06 2024-09-15T03:46:46.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.778 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.778 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:46.778 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151330+0000", 2024-09-15T03:46:46.778 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2425e-05 2024-09-15T03:46:46.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.779 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.779 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:46.779 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151337+0000", 2024-09-15T03:46:46.779 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6139999999999997e-06 2024-09-15T03:46:46.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.779 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.779 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.779 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151372+0000", 2024-09-15T03:46:46.779 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5683999999999999e-05 2024-09-15T03:46:46.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.779 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.779 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:46.779 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151378+0000", 2024-09-15T03:46:46.780 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7880000000000003e-06 2024-09-15T03:46:46.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.780 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.780 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:46.780 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151438+0000", 2024-09-15T03:46:46.780 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9679999999999998e-05 2024-09-15T03:46:46.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.780 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.780 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:46.780 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151446+0000", 2024-09-15T03:46:46.780 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.909e-06 2024-09-15T03:46:46.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.780 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.781 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:46.781 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151481+0000", 2024-09-15T03:46:46.781 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5005000000000001e-05 2024-09-15T03:46:46.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.781 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.781 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:46.781 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151492+0000", 2024-09-15T03:46:46.781 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1133e-05 2024-09-15T03:46:46.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.781 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.781 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:46.781 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158306+0000", 2024-09-15T03:46:46.781 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0068140459999999998 2024-09-15T03:46:46.782 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.782 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:46.782 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.782 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.782 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.782 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es3 38/28 ECSubWrite(tid=45, reqid=client.4521.0:227, at_version=38'24, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:46:46.782 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.155560+0000", 2024-09-15T03:46:46.782 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.538518069, 2024-09-15T03:46:46.782 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0057773140000000004, 2024-09-15T03:46:46.782 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:46.782 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:46.782 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:46.782 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.783 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:46.783 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155560+0000", 2024-09-15T03:46:46.783 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.783 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.783 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.783 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:46.783 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155560+0000", 2024-09-15T03:46:46.783 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.783 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.783 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.783 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:46.783 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155561+0000", 2024-09-15T03:46:46.783 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3300000000000001e-07 2024-09-15T03:46:46.783 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.784 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.784 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:46.784 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156072+0000", 2024-09-15T03:46:46.784 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00051137999999999997 2024-09-15T03:46:46.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.784 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.784 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:46.784 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156082+0000", 2024-09-15T03:46:46.784 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6229999999999992e-06 2024-09-15T03:46:46.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.784 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.784 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:46.784 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156087+0000", 2024-09-15T03:46:46.785 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8599999999999998e-06 2024-09-15T03:46:46.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.785 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.785 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.785 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156133+0000", 2024-09-15T03:46:46.785 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5836e-05 2024-09-15T03:46:46.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.785 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.785 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:46.785 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156143+0000", 2024-09-15T03:46:46.785 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8689999999999999e-06 2024-09-15T03:46:46.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.785 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.786 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:46.786 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156392+0000", 2024-09-15T03:46:46.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000248526 2024-09-15T03:46:46.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.788 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.788 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:46.788 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156400+0000", 2024-09-15T03:46:46.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8189999999999992e-06 2024-09-15T03:46:46.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.788 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.788 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:46.788 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157090+0000", 2024-09-15T03:46:46.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00069055600000000005 2024-09-15T03:46:46.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.789 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.789 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:46.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157109+0000", 2024-09-15T03:46:46.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8490000000000001e-05 2024-09-15T03:46:46.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.789 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.789 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:46.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.161338+0000", 2024-09-15T03:46:46.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042291220000000001 2024-09-15T03:46:46.789 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.789 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:46.790 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.790 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.790 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.790 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:231 3.fs0 3:f40d4749:::smithi00132172-35:head [copy-from ver 6] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:46:46.790 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.155669+0000", 2024-09-15T03:46:46.790 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.5384098389999998, 2024-09-15T03:46:46.790 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010370569, 2024-09-15T03:46:46.790 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:46.790 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:46.790 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:46.790 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:46.790 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:46.790 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 231 2024-09-15T03:46:46.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.791 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:46.791 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.791 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:46.791 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155669+0000", 2024-09-15T03:46:46.791 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.791 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.791 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:46.791 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155669+0000", 2024-09-15T03:46:46.791 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.791 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.792 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:46.792 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155671+0000", 2024-09-15T03:46:46.792 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0829999999999998e-06 2024-09-15T03:46:46.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.792 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.792 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:46.792 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155676+0000", 2024-09-15T03:46:46.792 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6210000000000001e-06 2024-09-15T03:46:46.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.792 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.792 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:46.792 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155690+0000", 2024-09-15T03:46:46.792 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4166e-05 2024-09-15T03:46:46.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.793 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:46.793 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155698+0000", 2024-09-15T03:46:46.793 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5560000000000002e-06 2024-09-15T03:46:46.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.793 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.793 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155748+0000", 2024-09-15T03:46:46.793 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9851000000000001e-05 2024-09-15T03:46:46.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.793 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:46.793 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155872+0000", 2024-09-15T03:46:46.794 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000124091 2024-09-15T03:46:46.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.794 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.794 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:46.794 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163470+0000", 2024-09-15T03:46:46.794 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075976730000000001 2024-09-15T03:46:46.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.794 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.794 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:46.794 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163801+0000", 2024-09-15T03:46:46.794 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033157999999999999 2024-09-15T03:46:46.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.794 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.795 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:46.795 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163814+0000", 2024-09-15T03:46:46.795 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2659e-05 2024-09-15T03:46:46.795 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.795 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.795 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:46.795 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163935+0000", 2024-09-15T03:46:46.795 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000121155 2024-09-15T03:46:46.795 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.795 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.795 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:46.796 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.164131+0000", 2024-09-15T03:46:46.796 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019600499999999999 2024-09-15T03:46:46.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.796 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.796 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:46.796 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.165998+0000", 2024-09-15T03:46:46.796 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001866882 2024-09-15T03:46:46.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.796 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.796 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:46.796 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.166039+0000", 2024-09-15T03:46:46.796 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1247e-05 2024-09-15T03:46:46.796 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.797 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:46.797 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.797 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.797 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:232 3.fs0 3:f40d4749:::smithi00132172-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:46:46.797 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:46:46.797 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.5383725500000001, 2024-09-15T03:46:46.797 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084205730000000006, 2024-09-15T03:46:46.797 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:46.797 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:46:46.797 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:46.797 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:46.797 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:46.798 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 232 2024-09-15T03:46:46.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.798 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:46.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.798 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:46.798 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:46:46.798 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.798 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:46.798 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:46:46.798 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:46.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:46:46.799 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7399999999999999e-07 2024-09-15T03:46:46.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.799 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:46.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155708+0000", 2024-09-15T03:46:46.799 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6500000000000001e-06 2024-09-15T03:46:46.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.799 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:46.800 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155713+0000", 2024-09-15T03:46:46.800 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2519999999999999e-06 2024-09-15T03:46:46.800 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.800 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.800 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:46.800 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155716+0000", 2024-09-15T03:46:46.800 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4430000000000002e-06 2024-09-15T03:46:46.800 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.800 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.800 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.801 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156060+0000", 2024-09-15T03:46:46.801 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034440000000000002 2024-09-15T03:46:46.801 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.801 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.801 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2024-09-15T03:46:46.801 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156112+0000", 2024-09-15T03:46:46.801 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1928000000000002e-05 2024-09-15T03:46:46.801 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.801 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.801 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.801 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163929+0000", 2024-09-15T03:46:46.801 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0078168999999999999 2024-09-15T03:46:46.801 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.801 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.802 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:46.802 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.164027+0000", 2024-09-15T03:46:46.802 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8109999999999994e-05 2024-09-15T03:46:46.802 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.802 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.802 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:46.802 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.164126+0000", 2024-09-15T03:46:46.802 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9516000000000001e-05 2024-09-15T03:46:46.802 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.802 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:46.802 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.802 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.802 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.803 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es3 38/28 ECSubWrite(tid=46, reqid=client.4521.0:228, at_version=38'25, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:46:46.803 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.157625+0000", 2024-09-15T03:46:46.803 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.536452994, 2024-09-15T03:46:46.803 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0040588689999999997, 2024-09-15T03:46:46.803 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:46.803 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:46.803 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:46.803 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.803 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:46.803 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157625+0000", 2024-09-15T03:46:46.803 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.803 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.803 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.803 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:46.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157625+0000", 2024-09-15T03:46:46.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.804 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:46.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157626+0000", 2024-09-15T03:46:46.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1699999999999998e-07 2024-09-15T03:46:46.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.804 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:46.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157845+0000", 2024-09-15T03:46:46.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021945600000000001 2024-09-15T03:46:46.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.805 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:46.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157859+0000", 2024-09-15T03:46:46.805 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3638999999999999e-05 2024-09-15T03:46:46.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.805 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:46.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157865+0000", 2024-09-15T03:46:46.805 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2469999999999997e-06 2024-09-15T03:46:46.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.805 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.806 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157928+0000", 2024-09-15T03:46:46.806 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3040000000000006e-05 2024-09-15T03:46:46.806 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.806 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.806 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:46.806 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157937+0000", 2024-09-15T03:46:46.806 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6449999999999994e-06 2024-09-15T03:46:46.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.808 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:46.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158042+0000", 2024-09-15T03:46:46.809 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010501399999999999 2024-09-15T03:46:46.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:46.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158061+0000", 2024-09-15T03:46:46.809 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9349999999999999e-05 2024-09-15T03:46:46.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:46.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158343+0000", 2024-09-15T03:46:46.809 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00028139000000000001 2024-09-15T03:46:46.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.810 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.810 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:46.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158387+0000", 2024-09-15T03:46:46.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4601000000000002e-05 2024-09-15T03:46:46.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.810 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.810 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:46.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.161684+0000", 2024-09-15T03:46:46.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032969700000000002 2024-09-15T03:46:46.810 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.810 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:46.810 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.811 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:255 3.as0 3:5c0337d9:::smithi00132172-38:head [rollback 2] snapc 7=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:46:46.811 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.169130+0000", 2024-09-15T03:46:46.811 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.52494844799999996, 2024-09-15T03:46:46.811 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0050691950000000003, 2024-09-15T03:46:46.811 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:46.811 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:46.811 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:46.811 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:46.811 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:46.811 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 255 2024-09-15T03:46:46.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.812 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:46.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:46.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169130+0000", 2024-09-15T03:46:46.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:46.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169130+0000", 2024-09-15T03:46:46.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:46.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169131+0000", 2024-09-15T03:46:46.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9000000000000005e-07 2024-09-15T03:46:46.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:46.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169136+0000", 2024-09-15T03:46:46.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3859999999999998e-06 2024-09-15T03:46:46.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:46.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169153+0000", 2024-09-15T03:46:46.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6889999999999999e-05 2024-09-15T03:46:46.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.814 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:46.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169164+0000", 2024-09-15T03:46:46.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.025e-05 2024-09-15T03:46:46.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.814 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170098+0000", 2024-09-15T03:46:46.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00093477700000000003 2024-09-15T03:46:46.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.815 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.815 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:46.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170185+0000", 2024-09-15T03:46:46.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6489999999999994e-05 2024-09-15T03:46:46.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.815 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.815 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:46.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170689+0000", 2024-09-15T03:46:46.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00050395300000000002 2024-09-15T03:46:46.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.815 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.815 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:46.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170775+0000", 2024-09-15T03:46:46.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6281999999999998e-05 2024-09-15T03:46:46.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:46.816 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170818+0000", 2024-09-15T03:46:46.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2608000000000003e-05 2024-09-15T03:46:46.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:46.816 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171560+0000", 2024-09-15T03:46:46.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00074203500000000005 2024-09-15T03:46:46.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:46.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171628+0000", 2024-09-15T03:46:46.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8478000000000004e-05 2024-09-15T03:46:46.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:46.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173860+0000", 2024-09-15T03:46:46.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022317499999999998 2024-09-15T03:46:46.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:46.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174199+0000", 2024-09-15T03:46:46.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033930599999999999 2024-09-15T03:46:46.818 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.818 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:46.818 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.818 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:256 3.as0 3:5c0337d9:::smithi00132172-38:head [append 0~0] snapc 7=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:46:46.818 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.169218+0000", 2024-09-15T03:46:46.818 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.52486011399999999, 2024-09-15T03:46:46.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0051095940000000003, 2024-09-15T03:46:46.818 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:46.818 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:46.818 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:46.819 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:46.819 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:46.819 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 256 2024-09-15T03:46:46.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.819 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:46.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:46.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169218+0000", 2024-09-15T03:46:46.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:46.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169218+0000", 2024-09-15T03:46:46.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:46.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169220+0000", 2024-09-15T03:46:46.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2699999999999999e-06 2024-09-15T03:46:46.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:46.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169225+0000", 2024-09-15T03:46:46.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4199999999999998e-06 2024-09-15T03:46:46.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.821 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:46.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169247+0000", 2024-09-15T03:46:46.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1766000000000001e-05 2024-09-15T03:46:46.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.821 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:46.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169257+0000", 2024-09-15T03:46:46.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0243e-05 2024-09-15T03:46:46.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.821 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170842+0000", 2024-09-15T03:46:46.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001585003 2024-09-15T03:46:46.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:46.822 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170892+0000", 2024-09-15T03:46:46.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9496000000000002e-05 2024-09-15T03:46:46.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:46.822 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171194+0000", 2024-09-15T03:46:46.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030285099999999998 2024-09-15T03:46:46.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:46.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171283+0000", 2024-09-15T03:46:46.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8219000000000004e-05 2024-09-15T03:46:46.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:46.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171563+0000", 2024-09-15T03:46:46.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00028080100000000001 2024-09-15T03:46:46.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:46.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171595+0000", 2024-09-15T03:46:46.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1322999999999998e-05 2024-09-15T03:46:46.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:46.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171633+0000", 2024-09-15T03:46:46.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8293000000000001e-05 2024-09-15T03:46:46.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:46.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174005+0000", 2024-09-15T03:46:46.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023715300000000002 2024-09-15T03:46:46.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:46.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174328+0000", 2024-09-15T03:46:46.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000323379 2024-09-15T03:46:46.825 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.825 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:46.825 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.825 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 39/31 ECSubWrite(tid=43, reqid=client.4521.0:252, at_version=39'27, trim_to=0'0, roll_forward_to=39'25))", 2024-09-15T03:46:46.825 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:46:46.825 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.51573285700000004, 2024-09-15T03:46:46.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082900309999999998, 2024-09-15T03:46:46.826 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:46.826 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:46.826 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:46.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:46.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:46:46.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:46.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:46:46.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:46.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:46:46.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6199999999999998e-07 2024-09-15T03:46:46.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:46.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178642+0000", 2024-09-15T03:46:46.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029559399999999999 2024-09-15T03:46:46.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:46.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178653+0000", 2024-09-15T03:46:46.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0865e-05 2024-09-15T03:46:46.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:46.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178658+0000", 2024-09-15T03:46:46.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3720000000000004e-06 2024-09-15T03:46:46.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178695+0000", 2024-09-15T03:46:46.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6708999999999997e-05 2024-09-15T03:46:46.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.829 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:46.829 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178701+0000", 2024-09-15T03:46:46.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7989999999999996e-06 2024-09-15T03:46:46.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.829 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:46.829 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178926+0000", 2024-09-15T03:46:46.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022405299999999999 2024-09-15T03:46:46.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.829 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:46.830 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178935+0000", 2024-09-15T03:46:46.830 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9649999999999997e-06 2024-09-15T03:46:46.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.830 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.830 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:46.830 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179230+0000", 2024-09-15T03:46:46.830 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029551000000000002 2024-09-15T03:46:46.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.830 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.830 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:46.830 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179247+0000", 2024-09-15T03:46:46.830 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6685999999999999e-05 2024-09-15T03:46:46.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:46.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.186636+0000", 2024-09-15T03:46:46.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0073889159999999997 2024-09-15T03:46:46.831 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.831 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:46.831 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.831 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 39/31 ECSubWrite(tid=44, reqid=client.4521.0:252, at_version=0'0, trim_to=0'0, roll_forward_to=39'27))", 2024-09-15T03:46:46.831 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.180059+0000", 2024-09-15T03:46:46.831 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.51401959399999997, 2024-09-15T03:46:46.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00676445, 2024-09-15T03:46:46.832 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:46.832 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:46.832 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:46.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.832 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:46.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180059+0000", 2024-09-15T03:46:46.832 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.832 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:46.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180059+0000", 2024-09-15T03:46:46.832 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:46.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.833 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:46.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180059+0000", 2024-09-15T03:46:46.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:46:46.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.833 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:46.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180070+0000", 2024-09-15T03:46:46.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0407e-05 2024-09-15T03:46:46.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.833 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:46.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180083+0000", 2024-09-15T03:46:46.834 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3341e-05 2024-09-15T03:46:46.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.834 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.834 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:46.834 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180089+0000", 2024-09-15T03:46:46.834 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8000000000000004e-06 2024-09-15T03:46:46.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.834 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.834 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:46.834 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180126+0000", 2024-09-15T03:46:46.834 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7151999999999999e-05 2024-09-15T03:46:46.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.834 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.835 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:46.835 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180136+0000", 2024-09-15T03:46:46.835 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8349999999999999e-06 2024-09-15T03:46:46.835 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.835 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.835 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:46.835 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180200+0000", 2024-09-15T03:46:46.835 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3612e-05 2024-09-15T03:46:46.835 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.835 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.835 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:46.835 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180206+0000", 2024-09-15T03:46:46.835 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3749999999999999e-06 2024-09-15T03:46:46.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.836 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.836 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:46.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180235+0000", 2024-09-15T03:46:46.836 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9068999999999999e-05 2024-09-15T03:46:46.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.836 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.836 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:46.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180244+0000", 2024-09-15T03:46:46.836 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1030000000000008e-06 2024-09-15T03:46:46.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:46.836 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:46.836 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:46.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.186823+0000", 2024-09-15T03:46:46.837 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0065792130000000004 2024-09-15T03:46:46.837 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.837 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:46.837 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.837 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:46.837 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:46.837 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:46.838 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:46:46.839 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:46.838+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:46:46.939 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:46.938+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:46.943 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:46.944 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:46.952 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:46:47.040 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:47.039+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:47.094 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:47.094 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:47.102 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:46:47.141 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:47.140+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:47.145 INFO:tasks.rados.rados.0.smithi001.stdout:98: done (6 left) 2024-09-15T03:46:47.146 INFO:tasks.rados.rados.0.smithi001.stdout:99: delete oid 7 current snap is 6 2024-09-15T03:46:47.146 INFO:tasks.rados.rados.0.smithi001.stdout:97: finishing rollback tid 0 to smithi00132172-38 2024-09-15T03:46:47.146 INFO:tasks.rados.rados.0.smithi001.stdout:97: finishing rollback tid 1 to smithi00132172-38 2024-09-15T03:46:47.146 INFO:tasks.rados.rados.0.smithi001.stdout:97: finishing rollback tid 2 to smithi00132172-38 2024-09-15T03:46:47.146 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 38 v 20 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-15T03:46:47.146 INFO:tasks.rados.rados.0.smithi001.stdout:93: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-15T03:46:47.153 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T03:46:47.159 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 25 v 17 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-15T03:46:47.159 INFO:tasks.rados.rados.0.smithi001.stdout:95: expect (ObjNum 47 snap 0 seq_num 47) 2024-09-15T03:46:47.182 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:46:47.182 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: filestore_inject_stall = '' (not observed, change may require restart) osd_enable_op_tracker = '' 2024-09-15T03:46:47.188 INFO:tasks.rados.rados.0.smithi001.stdout:94: finishing copy_from to smithi00132172-40 2024-09-15T03:46:47.188 INFO:tasks.rados.rados.0.smithi001.stdout:94: got expected ENOENT (src dne) 2024-09-15T03:46:47.188 INFO:tasks.rados.rados.0.smithi001.stdout:94: finishing copy_from racing read to smithi00132172-40 2024-09-15T03:46:47.189 INFO:tasks.rados.rados.0.smithi001.stdout:96: finishing copy_from racing read to smithi00132172-29 2024-09-15T03:46:47.189 INFO:tasks.rados.rados.0.smithi001.stdout:96: finishing copy_from to smithi00132172-29 2024-09-15T03:46:47.189 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 29 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-15T03:46:47.189 INFO:tasks.rados.rados.0.smithi001.stdout:92: done (6 left) 2024-09-15T03:46:47.189 INFO:tasks.rados.rados.0.smithi001.stdout:93: done (5 left) 2024-09-15T03:46:47.189 INFO:tasks.rados.rados.0.smithi001.stdout:94: done (4 left) 2024-09-15T03:46:47.189 INFO:tasks.rados.rados.0.smithi001.stdout:95: done (3 left) 2024-09-15T03:46:47.189 INFO:tasks.rados.rados.0.smithi001.stdout:96: done (2 left) 2024-09-15T03:46:47.189 INFO:tasks.rados.rados.0.smithi001.stdout:97: done (1 left) 2024-09-15T03:46:47.189 INFO:tasks.rados.rados.0.smithi001.stdout:99: done (0 left) 2024-09-15T03:46:47.189 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 31 current snap is 6 2024-09-15T03:46:47.190 INFO:tasks.rados.rados.0.smithi001.stdout:100: seq_num 66 ranges {294912=737280,1032192=376832} 2024-09-15T03:46:47.196 INFO:tasks.rados.rados.0.smithi001.stdout:100: writing smithi00132172-31 from 294912 to 1032192 tid 1 2024-09-15T03:46:47.197 INFO:tasks.rados.rados.0.smithi001.stdout:100: writing smithi00132172-31 from 1032192 to 1409024 tid 2 2024-09-15T03:46:47.197 INFO:tasks.rados.rados.0.smithi001.stdout:101: read oid 4 snap -1 2024-09-15T03:46:47.198 INFO:tasks.rados.rados.0.smithi001.stdout:101: expect (ObjNum 64 snap 6 seq_num 64) 2024-09-15T03:46:47.198 INFO:tasks.rados.rados.0.smithi001.stdout:102: rollback oid 26 current snap is 6 2024-09-15T03:46:47.198 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 26 to 1 2024-09-15T03:46:47.198 INFO:tasks.rados.rados.0.smithi001.stdout:103: setattr oid 25 current snap is 6 2024-09-15T03:46:47.199 INFO:tasks.rados.rados.0.smithi001.stdout:104: snap_remove snap 4 2024-09-15T03:46:47.201 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:46:47.201 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:46:47.220 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:46:47.220 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:46:47.225 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:47.225 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:47.234 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:46:47.235 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:46:47.235 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:46:47.242 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:47.242+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:46:47.263 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:46:47.263 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:46:47.278 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:46:47.278 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:46:47.295 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:46:47.295 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:46:47.343 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:47.342+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:47.351 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:47.351 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:47.360 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:46:47.444 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:47.443+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:47.462 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:47.462 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:47.470 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:46:47.545 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:47.544+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:46:47.571 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:47.571 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:47.579 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:46:47.645 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:47.644+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:46:47.680 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:47.680 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:47.688 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:46:47.746 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:47.745+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:46:47.788 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:47.788 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:47.796 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:46:47.881 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:47.880+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:46:47.938 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:47.938 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:47.946 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:46:47.948 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:47.947+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:48.049 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:48.048+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:48.088 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:48.088 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:48.097 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:46:48.149 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:46:48.149+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:46:48.150 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-15T03:46:48.151 INFO:tasks.rados.rados.0.smithi001.stdout:104: done (4 left) 2024-09-15T03:46:48.151 INFO:tasks.rados.rados.0.smithi001.stdout:105: read oid 22 snap 0 2024-09-15T03:46:48.151 INFO:tasks.rados.rados.0.smithi001.stdout:105: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-15T03:46:48.151 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 40 current snap is 6 2024-09-15T03:46:48.151 INFO:tasks.rados.rados.0.smithi001.stdout:106: seq_num 67 ranges {671744=753664,1425408=376832} 2024-09-15T03:46:48.161 INFO:tasks.rados.rados.0.smithi001.stdout:106: writing smithi00132172-40 from 671744 to 1425408 tid 1 2024-09-15T03:46:48.164 INFO:tasks.rados.rados.0.smithi001.stdout:106: writing smithi00132172-40 from 1425408 to 1802240 tid 2 2024-09-15T03:46:48.164 INFO:tasks.rados.rados.0.smithi001.stdout:107: read oid 8 snap -1 2024-09-15T03:46:48.164 INFO:tasks.rados.rados.0.smithi001.stdout:107: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-15T03:46:48.164 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 13 current snap is 6 2024-09-15T03:46:48.164 INFO:tasks.rados.rados.0.smithi001.stdout:108: seq_num 68 ranges {0=770048,770048=376832} 2024-09-15T03:46:48.169 INFO:tasks.rados.rados.0.smithi001.stdout:108: writing smithi00132172-13 from 0 to 770048 tid 1 2024-09-15T03:46:48.172 INFO:tasks.rados.rados.0.smithi001.stdout:108: writing smithi00132172-13 from 770048 to 1146880 tid 2 2024-09-15T03:46:48.172 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 38 current snap is 6 2024-09-15T03:46:48.172 INFO:tasks.rados.rados.0.smithi001.stdout:109: seq_num 69 ranges {622592=786432,1409024=376832} 2024-09-15T03:46:48.181 INFO:tasks.rados.rados.0.smithi001.stdout:109: writing smithi00132172-38 from 622592 to 1409024 tid 1 2024-09-15T03:46:48.183 INFO:tasks.rados.rados.0.smithi001.stdout:109: writing smithi00132172-38 from 1409024 to 1785856 tid 2 2024-09-15T03:46:48.183 INFO:tasks.rados.rados.0.smithi001.stdout:110: read oid 21 snap 1 2024-09-15T03:46:48.183 INFO:tasks.rados.rados.0.smithi001.stdout:110: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-15T03:46:48.184 INFO:tasks.rados.rados.0.smithi001.stdout:111: read oid 36 snap -1 2024-09-15T03:46:48.184 INFO:tasks.rados.rados.0.smithi001.stdout:111: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-15T03:46:48.184 INFO:tasks.rados.rados.0.smithi001.stdout:112: snap_create 2024-09-15T03:46:48.184 INFO:tasks.rados.rados.0.smithi001.stdout:100: finishing write tid 1 to smithi00132172-31 2024-09-15T03:46:48.184 INFO:tasks.rados.rados.0.smithi001.stdout:102: finishing rollback tid 0 to smithi00132172-26 2024-09-15T03:46:48.184 INFO:tasks.rados.rados.0.smithi001.stdout:102: finishing rollback tid 1 to smithi00132172-26 2024-09-15T03:46:48.184 INFO:tasks.rados.rados.0.smithi001.stdout:100: finishing write tid 2 to smithi00132172-31 2024-09-15T03:46:48.184 INFO:tasks.rados.rados.0.smithi001.stdout:100: finishing write tid 3 to smithi00132172-31 2024-09-15T03:46:48.184 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 25 v 22 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-15T03:46:48.184 INFO:tasks.rados.rados.0.smithi001.stdout:102: finishing rollback tid 2 to smithi00132172-26 2024-09-15T03:46:48.184 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 26 v 8 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-15T03:46:48.184 INFO:tasks.rados.rados.0.smithi001.stdout:100: finishing write tid 4 to smithi00132172-31 2024-09-15T03:46:48.184 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 31 v 38 (ObjNum 66 snap 6 seq_num 66) dirty exists 2024-09-15T03:46:48.185 INFO:tasks.rados.rados.0.smithi001.stdout:100: left oid 31 (ObjNum 66 snap 6 seq_num 66) 2024-09-15T03:46:48.185 INFO:tasks.rados.rados.0.smithi001.stdout:101: expect (ObjNum 64 snap 6 seq_num 64) 2024-09-15T03:46:48.214 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:48.214 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:48.216 INFO:tasks.rados.rados.0.smithi001.stdout:105: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-15T03:46:48.223 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:46:48.228 INFO:tasks.rados.rados.0.smithi001.stdout:106: finishing write tid 1 to smithi00132172-40 2024-09-15T03:46:48.228 INFO:tasks.rados.rados.0.smithi001.stdout:107: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-15T03:46:48.229 INFO:tasks.rados.rados.0.smithi001.stdout:106: finishing write tid 2 to smithi00132172-40 2024-09-15T03:46:48.229 INFO:tasks.rados.rados.0.smithi001.stdout:106: finishing write tid 3 to smithi00132172-40 2024-09-15T03:46:48.229 INFO:tasks.rados.rados.0.smithi001.stdout:108: finishing write tid 1 to smithi00132172-13 2024-09-15T03:46:48.229 INFO:tasks.rados.rados.0.smithi001.stdout:106: finishing write tid 4 to smithi00132172-40 2024-09-15T03:46:48.229 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 40 v 39 (ObjNum 67 snap 6 seq_num 67) dirty exists 2024-09-15T03:46:48.229 INFO:tasks.rados.rados.0.smithi001.stdout:106: left oid 40 (ObjNum 67 snap 6 seq_num 67) 2024-09-15T03:46:48.229 INFO:tasks.rados.rados.0.smithi001.stdout:108: finishing write tid 2 to smithi00132172-13 2024-09-15T03:46:48.230 INFO:tasks.rados.rados.0.smithi001.stdout:108: finishing write tid 3 to smithi00132172-13 2024-09-15T03:46:48.230 INFO:tasks.rados.rados.0.smithi001.stdout:108: finishing write tid 4 to smithi00132172-13 2024-09-15T03:46:48.230 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 13 v 31 (ObjNum 68 snap 6 seq_num 68) dirty exists 2024-09-15T03:46:48.230 INFO:tasks.rados.rados.0.smithi001.stdout:108: left oid 13 (ObjNum 68 snap 6 seq_num 68) 2024-09-15T03:46:48.230 INFO:tasks.rados.rados.0.smithi001.stdout:111: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-15T03:46:48.245 INFO:tasks.rados.rados.0.smithi001.stdout:109: finishing write tid 1 to smithi00132172-38 2024-09-15T03:46:48.245 INFO:tasks.rados.rados.0.smithi001.stdout:110: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-15T03:46:48.249 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-15T03:46:48.249 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T03:46:48.249 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-15T03:46:48.250 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-15T03:46:48.251 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:48.250+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:48.254 INFO:tasks.rados.rados.0.smithi001.stdout:109: finishing write tid 2 to smithi00132172-38 2024-09-15T03:46:48.254 INFO:tasks.rados.rados.0.smithi001.stdout:109: finishing write tid 3 to smithi00132172-38 2024-09-15T03:46:48.254 INFO:tasks.rados.rados.0.smithi001.stdout:109: finishing write tid 4 to smithi00132172-38 2024-09-15T03:46:48.254 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 38 v 34 (ObjNum 69 snap 6 seq_num 69) dirty exists 2024-09-15T03:46:48.254 INFO:tasks.rados.rados.0.smithi001.stdout:109: left oid 38 (ObjNum 69 snap 6 seq_num 69) 2024-09-15T03:46:48.330 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:48.330 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:48.340 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:46:48.351 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:48.351+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:48.452 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:48.452+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:46:48.453 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:48.454 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:48.463 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:46:48.523 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:46:48.523 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:46:48.523 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 41, 2024-09-15T03:46:48.524 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "f27b540e-e5e3-4e95-afe8-c95ee66b0a62", 2024-09-15T03:46:48.524 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-09-15T03:46:09.843094+0000", 2024-09-15T03:46:48.524 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-09-15T03:46:48.144729+0000", 2024-09-15T03:46:48.524 INFO:teuthology.orchestra.run.smithi001.stdout: "last_up_change": "2024-09-15T03:46:15.871985+0000", 2024-09-15T03:46:48.524 INFO:teuthology.orchestra.run.smithi001.stdout: "last_in_change": "2024-09-15T03:46:38.089065+0000", 2024-09-15T03:46:48.524 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-15T03:46:48.525 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_num": 5806080, 2024-09-15T03:46:48.525 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_set": [ 2024-09-15T03:46:48.525 INFO:teuthology.orchestra.run.smithi001.stdout: "nodeep-scrub", 2024-09-15T03:46:48.525 INFO:teuthology.orchestra.run.smithi001.stdout: "noscrub", 2024-09-15T03:46:48.525 INFO:teuthology.orchestra.run.smithi001.stdout: "pglog_hardlimit", 2024-09-15T03:46:48.525 INFO:teuthology.orchestra.run.smithi001.stdout: "purged_snapdirs", 2024-09-15T03:46:48.527 INFO:teuthology.orchestra.run.smithi001.stdout: "recovery_deletes", 2024-09-15T03:46:48.527 INFO:teuthology.orchestra.run.smithi001.stdout: "sortbitwise" 2024-09-15T03:46:48.527 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:46:48.527 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_version": 5, 2024-09-15T03:46:48.527 INFO:teuthology.orchestra.run.smithi001.stdout: "full_ratio": 0.94999998807907104, 2024-09-15T03:46:48.527 INFO:teuthology.orchestra.run.smithi001.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-15T03:46:48.527 INFO:teuthology.orchestra.run.smithi001.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-15T03:46:48.527 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_snapshot": "", 2024-09-15T03:46:48.528 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_max": 3, 2024-09-15T03:46:48.528 INFO:teuthology.orchestra.run.smithi001.stdout: "max_osd": 12, 2024-09-15T03:46:48.528 INFO:teuthology.orchestra.run.smithi001.stdout: "require_min_compat_client": "luminous", 2024-09-15T03:46:48.528 INFO:teuthology.orchestra.run.smithi001.stdout: "min_compat_client": "jewel", 2024-09-15T03:46:48.528 INFO:teuthology.orchestra.run.smithi001.stdout: "require_osd_release": "quincy", 2024-09-15T03:46:48.528 INFO:teuthology.orchestra.run.smithi001.stdout: "pools": [ 2024-09-15T03:46:48.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.528 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 1, 2024-09-15T03:46:48.528 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": ".mgr", 2024-09-15T03:46:48.529 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-15T03:46:16.185530+0000", 2024-09-15T03:46:48.529 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-09-15T03:46:48.529 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool", 2024-09-15T03:46:48.529 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-15T03:46:48.529 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-15T03:46:48.529 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 1, 2024-09-15T03:46:48.529 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 1, 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 1, 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 1, 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 1, 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 1, 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-15T03:46:48.530 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "20", 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 0, 2024-09-15T03:46:48.531 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 0, 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T03:46:48.532 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T03:46:48.533 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: "options": { 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_max": 32, 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_min": 1 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": {} 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.534 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 2, 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": "rbd", 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-15T03:46:22.284406+0000", 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 8193, 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 1, 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-15T03:46:48.535 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 8, 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 8, 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 8, 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 8, 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 8, 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-15T03:46:48.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "24", 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 3, 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 24, 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-15T03:46:48.537 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-15T03:46:48.538 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-15T03:46:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-15T03:46:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-15T03:46:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-15T03:46:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-15T03:46:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-15T03:46:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-15T03:46:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-15T03:46:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-15T03:46:48.539 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: "options": {}, 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd": {} 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 3, 2024-09-15T03:46:48.540 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": "unique_pool_0", 2024-09-15T03:46:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-15T03:46:34.105566+0000", 2024-09-15T03:46:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 8193, 2024-09-15T03:46:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-15T03:46:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 3, 2024-09-15T03:46:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 8, 2024-09-15T03:46:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 5, 2024-09-15T03:46:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 1, 2024-09-15T03:46:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-15T03:46:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-15T03:46:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T03:46:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T03:46:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-15T03:46:48.541 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-15T03:46:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 16, 2024-09-15T03:46:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 16, 2024-09-15T03:46:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 16, 2024-09-15T03:46:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 16, 2024-09-15T03:46:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 16, 2024-09-15T03:46:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-15T03:46:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-15T03:46:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-15T03:46:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-15T03:46:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-15T03:46:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-15T03:46:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-15T03:46:48.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "41", 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 10, 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 41, 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-15T03:46:48.543 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-15T03:46:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-15T03:46:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-15T03:46:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-15T03:46:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-15T03:46:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-15T03:46:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T03:46:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T03:46:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T03:46:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-15T03:46:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-15T03:46:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "lrcprofile", 2024-09-15T03:46:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-15T03:46:48.544 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-15T03:46:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-15T03:46:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-15T03:46:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-15T03:46:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-15T03:46:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-15T03:46:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T03:46:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-15T03:46:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-15T03:46:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 16384, 2024-09-15T03:46:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-15T03:46:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-15T03:46:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: "options": {}, 2024-09-15T03:46:48.545 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-15T03:46:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: "rados": {} 2024-09-15T03:46:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:46:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-15T03:46:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 0, 2024-09-15T03:46:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "cd433a6a-c875-4e6e-b89c-938caecbae3a", 2024-09-15T03:46:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:46:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:46:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:46:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:46:48.546 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:46:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:46:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:46:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 0, 2024-09-15T03:46:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:46:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:46:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:46:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6800", 2024-09-15T03:46:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:46:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.547 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.548 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6801", 2024-09-15T03:46:48.548 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:46:48.548 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.548 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.548 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:46:48.548 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.548 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.548 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6802", 2024-09-15T03:46:48.548 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:46:48.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.549 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.549 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6803", 2024-09-15T03:46:48.549 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:46:48.549 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.549 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.549 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:46:48.549 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.549 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.549 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6806", 2024-09-15T03:46:48.549 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:46:48.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.550 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.550 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6807", 2024-09-15T03:46:48.550 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:46:48.550 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.550 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.550 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:46:48.550 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.550 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.550 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6804", 2024-09-15T03:46:48.550 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:46:48.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.551 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.551 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6805", 2024-09-15T03:46:48.551 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:46:48.551 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.551 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.551 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6801/28908", 2024-09-15T03:46:48.551 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6803/28908", 2024-09-15T03:46:48.551 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6807/28908", 2024-09-15T03:46:48.551 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6805/28908", 2024-09-15T03:46:48.551 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:46:48.551 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:46:48.551 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:46:48.552 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.552 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 1, 2024-09-15T03:46:48.552 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "789930f9-42bf-476b-9f09-a21bffe33a79", 2024-09-15T03:46:48.552 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:46:48.552 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:46:48.552 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:46:48.552 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:46:48.552 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:46:48.552 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:46:48.552 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:46:48.552 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 31, 2024-09-15T03:46:48.553 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:46:48.553 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:46:48.553 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:46:48.553 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.553 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.553 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6808", 2024-09-15T03:46:48.553 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:46:48.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.553 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.553 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6809", 2024-09-15T03:46:48.553 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:46:48.553 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.554 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.554 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:46:48.554 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.554 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.554 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6810", 2024-09-15T03:46:48.554 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:46:48.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.554 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.554 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6811", 2024-09-15T03:46:48.554 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:46:48.554 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.555 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.555 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:46:48.555 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.555 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.555 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6814", 2024-09-15T03:46:48.555 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:46:48.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.555 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.555 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6815", 2024-09-15T03:46:48.555 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:46:48.556 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.556 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.556 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:46:48.556 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.556 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.556 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6812", 2024-09-15T03:46:48.556 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:46:48.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.556 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.556 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6813", 2024-09-15T03:46:48.556 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:46:48.557 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.557 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.557 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6809/28913", 2024-09-15T03:46:48.557 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6811/28913", 2024-09-15T03:46:48.557 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6815/28913", 2024-09-15T03:46:48.557 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6813/28913", 2024-09-15T03:46:48.557 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:46:48.557 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:46:48.557 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:46:48.557 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.557 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 2, 2024-09-15T03:46:48.558 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "9a83fec5-0d63-461c-b516-289443b3882b", 2024-09-15T03:46:48.558 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:46:48.558 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:46:48.558 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:46:48.558 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:46:48.558 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:46:48.558 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:46:48.558 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:46:48.558 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 28, 2024-09-15T03:46:48.558 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:46:48.558 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:46:48.558 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:46:48.558 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.559 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.559 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6821", 2024-09-15T03:46:48.559 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:46:48.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.559 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.559 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6823", 2024-09-15T03:46:48.559 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:46:48.559 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.559 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.559 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:46:48.559 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.560 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.560 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6825", 2024-09-15T03:46:48.560 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:46:48.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.560 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.560 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6827", 2024-09-15T03:46:48.560 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:46:48.560 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.560 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.560 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:46:48.561 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.561 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.561 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6830", 2024-09-15T03:46:48.561 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:46:48.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.561 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.561 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6831", 2024-09-15T03:46:48.561 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:46:48.561 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.561 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.561 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:46:48.562 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.562 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.562 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6828", 2024-09-15T03:46:48.562 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:46:48.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.563 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.563 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6829", 2024-09-15T03:46:48.563 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:46:48.563 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.563 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.563 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6823/28911", 2024-09-15T03:46:48.563 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6827/28911", 2024-09-15T03:46:48.563 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6831/28911", 2024-09-15T03:46:48.564 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6829/28911", 2024-09-15T03:46:48.564 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:46:48.564 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:46:48.564 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:46:48.564 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.564 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 3, 2024-09-15T03:46:48.564 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "d944bf89-e183-496a-9ca3-7949741b4be4", 2024-09-15T03:46:48.564 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:46:48.564 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:46:48.564 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:46:48.564 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:46:48.565 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:46:48.565 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:46:48.565 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:46:48.565 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 31, 2024-09-15T03:46:48.565 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:46:48.565 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:46:48.565 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:46:48.565 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.565 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.565 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6816", 2024-09-15T03:46:48.565 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:46:48.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.566 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.566 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6817", 2024-09-15T03:46:48.566 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:46:48.566 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.566 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.566 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:46:48.566 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.567 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.567 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6818", 2024-09-15T03:46:48.567 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:46:48.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.567 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.567 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6819", 2024-09-15T03:46:48.567 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:46:48.567 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.567 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.567 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:46:48.567 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.568 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.568 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6824", 2024-09-15T03:46:48.568 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:46:48.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.568 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.568 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6826", 2024-09-15T03:46:48.568 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:46:48.568 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.568 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.568 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:46:48.568 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.569 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.569 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6820", 2024-09-15T03:46:48.569 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:46:48.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.569 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.569 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6822", 2024-09-15T03:46:48.569 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:46:48.569 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.569 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.569 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6817/28914", 2024-09-15T03:46:48.570 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6819/28914", 2024-09-15T03:46:48.570 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6826/28914", 2024-09-15T03:46:48.570 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6822/28914", 2024-09-15T03:46:48.570 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:46:48.570 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:46:48.570 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:46:48.570 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.570 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 4, 2024-09-15T03:46:48.570 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "15ec2dbd-d099-4aff-94b2-1bf9dbb00b56", 2024-09-15T03:46:48.570 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:46:48.570 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:46:48.571 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:46:48.571 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:46:48.571 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:46:48.571 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:46:48.571 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:46:48.571 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 31, 2024-09-15T03:46:48.571 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:46:48.571 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:46:48.571 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:46:48.571 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.571 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.571 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6803", 2024-09-15T03:46:48.572 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:46:48.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.572 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.572 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6805", 2024-09-15T03:46:48.572 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:46:48.572 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.572 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.572 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:46:48.572 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.572 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.573 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6807", 2024-09-15T03:46:48.573 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:46:48.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.573 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.573 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6809", 2024-09-15T03:46:48.573 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:46:48.573 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.573 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.573 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:46:48.573 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.574 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.574 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6814", 2024-09-15T03:46:48.574 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:46:48.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.574 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.574 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6815", 2024-09-15T03:46:48.574 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:46:48.574 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.574 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.574 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:46:48.574 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.575 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.575 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6811", 2024-09-15T03:46:48.575 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:46:48.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.575 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.575 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6813", 2024-09-15T03:46:48.575 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:46:48.575 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.575 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.575 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.67:6805/28468", 2024-09-15T03:46:48.575 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.67:6809/28468", 2024-09-15T03:46:48.575 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.67:6815/28468", 2024-09-15T03:46:48.576 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.67:6813/28468", 2024-09-15T03:46:48.576 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:46:48.576 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:46:48.576 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:46:48.576 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.576 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 5, 2024-09-15T03:46:48.576 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "1e45de07-746b-4612-9a52-d05509cae9a3", 2024-09-15T03:46:48.576 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:46:48.576 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:46:48.576 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:46:48.577 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:46:48.577 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:46:48.577 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:46:48.577 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:46:48.577 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 21, 2024-09-15T03:46:48.577 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:46:48.577 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:46:48.577 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:46:48.577 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.577 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.577 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6824", 2024-09-15T03:46:48.577 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:46:48.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.578 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.578 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6825", 2024-09-15T03:46:48.578 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:46:48.578 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.578 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.578 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:46:48.578 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.578 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.578 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6826", 2024-09-15T03:46:48.578 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:46:48.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.579 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.579 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6827", 2024-09-15T03:46:48.579 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:46:48.579 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.579 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.579 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:46:48.579 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.579 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.579 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6830", 2024-09-15T03:46:48.579 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:46:48.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.580 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.580 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6831", 2024-09-15T03:46:48.580 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:46:48.580 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.580 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.580 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:46:48.580 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.580 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.580 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6828", 2024-09-15T03:46:48.581 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:46:48.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.581 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.581 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6829", 2024-09-15T03:46:48.581 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:46:48.581 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.581 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.581 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.67:6825/28495", 2024-09-15T03:46:48.581 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.67:6827/28495", 2024-09-15T03:46:48.581 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.67:6831/28495", 2024-09-15T03:46:48.582 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.67:6829/28495", 2024-09-15T03:46:48.582 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:46:48.582 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:46:48.582 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:46:48.582 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.582 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 6, 2024-09-15T03:46:48.582 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "ccf785f8-751d-473e-b17c-3df9346be9f8", 2024-09-15T03:46:48.582 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:46:48.582 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:46:48.582 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:46:48.582 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:46:48.583 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:46:48.583 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:46:48.583 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:46:48.583 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 31, 2024-09-15T03:46:48.583 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:46:48.583 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:46:48.583 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:46:48.583 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.583 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.583 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6800", 2024-09-15T03:46:48.583 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:46:48.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.584 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.584 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6801", 2024-09-15T03:46:48.584 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:46:48.584 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.584 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.584 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:46:48.584 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.584 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.584 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6802", 2024-09-15T03:46:48.585 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:46:48.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.585 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.585 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6804", 2024-09-15T03:46:48.585 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:46:48.585 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.585 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.585 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:46:48.585 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.585 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.586 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6810", 2024-09-15T03:46:48.586 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:46:48.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.586 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.586 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6812", 2024-09-15T03:46:48.586 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:46:48.586 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.586 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.586 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:46:48.586 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.587 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.587 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6806", 2024-09-15T03:46:48.587 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:46:48.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.587 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.587 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6808", 2024-09-15T03:46:48.587 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:46:48.587 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.587 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.587 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.67:6801/28469", 2024-09-15T03:46:48.587 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.67:6804/28469", 2024-09-15T03:46:48.588 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.67:6812/28469", 2024-09-15T03:46:48.588 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.67:6808/28469", 2024-09-15T03:46:48.588 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:46:48.588 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:46:48.588 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:46:48.588 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.588 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 7, 2024-09-15T03:46:48.588 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "f9f59987-869d-42eb-9eb6-08c39e3afe6d", 2024-09-15T03:46:48.588 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:46:48.588 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:46:48.588 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:46:48.588 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:46:48.589 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:46:48.589 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:46:48.589 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:46:48.589 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 31, 2024-09-15T03:46:48.589 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:46:48.589 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:46:48.589 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:46:48.589 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.589 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.589 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6816", 2024-09-15T03:46:48.589 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:46:48.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.590 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.590 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6817", 2024-09-15T03:46:48.590 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:46:48.590 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.590 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.590 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:46:48.590 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.590 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.590 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6818", 2024-09-15T03:46:48.590 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:46:48.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.591 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.591 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6819", 2024-09-15T03:46:48.591 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:46:48.591 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.591 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.591 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:46:48.591 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.591 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.591 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6822", 2024-09-15T03:46:48.591 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:46:48.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.592 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.592 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6823", 2024-09-15T03:46:48.592 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:46:48.592 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.592 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.592 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:46:48.592 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.592 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.592 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6820", 2024-09-15T03:46:48.593 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:46:48.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.593 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.593 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6821", 2024-09-15T03:46:48.593 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:46:48.593 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.593 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.593 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.67:6817/28470", 2024-09-15T03:46:48.593 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.67:6819/28470", 2024-09-15T03:46:48.593 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.67:6823/28470", 2024-09-15T03:46:48.593 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.67:6821/28470", 2024-09-15T03:46:48.594 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:46:48.594 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:46:48.594 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:46:48.594 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.594 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 8, 2024-09-15T03:46:48.594 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "69ddfe52-11df-4608-9a10-e8907c4cd87f", 2024-09-15T03:46:48.594 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:46:48.594 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:46:48.594 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:46:48.594 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:46:48.594 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:46:48.595 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:46:48.595 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:46:48.595 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 31, 2024-09-15T03:46:48.595 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:46:48.595 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:46:48.595 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:46:48.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.595 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6808", 2024-09-15T03:46:48.595 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:46:48.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.595 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.596 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6809", 2024-09-15T03:46:48.596 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:46:48.596 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.596 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.596 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:46:48.596 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.596 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.596 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6810", 2024-09-15T03:46:48.596 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:46:48.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.597 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.597 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6811", 2024-09-15T03:46:48.597 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:46:48.597 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.597 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.597 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:46:48.597 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.597 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.597 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6814", 2024-09-15T03:46:48.597 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:46:48.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.598 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.598 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6815", 2024-09-15T03:46:48.598 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:46:48.598 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.598 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.598 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:46:48.598 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.598 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.598 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6812", 2024-09-15T03:46:48.598 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:46:48.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.599 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.599 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6813", 2024-09-15T03:46:48.599 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:46:48.599 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.599 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.599 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.190:6809/29472", 2024-09-15T03:46:48.599 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.190:6811/29472", 2024-09-15T03:46:48.599 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.190:6815/29472", 2024-09-15T03:46:48.599 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.190:6813/29472", 2024-09-15T03:46:48.599 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:46:48.599 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:46:48.600 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:46:48.600 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.600 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 9, 2024-09-15T03:46:48.600 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "f92e1609-2a1a-4814-9ac0-44a73373d229", 2024-09-15T03:46:48.600 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:46:48.600 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:46:48.600 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:46:48.600 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:46:48.600 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:46:48.600 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:46:48.600 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:46:48.600 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 28, 2024-09-15T03:46:48.601 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:46:48.601 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:46:48.601 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:46:48.601 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.601 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.601 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6800", 2024-09-15T03:46:48.601 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:46:48.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.601 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.601 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6801", 2024-09-15T03:46:48.601 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:46:48.602 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.602 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.602 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:46:48.602 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.602 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.602 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6802", 2024-09-15T03:46:48.602 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:46:48.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.602 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.602 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6803", 2024-09-15T03:46:48.602 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:46:48.603 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.603 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.603 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:46:48.603 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.603 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.603 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6806", 2024-09-15T03:46:48.603 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:46:48.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.603 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.603 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6807", 2024-09-15T03:46:48.604 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:46:48.604 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.604 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.604 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:46:48.604 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.604 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.604 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6804", 2024-09-15T03:46:48.604 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:46:48.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.604 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.605 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6805", 2024-09-15T03:46:48.605 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:46:48.605 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.605 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.605 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.190:6801/29473", 2024-09-15T03:46:48.605 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.190:6803/29473", 2024-09-15T03:46:48.605 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.190:6807/29473", 2024-09-15T03:46:48.605 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.190:6805/29473", 2024-09-15T03:46:48.605 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:46:48.605 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:46:48.605 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:46:48.606 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.606 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 10, 2024-09-15T03:46:48.606 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "5f2e8ca3-8c65-4595-8e82-27dc3768ab99", 2024-09-15T03:46:48.606 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:46:48.606 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 0, 2024-09-15T03:46:48.606 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-09-15T03:46:48.606 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:46:48.606 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:46:48.606 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:46:48.606 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:46:48.606 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 28, 2024-09-15T03:46:48.606 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:46:48.607 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:46:48.607 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:46:48.607 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.607 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.607 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6816", 2024-09-15T03:46:48.607 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:46:48.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.607 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.607 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6817", 2024-09-15T03:46:48.607 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:46:48.607 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.608 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.608 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:46:48.608 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.608 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.608 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6818", 2024-09-15T03:46:48.608 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:46:48.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.608 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.608 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6819", 2024-09-15T03:46:48.609 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:46:48.609 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.609 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.609 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:46:48.609 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.609 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.609 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6822", 2024-09-15T03:46:48.609 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:46:48.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.610 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.610 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6823", 2024-09-15T03:46:48.610 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:46:48.610 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.610 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.610 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:46:48.610 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.610 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.610 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6820", 2024-09-15T03:46:48.610 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:46:48.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.611 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.611 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6821", 2024-09-15T03:46:48.611 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:46:48.611 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.611 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.611 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.190:6817/29474", 2024-09-15T03:46:48.611 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.190:6819/29474", 2024-09-15T03:46:48.611 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.190:6823/29474", 2024-09-15T03:46:48.611 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.190:6821/29474", 2024-09-15T03:46:48.611 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:46:48.611 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:46:48.611 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:46:48.612 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.612 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 11, 2024-09-15T03:46:48.612 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "cedf7df4-f875-42f5-a8cf-13cb82f05cc2", 2024-09-15T03:46:48.612 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:46:48.612 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:46:48.612 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:46:48.612 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:46:48.612 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:46:48.612 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:46:48.612 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:46:48.612 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 31, 2024-09-15T03:46:48.613 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:46:48.613 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:46:48.613 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:46:48.613 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.613 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.613 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6824", 2024-09-15T03:46:48.613 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:46:48.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.613 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.613 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6825", 2024-09-15T03:46:48.613 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:46:48.614 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.614 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.614 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:46:48.614 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.614 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.614 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6826", 2024-09-15T03:46:48.614 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:46:48.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.614 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.614 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6827", 2024-09-15T03:46:48.615 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:46:48.615 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.615 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.615 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:46:48.615 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.615 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.615 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6830", 2024-09-15T03:46:48.615 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:46:48.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.615 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.616 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6831", 2024-09-15T03:46:48.616 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:46:48.616 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.616 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.616 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:46:48.616 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:46:48.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.616 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:46:48.616 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6828", 2024-09-15T03:46:48.616 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:46:48.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.617 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:46:48.617 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6829", 2024-09-15T03:46:48.617 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:46:48.617 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.617 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.617 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.190:6825/29475", 2024-09-15T03:46:48.617 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.190:6827/29475", 2024-09-15T03:46:48.617 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.190:6831/29475", 2024-09-15T03:46:48.617 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.190:6829/29475", 2024-09-15T03:46:48.617 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:46:48.617 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:46:48.618 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:46:48.618 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.618 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.618 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:46:48.618 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_xinfo": [ 2024-09-15T03:46:48.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.618 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 0, 2024-09-15T03:46:48.618 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:46:48.618 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:46:48.618 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:46:48.618 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:46:48.618 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:46:48.618 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.943165+0000", 2024-09-15T03:46:48.619 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:46:48.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.619 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 1, 2024-09-15T03:46:48.619 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:46:48.619 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:46:48.619 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:46:48.619 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:46:48.619 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:46:48.619 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.946049+0000", 2024-09-15T03:46:48.619 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:46:48.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.619 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 2, 2024-09-15T03:46:48.620 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:46:48.620 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:46:48.620 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:46:48.620 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:46:48.620 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:46:48.620 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.981045+0000", 2024-09-15T03:46:48.620 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:46:48.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.620 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 3, 2024-09-15T03:46:48.620 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:46:48.620 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:46:48.620 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:46:48.621 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:46:48.621 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:46:48.621 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:14.019932+0000", 2024-09-15T03:46:48.621 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:46:48.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.621 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 4, 2024-09-15T03:46:48.621 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:46:48.621 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:46:48.621 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:46:48.621 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:46:48.621 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:46:48.621 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.936197+0000", 2024-09-15T03:46:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:46:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 5, 2024-09-15T03:46:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:46:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:46:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:46:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:46:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:46:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.982429+0000", 2024-09-15T03:46:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:46:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 6, 2024-09-15T03:46:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:46:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:46:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:46:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:46:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:46:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.953833+0000", 2024-09-15T03:46:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:46:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 7, 2024-09-15T03:46:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:46:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:46:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:46:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:46:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:46:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.967611+0000", 2024-09-15T03:46:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:46:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 8, 2024-09-15T03:46:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:46:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:46:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:46:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:46:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:46:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.996438+0000", 2024-09-15T03:46:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:46:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 9, 2024-09-15T03:46:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:46:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:46:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:46:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:46:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:46:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.926618+0000", 2024-09-15T03:46:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:46:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 10, 2024-09-15T03:46:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:46:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:46:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:46:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:46:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 65536, 2024-09-15T03:46:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.993934+0000", 2024-09-15T03:46:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:46:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 11, 2024-09-15T03:46:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:46:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:46:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:46:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:46:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:46:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:14.022484+0000", 2024-09-15T03:46:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:46:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:46:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap": [], 2024-09-15T03:46:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap_items": [], 2024-09-15T03:46:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_temp": [], 2024-09-15T03:46:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_temp": [], 2024-09-15T03:46:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklist": {}, 2024-09-15T03:46:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "range_blocklist": {}, 2024-09-15T03:46:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profiles": { 2024-09-15T03:46:48.628 INFO:teuthology.orchestra.run.smithi001.stdout: "default": { 2024-09-15T03:46:48.628 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-failure-domain": "osd", 2024-09-15T03:46:48.628 INFO:teuthology.orchestra.run.smithi001.stdout: "k": "2", 2024-09-15T03:46:48.628 INFO:teuthology.orchestra.run.smithi001.stdout: "m": "1", 2024-09-15T03:46:48.628 INFO:teuthology.orchestra.run.smithi001.stdout: "plugin": "jerasure", 2024-09-15T03:46:48.628 INFO:teuthology.orchestra.run.smithi001.stdout: "technique": "reed_sol_van" 2024-09-15T03:46:48.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.628 INFO:teuthology.orchestra.run.smithi001.stdout: "lrcprofile": { 2024-09-15T03:46:48.628 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-device-class": "", 2024-09-15T03:46:48.628 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-failure-domain": "osd", 2024-09-15T03:46:48.628 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-root": "default", 2024-09-15T03:46:48.628 INFO:teuthology.orchestra.run.smithi001.stdout: "k": "4", 2024-09-15T03:46:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: "l": "3", 2024-09-15T03:46:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: "m": "2", 2024-09-15T03:46:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "lrcprofile", 2024-09-15T03:46:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: "plugin": "lrc" 2024-09-15T03:46:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps_queue": [ 2024-09-15T03:46:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 3, 2024-09-15T03:46:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: "snaps": [ 2024-09-15T03:46:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: "begin": 4, 2024-09-15T03:46:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: "length": 1 2024-09-15T03:46:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: "begin": 6, 2024-09-15T03:46:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: "length": 2 2024-09-15T03:46:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:46:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: "new_removed_snaps": [ 2024-09-15T03:46:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 3, 2024-09-15T03:46:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: "snaps": [ 2024-09-15T03:46:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: "begin": 6, 2024-09-15T03:46:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: "length": 1 2024-09-15T03:46:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:46:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: "new_purged_snaps": [], 2024-09-15T03:46:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_node_flags": {}, 2024-09-15T03:46:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: "device_class_flags": {}, 2024-09-15T03:46:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": { 2024-09-15T03:46:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode_enabled": false, 2024-09-15T03:46:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_bucket_count": 0, 2024-09-15T03:46:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: "degraded_stretch_mode": 0, 2024-09-15T03:46:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: "recovering_stretch_mode": 0, 2024-09-15T03:46:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode_bucket": 0 2024-09-15T03:46:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:48.632 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:46:48.633 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:48.634 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:48.634 INFO:tasks.thrashosds.thrasher:j is {'epoch': 41, 'fsid': 'f27b540e-e5e3-4e95-afe8-c95ee66b0a62', 'created': '2024-09-15T03:46:09.843094+0000', 'modified': '2024-09-15T03:46:48.144729+0000', 'last_up_change': '2024-09-15T03:46:15.871985+0000', 'last_in_change': '2024-09-15T03:46:38.089065+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-15T03:46:16.185530+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-15T03:46:22.284406+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 24, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-15T03:46:34.105566+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 8, 'min_size': 5, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 10, 'snap_epoch': 41, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'lrcprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 16384, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'cd433a6a-c875-4e6e-b89c-938caecbae3a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6800', 'nonce': 28908}, {'type': 'v1', 'addr': '172.21.15.1:6801', 'nonce': 28908}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6802', 'nonce': 28908}, {'type': 'v1', 'addr': '172.21.15.1:6803', 'nonce': 28908}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6806', 'nonce': 28908}, {'type': 'v1', 'addr': '172.21.15.1:6807', 'nonce': 28908}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6804', 'nonce': 28908}, {'type': 'v1', 'addr': '172.21.15.1:6805', 'nonce': 28908}]}, 'public_addr': '172.21.15.1:6801/28908', 'cluster_addr': '172.21.15.1:6803/28908', 'heartbeat_back_addr': '172.21.15.1:6807/28908', 'heartbeat_front_addr': '172.21.15.1:6805/28908', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '789930f9-42bf-476b-9f09-a21bffe33a79', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6808', 'nonce': 28913}, {'type': 'v1', 'addr': '172.21.15.1:6809', 'nonce': 28913}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6810', 'nonce': 28913}, {'type': 'v1', 'addr': '172.21.15.1:6811', 'nonce': 28913}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6814', 'nonce': 28913}, {'type': 'v1', 'addr': '172.21.15.1:6815', 'nonce': 28913}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6812', 'nonce': 28913}, {'type': 'v1', 'addr': '172.21.15.1:6813', 'nonce': 28913}]}, 'public_addr': '172.21.15.1:6809/28913', 'cluster_addr': '172.21.15.1:6811/28913', 'heartbeat_back_addr': '172.21.15.1:6815/28913', 'heartbeat_front_addr': '172.21.15.1:6813/28913', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '9a83fec5-0d63-461c-b516-289443b3882b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 28, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6821', 'nonce': 28911}, {'type': 'v1', 'addr': '172.21.15.1:6823', 'nonce': 28911}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6825', 'nonce': 28911}, {'type': 'v1', 'addr': '172.21.15.1:6827', 'nonce': 28911}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6830', 'nonce': 28911}, {'type': 'v1', 'addr': '172.21.15.1:6831', 'nonce': 28911}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6828', 'nonce': 28911}, {'type': 'v1', 'addr': '172.21.15.1:6829', 'nonce': 28911}]}, 'public_addr': '172.21.15.1:6823/28911', 'cluster_addr': '172.21.15.1:6827/28911', 'heartbeat_back_addr': '172.21.15.1:6831/28911', 'heartbeat_front_addr': '172.21.15.1:6829/28911', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd944bf89-e183-496a-9ca3-7949741b4be4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6816', 'nonce': 28914}, {'type': 'v1', 'addr': '172.21.15.1:6817', 'nonce': 28914}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6818', 'nonce': 28914}, {'type': 'v1', 'addr': '172.21.15.1:6819', 'nonce': 28914}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6824', 'nonce': 28914}, {'type': 'v1', 'addr': '172.21.15.1:6826', 'nonce': 28914}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6820', 'nonce': 28914}, {'type': 'v1', 'addr': '172.21.15.1:6822', 'nonce': 28914}]}, 'public_addr': '172.21.15.1:6817/28914', 'cluster_addr': '172.21.15.1:6819/28914', 'heartbeat_back_addr': '172.21.15.1:6826/28914', 'heartbeat_front_addr': '172.21.15.1:6822/28914', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '15ec2dbd-d099-4aff-94b2-1bf9dbb00b56', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6803', 'nonce': 28468}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 28468}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6807', 'nonce': 28468}, {'type': 'v1', 'addr': '172.21.15.67:6809', 'nonce': 28468}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 28468}, {'type': 'v1', 'addr': '172.21.15.67:6815', 'nonce': 28468}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6811', 'nonce': 28468}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 28468}]}, 'public_addr': '172.21.15.67:6805/28468', 'cluster_addr': '172.21.15.67:6809/28468', 'heartbeat_back_addr': '172.21.15.67:6815/28468', 'heartbeat_front_addr': '172.21.15.67:6813/28468', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1e45de07-746b-4612-9a52-d05509cae9a3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6824', 'nonce': 28495}, {'type': 'v1', 'addr': '172.21.15.67:6825', 'nonce': 28495}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6826', 'nonce': 28495}, {'type': 'v1', 'addr': '172.21.15.67:6827', 'nonce': 28495}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6830', 'nonce': 28495}, {'type': 'v1', 'addr': '172.21.15.67:6831', 'nonce': 28495}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6828', 'nonce': 28495}, {'type': 'v1', 'addr': '172.21.15.67:6829', 'nonce': 28495}]}, 'public_addr': '172.21.15.67:6825/28495', 'cluster_addr': '172.21.15.67:6827/28495', 'heartbeat_back_addr': '172.21.15.67:6831/28495', 'heartbeat_front_addr': '172.21.15.67:6829/28495', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ccf785f8-751d-473e-b17c-3df9346be9f8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 28469}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 28469}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 28469}, {'type': 'v1', 'addr': '172.21.15.67:6804', 'nonce': 28469}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 28469}, {'type': 'v1', 'addr': '172.21.15.67:6812', 'nonce': 28469}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 28469}, {'type': 'v1', 'addr': '172.21.15.67:6808', 'nonce': 28469}]}, 'public_addr': '172.21.15.67:6801/28469', 'cluster_addr': '172.21.15.67:6804/28469', 'heartbeat_back_addr': '172.21.15.67:6812/28469', 'heartbeat_front_addr': '172.21.15.67:6808/28469', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f9f59987-869d-42eb-9eb6-08c39e3afe6d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 28470}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 28470}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6818', 'nonce': 28470}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 28470}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6822', 'nonce': 28470}, {'type': 'v1', 'addr': '172.21.15.67:6823', 'nonce': 28470}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 28470}, {'type': 'v1', 'addr': '172.21.15.67:6821', 'nonce': 28470}]}, 'public_addr': '172.21.15.67:6817/28470', 'cluster_addr': '172.21.15.67:6819/28470', 'heartbeat_back_addr': '172.21.15.67:6823/28470', 'heartbeat_front_addr': '172.21.15.67:6821/28470', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '69ddfe52-11df-4608-9a10-e8907c4cd87f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6808', 'nonce': 29472}, {'type': 'v1', 'addr': '172.21.15.190:6809', 'nonce': 29472}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6810', 'nonce': 29472}, {'type': 'v1', 'addr': '172.21.15.190:6811', 'nonce': 29472}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6814', 'nonce': 29472}, {'type': 'v1', 'addr': '172.21.15.190:6815', 'nonce': 29472}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6812', 'nonce': 29472}, {'type': 'v1', 'addr': '172.21.15.190:6813', 'nonce': 29472}]}, 'public_addr': '172.21.15.190:6809/29472', 'cluster_addr': '172.21.15.190:6811/29472', 'heartbeat_back_addr': '172.21.15.190:6815/29472', 'heartbeat_front_addr': '172.21.15.190:6813/29472', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'f92e1609-2a1a-4814-9ac0-44a73373d229', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 28, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6800', 'nonce': 29473}, {'type': 'v1', 'addr': '172.21.15.190:6801', 'nonce': 29473}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6802', 'nonce': 29473}, {'type': 'v1', 'addr': '172.21.15.190:6803', 'nonce': 29473}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6806', 'nonce': 29473}, {'type': 'v1', 'addr': '172.21.15.190:6807', 'nonce': 29473}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6804', 'nonce': 29473}, {'type': 'v1', 'addr': '172.21.15.190:6805', 'nonce': 29473}]}, 'public_addr': '172.21.15.190:6801/29473', 'cluster_addr': '172.21.15.190:6803/29473', 'heartbeat_back_addr': '172.21.15.190:6807/29473', 'heartbeat_front_addr': '172.21.15.190:6805/29473', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '5f2e8ca3-8c65-4595-8e82-27dc3768ab99', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 28, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6816', 'nonce': 29474}, {'type': 'v1', 'addr': '172.21.15.190:6817', 'nonce': 29474}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6818', 'nonce': 29474}, {'type': 'v1', 'addr': '172.21.15.190:6819', 'nonce': 29474}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6822', 'nonce': 29474}, {'type': 'v1', 'addr': '172.21.15.190:6823', 'nonce': 29474}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6820', 'nonce': 29474}, {'type': 'v1', 'addr': '172.21.15.190:6821', 'nonce': 29474}]}, 'public_addr': '172.21.15.190:6817/29474', 'cluster_addr': '172.21.15.190:6819/29474', 'heartbeat_back_addr': '172.21.15.190:6823/29474', 'heartbeat_front_addr': '172.21.15.190:6821/29474', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'cedf7df4-f875-42f5-a8cf-13cb82f05cc2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6824', 'nonce': 29475}, {'type': 'v1', 'addr': '172.21.15.190:6825', 'nonce': 29475}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6826', 'nonce': 29475}, {'type': 'v1', 'addr': '172.21.15.190:6827', 'nonce': 29475}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6830', 'nonce': 29475}, {'type': 'v1', 'addr': '172.21.15.190:6831', 'nonce': 29475}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6828', 'nonce': 29475}, {'type': 'v1', 'addr': '172.21.15.190:6829', 'nonce': 29475}]}, 'public_addr': '172.21.15.190:6825/29475', 'cluster_addr': '172.21.15.190:6827/29475', 'heartbeat_back_addr': '172.21.15.190:6831/29475', 'heartbeat_front_addr': '172.21.15.190:6829/29475', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.943165+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.946049+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.981045+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:14.019932+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.936197+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.982429+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.953833+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.967611+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.996438+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.926618+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.993934+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:14.022484+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'lrcprofile': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'k': '4', 'l': '3', 'm': '2', 'name': 'lrcprofile', 'plugin': 'lrc'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 4, 'length': 1}, {'begin': 6, 'length': 2}]}], 'new_removed_snaps': [{'pool': 3, 'snaps': [{'begin': 6, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-15T03:46:48.635 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:46:48.635 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:46:48.636 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:48.632+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:46:48.733 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:48.732+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:48.735 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:48.736 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:48.744 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:46:48.834 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:48.833+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:48.845 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:48.845 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:48.854 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:46:48.909 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:46:48.909 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-15T03:46:48.921 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-09-15T03:46:48.149466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29671987,"num_objects":71,"num_object_clones":19,"num_object_copies":544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":6,"num_read":212,"num_read_kb":8143,"num_write":294,"num_write_kb":32584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275,"ondisk_log_size":275,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":11,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":1030475776,"kb_used":8853820,"kb_used_data":1021631364,"kb_used_omap":4518,"kb_used_meta":0,"kb_avail":1021621956,"statfs":{"total":1055207194624,"available":1046140882944,"internally_reserved":9633792,"allocated":1046150516736,"data_stored":1046150516736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4626523,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,10],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":16,"commit_latency_ns":0,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6094848,"num_objects":13,"num_object_clones":13,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":4,"num_read":16,"num_read_kb":4134,"num_write":24,"num_write_kb":5952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001567"},"pg_stats":[{"pgid":"3.f","version":"39'14","reported_seq":59,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:46:46.148209+0000","last_change":"2024-09-15T03:46:46.148838+0000","last_active":"2024-09-15T03:46:46.148209+0000","last_peered":"2024-09-15T03:46:46.148209+0000","last_clean":"2024-09-15T03:46:46.148209+0000","last_became_active":"2024-09-15T03:46:39.379596+0000","last_became_peered":"2024-09-15T03:46:39.379596+0000","last_unstale":"2024-09-15T03:46:46.148209+0000","last_undegraded":"2024-09-15T03:46:46.148209+0000","last_fullsized":"2024-09-15T03:46:46.148209+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:40.918504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065190919999999998,"stat_sum":{"num_bytes":2686976,"num_objects":5,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":9,"num_write_kb":2624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0,7,2,8,4,6],"acting":[11,1,0,7,2,8,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"3.e","version":"38'26","reported_seq":68,"reported_epoch":39,"state":"active+clean+snaptrim","last_fresh":"2024-09-15T03:46:46.142671+0000","last_change":"2024-09-15T03:46:46.142762+0000","last_active":"2024-09-15T03:46:46.142671+0000","last_peered":"2024-09-15T03:46:46.142671+0000","last_clean":"2024-09-15T03:46:46.142671+0000","last_became_active":"2024-09-15T03:46:36.170536+0000","last_became_peered":"2024-09-15T03:46:36.170536+0000","last_unstale":"2024-09-15T03:46:46.142671+0000","last_undegraded":"2024-09-15T03:46:46.142671+0000","last_fullsized":"2024-09-15T03:46:46.142671+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:40.141809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4997120,"num_objects":11,"num_object_clones":4,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":25,"num_read_kb":1835,"num_write":22,"num_write_kb":4880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,11,7,8,5,9],"acting":[2,0,1,11,7,8,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"32'5","reported_seq":50,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:46:46.142767+0000","last_change":"2024-09-15T03:46:46.142984+0000","last_active":"2024-09-15T03:46:46.142767+0000","last_peered":"2024-09-15T03:46:46.142767+0000","last_clean":"2024-09-15T03:46:46.142767+0000","last_became_active":"2024-09-15T03:46:39.226972+0000","last_became_peered":"2024-09-15T03:46:39.226972+0000","last_unstale":"2024-09-15T03:46:46.142767+0000","last_undegraded":"2024-09-15T03:46:46.142767+0000","last_fullsized":"2024-09-15T03:46:46.142767+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:58.755792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028667999999999999,"stat_sum":{"num_bytes":1163264,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":5,"num_write_kb":1136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,1,3,0,7,9,4],"acting":[8,11,1,3,0,7,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"3.c","version":"32'5","reported_seq":35,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:46:46.140541+0000","last_change":"2024-09-15T03:46:46.140809+0000","last_active":"2024-09-15T03:46:46.140541+0000","last_peered":"2024-09-15T03:46:46.140541+0000","last_clean":"2024-09-15T03:46:46.140541+0000","last_became_active":"2024-09-15T03:46:36.171015+0000","last_became_peered":"2024-09-15T03:46:36.171015+0000","last_unstale":"2024-09-15T03:46:46.140541+0000","last_undegraded":"2024-09-15T03:46:46.140541+0000","last_fullsized":"2024-09-15T03:46:46.140541+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:36.183909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000373185,"stat_sum":{"num_bytes":540672,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":5,"num_write_kb":1072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,11,1,0,6,5,2],"acting":[9,7,11,1,0,6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"3.b","version":"38'23","reported_seq":73,"reported_epoch":39,"state":"active+clean+snaptrim","last_fresh":"2024-09-15T03:46:46.140484+0000","last_change":"2024-09-15T03:46:46.140504+0000","last_active":"2024-09-15T03:46:46.140484+0000","last_peered":"2024-09-15T03:46:46.140484+0000","last_clean":"2024-09-15T03:46:46.140484+0000","last_became_active":"2024-09-15T03:46:39.379542+0000","last_became_peered":"2024-09-15T03:46:39.379542+0000","last_unstale":"2024-09-15T03:46:46.140484+0000","last_undegraded":"2024-09-15T03:46:46.140484+0000","last_fullsized":"2024-09-15T03:46:46.140484+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:01.515626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4063232,"num_objects":10,"num_object_clones":3,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":16,"num_read_kb":8,"num_write":38,"num_write_kb":3968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,8,4,1,2,11,3],"acting":[7,6,8,4,1,2,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"39'16","reported_seq":43,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:46:46.147870+0000","last_change":"2024-09-15T03:46:46.148431+0000","last_active":"2024-09-15T03:46:46.147870+0000","last_peered":"2024-09-15T03:46:46.147870+0000","last_clean":"2024-09-15T03:46:46.147870+0000","last_became_active":"2024-09-15T03:46:36.101726+0000","last_became_peered":"2024-09-15T03:46:36.101726+0000","last_unstale":"2024-09-15T03:46:46.147870+0000","last_undegraded":"2024-09-15T03:46:46.147870+0000","last_fullsized":"2024-09-15T03:46:46.147870+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:03.376087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0063994519999999999,"stat_sum":{"num_bytes":835584,"num_objects":5,"num_object_clones":3,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":4,"num_read_kb":2,"num_write":9,"num_write_kb":816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,9,3,1,6,5],"acting":[11,8,7,9,3,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"3.9","version":"35'8","reported_seq":54,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:46:46.143534+0000","last_change":"2024-09-15T03:46:46.145124+0000","last_active":"2024-09-15T03:46:46.143534+0000","last_peered":"2024-09-15T03:46:46.143534+0000","last_clean":"2024-09-15T03:46:46.143534+0000","last_became_active":"2024-09-15T03:46:39.379593+0000","last_became_peered":"2024-09-15T03:46:39.379593+0000","last_unstale":"2024-09-15T03:46:46.143534+0000","last_undegraded":"2024-09-15T03:46:46.143534+0000","last_fullsized":"2024-09-15T03:46:46.143534+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:45.647548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017149979999999999,"stat_sum":{"num_bytes":1048576,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":7,"num_read_kb":3,"num_write":7,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0,1,9,6,8,11],"acting":[3,2,0,1,9,6,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"3.8","version":"32'4","reported_seq":49,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:46:46.143037+0000","last_change":"2024-09-15T03:46:46.143470+0000","last_active":"2024-09-15T03:46:46.143037+0000","last_peered":"2024-09-15T03:46:46.143037+0000","last_clean":"2024-09-15T03:46:46.143037+0000","last_became_active":"2024-09-15T03:46:39.381566+0000","last_became_peered":"2024-09-15T03:46:39.381566+0000","last_unstale":"2024-09-15T03:46:46.143037+0000","last_undegraded":"2024-09-15T03:46:46.143037+0000","last_fullsized":"2024-09-15T03:46:46.143037+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:48.809799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061421100000000003,"stat_sum":{"num_bytes":917504,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":4,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,11,4,1,0,5],"acting":[3,7,2,11,4,1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"1.0","version":"20'92","reported_seq":141,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:46:45.129735+0000","last_change":"2024-09-15T03:46:18.011841+0000","last_active":"2024-09-15T03:46:45.129735+0000","last_peered":"2024-09-15T03:46:45.129735+0000","last_clean":"2024-09-15T03:46:45.129735+0000","last_became_active":"2024-09-15T03:46:18.011130+0000","last_became_peered":"2024-09-15T03:46:18.011130+0000","last_unstale":"2024-09-15T03:46:45.129735+0000","last_undegraded":"2024-09-15T03:46:45.129735+0000","last_fullsized":"2024-09-15T03:46:45.129735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_clean_scrub_stamp":"2024-09-15T03:46:16.886982+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:32.556840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"22'1","reported_seq":31,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:46:45.129908+0000","last_change":"2024-09-15T03:46:26.776623+0000","last_active":"2024-09-15T03:46:45.129908+0000","last_peered":"2024-09-15T03:46:45.129908+0000","last_clean":"2024-09-15T03:46:45.129908+0000","last_became_active":"2024-09-15T03:46:24.038633+0000","last_became_peered":"2024-09-15T03:46:24.038633+0000","last_unstale":"2024-09-15T03:46:45.129908+0000","last_undegraded":"2024-09-15T03:46:45.129908+0000","last_fullsized":"2024-09-15T03:46:45.129908+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:45.125997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030178200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"38'10","reported_seq":62,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:46:46.143970+0000","last_change":"2024-09-15T03:46:46.144435+0000","last_active":"2024-09-15T03:46:46.143970+0000","last_peered":"2024-09-15T03:46:46.143970+0000","last_clean":"2024-09-15T03:46:46.143970+0000","last_became_active":"2024-09-15T03:46:39.381493+0000","last_became_peered":"2024-09-15T03:46:39.381493+0000","last_unstale":"2024-09-15T03:46:46.143970+0000","last_undegraded":"2024-09-15T03:46:46.143970+0000","last_fullsized":"2024-09-15T03:46:46.143970+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:41.910395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010791900000000001,"stat_sum":{"num_bytes":1114112,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":15,"num_read_kb":4133,"num_write":8,"num_write_kb":1168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1,7,8,6,11,9],"acting":[4,2,1,7,8,6,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":30,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:46:45.129651+0000","last_change":"2024-09-15T03:46:26.044503+0000","last_active":"2024-09-15T03:46:45.129651+0000","last_peered":"2024-09-15T03:46:45.129651+0000","last_clean":"2024-09-15T03:46:45.129651+0000","last_became_active":"2024-09-15T03:46:24.063106+0000","last_became_peered":"2024-09-15T03:46:24.063106+0000","last_unstale":"2024-09-15T03:46:45.129651+0000","last_undegraded":"2024-09-15T03:46:45.129651+0000","last_fullsized":"2024-09-15T03:46:45.129651+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:24.101828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000393075,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"32'2","reported_seq":46,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:46:46.142797+0000","last_change":"2024-09-15T03:46:46.144527+0000","last_active":"2024-09-15T03:46:46.142797+0000","last_peered":"2024-09-15T03:46:46.142797+0000","last_clean":"2024-09-15T03:46:46.142797+0000","last_became_active":"2024-09-15T03:46:39.381489+0000","last_became_peered":"2024-09-15T03:46:39.381489+0000","last_unstale":"2024-09-15T03:46:46.142797+0000","last_undegraded":"2024-09-15T03:46:46.142797+0000","last_fullsized":"2024-09-15T03:46:46.142797+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:41.290445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020105980000000002,"stat_sum":{"num_bytes":524288,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3,8,5,2,9,7],"acting":[6,11,3,8,5,2,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":30,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:46:45.130098+0000","last_change":"2024-09-15T03:46:26.042740+0000","last_active":"2024-09-15T03:46:45.130098+0000","last_peered":"2024-09-15T03:46:45.130098+0000","last_clean":"2024-09-15T03:46:45.130098+0000","last_became_active":"2024-09-15T03:46:24.037471+0000","last_became_peered":"2024-09-15T03:46:24.037471+0000","last_unstale":"2024-09-15T03:46:45.130098+0000","last_undegraded":"2024-09-15T03:46:45.130098+0000","last_fullsized":"2024-09-15T03:46:45.130098+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:29.654520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026490299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"39'18","reported_seq":65,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:46:46.150989+0000","last_change":"2024-09-15T03:46:46.151998+0000","last_active":"2024-09-15T03:46:46.150989+0000","last_peered":"2024-09-15T03:46:46.150989+0000","last_clean":"2024-09-15T03:46:46.150989+0000","last_became_active":"2024-09-15T03:46:39.378701+0000","last_became_peered":"2024-09-15T03:46:39.378701+0000","last_unstale":"2024-09-15T03:46:46.150989+0000","last_undegraded":"2024-09-15T03:46:46.150989+0000","last_fullsized":"2024-09-15T03:46:46.150989+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:46.925705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059149479999999997,"stat_sum":{"num_bytes":2752512,"num_objects":5,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":13,"num_read_kb":2070,"num_write":15,"num_write_kb":2688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,9,4,3,6,0,5],"acting":[1,2,9,4,3,6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"2.2","version":"24'2","reported_seq":32,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:46:45.129846+0000","last_change":"2024-09-15T03:46:26.045048+0000","last_active":"2024-09-15T03:46:45.129846+0000","last_peered":"2024-09-15T03:46:45.129846+0000","last_clean":"2024-09-15T03:46:45.129846+0000","last_became_active":"2024-09-15T03:46:24.062930+0000","last_became_peered":"2024-09-15T03:46:24.062930+0000","last_unstale":"2024-09-15T03:46:45.129846+0000","last_undegraded":"2024-09-15T03:46:45.129846+0000","last_fullsized":"2024-09-15T03:46:45.129846+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:24.188285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000944677,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"32'15","reported_seq":48,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:46:46.144629+0000","last_change":"2024-09-15T03:46:46.145079+0000","last_active":"2024-09-15T03:46:46.144629+0000","last_peered":"2024-09-15T03:46:46.144629+0000","last_clean":"2024-09-15T03:46:46.144629+0000","last_became_active":"2024-09-15T03:46:39.379990+0000","last_became_peered":"2024-09-15T03:46:39.379990+0000","last_unstale":"2024-09-15T03:46:46.144629+0000","last_undegraded":"2024-09-15T03:46:46.144629+0000","last_fullsized":"2024-09-15T03:46:46.144629+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:59.701705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022818840000000001,"stat_sum":{"num_bytes":2506752,"num_objects":6,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":7,"num_write":15,"num_write_kb":2448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,4,7,1,2,9,0],"acting":[6,8,4,7,1,2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"3.5","version":"38'10","reported_seq":36,"reported_epoch":39,"state":"active+clean+snaptrim","last_fresh":"2024-09-15T03:46:46.146932+0000","last_change":"2024-09-15T03:46:46.147461+0000","last_active":"2024-09-15T03:46:46.146932+0000","last_peered":"2024-09-15T03:46:46.146932+0000","last_clean":"2024-09-15T03:46:46.146932+0000","last_became_active":"2024-09-15T03:46:39.226376+0000","last_became_peered":"2024-09-15T03:46:39.226376+0000","last_unstale":"2024-09-15T03:46:46.146932+0000","last_undegraded":"2024-09-15T03:46:46.146932+0000","last_fullsized":"2024-09-15T03:46:46.146932+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:04.774618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2473984,"num_objects":3,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":8,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4,9,8,7,3,0],"acting":[1,6,4,9,8,7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":28,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-15T03:46:43.125462+0000","last_change":"2024-09-15T03:46:26.041613+0000","last_active":"2024-09-15T03:46:43.125462+0000","last_peered":"2024-09-15T03:46:43.125462+0000","last_clean":"2024-09-15T03:46:43.125462+0000","last_became_active":"2024-09-15T03:46:24.063070+0000","last_became_peered":"2024-09-15T03:46:24.063070+0000","last_unstale":"2024-09-15T03:46:43.125462+0000","last_undegraded":"2024-09-15T03:46:43.125462+0000","last_fullsized":"2024-09-15T03:46:43.125462+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:35.057577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034123999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"38'7","reported_seq":50,"reported_epoch":39,"state":"active+clean+snaptrim","last_fresh":"2024-09-15T03:46:46.142642+0000","last_change":"2024-09-15T03:46:46.142718+0000","last_active":"2024-09-15T03:46:46.142642+0000","last_peered":"2024-09-15T03:46:46.142642+0000","last_clean":"2024-09-15T03:46:46.142642+0000","last_became_active":"2024-09-15T03:46:39.382225+0000","last_became_peered":"2024-09-15T03:46:39.382225+0000","last_unstale":"2024-09-15T03:46:46.142642+0000","last_undegraded":"2024-09-15T03:46:46.142642+0000","last_fullsized":"2024-09-15T03:46:46.142642+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:42.367003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360448,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":5,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5,8,0,4,7,3],"acting":[6,2,5,8,0,4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":30,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:46:45.132274+0000","last_change":"2024-09-15T03:46:26.042111+0000","last_active":"2024-09-15T03:46:45.132274+0000","last_peered":"2024-09-15T03:46:45.132274+0000","last_clean":"2024-09-15T03:46:45.132274+0000","last_became_active":"2024-09-15T03:46:24.038945+0000","last_became_peered":"2024-09-15T03:46:24.038945+0000","last_unstale":"2024-09-15T03:46:45.132274+0000","last_undegraded":"2024-09-15T03:46:45.132274+0000","last_fullsized":"2024-09-15T03:46:45.132274+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:27.950416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032787499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"32'13","reported_seq":62,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:46:46.143586+0000","last_change":"2024-09-15T03:46:46.144142+0000","last_active":"2024-09-15T03:46:46.143586+0000","last_peered":"2024-09-15T03:46:46.143586+0000","last_clean":"2024-09-15T03:46:46.143586+0000","last_became_active":"2024-09-15T03:46:39.381096+0000","last_became_peered":"2024-09-15T03:46:39.381096+0000","last_unstale":"2024-09-15T03:46:46.143586+0000","last_undegraded":"2024-09-15T03:46:46.143586+0000","last_fullsized":"2024-09-15T03:46:46.143586+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:38.482324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001063476,"stat_sum":{"num_bytes":2539520,"num_objects":4,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":13,"num_write_kb":3152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,7,2,1,11,0],"acting":[4,3,9,7,2,1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":30,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:46:45.128675+0000","last_change":"2024-09-15T03:46:26.776405+0000","last_active":"2024-09-15T03:46:45.128675+0000","last_peered":"2024-09-15T03:46:45.128675+0000","last_clean":"2024-09-15T03:46:45.128675+0000","last_became_active":"2024-09-15T03:46:24.063380+0000","last_became_peered":"2024-09-15T03:46:24.063380+0000","last_unstale":"2024-09-15T03:46:45.128675+0000","last_undegraded":"2024-09-15T03:46:45.128675+0000","last_fullsized":"2024-09-15T03:46:45.128675+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:24.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034490600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"32'4","reported_seq":49,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:46:46.143078+0000","last_change":"2024-09-15T03:46:46.143845+0000","last_active":"2024-09-15T03:46:46.143078+0000","last_peered":"2024-09-15T03:46:46.143078+0000","last_clean":"2024-09-15T03:46:46.143078+0000","last_became_active":"2024-09-15T03:46:39.111863+0000","last_became_peered":"2024-09-15T03:46:39.111863+0000","last_unstale":"2024-09-15T03:46:46.143078+0000","last_undegraded":"2024-09-15T03:46:46.143078+0000","last_fullsized":"2024-09-15T03:46:46.143078+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:03.221382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088857300000000003,"stat_sum":{"num_bytes":557056,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":4,"num_write_kb":544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,9,8,3,6,1,11],"acting":[4,7,9,8,3,6,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":30,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:46:45.130001+0000","last_change":"2024-09-15T03:46:26.044851+0000","last_active":"2024-09-15T03:46:45.130001+0000","last_peered":"2024-09-15T03:46:45.130001+0000","last_clean":"2024-09-15T03:46:45.130001+0000","last_became_active":"2024-09-15T03:46:24.063335+0000","last_became_peered":"2024-09-15T03:46:24.063335+0000","last_unstale":"2024-09-15T03:46:45.130001+0000","last_undegraded":"2024-09-15T03:46:45.130001+0000","last_fullsized":"2024-09-15T03:46:45.130001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:34.631264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065928000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":29081600,"num_objects":67,"num_object_clones":19,"num_object_copies":536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":6,"num_read":136,"num_read_kb":8079,"num_write":169,"num_write_kb":30432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":180,"ondisk_log_size":180,"up":128,"acting":128,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":11,"up_from":17,"seq":73014444040,"num_pgs":13,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":803988,"kb_used_data":92876484,"kb_used_omap":416,"kb_used_meta":0,"kb_avail":92875628,"statfs":{"total":95927926784,"available":95104643072,"internally_reserved":876544,"allocated":95105519616,"data_stored":95105519616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":426978,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":10,"up_from":0,"seq":73014444040,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":9,"up_from":17,"seq":73014444040,"num_pgs":13,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":804532,"kb_used_data":92875716,"kb_used_omap":378,"kb_used_meta":0,"kb_avail":92875084,"statfs":{"total":95927926784,"available":95104086016,"internally_reserved":647168,"allocated":95104733184,"data_stored":95104733184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":387752,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]}]},{"osd":8,"up_from":17,"seq":73014444040,"num_pgs":16,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":806652,"kb_used_data":92874128,"kb_used_omap":502,"kb_used_meta":0,"kb_avail":92872964,"statfs":{"total":95927926784,"available":95101915136,"internally_reserved":1191936,"allocated":95103107072,"data_stored":95103107072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514279,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.87}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]}]},{"osd":7,"up_from":17,"seq":73014444040,"num_pgs":15,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":806448,"kb_used_data":92874332,"kb_used_omap":477,"kb_used_meta":0,"kb_avail":92873168,"statfs":{"total":95927926784,"available":95102124032,"internally_reserved":1191936,"allocated":95103315968,"data_stored":95103315968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":488959,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]}]},{"osd":6,"up_from":17,"seq":73014444040,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":804368,"kb_used_data":92876120,"kb_used_omap":406,"kb_used_meta":0,"kb_avail":92875248,"statfs":{"total":95927926784,"available":95104253952,"internally_reserved":892928,"allocated":95105146880,"data_stored":95105146880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":415864,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]}]},{"osd":5,"up_from":17,"seq":73014444040,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799880,"kb_used_data":92880084,"kb_used_omap":218,"kb_used_meta":0,"kb_avail":92879736,"statfs":{"total":95927926784,"available":95108849664,"internally_reserved":356352,"allocated":95109206016,"data_stored":95109206016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":223862,"internal_metadata":0},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,10],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]}]},{"osd":4,"up_from":17,"seq":73014444040,"num_pgs":13,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":804940,"kb_used_data":92875536,"kb_used_omap":409,"kb_used_meta":0,"kb_avail":92874676,"statfs":{"total":95927926784,"available":95103668224,"internally_reserved":880640,"allocated":95104548864,"data_stored":95104548864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":419358,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]}]},{"osd":3,"up_from":17,"seq":73014444040,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":803704,"kb_used_data":92876588,"kb_used_omap":375,"kb_used_meta":0,"kb_avail":92875912,"statfs":{"total":95927926784,"available":95104933888,"internally_reserved":692224,"allocated":95105626112,"data_stored":95105626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":384092,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]}]},{"osd":2,"up_from":17,"seq":73014444040,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":805356,"kb_used_data":92875124,"kb_used_omap":423,"kb_used_meta":0,"kb_avail":92874260,"statfs":{"total":95927926784,"available":95103242240,"internally_reserved":884736,"allocated":95104126976,"data_stored":95104126976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":433966,"internal_metadata":0},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81699999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]}]},{"osd":1,"up_from":17,"seq":73014444041,"num_pgs":15,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":807452,"kb_used_data":92873308,"kb_used_omap":493,"kb_used_meta":0,"kb_avail":92872164,"statfs":{"total":95927926784,"available":95101095936,"internally_reserved":1171456,"allocated":95102267392,"data_stored":95102267392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":505620,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]}]},{"osd":0,"up_from":17,"seq":73014444040,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":806500,"kb_used_data":92873944,"kb_used_omap":415,"kb_used_meta":0,"kb_avail":92873116,"statfs":{"total":95927926784,"available":95102070784,"internally_reserved":847872,"allocated":95102918656,"data_stored":95102918656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":425793,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]}],"pool_statfs":[]}} 2024-09-15T03:46:48.924 INFO:tasks.thrashosds.thrasher:Setting 3.6 to [7, 10, 3, 5, 0, 6, 1, 11, 8, 9, 2, 4] 2024-09-15T03:46:48.924 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.6', '7', '10', '3', '5', '0', '6', '1', '11', '8', '9', '2', '4'] 2024-09-15T03:46:48.924 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.6 7 10 3 5 0 6 1 11 8 9 2 4 2024-09-15T03:46:48.934 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:48.934+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:46:48.955 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:48.955 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:48.963 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:46:49.036 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:49.035+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:49.103 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:49.103 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:49.112 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:46:49.137 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:49.136+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:46:49.155 INFO:tasks.rados.rados.0.smithi001.stdout:100: done (11 left) 2024-09-15T03:46:49.155 INFO:tasks.rados.rados.0.smithi001.stdout:101: done (10 left) 2024-09-15T03:46:49.156 INFO:tasks.rados.rados.0.smithi001.stdout:102: done (9 left) 2024-09-15T03:46:49.156 INFO:tasks.rados.rados.0.smithi001.stdout:103: done (8 left) 2024-09-15T03:46:49.156 INFO:tasks.rados.rados.0.smithi001.stdout:105: done (7 left) 2024-09-15T03:46:49.156 INFO:tasks.rados.rados.0.smithi001.stdout:106: done (6 left) 2024-09-15T03:46:49.156 INFO:tasks.rados.rados.0.smithi001.stdout:107: done (5 left) 2024-09-15T03:46:49.156 INFO:tasks.rados.rados.0.smithi001.stdout:108: done (4 left) 2024-09-15T03:46:49.156 INFO:tasks.rados.rados.0.smithi001.stdout:109: done (3 left) 2024-09-15T03:46:49.156 INFO:tasks.rados.rados.0.smithi001.stdout:110: done (2 left) 2024-09-15T03:46:49.156 INFO:tasks.rados.rados.0.smithi001.stdout:111: done (1 left) 2024-09-15T03:46:49.157 INFO:tasks.rados.rados.0.smithi001.stdout:112: done (0 left) 2024-09-15T03:46:49.157 INFO:tasks.rados.rados.0.smithi001.stdout:113: read oid 20 snap 0 2024-09-15T03:46:49.157 INFO:tasks.rados.rados.0.smithi001.stdout:113: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-15T03:46:49.157 INFO:tasks.rados.rados.0.smithi001.stdout:114: snap_create 2024-09-15T03:46:49.160 INFO:tasks.rados.rados.0.smithi001.stdout:113: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-15T03:46:49.215 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:49.215 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:49.224 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:46:49.267 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:49.266+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:46:49.307 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:46:49.324 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:49.324 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:49.333 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:46:49.352 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:49.351+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:46:49.423 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:46:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:46:49.437 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:46:49.438 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:46:49.439 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:49.438+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:46:49.452 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:46:49.452 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:46:49.466 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:46:49.466 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:46:49.474 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:49.474 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:49.482 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:46:49.482 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:46:49.483 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:46:49.496 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:46:49.496 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:46:49.510 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:46:49.511 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:46:49.526 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:46:49.526 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:46:49.541 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:46:49.541 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:46:49.556 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:46:49.556 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:46:49.568 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:49.568+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:46:49.571 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:46:49.571 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:46:49.586 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:46:49.587 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:46:49.625 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:49.626 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:46:49.626 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:46:49.626 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:49.634 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:46:49.676 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:49.675+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:46:49.741 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:49.741+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:49.775 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:46:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:46:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:46:49.776 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:49.784 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:46:49.842 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:46:49.841+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:46:49.886 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:49.886 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:46:49.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:46:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:46:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs3 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108456+0000", 2024-09-15T03:46:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.7756047670000008, 2024-09-15T03:46:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015075257, 2024-09-15T03:46:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108456+0000", 2024-09-15T03:46:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108456+0000", 2024-09-15T03:46:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108457+0000", 2024-09-15T03:46:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9999999999999997e-07 2024-09-15T03:46:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108477+0000", 2024-09-15T03:46:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0344e-05 2024-09-15T03:46:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108497+0000", 2024-09-15T03:46:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0143999999999999e-05 2024-09-15T03:46:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108504+0000", 2024-09-15T03:46:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8499999999999996e-06 2024-09-15T03:46:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108554+0000", 2024-09-15T03:46:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9567999999999997e-05 2024-09-15T03:46:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108568+0000", 2024-09-15T03:46:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4326e-05 2024-09-15T03:46:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108654+0000", 2024-09-15T03:46:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6193999999999995e-05 2024-09-15T03:46:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108663+0000", 2024-09-15T03:46:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7479999999999999e-06 2024-09-15T03:46:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111234+0000", 2024-09-15T03:46:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025709079999999998 2024-09-15T03:46:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111259+0000", 2024-09-15T03:46:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4797999999999999e-05 2024-09-15T03:46:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123531+0000", 2024-09-15T03:46:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012272777 2024-09-15T03:46:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as2 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:46:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.7753761130000001, 2024-09-15T03:46:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0149108, 2024-09-15T03:46:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:46:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:46:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:46:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1699999999999999e-07 2024-09-15T03:46:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108710+0000", 2024-09-15T03:46:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5032999999999999e-05 2024-09-15T03:46:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108725+0000", 2024-09-15T03:46:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4788e-05 2024-09-15T03:46:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108733+0000", 2024-09-15T03:46:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4610000000000008e-06 2024-09-15T03:46:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108778+0000", 2024-09-15T03:46:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4224999999999998e-05 2024-09-15T03:46:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108800+0000", 2024-09-15T03:46:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2189999999999999e-05 2024-09-15T03:46:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108884+0000", 2024-09-15T03:46:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4569999999999998e-05 2024-09-15T03:46:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111237+0000", 2024-09-15T03:46:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002352313 2024-09-15T03:46:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114139+0000", 2024-09-15T03:46:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002902057 2024-09-15T03:46:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114173+0000", 2024-09-15T03:46:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4461000000000001e-05 2024-09-15T03:46:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123596+0000", 2024-09-15T03:46:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094222850000000007 2024-09-15T03:46:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs3 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:46:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.7753168349999999, 2024-09-15T03:46:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014886043, 2024-09-15T03:46:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:46:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:46:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:46:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5100000000000001e-07 2024-09-15T03:46:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108759+0000", 2024-09-15T03:46:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4945e-05 2024-09-15T03:46:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108768+0000", 2024-09-15T03:46:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2309999999999996e-06 2024-09-15T03:46:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108795+0000", 2024-09-15T03:46:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7829000000000001e-05 2024-09-15T03:46:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108841+0000", 2024-09-15T03:46:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5330999999999997e-05 2024-09-15T03:46:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108850+0000", 2024-09-15T03:46:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9959999999999999e-06 2024-09-15T03:46:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108940+0000", 2024-09-15T03:46:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9823999999999999e-05 2024-09-15T03:46:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111237+0000", 2024-09-15T03:46:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002297927 2024-09-15T03:46:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114139+0000", 2024-09-15T03:46:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029020029999999998 2024-09-15T03:46:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114212+0000", 2024-09-15T03:46:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2177999999999999e-05 2024-09-15T03:46:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123630+0000", 2024-09-15T03:46:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094184279999999995 2024-09-15T03:46:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as2 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.109397+0000", 2024-09-15T03:46:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.7746637770000007, 2024-09-15T03:46:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014322056, 2024-09-15T03:46:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109397+0000", 2024-09-15T03:46:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109397+0000", 2024-09-15T03:46:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109398+0000", 2024-09-15T03:46:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1600000000000001e-07 2024-09-15T03:46:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109410+0000", 2024-09-15T03:46:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2294e-05 2024-09-15T03:46:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109420+0000", 2024-09-15T03:46:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.764e-06 2024-09-15T03:46:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109426+0000", 2024-09-15T03:46:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9309999999999996e-06 2024-09-15T03:46:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109457+0000", 2024-09-15T03:46:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1047000000000003e-05 2024-09-15T03:46:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109462+0000", 2024-09-15T03:46:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1499999999999998e-06 2024-09-15T03:46:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109513+0000", 2024-09-15T03:46:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0722e-05 2024-09-15T03:46:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111238+0000", 2024-09-15T03:46:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017253380000000001 2024-09-15T03:46:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114140+0000", 2024-09-15T03:46:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029017370000000002 2024-09-15T03:46:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114201+0000", 2024-09-15T03:46:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1047e-05 2024-09-15T03:46:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123719+0000", 2024-09-15T03:46:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095184099999999997 2024-09-15T03:46:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs3 33/31 ECSubWrite(tid=27, reqid=client.4521.0:189, at_version=0'0, trim_to=0'0, roll_forward_to=33'10))", 2024-09-15T03:46:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.114638+0000", 2024-09-15T03:46:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.7694225800000005, 2024-09-15T03:46:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090975689999999998, 2024-09-15T03:46:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114638+0000", 2024-09-15T03:46:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114638+0000", 2024-09-15T03:46:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114639+0000", 2024-09-15T03:46:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7899999999999998e-07 2024-09-15T03:46:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114648+0000", 2024-09-15T03:46:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.969e-06 2024-09-15T03:46:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114663+0000", 2024-09-15T03:46:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4813e-05 2024-09-15T03:46:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114669+0000", 2024-09-15T03:46:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9710000000000001e-06 2024-09-15T03:46:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114728+0000", 2024-09-15T03:46:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9794999999999999e-05 2024-09-15T03:46:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114737+0000", 2024-09-15T03:46:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3639999999999999e-06 2024-09-15T03:46:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114862+0000", 2024-09-15T03:46:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124826 2024-09-15T03:46:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114881+0000", 2024-09-15T03:46:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8901000000000001e-05 2024-09-15T03:46:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114920+0000", 2024-09-15T03:46:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9258000000000003e-05 2024-09-15T03:46:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114936+0000", 2024-09-15T03:46:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5665999999999999e-05 2024-09-15T03:46:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123736+0000", 2024-09-15T03:46:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088003270000000001 2024-09-15T03:46:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as2 33/28 ECSubWrite(tid=28, reqid=client.4521.0:186, at_version=0'0, trim_to=0'0, roll_forward_to=33'8))", 2024-09-15T03:46:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.114840+0000", 2024-09-15T03:46:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.769221044, 2024-09-15T03:46:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090295170000000008, 2024-09-15T03:46:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114840+0000", 2024-09-15T03:46:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114840+0000", 2024-09-15T03:46:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114840+0000", 2024-09-15T03:46:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6899999999999997e-07 2024-09-15T03:46:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114850+0000", 2024-09-15T03:46:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0023e-05 2024-09-15T03:46:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:49.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114862+0000", 2024-09-15T03:46:49.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1615e-05 2024-09-15T03:46:49.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:49.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114868+0000", 2024-09-15T03:46:49.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3570000000000003e-06 2024-09-15T03:46:49.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:49.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114917+0000", 2024-09-15T03:46:49.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8838999999999998e-05 2024-09-15T03:46:49.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:49.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114926+0000", 2024-09-15T03:46:49.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7390000000000005e-06 2024-09-15T03:46:49.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:49.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.115039+0000", 2024-09-15T03:46:49.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113114 2024-09-15T03:46:49.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:49.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.115048+0000", 2024-09-15T03:46:49.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5080000000000005e-06 2024-09-15T03:46:49.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:49.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.115077+0000", 2024-09-15T03:46:49.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.97e-05 2024-09-15T03:46:49.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:49.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.115104+0000", 2024-09-15T03:46:49.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6197999999999999e-05 2024-09-15T03:46:49.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:49.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123869+0000", 2024-09-15T03:46:49.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087657550000000001 2024-09-15T03:46:49.924 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.924 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:49.924 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.924 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:193 3.bs0 3:df8b18a3:::smithi00132172-17:head [rollback 3] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:46:49.924 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.112181+0000", 2024-09-15T03:46:49.924 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.7718797669999997, 2024-09-15T03:46:49.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010302426999999999, 2024-09-15T03:46:49.924 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:49.924 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:49.925 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:49.925 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:49.925 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:49.925 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 193 2024-09-15T03:46:49.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.925 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:49.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:49.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112181+0000", 2024-09-15T03:46:49.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:49.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112181+0000", 2024-09-15T03:46:49.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:49.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112182+0000", 2024-09-15T03:46:49.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6000000000000004e-07 2024-09-15T03:46:49.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:49.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112185+0000", 2024-09-15T03:46:49.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.892e-06 2024-09-15T03:46:49.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:49.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112192+0000", 2024-09-15T03:46:49.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8110000000000003e-06 2024-09-15T03:46:49.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:49.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112197+0000", 2024-09-15T03:46:49.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3640000000000001e-06 2024-09-15T03:46:49.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:49.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.113873+0000", 2024-09-15T03:46:49.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016760760000000001 2024-09-15T03:46:49.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:49.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.113974+0000", 2024-09-15T03:46:49.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101303 2024-09-15T03:46:49.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:49.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114575+0000", 2024-09-15T03:46:49.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00060097600000000003 2024-09-15T03:46:49.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:49.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114707+0000", 2024-09-15T03:46:49.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000131543 2024-09-15T03:46:49.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:49.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114723+0000", 2024-09-15T03:46:49.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6449000000000001e-05 2024-09-15T03:46:49.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:49.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114771+0000", 2024-09-15T03:46:49.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8312000000000001e-05 2024-09-15T03:46:49.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:49.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.115309+0000", 2024-09-15T03:46:49.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053803300000000004 2024-09-15T03:46:49.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:49.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.115740+0000", 2024-09-15T03:46:49.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043093100000000001 2024-09-15T03:46:49.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:49.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.122484+0000", 2024-09-15T03:46:49.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0067431770000000004 2024-09-15T03:46:49.932 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.932 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:49.932 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.932 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:194 3.bs0 3:df8b18a3:::smithi00132172-17:head [append 0~0] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:46:49.932 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.112221+0000", 2024-09-15T03:46:49.932 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.7718397380000006, 2024-09-15T03:46:49.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013410515, 2024-09-15T03:46:49.932 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:49.933 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:49.933 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:49.933 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:49.933 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:49.933 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 194 2024-09-15T03:46:49.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.933 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:49.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:49.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112221+0000", 2024-09-15T03:46:49.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:49.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112221+0000", 2024-09-15T03:46:49.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:49.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112222+0000", 2024-09-15T03:46:49.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:46:49.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:49.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112226+0000", 2024-09-15T03:46:49.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6840000000000004e-06 2024-09-15T03:46:49.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:49.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112232+0000", 2024-09-15T03:46:49.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9660000000000001e-06 2024-09-15T03:46:49.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:49.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112236+0000", 2024-09-15T03:46:49.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2740000000000001e-06 2024-09-15T03:46:49.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:49.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116143+0000", 2024-09-15T03:46:49.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039065549999999999 2024-09-15T03:46:49.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:49.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116257+0000", 2024-09-15T03:46:49.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113655 2024-09-15T03:46:49.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:49.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.122980+0000", 2024-09-15T03:46:49.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0067229860000000002 2024-09-15T03:46:49.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:49.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123101+0000", 2024-09-15T03:46:49.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121143 2024-09-15T03:46:49.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:49.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123117+0000", 2024-09-15T03:46:49.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6279000000000001e-05 2024-09-15T03:46:49.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:49.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123185+0000", 2024-09-15T03:46:49.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7613000000000005e-05 2024-09-15T03:46:49.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:49.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124310+0000", 2024-09-15T03:46:49.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011254679999999999 2024-09-15T03:46:49.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:49.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125591+0000", 2024-09-15T03:46:49.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012803669999999999 2024-09-15T03:46:49.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:49.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125632+0000", 2024-09-15T03:46:49.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1115000000000002e-05 2024-09-15T03:46:49.939 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.939 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:49.939 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.939 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:197 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [delete] snapc 3=[3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e34)", 2024-09-15T03:46:49.939 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:46:49.939 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.7717956010000009, 2024-09-15T03:46:49.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021934704999999999, 2024-09-15T03:46:49.940 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:49.940 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:49.940 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:49.940 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:49.940 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:49.940 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197 2024-09-15T03:46:49.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.940 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:49.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:49.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:46:49.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:49.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:46:49.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:49.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112266+0000", 2024-09-15T03:46:49.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5000000000000002e-07 2024-09-15T03:46:49.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:49.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112269+0000", 2024-09-15T03:46:49.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.066e-06 2024-09-15T03:46:49.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:49.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112276+0000", 2024-09-15T03:46:49.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-09-15T03:46:49.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:49.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112281+0000", 2024-09-15T03:46:49.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3970000000000004e-06 2024-09-15T03:46:49.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:49.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123281+0000", 2024-09-15T03:46:49.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011000575 2024-09-15T03:46:49.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:49.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123395+0000", 2024-09-15T03:46:49.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113398 2024-09-15T03:46:49.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:49.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123963+0000", 2024-09-15T03:46:49.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000568101 2024-09-15T03:46:49.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:49.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124220+0000", 2024-09-15T03:46:49.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025704900000000003 2024-09-15T03:46:49.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:49.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124229+0000", 2024-09-15T03:46:49.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4860000000000003e-06 2024-09-15T03:46:49.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:49.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124277+0000", 2024-09-15T03:46:49.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7574000000000002e-05 2024-09-15T03:46:49.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:49.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124312+0000", 2024-09-15T03:46:49.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5256000000000003e-05 2024-09-15T03:46:49.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:49.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125716+0000", 2024-09-15T03:46:49.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014043720000000001 2024-09-15T03:46:49.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:49.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.134200+0000", 2024-09-15T03:46:49.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084835120000000003 2024-09-15T03:46:49.946 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.946 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:49.946 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.946 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es4 34/28 ECSubWrite(tid=33, reqid=client.4521.0:195, at_version=34'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-15T03:46:49.946 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:46:49.947 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.7659682419999996, 2024-09-15T03:46:49.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015907482000000001, 2024-09-15T03:46:49.947 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:49.947 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:49.947 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:49.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:49.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:46:49.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:49.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:46:49.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:49.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:46:49.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9400000000000004e-07 2024-09-15T03:46:49.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:49.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118299+0000", 2024-09-15T03:46:49.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000205677 2024-09-15T03:46:49.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:49.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118315+0000", 2024-09-15T03:46:49.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5429000000000001e-05 2024-09-15T03:46:49.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:49.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118322+0000", 2024-09-15T03:46:49.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8290000000000008e-06 2024-09-15T03:46:49.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:49.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118416+0000", 2024-09-15T03:46:49.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3140000000000006e-05 2024-09-15T03:46:49.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:49.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118431+0000", 2024-09-15T03:46:49.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5648e-05 2024-09-15T03:46:49.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:49.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118678+0000", 2024-09-15T03:46:49.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024701200000000001 2024-09-15T03:46:49.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:49.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118691+0000", 2024-09-15T03:46:49.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2653000000000001e-05 2024-09-15T03:46:49.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:49.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.119431+0000", 2024-09-15T03:46:49.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00074060399999999996 2024-09-15T03:46:49.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:49.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124280+0000", 2024-09-15T03:46:49.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0048484410000000002 2024-09-15T03:46:49.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:49.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.134000+0000", 2024-09-15T03:46:49.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0097201549999999994 2024-09-15T03:46:49.952 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.952 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:49.952 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.952 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es4 34/28 ECSubWrite(tid=34, reqid=client.4521.0:195, at_version=0'0, trim_to=0'0, roll_forward_to=34'16))", 2024-09-15T03:46:49.952 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.124887+0000", 2024-09-15T03:46:49.952 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.7591735899999996, 2024-09-15T03:46:49.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092852760000000003, 2024-09-15T03:46:49.953 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:49.953 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:49.953 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:49.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:49.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124887+0000", 2024-09-15T03:46:49.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:49.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124887+0000", 2024-09-15T03:46:49.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:49.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124888+0000", 2024-09-15T03:46:49.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2600000000000001e-07 2024-09-15T03:46:49.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:49.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124899+0000", 2024-09-15T03:46:49.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1256e-05 2024-09-15T03:46:49.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:49.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124910+0000", 2024-09-15T03:46:49.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0986e-05 2024-09-15T03:46:49.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:49.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124917+0000", 2024-09-15T03:46:49.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.297e-06 2024-09-15T03:46:49.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:49.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124985+0000", 2024-09-15T03:46:49.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7838999999999999e-05 2024-09-15T03:46:49.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:49.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124999+0000", 2024-09-15T03:46:49.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4201e-05 2024-09-15T03:46:49.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:49.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125095+0000", 2024-09-15T03:46:49.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5797000000000005e-05 2024-09-15T03:46:49.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:49.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125107+0000", 2024-09-15T03:46:49.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2194999999999999e-05 2024-09-15T03:46:49.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:49.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125145+0000", 2024-09-15T03:46:49.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8355000000000001e-05 2024-09-15T03:46:49.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:49.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125189+0000", 2024-09-15T03:46:49.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3741e-05 2024-09-15T03:46:49.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:49.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.134173+0000", 2024-09-15T03:46:49.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089836829999999993 2024-09-15T03:46:49.958 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.958 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:49.958 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.958 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:220 3.bs0 3:d99ddedc:::smithi00132172-44:head [copy-from ver 22] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:46:49.958 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:46:49.958 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7564159310000003, 2024-09-15T03:46:49.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024273429999999999, 2024-09-15T03:46:49.958 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:49.958 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:49.959 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:49.959 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:49.959 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:49.959 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 220 2024-09-15T03:46:49.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.959 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:49.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:49.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:46:49.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:49.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:46:49.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:49.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127647+0000", 2024-09-15T03:46:49.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1799999999999999e-06 2024-09-15T03:46:49.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:49.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127656+0000", 2024-09-15T03:46:49.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3759999999999997e-06 2024-09-15T03:46:49.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:49.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127674+0000", 2024-09-15T03:46:49.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7374e-05 2024-09-15T03:46:49.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:49.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127680+0000", 2024-09-15T03:46:49.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6949999999999997e-06 2024-09-15T03:46:49.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:49.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127736+0000", 2024-09-15T03:46:49.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6011999999999998e-05 2024-09-15T03:46:49.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:46:49.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127744+0000", 2024-09-15T03:46:49.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2870000000000004e-06 2024-09-15T03:46:49.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:49.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130338+0000", 2024-09-15T03:46:49.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025940249999999998 2024-09-15T03:46:49.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:49.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130423+0000", 2024-09-15T03:46:49.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5303000000000006e-05 2024-09-15T03:46:49.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:49.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.147840+0000", 2024-09-15T03:46:49.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017416580000000001 2024-09-15T03:46:49.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:49.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148318+0000", 2024-09-15T03:46:49.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047822899999999999 2024-09-15T03:46:49.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:49.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148329+0000", 2024-09-15T03:46:49.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0611000000000001e-05 2024-09-15T03:46:49.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:49.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148488+0000", 2024-09-15T03:46:49.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015938399999999999 2024-09-15T03:46:49.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:49.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148870+0000", 2024-09-15T03:46:49.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000382405 2024-09-15T03:46:49.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:49.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.151092+0000", 2024-09-15T03:46:49.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022212529999999999 2024-09-15T03:46:49.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:49.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.151918+0000", 2024-09-15T03:46:49.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082671600000000002 2024-09-15T03:46:49.966 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.966 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:49.966 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.966 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:221 3.bs0 3:d99ddedc:::smithi00132172-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:46:49.966 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:46:49.966 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7563717759999999, 2024-09-15T03:46:49.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021194474000000001, 2024-09-15T03:46:49.966 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:49.966 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:46:49.966 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:49.966 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:49.967 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:49.967 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 221 2024-09-15T03:46:49.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.967 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:49.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:49.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:46:49.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:49.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:46:49.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:49.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127690+0000", 2024-09-15T03:46:49.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:46:49.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:49.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127692+0000", 2024-09-15T03:46:49.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6589999999999999e-06 2024-09-15T03:46:49.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:49.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127700+0000", 2024-09-15T03:46:49.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4769999999999997e-06 2024-09-15T03:46:49.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:49.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127704+0000", 2024-09-15T03:46:49.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3819999999999997e-06 2024-09-15T03:46:49.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:49.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127805+0000", 2024-09-15T03:46:49.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100639 2024-09-15T03:46:49.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:49.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127809+0000", 2024-09-15T03:46:49.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8870000000000002e-06 2024-09-15T03:46:49.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:49.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130527+0000", 2024-09-15T03:46:49.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002718579 2024-09-15T03:46:49.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:46:49.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130558+0000", 2024-09-15T03:46:49.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0827999999999997e-05 2024-09-15T03:46:49.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:49.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148546+0000", 2024-09-15T03:46:49.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017988356 2024-09-15T03:46:49.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:49.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148683+0000", 2024-09-15T03:46:49.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013650899999999999 2024-09-15T03:46:49.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:49.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148884+0000", 2024-09-15T03:46:49.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020061499999999999 2024-09-15T03:46:49.972 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.972 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:49.972 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.972 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:252 3.bs0 3:d6f79b7b:::smithi00132172-29:head [copy-from ver 6] snapc 7=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:46:49.973 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.169146+0000", 2024-09-15T03:46:49.973 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.714914507, 2024-09-15T03:46:49.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016731068000000002, 2024-09-15T03:46:49.973 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:49.973 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:49.973 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:49.973 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:49.973 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:49.973 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 252 2024-09-15T03:46:49.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.974 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:49.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:49.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169146+0000", 2024-09-15T03:46:49.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:49.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169146+0000", 2024-09-15T03:46:49.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:49.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169147+0000", 2024-09-15T03:46:49.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1400000000000001e-07 2024-09-15T03:46:49.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:49.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169149+0000", 2024-09-15T03:46:49.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0279999999999999e-06 2024-09-15T03:46:49.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:49.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169152+0000", 2024-09-15T03:46:49.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0800000000000002e-06 2024-09-15T03:46:49.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:49.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169154+0000", 2024-09-15T03:46:49.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4679999999999999e-06 2024-09-15T03:46:49.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:49.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.174427+0000", 2024-09-15T03:46:49.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052724349999999998 2024-09-15T03:46:49.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:49.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.174543+0000", 2024-09-15T03:46:49.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116803 2024-09-15T03:46:49.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:49.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.178297+0000", 2024-09-15T03:46:49.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037531539999999999 2024-09-15T03:46:49.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:49.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179112+0000", 2024-09-15T03:46:49.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081548600000000003 2024-09-15T03:46:49.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:49.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179151+0000", 2024-09-15T03:46:49.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9158e-05 2024-09-15T03:46:49.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:49.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179272+0000", 2024-09-15T03:46:49.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120802 2024-09-15T03:46:49.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:49.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179531+0000", 2024-09-15T03:46:49.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000259351 2024-09-15T03:46:49.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:49.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179723+0000", 2024-09-15T03:46:49.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000191447 2024-09-15T03:46:49.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:49.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.185877+0000", 2024-09-15T03:46:49.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061546420000000001 2024-09-15T03:46:49.980 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.980 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:49.980 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.980 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:253 3.bs0 3:d6f79b7b:::smithi00132172-29:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:46:49.980 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.169161+0000", 2024-09-15T03:46:49.980 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.714900144, 2024-09-15T03:46:49.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010325136, 2024-09-15T03:46:49.981 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:49.981 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:46:49.981 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:49.981 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:49.981 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:49.981 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 253 2024-09-15T03:46:49.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.981 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:49.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:49.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169161+0000", 2024-09-15T03:46:49.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:49.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169161+0000", 2024-09-15T03:46:49.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:49.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169161+0000", 2024-09-15T03:46:49.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5899999999999998e-07 2024-09-15T03:46:49.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:49.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169162+0000", 2024-09-15T03:46:49.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1880000000000001e-06 2024-09-15T03:46:49.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:49.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169166+0000", 2024-09-15T03:46:49.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3629999999999998e-06 2024-09-15T03:46:49.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:49.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169168+0000", 2024-09-15T03:46:49.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3690000000000001e-06 2024-09-15T03:46:49.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:49.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.174740+0000", 2024-09-15T03:46:49.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055717470000000002 2024-09-15T03:46:49.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:46:49.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.174814+0000", 2024-09-15T03:46:49.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3965000000000001e-05 2024-09-15T03:46:49.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:49.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179305+0000", 2024-09-15T03:46:49.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0044917120000000001 2024-09-15T03:46:49.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:49.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179410+0000", 2024-09-15T03:46:49.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104734 2024-09-15T03:46:49.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:49.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179486+0000", 2024-09-15T03:46:49.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5798999999999997e-05 2024-09-15T03:46:49.985 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.986 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:49.986 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.986 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.3s3 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:46:49.986 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.171321+0000", 2024-09-15T03:46:49.986 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.7127401369999999, 2024-09-15T03:46:49.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0091209930000000008, 2024-09-15T03:46:49.986 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:49.986 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:49.986 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:49.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:49.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171321+0000", 2024-09-15T03:46:49.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:49.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171321+0000", 2024-09-15T03:46:49.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:49.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171322+0000", 2024-09-15T03:46:49.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.034e-06 2024-09-15T03:46:49.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:49.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171433+0000", 2024-09-15T03:46:49.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111079 2024-09-15T03:46:49.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:49.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171446+0000", 2024-09-15T03:46:49.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2802e-05 2024-09-15T03:46:49.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:49.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171453+0000", 2024-09-15T03:46:49.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9070000000000001e-06 2024-09-15T03:46:49.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:49.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171709+0000", 2024-09-15T03:46:49.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000256056 2024-09-15T03:46:49.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:49.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171717+0000", 2024-09-15T03:46:49.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9340000000000006e-06 2024-09-15T03:46:49.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:49.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172017+0000", 2024-09-15T03:46:49.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030057899999999998 2024-09-15T03:46:49.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:49.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172260+0000", 2024-09-15T03:46:49.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024294599999999999 2024-09-15T03:46:49.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:49.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172354+0000", 2024-09-15T03:46:49.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3660999999999997e-05 2024-09-15T03:46:49.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:49.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172385+0000", 2024-09-15T03:46:49.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1511999999999999e-05 2024-09-15T03:46:49.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:49.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.180442+0000", 2024-09-15T03:46:49.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0080564829999999997 2024-09-15T03:46:49.991 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.991 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:49.991 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.992 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.6s1 40/31 ECSubWrite(tid=61, reqid=client.4521.0:267, at_version=40'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:49.992 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:47.199701+0000", 2024-09-15T03:46:49.992 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 2.6843594299999998, 2024-09-15T03:46:49.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012707423000000001, 2024-09-15T03:46:49.992 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:49.992 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:49.992 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:49.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:49.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.199701+0000", 2024-09-15T03:46:49.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:49.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.199701+0000", 2024-09-15T03:46:49.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:49.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.199702+0000", 2024-09-15T03:46:49.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9299999999999997e-07 2024-09-15T03:46:49.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:49.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.199722+0000", 2024-09-15T03:46:49.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9417999999999999e-05 2024-09-15T03:46:49.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:49.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.199734+0000", 2024-09-15T03:46:49.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2621e-05 2024-09-15T03:46:49.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:49.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.199751+0000", 2024-09-15T03:46:49.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7223000000000001e-05 2024-09-15T03:46:49.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:49.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.199816+0000", 2024-09-15T03:46:49.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4974000000000005e-05 2024-09-15T03:46:49.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:49.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.199826+0000", 2024-09-15T03:46:49.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0074999999999999e-05 2024-09-15T03:46:49.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:49.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.199908+0000", 2024-09-15T03:46:49.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1561000000000001e-05 2024-09-15T03:46:49.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:49.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.199951+0000", 2024-09-15T03:46:49.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3055e-05 2024-09-15T03:46:49.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:49.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.199998+0000", 2024-09-15T03:46:49.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6795000000000002e-05 2024-09-15T03:46:49.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:49.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.200016+0000", 2024-09-15T03:46:49.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7689e-05 2024-09-15T03:46:49.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:49.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.212409+0000", 2024-09-15T03:46:49.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012393319 2024-09-15T03:46:49.997 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.997 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:49.997 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:49.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.997 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.6s1 40/31 ECSubWrite(tid=62, reqid=client.4521.0:268, at_version=40'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:49.998 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:47.200573+0000", 2024-09-15T03:46:49.998 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 2.6834878670000002, 2024-09-15T03:46:49.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012006539, 2024-09-15T03:46:49.998 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:49.998 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:49.998 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:49.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:49.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.200573+0000", 2024-09-15T03:46:49.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:49.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.200573+0000", 2024-09-15T03:46:49.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:49.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:49.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.200574+0000", 2024-09-15T03:46:49.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0399999999999995e-07 2024-09-15T03:46:49.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:49.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.200593+0000", 2024-09-15T03:46:49.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9165999999999999e-05 2024-09-15T03:46:49.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:49.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:49.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:50.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.200604+0000", 2024-09-15T03:46:50.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1610999999999999e-05 2024-09-15T03:46:50.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.200610+0000", 2024-09-15T03:46:50.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0390000000000001e-06 2024-09-15T03:46:50.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:50.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201000+0000", 2024-09-15T03:46:50.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038947999999999999 2024-09-15T03:46:50.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201009+0000", 2024-09-15T03:46:50.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.516e-06 2024-09-15T03:46:50.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201125+0000", 2024-09-15T03:46:50.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115114 2024-09-15T03:46:50.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201363+0000", 2024-09-15T03:46:50.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023833400000000001 2024-09-15T03:46:50.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203349+0000", 2024-09-15T03:46:50.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019857479999999999 2024-09-15T03:46:50.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203369+0000", 2024-09-15T03:46:50.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0404999999999999e-05 2024-09-15T03:46:50.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:50.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.212580+0000", 2024-09-15T03:46:50.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092104219999999994 2024-09-15T03:46:50.003 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:50.003 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:50.003 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:50.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.003 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.3s3 40/31 ECSubWrite(tid=50, reqid=client.4521.0:269, at_version=40'22, trim_to=0'0, roll_forward_to=40'21))", 2024-09-15T03:46:50.003 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:47.201224+0000", 2024-09-15T03:46:50.003 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 2.682837057, 2024-09-15T03:46:50.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011822386000000001, 2024-09-15T03:46:50.003 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:50.003 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.003 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:50.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:50.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201224+0000", 2024-09-15T03:46:50.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:50.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:50.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201224+0000", 2024-09-15T03:46:50.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:50.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:50.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201225+0000", 2024-09-15T03:46:50.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7599999999999996e-07 2024-09-15T03:46:50.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:50.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201263+0000", 2024-09-15T03:46:50.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8495000000000003e-05 2024-09-15T03:46:50.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:50.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201281+0000", 2024-09-15T03:46:50.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8396000000000001e-05 2024-09-15T03:46:50.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201289+0000", 2024-09-15T03:46:50.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2019999999999998e-06 2024-09-15T03:46:50.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:50.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201342+0000", 2024-09-15T03:46:50.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3779999999999997e-05 2024-09-15T03:46:50.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201352+0000", 2024-09-15T03:46:50.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6500000000000008e-06 2024-09-15T03:46:50.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201461+0000", 2024-09-15T03:46:50.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109146 2024-09-15T03:46:50.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203352+0000", 2024-09-15T03:46:50.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018905770000000001 2024-09-15T03:46:50.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203412+0000", 2024-09-15T03:46:50.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9734000000000001e-05 2024-09-15T03:46:50.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203445+0000", 2024-09-15T03:46:50.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3247999999999997e-05 2024-09-15T03:46:50.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:50.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.213046+0000", 2024-09-15T03:46:50.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096013820000000003 2024-09-15T03:46:50.008 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:50.008 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:50.008 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:50.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.009 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es4 40/28 ECSubWrite(tid=62, reqid=client.4521.0:261, at_version=0'0, trim_to=0'0, roll_forward_to=40'38))", 2024-09-15T03:46:50.009 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:47.203794+0000", 2024-09-15T03:46:50.009 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 2.6802671450000002, 2024-09-15T03:46:50.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088409040000000001, 2024-09-15T03:46:50.009 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:50.009 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.009 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:50.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:50.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203794+0000", 2024-09-15T03:46:50.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:50.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:50.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203794+0000", 2024-09-15T03:46:50.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:50.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:50.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203795+0000", 2024-09-15T03:46:50.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5999999999999991e-07 2024-09-15T03:46:50.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:50.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203806+0000", 2024-09-15T03:46:50.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0981e-05 2024-09-15T03:46:50.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:50.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203824+0000", 2024-09-15T03:46:50.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8391000000000001e-05 2024-09-15T03:46:50.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203832+0000", 2024-09-15T03:46:50.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8310000000000001e-06 2024-09-15T03:46:50.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:50.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203907+0000", 2024-09-15T03:46:50.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5160999999999994e-05 2024-09-15T03:46:50.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203917+0000", 2024-09-15T03:46:50.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.91e-06 2024-09-15T03:46:50.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.204019+0000", 2024-09-15T03:46:50.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101928 2024-09-15T03:46:50.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.204034+0000", 2024-09-15T03:46:50.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5078e-05 2024-09-15T03:46:50.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.204127+0000", 2024-09-15T03:46:50.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2918999999999996e-05 2024-09-15T03:46:50.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.204217+0000", 2024-09-15T03:46:50.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9738e-05 2024-09-15T03:46:50.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:50.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:50.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:50.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.212635+0000", 2024-09-15T03:46:50.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084180069999999999 2024-09-15T03:46:50.014 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:50.014 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:50.014 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:50.014 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:50.014 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:50.015 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:50.016 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:46:50.017 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:50.015+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:46:50.116 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:50.116+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:46:50.124 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:50.124 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:50.124 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:46:50.125 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:50.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:46:50.156 INFO:teuthology.orchestra.run.smithi001.stderr:set 3.6 pg_upmap_items mapping to [7->10,3->5,0->6,1->11,8->9,2->4] 2024-09-15T03:46:50.157 INFO:tasks.rados.rados.0.smithi001.stdout:113: done (1 left) 2024-09-15T03:46:50.157 INFO:tasks.rados.rados.0.smithi001.stdout:114: done (0 left) 2024-09-15T03:46:50.157 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 34 current snap is 8 2024-09-15T03:46:50.157 INFO:tasks.rados.rados.0.smithi001.stdout:115: seq_num 70 ranges {0=802816,802816=376832} 2024-09-15T03:46:50.162 INFO:tasks.rados.rados.0.smithi001.stdout:115: writing smithi00132172-34 from 0 to 802816 tid 1 2024-09-15T03:46:50.164 INFO:tasks.rados.rados.0.smithi001.stdout:115: writing smithi00132172-34 from 802816 to 1179648 tid 2 2024-09-15T03:46:50.165 INFO:tasks.rados.rados.0.smithi001.stdout:116: copy_from oid 26 from oid 46 current snap is 8 2024-09-15T03:46:50.165 INFO:tasks.rados.rados.0.smithi001.stdout:117: snap_create 2024-09-15T03:46:50.168 INFO:tasks.rados.rados.0.smithi001.stdout:115: finishing write tid 1 to smithi00132172-34 2024-09-15T03:46:50.168 INFO:tasks.rados.rados.0.smithi001.stdout:115: finishing write tid 2 to smithi00132172-34 2024-09-15T03:46:50.168 INFO:tasks.rados.rados.0.smithi001.stdout:115: finishing write tid 3 to smithi00132172-34 2024-09-15T03:46:50.170 INFO:tasks.rados.rados.0.smithi001.stdout:115: finishing write tid 4 to smithi00132172-34 2024-09-15T03:46:50.170 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 34 v 8 (ObjNum 70 snap 8 seq_num 70) dirty exists 2024-09-15T03:46:50.170 INFO:tasks.rados.rados.0.smithi001.stdout:115: left oid 34 (ObjNum 70 snap 8 seq_num 70) 2024-09-15T03:46:50.180 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T03:46:50.217 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:50.217+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:46:50.233 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:50.233 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:50.233 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:46:50.233 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:46:50.233 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:50.241 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:46:50.319 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:50.318+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:46:50.340 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:50.340 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:46:50.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:46:50.340 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:46:50.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.341 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:50.341 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:46:50.341 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 32.435519169000003, 2024-09-15T03:46:50.341 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10988437500000001, 2024-09-15T03:46:50.341 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.341 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:50.341 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:50.341 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:50.341 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:50.341 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1 2024-09-15T03:46:50.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.341 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.341 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.342 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.342 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:46:50.342 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.342 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.342 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:46:50.342 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.342 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.342 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902355+0000", 2024-09-15T03:46:50.342 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.429e-06 2024-09-15T03:46:50.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902362+0000", 2024-09-15T03:46:50.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7290000000000003e-06 2024-09-15T03:46:50.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902373+0000", 2024-09-15T03:46:50.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0533999999999999e-05 2024-09-15T03:46:50.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902379+0000", 2024-09-15T03:46:50.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4350000000000002e-06 2024-09-15T03:46:50.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902428+0000", 2024-09-15T03:46:50.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8571e-05 2024-09-15T03:46:50.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:46:50.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902434+0000", 2024-09-15T03:46:50.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5610000000000004e-06 2024-09-15T03:46:50.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011307+0000", 2024-09-15T03:46:50.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.108872306 2024-09-15T03:46:50.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:50.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011448+0000", 2024-09-15T03:46:50.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014137199999999999 2024-09-15T03:46:50.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:50.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011569+0000", 2024-09-15T03:46:50.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120925 2024-09-15T03:46:50.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011695+0000", 2024-09-15T03:46:50.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012586800000000001 2024-09-15T03:46:50.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011703+0000", 2024-09-15T03:46:50.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4190000000000002e-06 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011739+0000", 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5474e-05 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011940+0000", 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020108699999999999 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012179+0000", 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023950299999999999 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012215+0000", 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5332e-05 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012238+0000", 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3830000000000001e-05 2024-09-15T03:46:50.349 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4263.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 26.294122482999999, 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019495788999999999, 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4263", 2024-09-15T03:46:50.350 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/646873808", 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043752+0000", 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.815e-06 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043757+0000", 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4240000000000003e-06 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043766+0000", 2024-09-15T03:46:50.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8629999999999997e-06 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043773+0000", 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0099999999999998e-06 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043830+0000", 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7089999999999997e-05 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043837+0000", 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.08e-06 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063103+0000", 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019265619000000001 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063246+0000", 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000143888 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.354 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.fs5 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.230274576999999, 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098369180000000001, 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4499999999999998e-07 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107630+0000", 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0877999999999998e-05 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107642+0000", 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2567e-05 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107648+0000", 2024-09-15T03:46:50.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8690000000000001e-06 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107894+0000", 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024540400000000001 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107900+0000", 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5710000000000003e-06 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107968+0000", 2024-09-15T03:46:50.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7957e-05 2024-09-15T03:46:50.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108274+0000", 2024-09-15T03:46:50.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030591700000000001 2024-09-15T03:46:50.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108504+0000", 2024-09-15T03:46:50.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022952199999999999 2024-09-15T03:46:50.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109438+0000", 2024-09-15T03:46:50.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00093439100000000002 2024-09-15T03:46:50.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117435+0000", 2024-09-15T03:46:50.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0079974969999999992 2024-09-15T03:46:50.360 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.360 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.360 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.361 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:50.361 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:46:50.361 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.230217325, 2024-09-15T03:46:50.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084894540000000004, 2024-09-15T03:46:50.361 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.361 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.361 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:46:50.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:46:50.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:46:50.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4700000000000002e-07 2024-09-15T03:46:50.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107670+0000", 2024-09-15T03:46:50.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3892e-05 2024-09-15T03:46:50.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.363 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107678+0000", 2024-09-15T03:46:50.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9589999999999995e-06 2024-09-15T03:46:50.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.363 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107683+0000", 2024-09-15T03:46:50.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7199999999999997e-06 2024-09-15T03:46:50.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.363 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107768+0000", 2024-09-15T03:46:50.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4968000000000004e-05 2024-09-15T03:46:50.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.364 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.364 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107776+0000", 2024-09-15T03:46:50.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9109999999999993e-06 2024-09-15T03:46:50.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.364 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.364 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107858+0000", 2024-09-15T03:46:50.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2349999999999996e-05 2024-09-15T03:46:50.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108273+0000", 2024-09-15T03:46:50.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00041525600000000002 2024-09-15T03:46:50.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108503+0000", 2024-09-15T03:46:50.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022990599999999999 2024-09-15T03:46:50.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108705+0000", 2024-09-15T03:46:50.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020173000000000001 2024-09-15T03:46:50.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.366 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.116145+0000", 2024-09-15T03:46:50.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0074404149999999997 2024-09-15T03:46:50.366 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.366 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.366 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.366 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.fs5 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:50.366 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.109087+0000", 2024-09-15T03:46:50.366 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.228786331, 2024-09-15T03:46:50.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084235809999999994, 2024-09-15T03:46:50.367 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.367 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.367 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109087+0000", 2024-09-15T03:46:50.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109087+0000", 2024-09-15T03:46:50.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109087+0000", 2024-09-15T03:46:50.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7599999999999997e-07 2024-09-15T03:46:50.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109107+0000", 2024-09-15T03:46:50.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9511999999999999e-05 2024-09-15T03:46:50.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109127+0000", 2024-09-15T03:46:50.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0268e-05 2024-09-15T03:46:50.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109136+0000", 2024-09-15T03:46:50.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1330000000000005e-06 2024-09-15T03:46:50.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109594+0000", 2024-09-15T03:46:50.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045802100000000002 2024-09-15T03:46:50.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109602+0000", 2024-09-15T03:46:50.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7020000000000001e-06 2024-09-15T03:46:50.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109678+0000", 2024-09-15T03:46:50.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5428999999999999e-05 2024-09-15T03:46:50.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109699+0000", 2024-09-15T03:46:50.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1759999999999998e-05 2024-09-15T03:46:50.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109728+0000", 2024-09-15T03:46:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8886000000000001e-05 2024-09-15T03:46:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109736+0000", 2024-09-15T03:46:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6310000000000003e-06 2024-09-15T03:46:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117510+0000", 2024-09-15T03:46:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.007774563 2024-09-15T03:46:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.109145+0000", 2024-09-15T03:46:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.228728423, 2024-09-15T03:46:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075088480000000003, 2024-09-15T03:46:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109145+0000", 2024-09-15T03:46:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109145+0000", 2024-09-15T03:46:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109145+0000", 2024-09-15T03:46:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7399999999999999e-07 2024-09-15T03:46:50.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109160+0000", 2024-09-15T03:46:50.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4544e-05 2024-09-15T03:46:50.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109168+0000", 2024-09-15T03:46:50.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9200000000000004e-06 2024-09-15T03:46:50.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109172+0000", 2024-09-15T03:46:50.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.498e-06 2024-09-15T03:46:50.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109221+0000", 2024-09-15T03:46:50.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9348000000000003e-05 2024-09-15T03:46:50.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109230+0000", 2024-09-15T03:46:50.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2020000000000004e-06 2024-09-15T03:46:50.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109314+0000", 2024-09-15T03:46:50.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4445999999999998e-05 2024-09-15T03:46:50.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109570+0000", 2024-09-15T03:46:50.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00025616300000000001 2024-09-15T03:46:50.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109642+0000", 2024-09-15T03:46:50.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1600999999999994e-05 2024-09-15T03:46:50.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109653+0000", 2024-09-15T03:46:50.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1048e-05 2024-09-15T03:46:50.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.116654+0000", 2024-09-15T03:46:50.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.007000704 2024-09-15T03:46:50.377 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.377 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.377 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.378 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es5 34/28 ECSubWrite(tid=33, reqid=client.4521.0:195, at_version=34'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-15T03:46:50.378 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.118140+0000", 2024-09-15T03:46:50.378 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.2197330290000004, 2024-09-15T03:46:50.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069410629999999999, 2024-09-15T03:46:50.378 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.378 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.378 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.378 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118140+0000", 2024-09-15T03:46:50.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118140+0000", 2024-09-15T03:46:50.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118140+0000", 2024-09-15T03:46:50.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.29e-07 2024-09-15T03:46:50.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118354+0000", 2024-09-15T03:46:50.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000213291 2024-09-15T03:46:50.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118363+0000", 2024-09-15T03:46:50.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4739999999999994e-06 2024-09-15T03:46:50.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118368+0000", 2024-09-15T03:46:50.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2499999999999997e-06 2024-09-15T03:46:50.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118418+0000", 2024-09-15T03:46:50.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0166e-05 2024-09-15T03:46:50.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118427+0000", 2024-09-15T03:46:50.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0459999999999994e-06 2024-09-15T03:46:50.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118648+0000", 2024-09-15T03:46:50.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022084599999999999 2024-09-15T03:46:50.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118656+0000", 2024-09-15T03:46:50.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9899999999999997e-06 2024-09-15T03:46:50.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118725+0000", 2024-09-15T03:46:50.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8569e-05 2024-09-15T03:46:50.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118742+0000", 2024-09-15T03:46:50.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6776999999999999e-05 2024-09-15T03:46:50.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125081+0000", 2024-09-15T03:46:50.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0063395250000000004 2024-09-15T03:46:50.383 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.383 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.383 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.383 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 34/31 ECSubWrite(tid=34, reqid=client.4521.0:197, at_version=34'21, trim_to=0'0, roll_forward_to=34'18))", 2024-09-15T03:46:50.383 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.124009+0000", 2024-09-15T03:46:50.384 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.2138639799999993, 2024-09-15T03:46:50.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0066902589999999996, 2024-09-15T03:46:50.384 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.384 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.384 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124009+0000", 2024-09-15T03:46:50.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124009+0000", 2024-09-15T03:46:50.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124010+0000", 2024-09-15T03:46:50.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.63e-07 2024-09-15T03:46:50.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124031+0000", 2024-09-15T03:46:50.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1043000000000002e-05 2024-09-15T03:46:50.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124043+0000", 2024-09-15T03:46:50.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2500999999999999e-05 2024-09-15T03:46:50.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124049+0000", 2024-09-15T03:46:50.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.331e-06 2024-09-15T03:46:50.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124090+0000", 2024-09-15T03:46:50.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0692999999999998e-05 2024-09-15T03:46:50.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124097+0000", 2024-09-15T03:46:50.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.216e-06 2024-09-15T03:46:50.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124657+0000", 2024-09-15T03:46:50.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00055963300000000003 2024-09-15T03:46:50.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124664+0000", 2024-09-15T03:46:50.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0310000000000001e-06 2024-09-15T03:46:50.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124694+0000", 2024-09-15T03:46:50.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0383999999999999e-05 2024-09-15T03:46:50.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124707+0000", 2024-09-15T03:46:50.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2098e-05 2024-09-15T03:46:50.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.130699+0000", 2024-09-15T03:46:50.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0059928660000000003 2024-09-15T03:46:50.389 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.389 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.389 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.389 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/31 ECSubWrite(tid=38, reqid=client.4521.0:223, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:46:50.389 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:46:50.389 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.1906661459999999, 2024-09-15T03:46:50.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010504493, 2024-09-15T03:46:50.390 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.390 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.390 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:46:50.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:46:50.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:46:50.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4999999999999999e-07 2024-09-15T03:46:50.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147648+0000", 2024-09-15T03:46:50.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00044030399999999998 2024-09-15T03:46:50.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147660+0000", 2024-09-15T03:46:50.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2517000000000001e-05 2024-09-15T03:46:50.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147667+0000", 2024-09-15T03:46:50.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.737e-06 2024-09-15T03:46:50.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147713+0000", 2024-09-15T03:46:50.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6405e-05 2024-09-15T03:46:50.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147719+0000", 2024-09-15T03:46:50.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0580000000000003e-06 2024-09-15T03:46:50.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147905+0000", 2024-09-15T03:46:50.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018574600000000001 2024-09-15T03:46:50.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147917+0000", 2024-09-15T03:46:50.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1749999999999999e-05 2024-09-15T03:46:50.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148022+0000", 2024-09-15T03:46:50.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104863 2024-09-15T03:46:50.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148037+0000", 2024-09-15T03:46:50.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5302999999999999e-05 2024-09-15T03:46:50.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157711+0000", 2024-09-15T03:46:50.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0096745600000000005 2024-09-15T03:46:50.395 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.395 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.395 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.395 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/31 ECSubWrite(tid=39, reqid=client.4521.0:224, at_version=38'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:46:50.395 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.149886+0000", 2024-09-15T03:46:50.395 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.1879867190000004, 2024-09-15T03:46:50.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.008019155, 2024-09-15T03:46:50.395 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.395 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.395 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149886+0000", 2024-09-15T03:46:50.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149886+0000", 2024-09-15T03:46:50.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149887+0000", 2024-09-15T03:46:50.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3399999999999999e-07 2024-09-15T03:46:50.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150075+0000", 2024-09-15T03:46:50.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018829200000000001 2024-09-15T03:46:50.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150089+0000", 2024-09-15T03:46:50.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3886000000000001e-05 2024-09-15T03:46:50.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150096+0000", 2024-09-15T03:46:50.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7429999999999996e-06 2024-09-15T03:46:50.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150137+0000", 2024-09-15T03:46:50.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1170000000000001e-05 2024-09-15T03:46:50.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150146+0000", 2024-09-15T03:46:50.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8610000000000004e-06 2024-09-15T03:46:50.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150218+0000", 2024-09-15T03:46:50.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2588000000000004e-05 2024-09-15T03:46:50.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150224+0000", 2024-09-15T03:46:50.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9390000000000002e-06 2024-09-15T03:46:50.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150308+0000", 2024-09-15T03:46:50.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3222999999999999e-05 2024-09-15T03:46:50.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150318+0000", 2024-09-15T03:46:50.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0407e-05 2024-09-15T03:46:50.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157906+0000", 2024-09-15T03:46:50.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075875120000000002 2024-09-15T03:46:50.400 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.400 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.401 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.401 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 38/31 ECSubWrite(tid=37, reqid=client.4521.0:220, at_version=0'0, trim_to=0'0, roll_forward_to=38'23))", 2024-09-15T03:46:50.401 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.151298+0000", 2024-09-15T03:46:50.401 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.1865751500000004, 2024-09-15T03:46:50.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0066381979999999997, 2024-09-15T03:46:50.401 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.401 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.401 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.402 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151298+0000", 2024-09-15T03:46:50.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.402 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.402 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151298+0000", 2024-09-15T03:46:50.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151298+0000", 2024-09-15T03:46:50.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5400000000000002e-07 2024-09-15T03:46:50.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151306+0000", 2024-09-15T03:46:50.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5689999999999999e-06 2024-09-15T03:46:50.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151315+0000", 2024-09-15T03:46:50.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4239999999999999e-06 2024-09-15T03:46:50.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151321+0000", 2024-09-15T03:46:50.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.118e-06 2024-09-15T03:46:50.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151364+0000", 2024-09-15T03:46:50.405 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3509e-05 2024-09-15T03:46:50.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151372+0000", 2024-09-15T03:46:50.405 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2239999999999995e-06 2024-09-15T03:46:50.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151461+0000", 2024-09-15T03:46:50.405 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8721000000000007e-05 2024-09-15T03:46:50.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.406 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.406 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151468+0000", 2024-09-15T03:46:50.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0330000000000003e-06 2024-09-15T03:46:50.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.406 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.406 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.153509+0000", 2024-09-15T03:46:50.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020411549999999998 2024-09-15T03:46:50.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.406 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.406 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.153529+0000", 2024-09-15T03:46:50.407 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0072e-05 2024-09-15T03:46:50.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.407 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.407 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157936+0000", 2024-09-15T03:46:50.407 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044069189999999996 2024-09-15T03:46:50.407 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.407 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.407 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.407 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/31 ECSubWrite(tid=40, reqid=client.4521.0:225, at_version=38'10, trim_to=0'0, roll_forward_to=38'8))", 2024-09-15T03:46:50.407 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.151586+0000", 2024-09-15T03:46:50.408 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.1862868710000001, 2024-09-15T03:46:50.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064316060000000003, 2024-09-15T03:46:50.408 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.408 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.408 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.408 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151586+0000", 2024-09-15T03:46:50.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.408 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151586+0000", 2024-09-15T03:46:50.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151587+0000", 2024-09-15T03:46:50.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6400000000000003e-07 2024-09-15T03:46:50.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151600+0000", 2024-09-15T03:46:50.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3662999999999999e-05 2024-09-15T03:46:50.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151609+0000", 2024-09-15T03:46:50.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9120000000000004e-06 2024-09-15T03:46:50.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151619+0000", 2024-09-15T03:46:50.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0107e-05 2024-09-15T03:46:50.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151654+0000", 2024-09-15T03:46:50.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4295000000000003e-05 2024-09-15T03:46:50.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151662+0000", 2024-09-15T03:46:50.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6959999999999994e-06 2024-09-15T03:46:50.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151761+0000", 2024-09-15T03:46:50.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8356000000000005e-05 2024-09-15T03:46:50.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.153516+0000", 2024-09-15T03:46:50.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001754986 2024-09-15T03:46:50.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.153546+0000", 2024-09-15T03:46:50.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0028000000000001e-05 2024-09-15T03:46:50.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.153557+0000", 2024-09-15T03:46:50.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1561e-05 2024-09-15T03:46:50.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.413 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.413 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158018+0000", 2024-09-15T03:46:50.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.004460538 2024-09-15T03:46:50.413 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.413 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.413 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.413 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es5 38/28 ECSubWrite(tid=45, reqid=client.4521.0:227, at_version=38'24, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:46:50.413 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.155440+0000", 2024-09-15T03:46:50.413 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.1824328140000002, 2024-09-15T03:46:50.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0059398020000000001, 2024-09-15T03:46:50.413 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.413 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.414 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155440+0000", 2024-09-15T03:46:50.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155440+0000", 2024-09-15T03:46:50.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155441+0000", 2024-09-15T03:46:50.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.077e-06 2024-09-15T03:46:50.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155981+0000", 2024-09-15T03:46:50.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00053971599999999998 2024-09-15T03:46:50.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155994+0000", 2024-09-15T03:46:50.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3339000000000001e-05 2024-09-15T03:46:50.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156000+0000", 2024-09-15T03:46:50.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9529999999999996e-06 2024-09-15T03:46:50.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156048+0000", 2024-09-15T03:46:50.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7981e-05 2024-09-15T03:46:50.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156054+0000", 2024-09-15T03:46:50.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5169999999999999e-06 2024-09-15T03:46:50.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157320+0000", 2024-09-15T03:46:50.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00126636 2024-09-15T03:46:50.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157331+0000", 2024-09-15T03:46:50.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0577000000000001e-05 2024-09-15T03:46:50.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157434+0000", 2024-09-15T03:46:50.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010285000000000001 2024-09-15T03:46:50.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.418 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157449+0000", 2024-09-15T03:46:50.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5545000000000001e-05 2024-09-15T03:46:50.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.418 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.161380+0000", 2024-09-15T03:46:50.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0039308870000000001 2024-09-15T03:46:50.418 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.418 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.418 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.419 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 39/28 ECSubWrite(tid=42, reqid=client.4521.0:255, at_version=39'19, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:46:50.419 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.170708+0000", 2024-09-15T03:46:50.419 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.1671653590000002, 2024-09-15T03:46:50.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0093655969999999998, 2024-09-15T03:46:50.419 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.419 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.419 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.419 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170708+0000", 2024-09-15T03:46:50.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170708+0000", 2024-09-15T03:46:50.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170708+0000", 2024-09-15T03:46:50.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8899999999999999e-07 2024-09-15T03:46:50.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170728+0000", 2024-09-15T03:46:50.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9184000000000001e-05 2024-09-15T03:46:50.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170741+0000", 2024-09-15T03:46:50.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3696999999999999e-05 2024-09-15T03:46:50.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170747+0000", 2024-09-15T03:46:50.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1569999999999997e-06 2024-09-15T03:46:50.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170803+0000", 2024-09-15T03:46:50.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5974000000000003e-05 2024-09-15T03:46:50.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170811+0000", 2024-09-15T03:46:50.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8259999999999993e-06 2024-09-15T03:46:50.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170902+0000", 2024-09-15T03:46:50.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1116e-05 2024-09-15T03:46:50.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170909+0000", 2024-09-15T03:46:50.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3489999999999997e-06 2024-09-15T03:46:50.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173637+0000", 2024-09-15T03:46:50.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027282270000000002 2024-09-15T03:46:50.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.424 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.424 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.424 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173649+0000", 2024-09-15T03:46:50.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2561000000000001e-05 2024-09-15T03:46:50.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.424 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.424 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.424 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180073+0000", 2024-09-15T03:46:50.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064239170000000003 2024-09-15T03:46:50.424 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.425 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.425 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.425 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.425 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.426 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:46:50.426 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:46:50.426 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.1670490940000002, 2024-09-15T03:46:50.426 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011878227, 2024-09-15T03:46:50.426 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.426 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.426 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.426 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.426 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.426 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:46:50.426 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.426 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.426 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.426 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:46:50.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.427 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170825+0000", 2024-09-15T03:46:50.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9599999999999999e-07 2024-09-15T03:46:50.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.427 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171100+0000", 2024-09-15T03:46:50.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027548500000000001 2024-09-15T03:46:50.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171112+0000", 2024-09-15T03:46:50.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2152e-05 2024-09-15T03:46:50.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171119+0000", 2024-09-15T03:46:50.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5050000000000004e-06 2024-09-15T03:46:50.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171162+0000", 2024-09-15T03:46:50.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2908000000000003e-05 2024-09-15T03:46:50.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171169+0000", 2024-09-15T03:46:50.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9739999999999996e-06 2024-09-15T03:46:50.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171398+0000", 2024-09-15T03:46:50.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022954799999999999 2024-09-15T03:46:50.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173640+0000", 2024-09-15T03:46:50.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022417000000000001 2024-09-15T03:46:50.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173705+0000", 2024-09-15T03:46:50.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4907999999999996e-05 2024-09-15T03:46:50.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173742+0000", 2024-09-15T03:46:50.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6940999999999997e-05 2024-09-15T03:46:50.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.182702+0000", 2024-09-15T03:46:50.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0089604100000000002 2024-09-15T03:46:50.431 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.431 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.431 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.432 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 39/28 ECSubWrite(tid=43, reqid=client.4521.0:256, at_version=39'20, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:46:50.432 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.171196+0000", 2024-09-15T03:46:50.432 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.1666770089999998, 2024-09-15T03:46:50.432 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0090205690000000009, 2024-09-15T03:46:50.432 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.432 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.432 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171196+0000", 2024-09-15T03:46:50.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171196+0000", 2024-09-15T03:46:50.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171197+0000", 2024-09-15T03:46:50.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3e-07 2024-09-15T03:46:50.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171218+0000", 2024-09-15T03:46:50.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0775e-05 2024-09-15T03:46:50.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171241+0000", 2024-09-15T03:46:50.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3538999999999999e-05 2024-09-15T03:46:50.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171248+0000", 2024-09-15T03:46:50.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6220000000000003e-06 2024-09-15T03:46:50.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171291+0000", 2024-09-15T03:46:50.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3115000000000003e-05 2024-09-15T03:46:50.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171301+0000", 2024-09-15T03:46:50.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.662e-06 2024-09-15T03:46:50.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171395+0000", 2024-09-15T03:46:50.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4368000000000002e-05 2024-09-15T03:46:50.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173639+0000", 2024-09-15T03:46:50.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022444040000000002 2024-09-15T03:46:50.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173704+0000", 2024-09-15T03:46:50.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5031999999999997e-05 2024-09-15T03:46:50.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173718+0000", 2024-09-15T03:46:50.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3159000000000001e-05 2024-09-15T03:46:50.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180217+0000", 2024-09-15T03:46:50.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064991629999999996 2024-09-15T03:46:50.437 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.437 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.438 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.438 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.438 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 39/28 ECSubWrite(tid=44, reqid=client.4521.0:256, at_version=0'0, trim_to=0'0, roll_forward_to=39'20))", 2024-09-15T03:46:50.438 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.174176+0000", 2024-09-15T03:46:50.438 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.1636967890000003, 2024-09-15T03:46:50.438 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0061969080000000001, 2024-09-15T03:46:50.438 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.438 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.438 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.438 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.438 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.438 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174176+0000", 2024-09-15T03:46:50.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174176+0000", 2024-09-15T03:46:50.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174177+0000", 2024-09-15T03:46:50.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0999999999999998e-07 2024-09-15T03:46:50.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.440 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174187+0000", 2024-09-15T03:46:50.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0077e-05 2024-09-15T03:46:50.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.440 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174201+0000", 2024-09-15T03:46:50.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3499999999999999e-05 2024-09-15T03:46:50.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.440 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174207+0000", 2024-09-15T03:46:50.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7360000000000004e-06 2024-09-15T03:46:50.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174253+0000", 2024-09-15T03:46:50.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5223999999999999e-05 2024-09-15T03:46:50.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174262+0000", 2024-09-15T03:46:50.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8369999999999992e-06 2024-09-15T03:46:50.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174370+0000", 2024-09-15T03:46:50.442 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000107611 2024-09-15T03:46:50.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.442 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.442 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.442 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174377+0000", 2024-09-15T03:46:50.442 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.759e-06 2024-09-15T03:46:50.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.442 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.442 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.442 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174406+0000", 2024-09-15T03:46:50.442 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8816e-05 2024-09-15T03:46:50.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.442 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.442 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174413+0000", 2024-09-15T03:46:50.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6669999999999996e-06 2024-09-15T03:46:50.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180373+0000", 2024-09-15T03:46:50.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0059599709999999997 2024-09-15T03:46:50.443 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.443 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.443 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.443 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 39/31 ECSubWrite(tid=44, reqid=client.4521.0:243, at_version=0'0, trim_to=0'0, roll_forward_to=39'17))", 2024-09-15T03:46:50.444 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.176341+0000", 2024-09-15T03:46:50.444 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.161532148, 2024-09-15T03:46:50.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006411043, 2024-09-15T03:46:50.444 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.444 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.444 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176341+0000", 2024-09-15T03:46:50.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176341+0000", 2024-09-15T03:46:50.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176342+0000", 2024-09-15T03:46:50.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4000000000000001e-07 2024-09-15T03:46:50.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176356+0000", 2024-09-15T03:46:50.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4756999999999999e-05 2024-09-15T03:46:50.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.446 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176364+0000", 2024-09-15T03:46:50.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6939999999999995e-06 2024-09-15T03:46:50.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.446 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176371+0000", 2024-09-15T03:46:50.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6959999999999999e-06 2024-09-15T03:46:50.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.446 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176404+0000", 2024-09-15T03:46:50.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2972000000000002e-05 2024-09-15T03:46:50.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176413+0000", 2024-09-15T03:46:50.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2499999999999995e-06 2024-09-15T03:46:50.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176496+0000", 2024-09-15T03:46:50.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3287000000000004e-05 2024-09-15T03:46:50.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176524+0000", 2024-09-15T03:46:50.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7659999999999999e-05 2024-09-15T03:46:50.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.448 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176556+0000", 2024-09-15T03:46:50.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1687999999999999e-05 2024-09-15T03:46:50.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.448 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176573+0000", 2024-09-15T03:46:50.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7680999999999999e-05 2024-09-15T03:46:50.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.182752+0000", 2024-09-15T03:46:50.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0061787179999999997 2024-09-15T03:46:50.449 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.449 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.449 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.449 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 39/31 ECSubWrite(tid=43, reqid=client.4521.0:252, at_version=39'27, trim_to=0'0, roll_forward_to=39'25))", 2024-09-15T03:46:50.449 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:46:50.449 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.1593317760000001, 2024-09-15T03:46:50.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011286757, 2024-09-15T03:46:50.449 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.449 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.450 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:46:50.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:46:50.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178542+0000", 2024-09-15T03:46:50.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9400000000000004e-07 2024-09-15T03:46:50.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178754+0000", 2024-09-15T03:46:50.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021228800000000001 2024-09-15T03:46:50.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178765+0000", 2024-09-15T03:46:50.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0176e-05 2024-09-15T03:46:50.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178769+0000", 2024-09-15T03:46:50.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1359999999999999e-06 2024-09-15T03:46:50.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178810+0000", 2024-09-15T03:46:50.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0825999999999998e-05 2024-09-15T03:46:50.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178820+0000", 2024-09-15T03:46:50.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0237999999999999e-05 2024-09-15T03:46:50.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179033+0000", 2024-09-15T03:46:50.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021361500000000001 2024-09-15T03:46:50.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179040+0000", 2024-09-15T03:46:50.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0509999999999999e-06 2024-09-15T03:46:50.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179102+0000", 2024-09-15T03:46:50.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1977000000000006e-05 2024-09-15T03:46:50.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179114+0000", 2024-09-15T03:46:50.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1223000000000001e-05 2024-09-15T03:46:50.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.189828+0000", 2024-09-15T03:46:50.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010714433000000001 2024-09-15T03:46:50.454 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.454 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.454 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.455 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 39/31 ECSubWrite(tid=44, reqid=client.4521.0:252, at_version=0'0, trim_to=0'0, roll_forward_to=39'27))", 2024-09-15T03:46:50.455 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.179948+0000", 2024-09-15T03:46:50.455 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.1579246770000005, 2024-09-15T03:46:50.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010068739, 2024-09-15T03:46:50.455 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.455 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.455 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179948+0000", 2024-09-15T03:46:50.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179948+0000", 2024-09-15T03:46:50.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179949+0000", 2024-09-15T03:46:50.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6199999999999998e-07 2024-09-15T03:46:50.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179960+0000", 2024-09-15T03:46:50.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1216000000000001e-05 2024-09-15T03:46:50.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179970+0000", 2024-09-15T03:46:50.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8980000000000008e-06 2024-09-15T03:46:50.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179975+0000", 2024-09-15T03:46:50.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5650000000000003e-06 2024-09-15T03:46:50.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180004+0000", 2024-09-15T03:46:50.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9224999999999999e-05 2024-09-15T03:46:50.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180009+0000", 2024-09-15T03:46:50.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9250000000000003e-06 2024-09-15T03:46:50.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180061+0000", 2024-09-15T03:46:50.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1758000000000001e-05 2024-09-15T03:46:50.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.459 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180067+0000", 2024-09-15T03:46:50.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3269999999999997e-06 2024-09-15T03:46:50.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.459 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180101+0000", 2024-09-15T03:46:50.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3701000000000001e-05 2024-09-15T03:46:50.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.459 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180112+0000", 2024-09-15T03:46:50.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1287000000000001e-05 2024-09-15T03:46:50.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.460 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.190017+0000", 2024-09-15T03:46:50.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0099052749999999998 2024-09-15T03:46:50.460 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.460 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.460 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.460 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.460 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.460 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:50.462 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:46:50.462 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:50.461+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:46:50.563 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:46:50.562+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:46:50.599 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:50.599 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:50.599 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:46:50.599 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:50.609 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:46:50.663 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:50.663+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:46:50.705 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:50.705 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:50.705 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:46:50.705 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:46:50.705 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:50.713 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:46:50.764 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:50.764+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:46:50.812 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:50.812 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:46:50.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:46:50.812 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:46:50.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.812 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 32/31 ECSubWrite(tid=42, reqid=client.4521.0:166, at_version=32'13, trim_to=0'0, roll_forward_to=32'12))", 2024-09-15T03:46:50.812 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.532201+0000", 2024-09-15T03:46:50.812 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.277759598999999, 2024-09-15T03:46:50.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0036328060000000001, 2024-09-15T03:46:50.816 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.816 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.816 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.816 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532201+0000", 2024-09-15T03:46:50.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532201+0000", 2024-09-15T03:46:50.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532202+0000", 2024-09-15T03:46:50.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.44e-07 2024-09-15T03:46:50.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532214+0000", 2024-09-15T03:46:50.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2689e-05 2024-09-15T03:46:50.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.818 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532224+0000", 2024-09-15T03:46:50.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3549999999999994e-06 2024-09-15T03:46:50.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.818 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532228+0000", 2024-09-15T03:46:50.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8619999999999996e-06 2024-09-15T03:46:50.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532268+0000", 2024-09-15T03:46:50.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0899000000000003e-05 2024-09-15T03:46:50.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532277+0000", 2024-09-15T03:46:50.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6410000000000008e-06 2024-09-15T03:46:50.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532354+0000", 2024-09-15T03:46:50.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6904999999999996e-05 2024-09-15T03:46:50.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532361+0000", 2024-09-15T03:46:50.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.798e-06 2024-09-15T03:46:50.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532387+0000", 2024-09-15T03:46:50.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6346000000000002e-05 2024-09-15T03:46:50.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.821 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.532400+0000", 2024-09-15T03:46:50.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2655e-05 2024-09-15T03:46:50.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.821 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.535834+0000", 2024-09-15T03:46:50.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0034340120000000002 2024-09-15T03:46:50.821 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.821 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:175 3.cs0 3:3164c049:::smithi00132172-34:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e32)", 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.553220+0000", 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.256740382, 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0052725100000000002, 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:50.822 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 175 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553220+0000", 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553220+0000", 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553221+0000", 2024-09-15T03:46:50.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0860000000000001e-06 2024-09-15T03:46:50.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553226+0000", 2024-09-15T03:46:50.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1630000000000004e-06 2024-09-15T03:46:50.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553239+0000", 2024-09-15T03:46:50.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2187e-05 2024-09-15T03:46:50.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553244+0000", 2024-09-15T03:46:50.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8960000000000002e-06 2024-09-15T03:46:50.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553276+0000", 2024-09-15T03:46:50.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2237000000000003e-05 2024-09-15T03:46:50.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:50.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553345+0000", 2024-09-15T03:46:50.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9618999999999999e-05 2024-09-15T03:46:50.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553695+0000", 2024-09-15T03:46:50.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034979 2024-09-15T03:46:50.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553780+0000", 2024-09-15T03:46:50.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4633000000000001e-05 2024-09-15T03:46:50.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553787+0000", 2024-09-15T03:46:50.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2409999999999998e-06 2024-09-15T03:46:50.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553828+0000", 2024-09-15T03:46:50.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0564e-05 2024-09-15T03:46:50.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553841+0000", 2024-09-15T03:46:50.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3249e-05 2024-09-15T03:46:50.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:50.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.554266+0000", 2024-09-15T03:46:50.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00042530299999999998 2024-09-15T03:46:50.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.558493+0000", 2024-09-15T03:46:50.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042265419999999998 2024-09-15T03:46:50.829 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.829 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.829 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.829 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s7 32/31 ECSubWrite(tid=44, reqid=client.4521.0:173, at_version=32'7, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:50.829 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.562211+0000", 2024-09-15T03:46:50.829 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.247749675, 2024-09-15T03:46:50.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0038125260000000001, 2024-09-15T03:46:50.829 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.829 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.829 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.830 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.830 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.830 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562211+0000", 2024-09-15T03:46:50.830 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.830 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.830 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.830 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562211+0000", 2024-09-15T03:46:50.830 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.830 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.830 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.830 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562212+0000", 2024-09-15T03:46:50.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2099999999999996e-07 2024-09-15T03:46:50.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562643+0000", 2024-09-15T03:46:50.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043174199999999999 2024-09-15T03:46:50.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562654+0000", 2024-09-15T03:46:50.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0927999999999999e-05 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562666+0000", 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2261e-05 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562709+0000", 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2885e-05 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.832 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562716+0000", 2024-09-15T03:46:50.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6490000000000002e-06 2024-09-15T03:46:50.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.833 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562882+0000", 2024-09-15T03:46:50.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016565500000000001 2024-09-15T03:46:50.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.833 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562888+0000", 2024-09-15T03:46:50.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.703e-06 2024-09-15T03:46:50.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562972+0000", 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3724e-05 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.562984+0000", 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2361000000000001e-05 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.566023+0000", 2024-09-15T03:46:50.834 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030389969999999999 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.108773+0000", 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.701187763, 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0047381410000000004, 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.835 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108773+0000", 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108773+0000", 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.836 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108774+0000", 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8800000000000002e-07 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108801+0000", 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7339999999999999e-05 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108816+0000", 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5359999999999999e-05 2024-09-15T03:46:50.837 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108823+0000", 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0060000000000003e-06 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108930+0000", 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000106569 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108940+0000", 2024-09-15T03:46:50.838 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.679e-06 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109030+0000", 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0618999999999994e-05 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109041+0000", 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1011e-05 2024-09-15T03:46:50.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109073+0000", 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1427999999999998e-05 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109088+0000", 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4956e-05 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.840 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.113511+0000", 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044233850000000002 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.109181+0000", 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.700779864999999, 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044348879999999997, 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.841 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109181+0000", 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109181+0000", 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.842 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109181+0000", 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0900000000000001e-07 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109197+0000", 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5150999999999999e-05 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109211+0000", 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3998999999999999e-05 2024-09-15T03:46:50.843 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109217+0000", 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.686e-06 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109254+0000", 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6424999999999998e-05 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.844 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109262+0000", 2024-09-15T03:46:50.845 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8230000000000001e-06 2024-09-15T03:46:50.845 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.845 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.845 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.845 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109350+0000", 2024-09-15T03:46:50.845 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7120999999999995e-05 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109358+0000", 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3399999999999998e-06 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109388+0000", 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9807000000000001e-05 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.846 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109398+0000", 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0744000000000001e-05 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.113616+0000", 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042171830000000002 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es7 34/28 ECSubWrite(tid=33, reqid=client.4521.0:195, at_version=34'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-15T03:46:50.847 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.118211+0000", 2024-09-15T03:46:50.848 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.6917492719999991, 2024-09-15T03:46:50.848 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0070162009999999997, 2024-09-15T03:46:50.848 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.848 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.848 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.848 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.848 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.848 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118211+0000", 2024-09-15T03:46:50.848 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.848 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.848 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.848 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.848 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118211+0000", 2024-09-15T03:46:50.848 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.849 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.849 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.849 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.849 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118212+0000", 2024-09-15T03:46:50.849 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5599999999999999e-07 2024-09-15T03:46:50.849 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.849 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.849 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.849 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118422+0000", 2024-09-15T03:46:50.849 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021081 2024-09-15T03:46:50.849 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.849 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118433+0000", 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0529e-05 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118438+0000", 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8210000000000001e-06 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118479+0000", 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1493999999999999e-05 2024-09-15T03:46:50.850 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118490+0000", 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0550000000000001e-05 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118700+0000", 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021063500000000001 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.851 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118711+0000", 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0849000000000001e-05 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118776+0000", 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4702999999999994e-05 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118797+0000", 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1265999999999999e-05 2024-09-15T03:46:50.852 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125228+0000", 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064303880000000004 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.0s2 39/31 ECSubWrite(tid=43, reqid=client.4521.0:240, at_version=39'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.148448+0000", 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.6615123189999998, 2024-09-15T03:46:50.853 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0041543140000000001, 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.148448+0000", 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.148448+0000", 2024-09-15T03:46:50.854 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.148448+0000", 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7100000000000001e-07 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.148468+0000", 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9656999999999998e-05 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.855 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.148479+0000", 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.046e-05 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.148484+0000", 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5219999999999998e-06 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.148535+0000", 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0519000000000002e-05 2024-09-15T03:46:50.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.148543+0000", 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0830000000000004e-06 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.148763+0000", 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022076499999999999 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.857 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.148797+0000", 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3671999999999999e-05 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.148834+0000", 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7348000000000002e-05 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.148868+0000", 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3346999999999998e-05 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.858 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152603+0000", 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0037347700000000001 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s3 39/31 ECSubWrite(tid=44, reqid=osd.1.0:44, at_version=39'12, trim_to=0'0, roll_forward_to=38'10))", 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.150872+0000", 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.6590888350000004, 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075168539999999999, 2024-09-15T03:46:50.859 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150872+0000", 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150872+0000", 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.860 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150872+0000", 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3300000000000002e-07 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150909+0000", 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6294000000000002e-05 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150919+0000", 2024-09-15T03:46:50.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9690000000000006e-06 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150924+0000", 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7710000000000003e-06 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150958+0000", 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3531999999999997e-05 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.862 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.863 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150970+0000", 2024-09-15T03:46:50.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1831000000000001e-05 2024-09-15T03:46:50.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.863 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.863 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.863 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152463+0000", 2024-09-15T03:46:50.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0014934239999999999 2024-09-15T03:46:50.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.863 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.863 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152471+0000", 2024-09-15T03:46:50.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2830000000000004e-06 2024-09-15T03:46:50.864 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.864 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.864 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152507+0000", 2024-09-15T03:46:50.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6739999999999997e-05 2024-09-15T03:46:50.864 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.864 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.864 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152517+0000", 2024-09-15T03:46:50.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8570000000000007e-06 2024-09-15T03:46:50.864 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.864 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.865 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.865 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.158389+0000", 2024-09-15T03:46:50.865 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058715199999999999 2024-09-15T03:46:50.865 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.865 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.865 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.865 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.0s2 39/31 ECSubWrite(tid=45, reqid=client.4521.0:241, at_version=39'17, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:46:50.865 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.150931+0000", 2024-09-15T03:46:50.865 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.659029716, 2024-09-15T03:46:50.865 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0072693200000000001, 2024-09-15T03:46:50.865 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.866 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.866 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.866 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.866 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.866 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150931+0000", 2024-09-15T03:46:50.866 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.866 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.866 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.866 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.866 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150931+0000", 2024-09-15T03:46:50.866 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.867 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.867 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.867 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150931+0000", 2024-09-15T03:46:50.867 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3699999999999999e-07 2024-09-15T03:46:50.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.867 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.867 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.867 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150942+0000", 2024-09-15T03:46:50.867 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1053000000000001e-05 2024-09-15T03:46:50.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.868 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.868 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.868 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150946+0000", 2024-09-15T03:46:50.868 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0890000000000002e-06 2024-09-15T03:46:50.868 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.868 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.868 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.868 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150949+0000", 2024-09-15T03:46:50.868 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5270000000000001e-06 2024-09-15T03:46:50.868 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.868 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.868 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.868 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150983+0000", 2024-09-15T03:46:50.868 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4258000000000003e-05 2024-09-15T03:46:50.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.869 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.869 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.869 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150994+0000", 2024-09-15T03:46:50.869 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0893999999999999e-05 2024-09-15T03:46:50.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.869 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.869 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.869 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.151057+0000", 2024-09-15T03:46:50.870 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3478000000000004e-05 2024-09-15T03:46:50.870 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.870 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.870 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.870 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.151063+0000", 2024-09-15T03:46:50.870 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7880000000000003e-06 2024-09-15T03:46:50.870 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.870 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.870 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.870 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152305+0000", 2024-09-15T03:46:50.870 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012418850000000001 2024-09-15T03:46:50.870 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.870 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.870 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152335+0000", 2024-09-15T03:46:50.871 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0049e-05 2024-09-15T03:46:50.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.871 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.871 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.158200+0000", 2024-09-15T03:46:50.871 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058650619999999999 2024-09-15T03:46:50.871 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.871 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.871 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.871 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.871 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.0s2 39/31 ECSubWrite(tid=46, reqid=client.4521.0:242, at_version=39'18, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:46:50.872 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.150954+0000", 2024-09-15T03:46:50.872 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.6590062139999997, 2024-09-15T03:46:50.872 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.007342459, 2024-09-15T03:46:50.872 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.872 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.872 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.872 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.872 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.872 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150954+0000", 2024-09-15T03:46:50.872 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.872 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.872 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.872 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150954+0000", 2024-09-15T03:46:50.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.873 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.873 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150955+0000", 2024-09-15T03:46:50.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2700000000000001e-07 2024-09-15T03:46:50.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.873 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.873 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150964+0000", 2024-09-15T03:46:50.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2820000000000002e-06 2024-09-15T03:46:50.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.873 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150970+0000", 2024-09-15T03:46:50.874 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0700000000000003e-06 2024-09-15T03:46:50.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.874 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150973+0000", 2024-09-15T03:46:50.874 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.576e-06 2024-09-15T03:46:50.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.874 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.151082+0000", 2024-09-15T03:46:50.874 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000109261 2024-09-15T03:46:50.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.875 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.875 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.151088+0000", 2024-09-15T03:46:50.875 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2480000000000001e-06 2024-09-15T03:46:50.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.875 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.875 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.151145+0000", 2024-09-15T03:46:50.875 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6886999999999999e-05 2024-09-15T03:46:50.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.875 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.875 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152307+0000", 2024-09-15T03:46:50.875 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0011623180000000001 2024-09-15T03:46:50.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.876 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.876 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152336+0000", 2024-09-15T03:46:50.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8677999999999999e-05 2024-09-15T03:46:50.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.876 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.876 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152365+0000", 2024-09-15T03:46:50.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9170999999999999e-05 2024-09-15T03:46:50.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.876 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.877 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.158297+0000", 2024-09-15T03:46:50.877 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0059317409999999999 2024-09-15T03:46:50.877 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.877 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.877 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.877 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.877 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es7 39/28 ECSubWrite(tid=51, reqid=osd.2.0:50, at_version=0'0, trim_to=0'0, roll_forward_to=39'28))", 2024-09-15T03:46:50.877 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.154447+0000", 2024-09-15T03:46:50.877 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.6555133580000003, 2024-09-15T03:46:50.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0039021279999999999, 2024-09-15T03:46:50.878 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.878 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.878 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.878 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.154447+0000", 2024-09-15T03:46:50.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.878 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.154447+0000", 2024-09-15T03:46:50.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.879 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.879 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.879 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.154449+0000", 2024-09-15T03:46:50.879 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5039999999999999e-06 2024-09-15T03:46:50.879 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.879 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.879 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.879 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.154468+0000", 2024-09-15T03:46:50.879 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9102000000000001e-05 2024-09-15T03:46:50.879 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.879 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.879 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.879 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.154494+0000", 2024-09-15T03:46:50.880 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.565e-05 2024-09-15T03:46:50.880 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.880 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.880 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.880 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.154498+0000", 2024-09-15T03:46:50.880 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1659999999999996e-06 2024-09-15T03:46:50.880 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.880 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.880 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.880 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.154530+0000", 2024-09-15T03:46:50.880 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2361999999999999e-05 2024-09-15T03:46:50.880 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.880 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.881 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.881 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.154536+0000", 2024-09-15T03:46:50.881 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7019999999999998e-06 2024-09-15T03:46:50.881 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.881 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.881 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.881 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.154596+0000", 2024-09-15T03:46:50.881 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9925e-05 2024-09-15T03:46:50.881 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.881 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.881 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.881 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.154601+0000", 2024-09-15T03:46:50.881 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4419999999999997e-06 2024-09-15T03:46:50.882 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.882 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.882 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.882 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.155190+0000", 2024-09-15T03:46:50.882 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00058893400000000005 2024-09-15T03:46:50.882 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.882 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.882 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.882 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.155201+0000", 2024-09-15T03:46:50.883 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0998e-05 2024-09-15T03:46:50.883 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.883 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.883 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.158349+0000", 2024-09-15T03:46:50.883 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031483430000000001 2024-09-15T03:46:50.883 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.883 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.883 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.883 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.884 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.884 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 39/28 ECSubWrite(tid=41, reqid=client.4521.0:254, at_version=39'18, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:46:50.884 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.169880+0000", 2024-09-15T03:46:50.884 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.6400810559999996, 2024-09-15T03:46:50.884 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0040297620000000001, 2024-09-15T03:46:50.884 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.884 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.884 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.884 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.884 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.884 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169880+0000", 2024-09-15T03:46:50.884 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.885 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.885 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.885 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.885 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169880+0000", 2024-09-15T03:46:50.885 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.885 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.885 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.885 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.885 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169880+0000", 2024-09-15T03:46:50.885 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8999999999999995e-07 2024-09-15T03:46:50.885 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.885 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.885 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.885 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169900+0000", 2024-09-15T03:46:50.886 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9891e-05 2024-09-15T03:46:50.886 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.886 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.886 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.886 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169913+0000", 2024-09-15T03:46:50.886 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.226e-05 2024-09-15T03:46:50.886 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.886 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.886 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.886 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169919+0000", 2024-09-15T03:46:50.886 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1820000000000003e-06 2024-09-15T03:46:50.886 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.886 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.887 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.887 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169965+0000", 2024-09-15T03:46:50.887 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6285999999999997e-05 2024-09-15T03:46:50.887 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.887 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.887 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.887 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169974+0000", 2024-09-15T03:46:50.887 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0189999999999995e-06 2024-09-15T03:46:50.887 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.887 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.887 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.887 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170200+0000", 2024-09-15T03:46:50.887 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022614600000000001 2024-09-15T03:46:50.888 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.888 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.888 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.888 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170208+0000", 2024-09-15T03:46:50.888 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8590000000000005e-06 2024-09-15T03:46:50.888 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.888 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.888 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.888 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170237+0000", 2024-09-15T03:46:50.888 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8864e-05 2024-09-15T03:46:50.888 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.888 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.888 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.889 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170254+0000", 2024-09-15T03:46:50.889 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7226000000000001e-05 2024-09-15T03:46:50.889 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.889 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.889 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.889 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173909+0000", 2024-09-15T03:46:50.889 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0036552389999999998 2024-09-15T03:46:50.889 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.889 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.889 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.889 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.889 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.889 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 39/28 ECSubWrite(tid=42, reqid=client.4521.0:255, at_version=39'19, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:46:50.890 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.170724+0000", 2024-09-15T03:46:50.890 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.6392366269999998, 2024-09-15T03:46:50.890 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0086604270000000001, 2024-09-15T03:46:50.890 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.890 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.890 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.890 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.890 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.890 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170724+0000", 2024-09-15T03:46:50.890 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.890 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.890 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.890 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.891 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170724+0000", 2024-09-15T03:46:50.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.891 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.891 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170725+0000", 2024-09-15T03:46:50.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8999999999999995e-07 2024-09-15T03:46:50.891 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.891 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170742+0000", 2024-09-15T03:46:50.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7181e-05 2024-09-15T03:46:50.891 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170757+0000", 2024-09-15T03:46:50.892 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4693e-05 2024-09-15T03:46:50.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170763+0000", 2024-09-15T03:46:50.892 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4509999999999997e-06 2024-09-15T03:46:50.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.893 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170807+0000", 2024-09-15T03:46:50.893 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3574e-05 2024-09-15T03:46:50.893 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.893 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.893 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.893 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170815+0000", 2024-09-15T03:46:50.893 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3459999999999994e-06 2024-09-15T03:46:50.893 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.893 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.893 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.893 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170900+0000", 2024-09-15T03:46:50.893 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5436e-05 2024-09-15T03:46:50.893 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.893 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.894 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.894 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170906+0000", 2024-09-15T03:46:50.894 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9190000000000004e-06 2024-09-15T03:46:50.894 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.894 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.894 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.894 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173681+0000", 2024-09-15T03:46:50.894 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00277437 2024-09-15T03:46:50.894 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.894 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.894 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.894 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173695+0000", 2024-09-15T03:46:50.894 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4446e-05 2024-09-15T03:46:50.895 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.895 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.895 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.895 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179384+0000", 2024-09-15T03:46:50.895 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0056892210000000004 2024-09-15T03:46:50.895 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.895 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.895 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.895 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.895 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.895 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:46:50.896 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:46:50.896 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.6388587919999997, 2024-09-15T03:46:50.896 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012183955999999999, 2024-09-15T03:46:50.896 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.896 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.896 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.896 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.896 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.896 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:46:50.896 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.897 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.897 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.897 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.897 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:46:50.897 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.897 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.897 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.897 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.897 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:46:50.897 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.17e-07 2024-09-15T03:46:50.897 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.897 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.897 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.898 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171454+0000", 2024-09-15T03:46:50.898 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00035226700000000002 2024-09-15T03:46:50.898 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.898 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.898 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.898 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171465+0000", 2024-09-15T03:46:50.898 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0943e-05 2024-09-15T03:46:50.898 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.898 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.898 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.898 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171471+0000", 2024-09-15T03:46:50.898 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7400000000000001e-06 2024-09-15T03:46:50.899 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.899 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.899 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171508+0000", 2024-09-15T03:46:50.899 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7147999999999997e-05 2024-09-15T03:46:50.899 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.899 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.899 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171517+0000", 2024-09-15T03:46:50.899 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2560000000000008e-06 2024-09-15T03:46:50.899 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.899 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.899 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171716+0000", 2024-09-15T03:46:50.900 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019899699999999999 2024-09-15T03:46:50.900 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.900 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.900 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.900 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173686+0000", 2024-09-15T03:46:50.900 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019696829999999999 2024-09-15T03:46:50.900 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.900 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.900 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.900 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173777+0000", 2024-09-15T03:46:50.900 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0637999999999995e-05 2024-09-15T03:46:50.900 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.901 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.901 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.901 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173814+0000", 2024-09-15T03:46:50.901 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7499000000000001e-05 2024-09-15T03:46:50.901 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.901 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.901 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.901 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.183286+0000", 2024-09-15T03:46:50.901 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094716680000000008 2024-09-15T03:46:50.901 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.901 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.901 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.902 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.902 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.902 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 39/28 ECSubWrite(tid=43, reqid=client.4521.0:256, at_version=39'20, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:46:50.902 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.171211+0000", 2024-09-15T03:46:50.902 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.6387496989999999, 2024-09-15T03:46:50.902 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082752069999999997, 2024-09-15T03:46:50.902 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.902 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.902 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.902 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.902 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.902 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171211+0000", 2024-09-15T03:46:50.902 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.903 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.903 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.903 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.903 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171211+0000", 2024-09-15T03:46:50.903 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.903 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.903 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.903 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.903 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171211+0000", 2024-09-15T03:46:50.903 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.39e-07 2024-09-15T03:46:50.903 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.903 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.903 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.903 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171228+0000", 2024-09-15T03:46:50.904 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6254999999999999e-05 2024-09-15T03:46:50.904 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.904 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.904 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.904 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171241+0000", 2024-09-15T03:46:50.904 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3482999999999999e-05 2024-09-15T03:46:50.904 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.904 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.904 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.904 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171247+0000", 2024-09-15T03:46:50.904 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.088e-06 2024-09-15T03:46:50.904 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.905 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.905 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.905 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171292+0000", 2024-09-15T03:46:50.905 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5062000000000003e-05 2024-09-15T03:46:50.905 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.905 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.905 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.905 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171302+0000", 2024-09-15T03:46:50.905 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.302e-06 2024-09-15T03:46:50.905 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.905 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.905 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.905 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171382+0000", 2024-09-15T03:46:50.905 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0717999999999995e-05 2024-09-15T03:46:50.906 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.906 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.906 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.906 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173686+0000", 2024-09-15T03:46:50.906 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023032320000000001 2024-09-15T03:46:50.906 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.906 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.906 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.906 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173776+0000", 2024-09-15T03:46:50.906 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0302000000000004e-05 2024-09-15T03:46:50.906 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.906 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.907 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.907 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173805+0000", 2024-09-15T03:46:50.907 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8979000000000002e-05 2024-09-15T03:46:50.907 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.907 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.907 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.907 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179486+0000", 2024-09-15T03:46:50.907 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0056813469999999998 2024-09-15T03:46:50.907 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.907 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.907 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.907 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.907 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.908 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 39/28 ECSubWrite(tid=44, reqid=client.4521.0:256, at_version=0'0, trim_to=0'0, roll_forward_to=39'20))", 2024-09-15T03:46:50.908 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.174185+0000", 2024-09-15T03:46:50.908 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.6357755940000001, 2024-09-15T03:46:50.908 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0054108059999999998, 2024-09-15T03:46:50.908 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.908 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.908 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.908 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.908 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.908 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174185+0000", 2024-09-15T03:46:50.908 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.908 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.908 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.908 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.909 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174185+0000", 2024-09-15T03:46:50.909 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.909 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.909 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.909 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.909 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174186+0000", 2024-09-15T03:46:50.909 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5099999999999999e-07 2024-09-15T03:46:50.909 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.909 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.909 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.909 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174197+0000", 2024-09-15T03:46:50.909 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0823e-05 2024-09-15T03:46:50.909 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.910 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.910 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.910 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174212+0000", 2024-09-15T03:46:50.910 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5268e-05 2024-09-15T03:46:50.910 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.910 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.910 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.910 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174218+0000", 2024-09-15T03:46:50.910 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9580000000000004e-06 2024-09-15T03:46:50.910 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.910 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.910 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.910 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174273+0000", 2024-09-15T03:46:50.910 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4982999999999999e-05 2024-09-15T03:46:50.911 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.911 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.911 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.911 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174281+0000", 2024-09-15T03:46:50.911 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0309999999999998e-06 2024-09-15T03:46:50.911 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.911 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.911 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.911 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174386+0000", 2024-09-15T03:46:50.911 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000105215 2024-09-15T03:46:50.911 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.911 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.911 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.912 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174392+0000", 2024-09-15T03:46:50.912 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2500000000000003e-06 2024-09-15T03:46:50.912 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.912 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.912 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.912 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174418+0000", 2024-09-15T03:46:50.912 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5259999999999999e-05 2024-09-15T03:46:50.912 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.912 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.912 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.912 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174428+0000", 2024-09-15T03:46:50.912 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0047999999999999e-05 2024-09-15T03:46:50.912 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.913 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.913 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.913 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179596+0000", 2024-09-15T03:46:50.913 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0051682189999999999 2024-09-15T03:46:50.913 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.913 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.913 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.913 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.913 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.913 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 39/31 ECSubWrite(tid=44, reqid=client.4521.0:243, at_version=0'0, trim_to=0'0, roll_forward_to=39'17))", 2024-09-15T03:46:50.913 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.176404+0000", 2024-09-15T03:46:50.913 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.6335569020000005, 2024-09-15T03:46:50.913 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069470829999999997, 2024-09-15T03:46:50.914 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.914 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.914 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.914 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.914 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.914 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176404+0000", 2024-09-15T03:46:50.914 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.914 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.914 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.914 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.914 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176404+0000", 2024-09-15T03:46:50.914 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.914 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.915 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.915 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.915 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176404+0000", 2024-09-15T03:46:50.915 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.54e-07 2024-09-15T03:46:50.915 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.915 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.915 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.915 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176417+0000", 2024-09-15T03:46:50.915 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2785999999999999e-05 2024-09-15T03:46:50.915 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.915 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.915 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.915 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176430+0000", 2024-09-15T03:46:50.916 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2622999999999999e-05 2024-09-15T03:46:50.916 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.916 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.916 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.916 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176436+0000", 2024-09-15T03:46:50.916 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3570000000000003e-06 2024-09-15T03:46:50.916 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.916 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.916 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.916 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176479+0000", 2024-09-15T03:46:50.916 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2939999999999999e-05 2024-09-15T03:46:50.916 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.916 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.917 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.917 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176487+0000", 2024-09-15T03:46:50.917 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2979999999999992e-06 2024-09-15T03:46:50.917 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.917 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.917 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.917 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176580+0000", 2024-09-15T03:46:50.917 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2349999999999995e-05 2024-09-15T03:46:50.917 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.917 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.917 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.917 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176588+0000", 2024-09-15T03:46:50.917 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2360000000000004e-06 2024-09-15T03:46:50.917 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.918 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.918 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.918 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176618+0000", 2024-09-15T03:46:50.918 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9459999999999999e-05 2024-09-15T03:46:50.918 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.918 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.918 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.918 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176629+0000", 2024-09-15T03:46:50.918 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1382999999999999e-05 2024-09-15T03:46:50.918 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.918 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.918 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.918 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.183351+0000", 2024-09-15T03:46:50.918 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0067218959999999998 2024-09-15T03:46:50.919 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.919 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.919 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.919 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.919 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.919 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.6s2 40/31 ECSubWrite(tid=60, reqid=client.4521.0:266, at_version=40'6, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:50.919 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:47.198527+0000", 2024-09-15T03:46:50.919 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 3.61143342, 2024-09-15T03:46:50.919 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0037991380000000001, 2024-09-15T03:46:50.919 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.919 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:50.919 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.919 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.920 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.920 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.198527+0000", 2024-09-15T03:46:50.920 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.920 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.920 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.920 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.920 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.198527+0000", 2024-09-15T03:46:50.920 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.920 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.920 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.920 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.920 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.198527+0000", 2024-09-15T03:46:50.920 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.11e-07 2024-09-15T03:46:50.920 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.921 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.921 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.921 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.198548+0000", 2024-09-15T03:46:50.921 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0831000000000001e-05 2024-09-15T03:46:50.921 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.921 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.921 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.921 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.198560+0000", 2024-09-15T03:46:50.921 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1834000000000001e-05 2024-09-15T03:46:50.921 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.921 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.921 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.922 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.198569+0000", 2024-09-15T03:46:50.922 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6950000000000006e-06 2024-09-15T03:46:50.922 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.922 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.922 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.922 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.198613+0000", 2024-09-15T03:46:50.922 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4434999999999997e-05 2024-09-15T03:46:50.922 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.922 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.922 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.922 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.198624+0000", 2024-09-15T03:46:50.922 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0528e-05 2024-09-15T03:46:50.922 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.922 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.923 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.923 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.198917+0000", 2024-09-15T03:46:50.923 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000293694 2024-09-15T03:46:50.923 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.923 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.923 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.923 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.198944+0000", 2024-09-15T03:46:50.923 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6454000000000001e-05 2024-09-15T03:46:50.923 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.923 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.923 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.923 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.198981+0000", 2024-09-15T03:46:50.923 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7098000000000003e-05 2024-09-15T03:46:50.923 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.924 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.924 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.924 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.199001+0000", 2024-09-15T03:46:50.924 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0479000000000002e-05 2024-09-15T03:46:50.924 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.924 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.924 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.924 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.202326+0000", 2024-09-15T03:46:50.924 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0033248790000000002 2024-09-15T03:46:50.924 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.924 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.924 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.924 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.925 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.925 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:300 3.cs0 3:3164c049:::smithi00132172-34:head [append 0~802816] snapc c=[c,b,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-15T03:46:50.925 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:50.162456+0000", 2024-09-15T03:46:50.925 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.647504776, 2024-09-15T03:46:50.925 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0057491620000000004, 2024-09-15T03:46:50.925 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.925 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:50.925 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:50.925 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:50.925 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:50.925 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 300 2024-09-15T03:46:50.925 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.925 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.926 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.162456+0000", 2024-09-15T03:46:50.926 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.926 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.926 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.162456+0000", 2024-09-15T03:46:50.926 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.926 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.926 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.162458+0000", 2024-09-15T03:46:50.926 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7710000000000001e-06 2024-09-15T03:46:50.926 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.927 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.927 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.927 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.163778+0000", 2024-09-15T03:46:50.927 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013201 2024-09-15T03:46:50.927 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.927 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.927 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.927 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.163798+0000", 2024-09-15T03:46:50.927 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0669000000000001e-05 2024-09-15T03:46:50.927 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.927 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.927 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.927 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.163806+0000", 2024-09-15T03:46:50.927 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8259999999999993e-06 2024-09-15T03:46:50.928 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.928 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.928 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.928 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.163851+0000", 2024-09-15T03:46:50.928 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4449000000000001e-05 2024-09-15T03:46:50.928 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.928 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.928 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:50.928 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.163921+0000", 2024-09-15T03:46:50.928 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0483999999999998e-05 2024-09-15T03:46:50.928 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.928 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.928 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:50.929 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.164900+0000", 2024-09-15T03:46:50.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0009785970000000001 2024-09-15T03:46:50.929 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.929 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.929 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:50.929 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.165000+0000", 2024-09-15T03:46:50.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9877999999999995e-05 2024-09-15T03:46:50.929 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.929 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.929 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:50.929 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.165014+0000", 2024-09-15T03:46:50.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.412e-05 2024-09-15T03:46:50.929 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.929 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.930 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:50.930 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.165142+0000", 2024-09-15T03:46:50.930 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000128622 2024-09-15T03:46:50.930 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.930 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.930 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:50.930 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.165179+0000", 2024-09-15T03:46:50.930 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6752999999999999e-05 2024-09-15T03:46:50.930 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.930 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.930 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:50.930 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.168155+0000", 2024-09-15T03:46:50.931 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002976027 2024-09-15T03:46:50.931 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.931 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.931 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.931 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.168205+0000", 2024-09-15T03:46:50.931 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9866e-05 2024-09-15T03:46:50.931 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.931 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.931 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.931 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.931 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.931 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:303 3.cs0 3:3164c049:::smithi00132172-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-15T03:46:50.931 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:50.165923+0000", 2024-09-15T03:46:50.932 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.64403714000000001, 2024-09-15T03:46:50.932 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0043065530000000003, 2024-09-15T03:46:50.932 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:50.932 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:46:50.932 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:50.932 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:50.932 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:50.932 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 303 2024-09-15T03:46:50.932 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.932 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:50.932 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.932 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:50.932 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.165923+0000", 2024-09-15T03:46:50.933 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.933 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.933 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.933 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:50.933 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.165923+0000", 2024-09-15T03:46:50.933 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:50.933 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.933 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.933 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:50.933 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.165924+0000", 2024-09-15T03:46:50.933 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.27e-07 2024-09-15T03:46:50.933 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.933 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.933 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:50.934 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.165928+0000", 2024-09-15T03:46:50.934 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9249999999999997e-06 2024-09-15T03:46:50.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.934 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:50.934 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.165935+0000", 2024-09-15T03:46:50.934 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.737e-06 2024-09-15T03:46:50.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.934 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:50.934 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.165939+0000", 2024-09-15T03:46:50.934 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9400000000000004e-06 2024-09-15T03:46:50.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.935 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:50.935 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.168823+0000", 2024-09-15T03:46:50.935 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002884644 2024-09-15T03:46:50.935 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.935 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.935 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:50.935 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.168907+0000", 2024-09-15T03:46:50.935 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3555999999999998e-05 2024-09-15T03:46:50.935 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:50.935 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:50.935 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:50.935 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.170230+0000", 2024-09-15T03:46:50.935 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013233240000000001 2024-09-15T03:46:50.936 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.936 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.936 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.936 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:50.936 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:50.936 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:50.937 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:46:50.938 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:46:50.937+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:46:51.034 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:51.035 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:51.035 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:46:51.035 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:51.038 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:51.037+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:51.043 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:46:51.139 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:51.139+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:46:51.162 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-15T03:46:51.162 INFO:tasks.rados.rados.0.smithi001.stdout:115: done (2 left) 2024-09-15T03:46:51.163 INFO:tasks.rados.rados.0.smithi001.stdout:117: done (1 left) 2024-09-15T03:46:51.163 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 44 current snap is 9 2024-09-15T03:46:51.163 INFO:tasks.rados.rados.0.smithi001.stdout:118: seq_num 71 ranges {1245184=425984,1671168=573440,2244608=196608} 2024-09-15T03:46:51.171 INFO:tasks.rados.rados.0.smithi001.stdout:118: writing smithi00132172-44 from 1245184 to 1671168 tid 1 2024-09-15T03:46:51.175 INFO:tasks.rados.rados.0.smithi001.stdout:118: writing smithi00132172-44 from 1671168 to 2244608 tid 2 2024-09-15T03:46:51.177 INFO:tasks.rados.rados.0.smithi001.stdout:118: writing smithi00132172-44 from 2244608 to 2441216 tid 3 2024-09-15T03:46:51.177 INFO:tasks.rados.rados.0.smithi001.stdout:119: snap_create 2024-09-15T03:46:51.177 INFO:tasks.rados.rados.0.smithi001.stdout:118: finishing write tid 1 to smithi00132172-44 2024-09-15T03:46:51.182 INFO:tasks.rados.rados.0.smithi001.stdout:118: finishing write tid 2 to smithi00132172-44 2024-09-15T03:46:51.182 INFO:tasks.rados.rados.0.smithi001.stdout:118: finishing write tid 3 to smithi00132172-44 2024-09-15T03:46:51.183 INFO:tasks.rados.rados.0.smithi001.stdout:118: finishing write tid 4 to smithi00132172-44 2024-09-15T03:46:51.184 INFO:tasks.rados.rados.0.smithi001.stdout:118: finishing write tid 5 to smithi00132172-44 2024-09-15T03:46:51.184 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 44 v 44 (ObjNum 71 snap 9 seq_num 71) dirty exists 2024-09-15T03:46:51.184 INFO:tasks.rados.rados.0.smithi001.stdout:118: left oid 44 (ObjNum 71 snap 9 seq_num 71) 2024-09-15T03:46:51.185 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:51.185 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:51.185 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:46:51.185 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:46:51.186 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:51.193 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:46:51.240 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:51.240+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:51.289 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:51.289 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:46:51.289 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:46:51.289 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [] 2024-09-15T03:46:51.289 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:51.301 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:46:51.342 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:51.341+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:51.438 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:51.438 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:51.438 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:46:51.438 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:51.441 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:51.440+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:51.446 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:46:51.543 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:51.544 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:51.544 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:46:51.544 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:46:51.544 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:51.545 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:51.542+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:51.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:46:51.598 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:46:51.646 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:51.644+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:46:51.691 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:51.692 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:46:51.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:46:51.692 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:46:51.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.692 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:184 3.as0 3:5515acd6:::smithi00132172-13:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:51.692 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104658+0000", 2024-09-15T03:46:51.692 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.584214101000001, 2024-09-15T03:46:51.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006212525, 2024-09-15T03:46:51.692 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:51.692 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:51.692 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:51.693 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:51.693 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:51.693 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 184 2024-09-15T03:46:51.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.693 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:51.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:51.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104658+0000", 2024-09-15T03:46:51.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:51.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104658+0000", 2024-09-15T03:46:51.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.694 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:51.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104660+0000", 2024-09-15T03:46:51.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.17e-06 2024-09-15T03:46:51.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:51.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104672+0000", 2024-09-15T03:46:51.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1161999999999999e-05 2024-09-15T03:46:51.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:51.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104693+0000", 2024-09-15T03:46:51.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0995000000000001e-05 2024-09-15T03:46:51.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:51.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104700+0000", 2024-09-15T03:46:51.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4189999999999996e-06 2024-09-15T03:46:51.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104752+0000", 2024-09-15T03:46:51.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2413000000000001e-05 2024-09-15T03:46:51.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-09-15T03:46:51.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104757+0000", 2024-09-15T03:46:51.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9599999999999999e-06 2024-09-15T03:46:51.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.105683+0000", 2024-09-15T03:46:51.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00092545599999999996 2024-09-15T03:46:51.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:51.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.105757+0000", 2024-09-15T03:46:51.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4567999999999995e-05 2024-09-15T03:46:51.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:51.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106531+0000", 2024-09-15T03:46:51.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00077397999999999996 2024-09-15T03:46:51.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:51.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106775+0000", 2024-09-15T03:46:51.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024402899999999999 2024-09-15T03:46:51.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:51.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106786+0000", 2024-09-15T03:46:51.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0825000000000001e-05 2024-09-15T03:46:51.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:51.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106845+0000", 2024-09-15T03:46:51.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8733999999999997e-05 2024-09-15T03:46:51.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:51.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109399+0000", 2024-09-15T03:46:51.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002553573 2024-09-15T03:46:51.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:51.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109844+0000", 2024-09-15T03:46:51.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00044492700000000002 2024-09-15T03:46:51.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:51.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.110871+0000", 2024-09-15T03:46:51.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010273140000000001 2024-09-15T03:46:51.700 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.700 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:51.700 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.700 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:185 3.as0 3:5515acd6:::smithi00132172-13:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:51.701 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:46:51.701 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.584163271, 2024-09-15T03:46:51.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01268643, 2024-09-15T03:46:51.701 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:51.701 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:51.701 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:51.701 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:51.701 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:51.701 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 185 2024-09-15T03:46:51.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.701 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:51.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:51.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:46:51.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:51.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:46:51.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:51.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104710+0000", 2024-09-15T03:46:51.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0200000000000003e-07 2024-09-15T03:46:51.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:51.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104713+0000", 2024-09-15T03:46:51.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2339999999999999e-06 2024-09-15T03:46:51.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:51.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104718+0000", 2024-09-15T03:46:51.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3449999999999996e-06 2024-09-15T03:46:51.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:51.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104722+0000", 2024-09-15T03:46:51.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8399999999999997e-06 2024-09-15T03:46:51.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104849+0000", 2024-09-15T03:46:51.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012709599999999999 2024-09-15T03:46:51.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:51.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104856+0000", 2024-09-15T03:46:51.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.652e-06 2024-09-15T03:46:51.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106851+0000", 2024-09-15T03:46:51.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019954840000000001 2024-09-15T03:46:51.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:51.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106935+0000", 2024-09-15T03:46:51.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4095e-05 2024-09-15T03:46:51.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:51.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107453+0000", 2024-09-15T03:46:51.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00051729499999999997 2024-09-15T03:46:51.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:51.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107554+0000", 2024-09-15T03:46:51.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000101251 2024-09-15T03:46:51.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:51.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108104+0000", 2024-09-15T03:46:51.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00054995100000000002 2024-09-15T03:46:51.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:51.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108503+0000", 2024-09-15T03:46:51.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039953500000000002 2024-09-15T03:46:51.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:51.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109405+0000", 2024-09-15T03:46:51.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00090170099999999998 2024-09-15T03:46:51.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:51.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114365+0000", 2024-09-15T03:46:51.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049602120000000003 2024-09-15T03:46:51.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.708 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:51.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117396+0000", 2024-09-15T03:46:51.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030303370000000001 2024-09-15T03:46:51.708 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.708 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:51.708 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.708 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:186 3.as0 3:5515acd6:::smithi00132172-13:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:51.708 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:46:51.708 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.584123481000001, 2024-09-15T03:46:51.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012817485999999999, 2024-09-15T03:46:51.709 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:51.709 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:51.709 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:51.709 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:51.709 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:51.709 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 186 2024-09-15T03:46:51.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.709 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:51.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:51.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:46:51.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.713 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:51.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:46:51.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.714 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:51.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104750+0000", 2024-09-15T03:46:51.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1500000000000004e-07 2024-09-15T03:46:51.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.714 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:51.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104754+0000", 2024-09-15T03:46:51.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.946e-06 2024-09-15T03:46:51.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.714 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:51.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104769+0000", 2024-09-15T03:46:51.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.507e-05 2024-09-15T03:46:51.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:51.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104776+0000", 2024-09-15T03:46:51.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7849999999999992e-06 2024-09-15T03:46:51.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104888+0000", 2024-09-15T03:46:51.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011164500000000001 2024-09-15T03:46:51.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:51.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104894+0000", 2024-09-15T03:46:51.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.942e-06 2024-09-15T03:46:51.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108172+0000", 2024-09-15T03:46:51.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032781939999999999 2024-09-15T03:46:51.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:51.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108257+0000", 2024-09-15T03:46:51.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4488999999999997e-05 2024-09-15T03:46:51.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:51.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108732+0000", 2024-09-15T03:46:51.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047547399999999998 2024-09-15T03:46:51.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:51.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108823+0000", 2024-09-15T03:46:51.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1106000000000005e-05 2024-09-15T03:46:51.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:51.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108961+0000", 2024-09-15T03:46:51.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013735099999999999 2024-09-15T03:46:51.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:51.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109065+0000", 2024-09-15T03:46:51.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104715 2024-09-15T03:46:51.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:51.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109406+0000", 2024-09-15T03:46:51.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034062300000000002 2024-09-15T03:46:51.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.719 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.719 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:51.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114529+0000", 2024-09-15T03:46:51.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.005123219 2024-09-15T03:46:51.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.719 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.719 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:51.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117566+0000", 2024-09-15T03:46:51.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030372120000000001 2024-09-15T03:46:51.719 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.719 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:51.719 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.720 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:187 3.fs0 3:fe9c5513:::smithi00132172-47:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:51.720 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104782+0000", 2024-09-15T03:46:51.720 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.584090462000001, 2024-09-15T03:46:51.720 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069218609999999996, 2024-09-15T03:46:51.720 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:51.720 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:51.720 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:51.720 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:51.720 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:51.720 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 187 2024-09-15T03:46:51.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.721 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:51.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:51.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104782+0000", 2024-09-15T03:46:51.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:51.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104782+0000", 2024-09-15T03:46:51.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:51.722 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104782+0000", 2024-09-15T03:46:51.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8599999999999998e-07 2024-09-15T03:46:51.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:51.722 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104785+0000", 2024-09-15T03:46:51.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.649e-06 2024-09-15T03:46:51.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:51.722 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104800+0000", 2024-09-15T03:46:51.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5305e-05 2024-09-15T03:46:51.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:51.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104805+0000", 2024-09-15T03:46:51.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5569999999999997e-06 2024-09-15T03:46:51.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104854+0000", 2024-09-15T03:46:51.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9039999999999998e-05 2024-09-15T03:46:51.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-09-15T03:46:51.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104860+0000", 2024-09-15T03:46:51.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2979999999999997e-06 2024-09-15T03:46:51.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.105869+0000", 2024-09-15T03:46:51.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001009036 2024-09-15T03:46:51.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:51.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.105954+0000", 2024-09-15T03:46:51.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4357000000000006e-05 2024-09-15T03:46:51.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:51.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106626+0000", 2024-09-15T03:46:51.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00067191099999999997 2024-09-15T03:46:51.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:51.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106786+0000", 2024-09-15T03:46:51.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000160469 2024-09-15T03:46:51.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:51.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106788+0000", 2024-09-15T03:46:51.726 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9630000000000001e-06 2024-09-15T03:46:51.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.726 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:51.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106846+0000", 2024-09-15T03:46:51.726 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7719000000000001e-05 2024-09-15T03:46:51.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.726 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:51.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108372+0000", 2024-09-15T03:46:51.726 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0015264499999999999 2024-09-15T03:46:51.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:51.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.111475+0000", 2024-09-15T03:46:51.727 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031028459999999998 2024-09-15T03:46:51.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:51.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.111704+0000", 2024-09-15T03:46:51.727 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022877499999999999 2024-09-15T03:46:51.727 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.727 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:51.727 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.728 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.728 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:188 3.fs0 3:fe9c5513:::smithi00132172-47:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:51.728 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:46:51.728 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.584040784000001, 2024-09-15T03:46:51.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012637061, 2024-09-15T03:46:51.728 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:51.728 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:51.728 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:51.728 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:51.728 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:51.728 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 188 2024-09-15T03:46:51.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.729 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:51.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.729 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:51.729 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:46:51.729 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.729 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:51.729 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:46:51.729 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.730 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:51.730 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:46:51.730 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2799999999999996e-07 2024-09-15T03:46:51.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.730 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.730 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:51.730 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104835+0000", 2024-09-15T03:46:51.730 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7190000000000001e-06 2024-09-15T03:46:51.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.730 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:51.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104842+0000", 2024-09-15T03:46:51.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4580000000000004e-06 2024-09-15T03:46:51.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:51.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104846+0000", 2024-09-15T03:46:51.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1250000000000003e-06 2024-09-15T03:46:51.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104884+0000", 2024-09-15T03:46:51.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7075e-05 2024-09-15T03:46:51.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.732 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:51.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104888+0000", 2024-09-15T03:46:51.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8520000000000003e-06 2024-09-15T03:46:51.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.732 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106860+0000", 2024-09-15T03:46:51.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019719440000000002 2024-09-15T03:46:51.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.732 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:51.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106938+0000", 2024-09-15T03:46:51.733 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7421000000000003e-05 2024-09-15T03:46:51.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:51.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107368+0000", 2024-09-15T03:46:51.733 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043042999999999998 2024-09-15T03:46:51.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:51.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107460+0000", 2024-09-15T03:46:51.733 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1392999999999997e-05 2024-09-15T03:46:51.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.734 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.734 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:51.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107473+0000", 2024-09-15T03:46:51.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3702e-05 2024-09-15T03:46:51.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.734 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.734 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:51.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107504+0000", 2024-09-15T03:46:51.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1000000000000001e-05 2024-09-15T03:46:51.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.734 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.734 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:51.735 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108376+0000", 2024-09-15T03:46:51.735 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00087162000000000003 2024-09-15T03:46:51.735 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.735 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.735 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:51.735 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.111657+0000", 2024-09-15T03:46:51.735 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032815589999999999 2024-09-15T03:46:51.735 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.735 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.735 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:51.735 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117469+0000", 2024-09-15T03:46:51.735 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058112349999999997 2024-09-15T03:46:51.735 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.736 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:51.736 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.736 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.736 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:189 3.fs0 3:fe9c5513:::smithi00132172-47:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:51.736 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:46:51.736 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.584021598, 2024-09-15T03:46:51.736 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012731009999999999, 2024-09-15T03:46:51.736 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:51.736 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:51.736 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:51.736 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:51.736 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:51.737 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 189 2024-09-15T03:46:51.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.737 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:51.737 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.737 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:51.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:46:51.737 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.737 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.737 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:51.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:46:51.737 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.737 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.738 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:51.738 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:46:51.738 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7899999999999999e-07 2024-09-15T03:46:51.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.738 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:51.738 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104853+0000", 2024-09-15T03:46:51.738 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.933e-06 2024-09-15T03:46:51.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.739 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.739 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:51.739 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104859+0000", 2024-09-15T03:46:51.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.592e-06 2024-09-15T03:46:51.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.739 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.739 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:51.739 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104863+0000", 2024-09-15T03:46:51.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4000000000000002e-06 2024-09-15T03:46:51.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.739 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.739 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104910+0000", 2024-09-15T03:46:51.740 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6678000000000003e-05 2024-09-15T03:46:51.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.740 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:51.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104916+0000", 2024-09-15T03:46:51.740 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1079999999999998e-06 2024-09-15T03:46:51.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.740 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107550+0000", 2024-09-15T03:46:51.740 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026344110000000001 2024-09-15T03:46:51.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.741 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.741 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:51.741 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107624+0000", 2024-09-15T03:46:51.741 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4134000000000005e-05 2024-09-15T03:46:51.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.741 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.741 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:51.741 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107988+0000", 2024-09-15T03:46:51.741 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036353600000000002 2024-09-15T03:46:51.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.741 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.741 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:51.742 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108066+0000", 2024-09-15T03:46:51.742 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8047999999999995e-05 2024-09-15T03:46:51.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.742 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.742 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:51.742 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108105+0000", 2024-09-15T03:46:51.742 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8868000000000001e-05 2024-09-15T03:46:51.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.742 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.742 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:51.742 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108504+0000", 2024-09-15T03:46:51.742 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039916600000000001 2024-09-15T03:46:51.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.743 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:51.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108955+0000", 2024-09-15T03:46:51.743 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045126200000000002 2024-09-15T03:46:51.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.743 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:51.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114394+0000", 2024-09-15T03:46:51.743 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0054382750000000002 2024-09-15T03:46:51.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.743 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:51.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117582+0000", 2024-09-15T03:46:51.744 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031882199999999999 2024-09-15T03:46:51.744 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.744 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:51.744 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.744 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.744 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es3 34/28 ECSubWrite(tid=33, reqid=client.4521.0:195, at_version=34'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-15T03:46:51.744 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.118213+0000", 2024-09-15T03:46:51.744 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.570659373, 2024-09-15T03:46:51.744 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006662701, 2024-09-15T03:46:51.745 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:51.745 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:51.745 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:51.745 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.745 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:51.745 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118213+0000", 2024-09-15T03:46:51.745 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.745 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.745 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:51.745 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118213+0000", 2024-09-15T03:46:51.745 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.746 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.746 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:51.746 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118213+0000", 2024-09-15T03:46:51.746 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6300000000000001e-07 2024-09-15T03:46:51.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.746 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.746 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:51.746 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118447+0000", 2024-09-15T03:46:51.746 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000234016 2024-09-15T03:46:51.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.746 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.746 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:51.747 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118465+0000", 2024-09-15T03:46:51.747 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7978e-05 2024-09-15T03:46:51.747 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.747 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.747 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:51.747 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118475+0000", 2024-09-15T03:46:51.747 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3179999999999996e-06 2024-09-15T03:46:51.747 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.747 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.747 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.747 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118524+0000", 2024-09-15T03:46:51.747 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9069000000000001e-05 2024-09-15T03:46:51.748 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.748 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.748 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:51.748 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118534+0000", 2024-09-15T03:46:51.748 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0222e-05 2024-09-15T03:46:51.748 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.748 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.748 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:51.748 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118727+0000", 2024-09-15T03:46:51.748 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000193124 2024-09-15T03:46:51.748 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.748 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.748 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:51.749 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118734+0000", 2024-09-15T03:46:51.749 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2080000000000003e-06 2024-09-15T03:46:51.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.749 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.749 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:51.749 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118804+0000", 2024-09-15T03:46:51.749 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9720000000000003e-05 2024-09-15T03:46:51.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.749 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.749 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:51.749 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118820+0000", 2024-09-15T03:46:51.749 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5705999999999998e-05 2024-09-15T03:46:51.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.750 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.750 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:51.750 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124876+0000", 2024-09-15T03:46:51.750 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006056077 2024-09-15T03:46:51.750 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.750 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:51.750 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.750 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.750 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.750 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 34/31 ECSubWrite(tid=34, reqid=client.4521.0:197, at_version=34'21, trim_to=0'0, roll_forward_to=34'18))", 2024-09-15T03:46:51.750 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.123980+0000", 2024-09-15T03:46:51.750 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.564892543999999, 2024-09-15T03:46:51.750 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0066547489999999997, 2024-09-15T03:46:51.751 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:51.751 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:51.751 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:51.751 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.751 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:51.751 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123980+0000", 2024-09-15T03:46:51.751 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.752 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.752 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.752 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:51.752 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123980+0000", 2024-09-15T03:46:51.752 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.752 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.752 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.752 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:51.752 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123980+0000", 2024-09-15T03:46:51.752 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8800000000000003e-07 2024-09-15T03:46:51.752 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.752 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.752 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:51.753 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123996+0000", 2024-09-15T03:46:51.753 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5838e-05 2024-09-15T03:46:51.753 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.753 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.753 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:51.753 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124005+0000", 2024-09-15T03:46:51.753 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1249999999999999e-06 2024-09-15T03:46:51.753 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.753 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.753 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:51.753 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124010+0000", 2024-09-15T03:46:51.753 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2640000000000002e-06 2024-09-15T03:46:51.753 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.754 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.754 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.754 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124045+0000", 2024-09-15T03:46:51.754 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5145000000000003e-05 2024-09-15T03:46:51.754 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.754 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.754 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:51.754 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124054+0000", 2024-09-15T03:46:51.754 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8389999999999996e-06 2024-09-15T03:46:51.754 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.754 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.754 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:51.754 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124696+0000", 2024-09-15T03:46:51.755 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00064247499999999999 2024-09-15T03:46:51.755 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.755 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.755 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:51.755 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124703+0000", 2024-09-15T03:46:51.755 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.905e-06 2024-09-15T03:46:51.755 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.755 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.755 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:51.755 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124736+0000", 2024-09-15T03:46:51.755 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2966999999999998e-05 2024-09-15T03:46:51.755 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.755 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.756 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:51.756 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124758+0000", 2024-09-15T03:46:51.756 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2569999999999999e-05 2024-09-15T03:46:51.756 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.756 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.756 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:51.756 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.130635+0000", 2024-09-15T03:46:51.756 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058761330000000004 2024-09-15T03:46:51.756 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.756 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:51.756 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.756 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.757 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.757 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 34/31 ECSubWrite(tid=35, reqid=client.4521.0:197, at_version=0'0, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:46:51.757 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.125913+0000", 2024-09-15T03:46:51.757 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.562959628, 2024-09-15T03:46:51.757 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0048403999999999999, 2024-09-15T03:46:51.757 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:51.757 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:51.757 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:51.757 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.757 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:51.757 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125913+0000", 2024-09-15T03:46:51.757 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.757 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.758 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.758 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:51.758 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125913+0000", 2024-09-15T03:46:51.758 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.758 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.758 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.758 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:51.758 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125913+0000", 2024-09-15T03:46:51.758 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3499999999999996e-07 2024-09-15T03:46:51.758 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.758 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.758 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:51.758 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125924+0000", 2024-09-15T03:46:51.759 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0669000000000001e-05 2024-09-15T03:46:51.759 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.759 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.759 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:51.759 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125935+0000", 2024-09-15T03:46:51.759 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1225e-05 2024-09-15T03:46:51.759 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.759 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.759 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:51.759 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125941+0000", 2024-09-15T03:46:51.759 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2519999999999996e-06 2024-09-15T03:46:51.759 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.759 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.760 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.760 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125979+0000", 2024-09-15T03:46:51.760 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7765999999999997e-05 2024-09-15T03:46:51.760 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.760 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.760 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:51.760 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125989+0000", 2024-09-15T03:46:51.760 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5950000000000005e-06 2024-09-15T03:46:51.760 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.760 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.760 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:51.760 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.126092+0000", 2024-09-15T03:46:51.760 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000103132 2024-09-15T03:46:51.761 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.761 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.761 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:51.761 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.126097+0000", 2024-09-15T03:46:51.761 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0220000000000004e-06 2024-09-15T03:46:51.761 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.761 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.761 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:51.761 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.126129+0000", 2024-09-15T03:46:51.761 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1668999999999998e-05 2024-09-15T03:46:51.761 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.761 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.761 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:51.762 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.126136+0000", 2024-09-15T03:46:51.762 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6179999999999997e-06 2024-09-15T03:46:51.762 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.762 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.762 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:51.762 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.130753+0000", 2024-09-15T03:46:51.762 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0046169169999999999 2024-09-15T03:46:51.762 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.762 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:51.762 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.762 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.762 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.762 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:222 3.as0 3:5515acd6:::smithi00132172-13:head [delete] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)", 2024-09-15T03:46:51.763 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.127651+0000", 2024-09-15T03:46:51.763 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.5612214130000002, 2024-09-15T03:46:51.763 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0063992839999999999, 2024-09-15T03:46:51.763 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:51.763 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:51.763 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:51.763 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:51.763 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:51.763 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 222 2024-09-15T03:46:51.763 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.763 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:51.763 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.764 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:51.764 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.127651+0000", 2024-09-15T03:46:51.764 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.764 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.764 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.764 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:51.764 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.127651+0000", 2024-09-15T03:46:51.764 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.764 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.764 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.764 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:51.764 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.127653+0000", 2024-09-15T03:46:51.764 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8190000000000001e-06 2024-09-15T03:46:51.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.765 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.765 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:51.765 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.127660+0000", 2024-09-15T03:46:51.765 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9500000000000004e-06 2024-09-15T03:46:51.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.765 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.765 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:51.765 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.127677+0000", 2024-09-15T03:46:51.766 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7255999999999999e-05 2024-09-15T03:46:51.766 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.766 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.766 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:51.766 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.127688+0000", 2024-09-15T03:46:51.766 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0762e-05 2024-09-15T03:46:51.766 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.766 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.766 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.766 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.127732+0000", 2024-09-15T03:46:51.766 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4308000000000003e-05 2024-09-15T03:46:51.766 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.766 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.766 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-09-15T03:46:51.767 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.127737+0000", 2024-09-15T03:46:51.767 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8099999999999997e-06 2024-09-15T03:46:51.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.767 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.767 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.767 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.130409+0000", 2024-09-15T03:46:51.767 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026722809999999999 2024-09-15T03:46:51.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.767 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.767 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:51.767 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.130471+0000", 2024-09-15T03:46:51.767 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1709000000000002e-05 2024-09-15T03:46:51.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.768 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.768 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:51.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.130960+0000", 2024-09-15T03:46:51.768 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00048949300000000002 2024-09-15T03:46:51.768 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.768 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.768 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:51.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.131233+0000", 2024-09-15T03:46:51.768 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000272627 2024-09-15T03:46:51.768 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.768 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.768 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:51.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.131242+0000", 2024-09-15T03:46:51.769 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.115e-06 2024-09-15T03:46:51.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.769 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:51.769 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.131280+0000", 2024-09-15T03:46:51.769 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8324000000000001e-05 2024-09-15T03:46:51.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.769 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:51.769 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.131294+0000", 2024-09-15T03:46:51.769 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3896999999999999e-05 2024-09-15T03:46:51.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.769 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:51.770 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.132132+0000", 2024-09-15T03:46:51.770 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00083733500000000003 2024-09-15T03:46:51.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.770 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.770 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:51.770 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.134050+0000", 2024-09-15T03:46:51.770 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019185980000000001 2024-09-15T03:46:51.770 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.770 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:51.771 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.771 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.771 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.771 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 38/31 ECSubWrite(tid=36, reqid=client.4521.0:220, at_version=38'23, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:46:51.771 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.148035+0000", 2024-09-15T03:46:51.771 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.5408374919999996, 2024-09-15T03:46:51.771 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010074534, 2024-09-15T03:46:51.771 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:51.771 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:51.771 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:51.771 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.771 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:51.771 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148035+0000", 2024-09-15T03:46:51.772 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.772 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.772 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.772 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:51.772 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148035+0000", 2024-09-15T03:46:51.772 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.772 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.772 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.773 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:51.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148036+0000", 2024-09-15T03:46:51.773 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3939999999999999e-06 2024-09-15T03:46:51.773 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.773 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.773 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:51.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148681+0000", 2024-09-15T03:46:51.773 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00064493899999999997 2024-09-15T03:46:51.773 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.773 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.773 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:51.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148695+0000", 2024-09-15T03:46:51.773 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4078e-05 2024-09-15T03:46:51.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.774 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.774 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:51.774 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148701+0000", 2024-09-15T03:46:51.774 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7230000000000001e-06 2024-09-15T03:46:51.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.774 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.774 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.774 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148742+0000", 2024-09-15T03:46:51.774 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0781000000000001e-05 2024-09-15T03:46:51.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.774 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.775 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:51.775 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148749+0000", 2024-09-15T03:46:51.775 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6109999999999996e-06 2024-09-15T03:46:51.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.775 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.775 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:51.775 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149077+0000", 2024-09-15T03:46:51.775 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000327597 2024-09-15T03:46:51.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.775 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.775 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:51.775 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149087+0000", 2024-09-15T03:46:51.775 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4739999999999994e-06 2024-09-15T03:46:51.776 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.776 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.776 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:51.776 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149230+0000", 2024-09-15T03:46:51.776 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014398300000000001 2024-09-15T03:46:51.776 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.776 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.776 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:51.776 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149247+0000", 2024-09-15T03:46:51.776 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6201e-05 2024-09-15T03:46:51.776 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.776 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.777 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:51.777 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158109+0000", 2024-09-15T03:46:51.777 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0088627529999999993 2024-09-15T03:46:51.777 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.777 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:51.777 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.777 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.777 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.777 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 38/31 ECSubWrite(tid=37, reqid=client.4521.0:220, at_version=0'0, trim_to=0'0, roll_forward_to=38'23))", 2024-09-15T03:46:51.777 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.151308+0000", 2024-09-15T03:46:51.777 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.5375642469999997, 2024-09-15T03:46:51.777 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069978310000000004, 2024-09-15T03:46:51.777 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:51.778 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:51.778 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:51.778 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.778 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:51.778 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151308+0000", 2024-09-15T03:46:51.778 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.778 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.778 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:51.778 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151308+0000", 2024-09-15T03:46:51.778 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.778 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.778 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:51.779 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151309+0000", 2024-09-15T03:46:51.779 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2200000000000003e-07 2024-09-15T03:46:51.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.779 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.779 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:51.779 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151318+0000", 2024-09-15T03:46:51.779 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7250000000000003e-06 2024-09-15T03:46:51.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.779 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.779 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:51.779 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151330+0000", 2024-09-15T03:46:51.779 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2425e-05 2024-09-15T03:46:51.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.780 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.780 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:51.780 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151337+0000", 2024-09-15T03:46:51.780 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6139999999999997e-06 2024-09-15T03:46:51.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.780 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.780 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.780 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151372+0000", 2024-09-15T03:46:51.780 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5683999999999999e-05 2024-09-15T03:46:51.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.780 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.780 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:51.780 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151378+0000", 2024-09-15T03:46:51.781 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7880000000000003e-06 2024-09-15T03:46:51.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.781 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.781 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:51.781 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151438+0000", 2024-09-15T03:46:51.781 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9679999999999998e-05 2024-09-15T03:46:51.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.781 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.781 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:51.781 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151446+0000", 2024-09-15T03:46:51.781 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.909e-06 2024-09-15T03:46:51.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.781 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.782 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:51.782 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151481+0000", 2024-09-15T03:46:51.782 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5005000000000001e-05 2024-09-15T03:46:51.782 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.782 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.782 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:51.782 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151492+0000", 2024-09-15T03:46:51.782 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1133e-05 2024-09-15T03:46:51.782 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.782 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.782 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:51.782 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158306+0000", 2024-09-15T03:46:51.782 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0068140459999999998 2024-09-15T03:46:51.782 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.783 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:51.783 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.783 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.783 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.783 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es3 38/28 ECSubWrite(tid=45, reqid=client.4521.0:227, at_version=38'24, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:46:51.783 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.155560+0000", 2024-09-15T03:46:51.783 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.5333120220000005, 2024-09-15T03:46:51.783 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0057773140000000004, 2024-09-15T03:46:51.783 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:51.783 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:51.783 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:51.783 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.783 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:51.784 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155560+0000", 2024-09-15T03:46:51.784 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.784 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.784 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:51.784 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155560+0000", 2024-09-15T03:46:51.784 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.784 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.784 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:51.784 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155561+0000", 2024-09-15T03:46:51.784 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3300000000000001e-07 2024-09-15T03:46:51.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.785 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.785 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:51.785 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156072+0000", 2024-09-15T03:46:51.785 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00051137999999999997 2024-09-15T03:46:51.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.785 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.785 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:51.785 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156082+0000", 2024-09-15T03:46:51.785 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6229999999999992e-06 2024-09-15T03:46:51.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.785 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.785 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:51.786 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156087+0000", 2024-09-15T03:46:51.786 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8599999999999998e-06 2024-09-15T03:46:51.786 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.786 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.786 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.786 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156133+0000", 2024-09-15T03:46:51.786 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5836e-05 2024-09-15T03:46:51.786 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.786 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.786 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:51.786 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156143+0000", 2024-09-15T03:46:51.786 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8689999999999999e-06 2024-09-15T03:46:51.786 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.787 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.787 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:51.787 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156392+0000", 2024-09-15T03:46:51.787 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000248526 2024-09-15T03:46:51.787 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.787 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.787 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:51.787 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156400+0000", 2024-09-15T03:46:51.787 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8189999999999992e-06 2024-09-15T03:46:51.787 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.787 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.787 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:51.788 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157090+0000", 2024-09-15T03:46:51.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00069055600000000005 2024-09-15T03:46:51.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.788 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.788 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:51.788 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157109+0000", 2024-09-15T03:46:51.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8490000000000001e-05 2024-09-15T03:46:51.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.788 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.788 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:51.788 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.161338+0000", 2024-09-15T03:46:51.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042291220000000001 2024-09-15T03:46:51.788 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.789 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:51.789 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.789 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.789 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:231 3.fs0 3:f40d4749:::smithi00132172-35:head [copy-from ver 6] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:46:51.789 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.155669+0000", 2024-09-15T03:46:51.789 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.5332037920000001, 2024-09-15T03:46:51.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010370569, 2024-09-15T03:46:51.789 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:51.789 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:51.789 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:51.789 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:51.790 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:51.790 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 231 2024-09-15T03:46:51.790 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.790 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:51.790 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.790 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:51.790 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155669+0000", 2024-09-15T03:46:51.790 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.790 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.790 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.790 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:51.790 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155669+0000", 2024-09-15T03:46:51.790 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.791 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.791 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:51.791 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155671+0000", 2024-09-15T03:46:51.791 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0829999999999998e-06 2024-09-15T03:46:51.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.791 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.791 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:51.791 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155676+0000", 2024-09-15T03:46:51.791 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6210000000000001e-06 2024-09-15T03:46:51.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.791 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.791 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:51.792 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155690+0000", 2024-09-15T03:46:51.792 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4166e-05 2024-09-15T03:46:51.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.792 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.792 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:51.792 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155698+0000", 2024-09-15T03:46:51.792 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5560000000000002e-06 2024-09-15T03:46:51.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.792 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.792 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.792 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155748+0000", 2024-09-15T03:46:51.792 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9851000000000001e-05 2024-09-15T03:46:51.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.793 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:51.793 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155872+0000", 2024-09-15T03:46:51.793 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000124091 2024-09-15T03:46:51.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.793 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:51.793 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163470+0000", 2024-09-15T03:46:51.793 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075976730000000001 2024-09-15T03:46:51.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.793 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:51.793 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163801+0000", 2024-09-15T03:46:51.794 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033157999999999999 2024-09-15T03:46:51.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.794 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.794 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:51.794 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163814+0000", 2024-09-15T03:46:51.794 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2659e-05 2024-09-15T03:46:51.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.794 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.794 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:51.794 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163935+0000", 2024-09-15T03:46:51.794 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000121155 2024-09-15T03:46:51.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.795 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.795 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:51.795 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.164131+0000", 2024-09-15T03:46:51.795 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019600499999999999 2024-09-15T03:46:51.795 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.795 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.795 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:51.795 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.165998+0000", 2024-09-15T03:46:51.795 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001866882 2024-09-15T03:46:51.795 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.795 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.795 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:51.795 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.166039+0000", 2024-09-15T03:46:51.796 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1247e-05 2024-09-15T03:46:51.796 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.796 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:51.796 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.796 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.796 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:232 3.fs0 3:f40d4749:::smithi00132172-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:46:51.796 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:46:51.796 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.5331665030000003, 2024-09-15T03:46:51.796 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084205730000000006, 2024-09-15T03:46:51.796 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:51.796 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:46:51.796 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:51.797 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:51.797 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:51.797 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 232 2024-09-15T03:46:51.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.797 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:51.797 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.797 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:51.797 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:46:51.797 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.797 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.797 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:51.797 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:46:51.798 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.798 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:51.798 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:46:51.798 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7399999999999999e-07 2024-09-15T03:46:51.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.798 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:51.798 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155708+0000", 2024-09-15T03:46:51.798 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6500000000000001e-06 2024-09-15T03:46:51.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:51.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155713+0000", 2024-09-15T03:46:51.799 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2519999999999999e-06 2024-09-15T03:46:51.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.799 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:51.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155716+0000", 2024-09-15T03:46:51.799 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4430000000000002e-06 2024-09-15T03:46:51.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.799 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156060+0000", 2024-09-15T03:46:51.799 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034440000000000002 2024-09-15T03:46:51.800 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.800 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.800 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2024-09-15T03:46:51.800 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156112+0000", 2024-09-15T03:46:51.800 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1928000000000002e-05 2024-09-15T03:46:51.800 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.800 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.800 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.800 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163929+0000", 2024-09-15T03:46:51.800 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0078168999999999999 2024-09-15T03:46:51.800 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.800 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.800 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:51.801 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.164027+0000", 2024-09-15T03:46:51.801 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8109999999999994e-05 2024-09-15T03:46:51.801 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.801 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.801 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:51.801 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.164126+0000", 2024-09-15T03:46:51.801 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9516000000000001e-05 2024-09-15T03:46:51.801 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.801 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:51.801 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.801 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.801 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.802 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:255 3.as0 3:5c0337d9:::smithi00132172-38:head [rollback 2] snapc 7=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:46:51.802 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.169130+0000", 2024-09-15T03:46:51.802 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.5197424010000002, 2024-09-15T03:46:51.802 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0050691950000000003, 2024-09-15T03:46:51.802 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:51.802 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:51.802 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:51.802 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:51.802 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:51.802 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 255 2024-09-15T03:46:51.802 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.802 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:51.802 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.803 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:51.803 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169130+0000", 2024-09-15T03:46:51.803 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.803 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.803 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.803 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:51.803 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169130+0000", 2024-09-15T03:46:51.803 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.803 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.803 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.803 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:51.803 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169131+0000", 2024-09-15T03:46:51.803 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9000000000000005e-07 2024-09-15T03:46:51.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.804 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:51.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169136+0000", 2024-09-15T03:46:51.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3859999999999998e-06 2024-09-15T03:46:51.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.804 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:51.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169153+0000", 2024-09-15T03:46:51.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6889999999999999e-05 2024-09-15T03:46:51.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.804 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:51.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169164+0000", 2024-09-15T03:46:51.805 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.025e-05 2024-09-15T03:46:51.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.805 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170098+0000", 2024-09-15T03:46:51.805 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00093477700000000003 2024-09-15T03:46:51.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.805 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:51.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170185+0000", 2024-09-15T03:46:51.805 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6489999999999994e-05 2024-09-15T03:46:51.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.806 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.806 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:51.806 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170689+0000", 2024-09-15T03:46:51.806 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00050395300000000002 2024-09-15T03:46:51.806 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.806 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.806 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:51.806 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170775+0000", 2024-09-15T03:46:51.806 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6281999999999998e-05 2024-09-15T03:46:51.806 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.806 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.806 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:51.806 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170818+0000", 2024-09-15T03:46:51.807 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2608000000000003e-05 2024-09-15T03:46:51.807 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.807 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.807 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:51.807 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171560+0000", 2024-09-15T03:46:51.807 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00074203500000000005 2024-09-15T03:46:51.807 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.807 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.807 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:51.807 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171628+0000", 2024-09-15T03:46:51.807 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8478000000000004e-05 2024-09-15T03:46:51.807 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.808 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.808 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:51.808 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173860+0000", 2024-09-15T03:46:51.808 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022317499999999998 2024-09-15T03:46:51.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.808 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.808 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:51.808 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174199+0000", 2024-09-15T03:46:51.808 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033930599999999999 2024-09-15T03:46:51.808 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.808 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:51.808 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.809 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:256 3.as0 3:5c0337d9:::smithi00132172-38:head [append 0~0] snapc 7=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:46:51.809 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.169218+0000", 2024-09-15T03:46:51.809 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.5196540670000003, 2024-09-15T03:46:51.809 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0051095940000000003, 2024-09-15T03:46:51.809 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:51.809 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:51.809 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:51.809 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:51.809 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:51.809 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 256 2024-09-15T03:46:51.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.809 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:51.810 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.810 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:51.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169218+0000", 2024-09-15T03:46:51.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.810 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.810 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:51.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169218+0000", 2024-09-15T03:46:51.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.810 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.810 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:51.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169220+0000", 2024-09-15T03:46:51.811 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2699999999999999e-06 2024-09-15T03:46:51.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.811 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:51.811 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169225+0000", 2024-09-15T03:46:51.811 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4199999999999998e-06 2024-09-15T03:46:51.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.811 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:51.811 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169247+0000", 2024-09-15T03:46:51.811 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1766000000000001e-05 2024-09-15T03:46:51.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.811 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:51.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.169257+0000", 2024-09-15T03:46:51.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0243e-05 2024-09-15T03:46:51.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170842+0000", 2024-09-15T03:46:51.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001585003 2024-09-15T03:46:51.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:51.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170892+0000", 2024-09-15T03:46:51.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9496000000000002e-05 2024-09-15T03:46:51.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:51.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171194+0000", 2024-09-15T03:46:51.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030285099999999998 2024-09-15T03:46:51.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:51.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171283+0000", 2024-09-15T03:46:51.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8219000000000004e-05 2024-09-15T03:46:51.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.814 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:51.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171563+0000", 2024-09-15T03:46:51.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00028080100000000001 2024-09-15T03:46:51.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.814 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:51.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171595+0000", 2024-09-15T03:46:51.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1322999999999998e-05 2024-09-15T03:46:51.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.814 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:51.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171633+0000", 2024-09-15T03:46:51.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8293000000000001e-05 2024-09-15T03:46:51.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.815 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.815 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:51.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174005+0000", 2024-09-15T03:46:51.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023715300000000002 2024-09-15T03:46:51.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.815 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.815 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:51.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174328+0000", 2024-09-15T03:46:51.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000323379 2024-09-15T03:46:51.815 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.816 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:51.816 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.816 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 39/31 ECSubWrite(tid=43, reqid=client.4521.0:252, at_version=39'27, trim_to=0'0, roll_forward_to=39'25))", 2024-09-15T03:46:51.816 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:46:51.816 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.51052681, 2024-09-15T03:46:51.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082900309999999998, 2024-09-15T03:46:51.816 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:51.816 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:51.816 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:51.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:51.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:46:51.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:51.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:46:51.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:51.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:46:51.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6199999999999998e-07 2024-09-15T03:46:51.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.818 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:51.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178642+0000", 2024-09-15T03:46:51.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029559399999999999 2024-09-15T03:46:51.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.818 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:51.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178653+0000", 2024-09-15T03:46:51.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0865e-05 2024-09-15T03:46:51.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.818 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:51.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178658+0000", 2024-09-15T03:46:51.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3720000000000004e-06 2024-09-15T03:46:51.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178695+0000", 2024-09-15T03:46:51.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6708999999999997e-05 2024-09-15T03:46:51.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:51.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178701+0000", 2024-09-15T03:46:51.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7989999999999996e-06 2024-09-15T03:46:51.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:51.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178926+0000", 2024-09-15T03:46:51.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022405299999999999 2024-09-15T03:46:51.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:51.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178935+0000", 2024-09-15T03:46:51.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9649999999999997e-06 2024-09-15T03:46:51.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:51.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179230+0000", 2024-09-15T03:46:51.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029551000000000002 2024-09-15T03:46:51.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.821 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:51.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179247+0000", 2024-09-15T03:46:51.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6685999999999999e-05 2024-09-15T03:46:51.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.821 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:51.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.186636+0000", 2024-09-15T03:46:51.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0073889159999999997 2024-09-15T03:46:51.821 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.822 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:51.822 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.822 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 39/31 ECSubWrite(tid=44, reqid=client.4521.0:252, at_version=0'0, trim_to=0'0, roll_forward_to=39'27))", 2024-09-15T03:46:51.822 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.180059+0000", 2024-09-15T03:46:51.822 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.5088135469999999, 2024-09-15T03:46:51.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00676445, 2024-09-15T03:46:51.822 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:51.822 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:51.822 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:51.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:51.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180059+0000", 2024-09-15T03:46:51.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:51.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180059+0000", 2024-09-15T03:46:51.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:51.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180059+0000", 2024-09-15T03:46:51.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:46:51.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:51.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180070+0000", 2024-09-15T03:46:51.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0407e-05 2024-09-15T03:46:51.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:51.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180083+0000", 2024-09-15T03:46:51.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3341e-05 2024-09-15T03:46:51.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:51.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180089+0000", 2024-09-15T03:46:51.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8000000000000004e-06 2024-09-15T03:46:51.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180126+0000", 2024-09-15T03:46:51.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7151999999999999e-05 2024-09-15T03:46:51.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:51.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180136+0000", 2024-09-15T03:46:51.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8349999999999999e-06 2024-09-15T03:46:51.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:51.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180200+0000", 2024-09-15T03:46:51.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3612e-05 2024-09-15T03:46:51.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:51.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180206+0000", 2024-09-15T03:46:51.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3749999999999999e-06 2024-09-15T03:46:51.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:51.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180235+0000", 2024-09-15T03:46:51.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9068999999999999e-05 2024-09-15T03:46:51.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:51.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180244+0000", 2024-09-15T03:46:51.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1030000000000008e-06 2024-09-15T03:46:51.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:51.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.186823+0000", 2024-09-15T03:46:51.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0065792130000000004 2024-09-15T03:46:51.827 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.827 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:51.828 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.828 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.6s7 40/31 ECSubWrite(tid=61, reqid=client.4521.0:267, at_version=40'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:51.828 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:47.199957+0000", 2024-09-15T03:46:51.828 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.4889154050000002, 2024-09-15T03:46:51.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.004752816, 2024-09-15T03:46:51.828 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:51.828 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:51.828 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:51.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:51.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.199957+0000", 2024-09-15T03:46:51.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.829 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:51.829 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.199957+0000", 2024-09-15T03:46:51.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:51.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.829 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:51.829 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.199958+0000", 2024-09-15T03:46:51.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3900000000000005e-07 2024-09-15T03:46:51.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.829 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:51.830 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.199970+0000", 2024-09-15T03:46:51.830 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2584e-05 2024-09-15T03:46:51.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.830 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.830 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:51.830 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.199980+0000", 2024-09-15T03:46:51.830 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6940000000000007e-06 2024-09-15T03:46:51.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.830 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.830 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:51.830 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.199984+0000", 2024-09-15T03:46:51.830 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0389999999999998e-06 2024-09-15T03:46:51.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:51.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.200011+0000", 2024-09-15T03:46:51.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7251e-05 2024-09-15T03:46:51.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:51.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.200017+0000", 2024-09-15T03:46:51.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0299999999999999e-06 2024-09-15T03:46:51.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:51.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.200071+0000", 2024-09-15T03:46:51.832 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4036999999999998e-05 2024-09-15T03:46:51.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.832 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:51.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.201934+0000", 2024-09-15T03:46:51.832 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001862697 2024-09-15T03:46:51.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.832 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:51.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.202023+0000", 2024-09-15T03:46:51.832 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8691999999999997e-05 2024-09-15T03:46:51.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.833 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:51.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.202048+0000", 2024-09-15T03:46:51.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5698999999999999e-05 2024-09-15T03:46:51.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:51.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:51.833 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:51.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:47.204710+0000", 2024-09-15T03:46:51.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002661554 2024-09-15T03:46:51.833 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.833 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:51.833 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.833 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:51.833 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:51.834 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:51.835 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:46:51.835 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:46:51.836 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:46:51.836 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:46:51.836 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:51.834+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:51.837 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:46:51.837 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:46:51.837 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:46:51.838 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:46:51.838 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:46:51.838 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:46:51.839 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:46:51.839 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:46:51.839 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:46:51.839 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:46:51.840 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:46:51.840 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:46:51.840 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:46:51.846 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:46:51.846 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:46:51.861 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:46:51.861 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:46:51.876 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:46:51.876 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:46:51.891 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:46:51.891 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:46:51.943 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:51.943 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:51.943 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:51.940+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:46:51.950 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:46:52.037 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:52.036+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:52.091 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:52.091 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:52.099 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:46:52.137 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:52.137+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:46:52.169 INFO:tasks.rados.rados.0.smithi001.stdout:118: done (2 left) 2024-09-15T03:46:52.169 INFO:tasks.rados.rados.0.smithi001.stdout:119: done (1 left) 2024-09-15T03:46:52.169 INFO:tasks.rados.rados.0.smithi001.stdout:120: snap_remove snap 6 2024-09-15T03:46:52.204 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:52.204 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:52.213 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:46:52.238 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:52.237+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:52.313 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:52.313 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:52.324 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:46:52.368 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:52.366+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:46:52.425 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:52.425 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:52.433 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:46:52.440 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:52.439+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:46:52.540 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:52.539+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:52.574 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:52.574 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:52.583 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:46:52.641 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:52.640+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:52.724 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:52.724 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:52.733 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:46:52.742 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:52.741+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:52.833 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:52.834 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:52.842 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:46:52.843 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:52.841+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:46:52.944 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:52.944 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:52.944 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:52.943+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:52.952 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:46:53.045 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:53.043+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:53.052 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:53.053 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:53.061 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:46:53.146 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:53.145+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:46:53.173 INFO:tasks.rados.rados.0.smithi001.stdout:120: done (1 left) 2024-09-15T03:46:53.173 INFO:tasks.rados.rados.0.smithi001.stdout:121: rollback oid 29 current snap is 10 2024-09-15T03:46:53.173 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 29 to 7 2024-09-15T03:46:53.173 INFO:tasks.rados.rados.0.smithi001.stdout:116: finishing copy_from racing read to smithi00132172-26 2024-09-15T03:46:53.173 INFO:tasks.rados.rados.0.smithi001.stdout:116: finishing copy_from to smithi00132172-26 2024-09-15T03:46:53.174 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 26 v 12 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-15T03:46:53.174 INFO:tasks.rados.rados.0.smithi001.stdout:116: done (1 left) 2024-09-15T03:46:53.174 INFO:tasks.rados.rados.0.smithi001.stdout:122: rmattr oid 46 current snap is 10 2024-09-15T03:46:53.174 INFO:tasks.rados.rados.0.smithi001.stdout:122: done (1 left) 2024-09-15T03:46:53.174 INFO:tasks.rados.rados.0.smithi001.stdout:123: snap_remove snap 3 2024-09-15T03:46:53.182 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T03:46:53.206 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:53.207 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:53.216 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:46:53.258 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:53.256+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:53.349 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:53.348+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:53.359 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:53.360 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:53.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:46:53.449 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:53.449+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:46:53.482 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:53.483 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:53.491 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:46:53.553 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:53.552+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:53.635 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:53.635 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:53.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:46:53.654 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:53.653+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:53.744 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:53.744 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:53.752 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:46:53.757 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:53.755+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:46:53.857 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:53.858 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:53.858 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:53.857+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:53.866 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:46:53.902 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:46:53.961 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:53.960+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:53.966 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:53.966 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:53.974 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:46:54.022 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:46:54.022 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:46:54.063 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:54.062+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:46:54.075 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:54.075 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:54.084 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:46:54.170 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:54.169+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:46:54.177 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-15T03:46:54.178 INFO:tasks.rados.rados.0.smithi001.stdout:123: done (1 left) 2024-09-15T03:46:54.178 INFO:tasks.rados.rados.0.smithi001.stdout:124: snap_create 2024-09-15T03:46:54.178 INFO:tasks.rados.rados.0.smithi001.stdout:121: finishing rollback tid 0 to smithi00132172-29 2024-09-15T03:46:54.178 INFO:tasks.rados.rados.0.smithi001.stdout:121: finishing rollback tid 1 to smithi00132172-29 2024-09-15T03:46:54.179 INFO:tasks.rados.rados.0.smithi001.stdout:121: finishing rollback tid 2 to smithi00132172-29 2024-09-15T03:46:54.179 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 29 v 48 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-15T03:46:54.236 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:54.236 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:54.237 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:46:54.237 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:46:54.244 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:46:54.251 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:46:54.252 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:46:54.266 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:46:54.266 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:46:54.280 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:46:54.280 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:46:54.286 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:54.285+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:46:54.295 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:46:54.295 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:46:54.310 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:46:54.310 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:46:54.324 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:46:54.324 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:46:54.339 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:46:54.339 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:46:54.354 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:46:54.355 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:46:54.365 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:54.364+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:54.372 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:46:54.372 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:46:54.386 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:54.386 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:46:54.386 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:46:54.386 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:46:54.386 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:54.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:46:54.466 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:54.465+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:46:54.497 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:54.497 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:46:54.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:46:54.497 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:46:54.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.497 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:171 3.4s0 3:2d1c787c:::smithi00132172-45:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-15T03:46:54.498 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.553202+0000", 2024-09-15T03:46:54.498 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.941887380000001, 2024-09-15T03:46:54.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0083718339999999999, 2024-09-15T03:46:54.498 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:54.498 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:54.498 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:54.498 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:54.498 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:54.498 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 171 2024-09-15T03:46:54.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.498 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:54.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:54.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553202+0000", 2024-09-15T03:46:54.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:54.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553202+0000", 2024-09-15T03:46:54.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:54.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553203+0000", 2024-09-15T03:46:54.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.122e-06 2024-09-15T03:46:54.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:54.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553207+0000", 2024-09-15T03:46:54.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2200000000000003e-06 2024-09-15T03:46:54.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:54.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553219+0000", 2024-09-15T03:46:54.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1316e-05 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553224+0000", 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6710000000000004e-06 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553265+0000", 2024-09-15T03:46:54.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0939000000000002e-05 2024-09-15T03:46:54.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:54.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553341+0000", 2024-09-15T03:46:54.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5939000000000006e-05 2024-09-15T03:46:54.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:54.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559019+0000", 2024-09-15T03:46:54.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056774299999999998 2024-09-15T03:46:54.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:54.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559137+0000", 2024-09-15T03:46:54.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117751 2024-09-15T03:46:54.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:54.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559213+0000", 2024-09-15T03:46:54.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6882e-05 2024-09-15T03:46:54.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:54.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559551+0000", 2024-09-15T03:46:54.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033777900000000002 2024-09-15T03:46:54.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:54.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559571+0000", 2024-09-15T03:46:54.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9728000000000002e-05 2024-09-15T03:46:54.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:54.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.560343+0000", 2024-09-15T03:46:54.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00077172899999999997 2024-09-15T03:46:54.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:54.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.561574+0000", 2024-09-15T03:46:54.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012313280000000001 2024-09-15T03:46:54.506 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.506 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:54.506 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.506 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:172 3.4s0 3:2d1c787c:::smithi00132172-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-15T03:46:54.506 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:39.553234+0000", 2024-09-15T03:46:54.506 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.941855533, 2024-09-15T03:46:54.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062080829999999997, 2024-09-15T03:46:54.507 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:54.507 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:46:54.507 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:54.507 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:54.507 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:54.507 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 172 2024-09-15T03:46:54.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.507 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:54.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:54.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553234+0000", 2024-09-15T03:46:54.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:54.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553234+0000", 2024-09-15T03:46:54.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:54.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553235+0000", 2024-09-15T03:46:54.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2600000000000002e-07 2024-09-15T03:46:54.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:54.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553237+0000", 2024-09-15T03:46:54.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4200000000000001e-06 2024-09-15T03:46:54.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:54.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553245+0000", 2024-09-15T03:46:54.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5870000000000004e-06 2024-09-15T03:46:54.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:54.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553249+0000", 2024-09-15T03:46:54.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2309999999999998e-06 2024-09-15T03:46:54.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:54.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553479+0000", 2024-09-15T03:46:54.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023039299999999999 2024-09-15T03:46:54.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:46:54.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.553549+0000", 2024-09-15T03:46:54.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9381999999999994e-05 2024-09-15T03:46:54.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:54.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559254+0000", 2024-09-15T03:46:54.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057053210000000002 2024-09-15T03:46:54.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:54.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559348+0000", 2024-09-15T03:46:54.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3715000000000007e-05 2024-09-15T03:46:54.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:54.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:39.559442+0000", 2024-09-15T03:46:54.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4507999999999997e-05 2024-09-15T03:46:54.513 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.513 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:54.513 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.513 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=21, reqid=client.4521.0:184, at_version=33'6, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:54.513 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.388580409999999, 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010901256999999999, 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:46:54.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0200000000000001e-07 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106529+0000", 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9539e-05 2024-09-15T03:46:54.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:54.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106542+0000", 2024-09-15T03:46:54.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2731999999999999e-05 2024-09-15T03:46:54.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:54.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106549+0000", 2024-09-15T03:46:54.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3339999999999996e-06 2024-09-15T03:46:54.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:54.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106613+0000", 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4527000000000001e-05 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106622+0000", 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.755e-06 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106841+0000", 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021906200000000001 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:54.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106869+0000", 2024-09-15T03:46:54.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.739e-05 2024-09-15T03:46:54.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:54.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106948+0000", 2024-09-15T03:46:54.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9422e-05 2024-09-15T03:46:54.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:54.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106965+0000", 2024-09-15T03:46:54.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7010000000000001e-05 2024-09-15T03:46:54.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.117410+0000", 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010445284000000001 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=22, reqid=client.4521.0:187, at_version=33'8, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.106795+0000", 2024-09-15T03:46:54.520 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.388294815, 2024-09-15T03:46:54.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0075031530000000003, 2024-09-15T03:46:54.521 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:54.521 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:54.521 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:54.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:54.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106795+0000", 2024-09-15T03:46:54.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:54.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106795+0000", 2024-09-15T03:46:54.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106795+0000", 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1900000000000002e-07 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106814+0000", 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8376000000000002e-05 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:54.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106825+0000", 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1096e-05 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106832+0000", 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4939999999999997e-06 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:54.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106877+0000", 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4799999999999998e-05 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106888+0000", 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0621e-05 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.107075+0000", 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000187104 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.107099+0000", 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4369e-05 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.107135+0000", 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.553e-05 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.107453+0000", 2024-09-15T03:46:54.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031841299999999998 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114298+0000", 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068447309999999997 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.386666923, 2024-09-15T03:46:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015296944, 2024-09-15T03:46:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:46:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:46:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:46:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5499999999999998e-07 2024-09-15T03:46:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108447+0000", 2024-09-15T03:46:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3314000000000001e-05 2024-09-15T03:46:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108460+0000", 2024-09-15T03:46:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3444e-05 2024-09-15T03:46:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108464+0000", 2024-09-15T03:46:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3610000000000003e-06 2024-09-15T03:46:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108494+0000", 2024-09-15T03:46:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9688000000000001e-05 2024-09-15T03:46:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108502+0000", 2024-09-15T03:46:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.678e-06 2024-09-15T03:46:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108601+0000", 2024-09-15T03:46:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9748999999999997e-05 2024-09-15T03:46:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108610+0000", 2024-09-15T03:46:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2490000000000001e-06 2024-09-15T03:46:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109977+0000", 2024-09-15T03:46:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013669450000000001 2024-09-15T03:46:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109993+0000", 2024-09-15T03:46:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6688e-05 2024-09-15T03:46:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123720+0000", 2024-09-15T03:46:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013726172999999999 2024-09-15T03:46:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:46:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.386617262, 2024-09-15T03:46:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015105210000000001, 2024-09-15T03:46:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:46:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:46:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108473+0000", 2024-09-15T03:46:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.84e-07 2024-09-15T03:46:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108486+0000", 2024-09-15T03:46:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3845999999999999e-05 2024-09-15T03:46:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108491+0000", 2024-09-15T03:46:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4719999999999997e-06 2024-09-15T03:46:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108495+0000", 2024-09-15T03:46:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.822e-06 2024-09-15T03:46:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108546+0000", 2024-09-15T03:46:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0853000000000003e-05 2024-09-15T03:46:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108555+0000", 2024-09-15T03:46:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3319999999999998e-06 2024-09-15T03:46:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108638+0000", 2024-09-15T03:46:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3580999999999998e-05 2024-09-15T03:46:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109983+0000", 2024-09-15T03:46:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001344392 2024-09-15T03:46:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110023+0000", 2024-09-15T03:46:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9895999999999999e-05 2024-09-15T03:46:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110040+0000", 2024-09-15T03:46:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.747e-05 2024-09-15T03:46:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123577+0000", 2024-09-15T03:46:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013537262 2024-09-15T03:46:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:46:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.386583383, 2024-09-15T03:46:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015311313999999999, 2024-09-15T03:46:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:46:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:46:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:46:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8500000000000002e-07 2024-09-15T03:46:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108517+0000", 2024-09-15T03:46:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0543000000000001e-05 2024-09-15T03:46:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108524+0000", 2024-09-15T03:46:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7170000000000002e-06 2024-09-15T03:46:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108528+0000", 2024-09-15T03:46:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3970000000000004e-06 2024-09-15T03:46:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108632+0000", 2024-09-15T03:46:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103687 2024-09-15T03:46:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108637+0000", 2024-09-15T03:46:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4389999999999999e-06 2024-09-15T03:46:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108713+0000", 2024-09-15T03:46:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6223000000000005e-05 2024-09-15T03:46:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109984+0000", 2024-09-15T03:46:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012700719999999999 2024-09-15T03:46:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110023+0000", 2024-09-15T03:46:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9864999999999999e-05 2024-09-15T03:46:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110041+0000", 2024-09-15T03:46:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8040999999999999e-05 2024-09-15T03:46:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123817+0000", 2024-09-15T03:46:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013776045000000001 2024-09-15T03:46:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:46:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.386089775, 2024-09-15T03:46:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014864502, 2024-09-15T03:46:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:46:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:46:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:46:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:46:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109016+0000", 2024-09-15T03:46:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5889999999999999e-05 2024-09-15T03:46:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109029+0000", 2024-09-15T03:46:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2812e-05 2024-09-15T03:46:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109035+0000", 2024-09-15T03:46:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0109999999999997e-06 2024-09-15T03:46:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109073+0000", 2024-09-15T03:46:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8260000000000003e-05 2024-09-15T03:46:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109080+0000", 2024-09-15T03:46:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0190000000000001e-06 2024-09-15T03:46:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109165+0000", 2024-09-15T03:46:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4957000000000007e-05 2024-09-15T03:46:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109984+0000", 2024-09-15T03:46:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081866399999999998 2024-09-15T03:46:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110024+0000", 2024-09-15T03:46:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9644999999999997e-05 2024-09-15T03:46:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110072+0000", 2024-09-15T03:46:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8418999999999998e-05 2024-09-15T03:46:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123864+0000", 2024-09-15T03:46:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013792234 2024-09-15T03:46:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=27, reqid=client.4521.0:189, at_version=0'0, trim_to=0'0, roll_forward_to=33'10))", 2024-09-15T03:46:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.114624+0000", 2024-09-15T03:46:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.380465786, 2024-09-15T03:46:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092946850000000004, 2024-09-15T03:46:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114624+0000", 2024-09-15T03:46:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114624+0000", 2024-09-15T03:46:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114624+0000", 2024-09-15T03:46:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3600000000000004e-07 2024-09-15T03:46:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114633+0000", 2024-09-15T03:46:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1579999999999994e-06 2024-09-15T03:46:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114644+0000", 2024-09-15T03:46:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.04e-05 2024-09-15T03:46:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114649+0000", 2024-09-15T03:46:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8760000000000004e-06 2024-09-15T03:46:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114687+0000", 2024-09-15T03:46:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8438e-05 2024-09-15T03:46:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114697+0000", 2024-09-15T03:46:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6479999999999998e-06 2024-09-15T03:46:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114818+0000", 2024-09-15T03:46:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121116 2024-09-15T03:46:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114830+0000", 2024-09-15T03:46:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1844e-05 2024-09-15T03:46:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114921+0000", 2024-09-15T03:46:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1461999999999999e-05 2024-09-15T03:46:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114934+0000", 2024-09-15T03:46:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2857000000000001e-05 2024-09-15T03:46:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123918+0000", 2024-09-15T03:46:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089843500000000003 2024-09-15T03:46:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=28, reqid=client.4521.0:186, at_version=0'0, trim_to=0'0, roll_forward_to=33'8))", 2024-09-15T03:46:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.114815+0000", 2024-09-15T03:46:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.380274929, 2024-09-15T03:46:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.009158846, 2024-09-15T03:46:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114815+0000", 2024-09-15T03:46:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114815+0000", 2024-09-15T03:46:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114815+0000", 2024-09-15T03:46:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.75e-07 2024-09-15T03:46:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114824+0000", 2024-09-15T03:46:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9819999999999997e-06 2024-09-15T03:46:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114839+0000", 2024-09-15T03:46:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4521999999999999e-05 2024-09-15T03:46:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114845+0000", 2024-09-15T03:46:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7399999999999998e-06 2024-09-15T03:46:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114910+0000", 2024-09-15T03:46:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4492999999999994e-05 2024-09-15T03:46:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114919+0000", 2024-09-15T03:46:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9299999999999992e-06 2024-09-15T03:46:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.115022+0000", 2024-09-15T03:46:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103019 2024-09-15T03:46:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.115030+0000", 2024-09-15T03:46:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6690000000000006e-06 2024-09-15T03:46:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.115056+0000", 2024-09-15T03:46:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6001000000000001e-05 2024-09-15T03:46:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.115070+0000", 2024-09-15T03:46:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4552999999999999e-05 2024-09-15T03:46:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123973+0000", 2024-09-15T03:46:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089033619999999997 2024-09-15T03:46:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=34, reqid=client.4521.0:197, at_version=34'21, trim_to=0'0, roll_forward_to=34'18))", 2024-09-15T03:46:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:46:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.371077894999999, 2024-09-15T03:46:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012992399, 2024-09-15T03:46:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:46:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:46:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124013+0000", 2024-09-15T03:46:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6299999999999993e-07 2024-09-15T03:46:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124044+0000", 2024-09-15T03:46:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0995999999999999e-05 2024-09-15T03:46:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124062+0000", 2024-09-15T03:46:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8413000000000001e-05 2024-09-15T03:46:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124069+0000", 2024-09-15T03:46:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3930000000000002e-06 2024-09-15T03:46:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124150+0000", 2024-09-15T03:46:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1097e-05 2024-09-15T03:46:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124161+0000", 2024-09-15T03:46:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0720000000000001e-05 2024-09-15T03:46:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124402+0000", 2024-09-15T03:46:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000240442 2024-09-15T03:46:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125141+0000", 2024-09-15T03:46:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00073915500000000002 2024-09-15T03:46:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125191+0000", 2024-09-15T03:46:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9951999999999999e-05 2024-09-15T03:46:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125251+0000", 2024-09-15T03:46:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9938000000000001e-05 2024-09-15T03:46:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.137004+0000", 2024-09-15T03:46:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011753329999999999 2024-09-15T03:46:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=35, reqid=client.4521.0:197, at_version=0'0, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:46:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:46:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.369137851, 2024-09-15T03:46:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011202456, 2024-09-15T03:46:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:46:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:46:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125953+0000", 2024-09-15T03:46:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9100000000000002e-07 2024-09-15T03:46:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125964+0000", 2024-09-15T03:46:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1399999999999999e-05 2024-09-15T03:46:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125979+0000", 2024-09-15T03:46:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4810000000000001e-05 2024-09-15T03:46:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126004+0000", 2024-09-15T03:46:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4913e-05 2024-09-15T03:46:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126059+0000", 2024-09-15T03:46:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5668999999999998e-05 2024-09-15T03:46:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126071+0000", 2024-09-15T03:46:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1219999999999999e-05 2024-09-15T03:46:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126151+0000", 2024-09-15T03:46:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0572e-05 2024-09-15T03:46:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126163+0000", 2024-09-15T03:46:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2101e-05 2024-09-15T03:46:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.127945+0000", 2024-09-15T03:46:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001781803 2024-09-15T03:46:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.127982+0000", 2024-09-15T03:46:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7129999999999999e-05 2024-09-15T03:46:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.137154+0000", 2024-09-15T03:46:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0091719469999999997 2024-09-15T03:46:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 38/31 ECSubWrite(tid=36, reqid=client.4521.0:220, at_version=38'23, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:46:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.148041+0000", 2024-09-15T03:46:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.3470482700000002, 2024-09-15T03:46:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007978232, 2024-09-15T03:46:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148041+0000", 2024-09-15T03:46:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148041+0000", 2024-09-15T03:46:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148042+0000", 2024-09-15T03:46:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5800000000000001e-07 2024-09-15T03:46:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148481+0000", 2024-09-15T03:46:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043990300000000001 2024-09-15T03:46:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148498+0000", 2024-09-15T03:46:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7033000000000001e-05 2024-09-15T03:46:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148505+0000", 2024-09-15T03:46:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.754e-06 2024-09-15T03:46:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148586+0000", 2024-09-15T03:46:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0706999999999998e-05 2024-09-15T03:46:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148597+0000", 2024-09-15T03:46:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0812000000000001e-05 2024-09-15T03:46:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.149131+0000", 2024-09-15T03:46:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053444699999999996 2024-09-15T03:46:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.149141+0000", 2024-09-15T03:46:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0027000000000001e-05 2024-09-15T03:46:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.149660+0000", 2024-09-15T03:46:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051922199999999996 2024-09-15T03:46:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.149680+0000", 2024-09-15T03:46:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9962e-05 2024-09-15T03:46:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156019+0000", 2024-09-15T03:46:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063391070000000001 2024-09-15T03:46:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:237 3.4s0 3:2d1c787c:::smithi00132172-45:head [rollback 3] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:46:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.155920+0000", 2024-09-15T03:46:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.3391691489999999, 2024-09-15T03:46:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069997130000000003, 2024-09-15T03:46:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 237 2024-09-15T03:46:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155920+0000", 2024-09-15T03:46:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155920+0000", 2024-09-15T03:46:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155921+0000", 2024-09-15T03:46:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2300000000000001e-07 2024-09-15T03:46:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155926+0000", 2024-09-15T03:46:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1789999999999999e-06 2024-09-15T03:46:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155935+0000", 2024-09-15T03:46:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0950000000000002e-06 2024-09-15T03:46:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155940+0000", 2024-09-15T03:46:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9180000000000002e-06 2024-09-15T03:46:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157278+0000", 2024-09-15T03:46:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013380320000000001 2024-09-15T03:46:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157379+0000", 2024-09-15T03:46:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100318 2024-09-15T03:46:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158144+0000", 2024-09-15T03:46:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00076588999999999997 2024-09-15T03:46:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158230+0000", 2024-09-15T03:46:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5791000000000006e-05 2024-09-15T03:46:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158288+0000", 2024-09-15T03:46:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7673000000000002e-05 2024-09-15T03:46:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158326+0000", 2024-09-15T03:46:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8034000000000002e-05 2024-09-15T03:46:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160003+0000", 2024-09-15T03:46:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016768900000000001 2024-09-15T03:46:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.161382+0000", 2024-09-15T03:46:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013790619999999999 2024-09-15T03:46:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162920+0000", 2024-09-15T03:46:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015382079999999999 2024-09-15T03:46:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:238 3.4s0 3:2d1c787c:::smithi00132172-45:head [append 0~0] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:46:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.155945+0000", 2024-09-15T03:46:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.3391444240000006, 2024-09-15T03:46:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071004450000000004, 2024-09-15T03:46:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 238 2024-09-15T03:46:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155945+0000", 2024-09-15T03:46:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155945+0000", 2024-09-15T03:46:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155946+0000", 2024-09-15T03:46:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4499999999999997e-07 2024-09-15T03:46:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155948+0000", 2024-09-15T03:46:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.295e-06 2024-09-15T03:46:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155954+0000", 2024-09-15T03:46:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0650000000000004e-06 2024-09-15T03:46:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155958+0000", 2024-09-15T03:46:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1819999999999999e-06 2024-09-15T03:46:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158313+0000", 2024-09-15T03:46:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023549030000000002 2024-09-15T03:46:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158431+0000", 2024-09-15T03:46:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011800599999999999 2024-09-15T03:46:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159002+0000", 2024-09-15T03:46:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00057089 2024-09-15T03:46:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159095+0000", 2024-09-15T03:46:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2749000000000003e-05 2024-09-15T03:46:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159110+0000", 2024-09-15T03:46:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5192e-05 2024-09-15T03:46:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159173+0000", 2024-09-15T03:46:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2947000000000006e-05 2024-09-15T03:46:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160004+0000", 2024-09-15T03:46:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083124000000000002 2024-09-15T03:46:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.161650+0000", 2024-09-15T03:46:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016463700000000001 2024-09-15T03:46:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.163046+0000", 2024-09-15T03:46:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001395161 2024-09-15T03:46:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:243 3.3s0 3:c7c3ef16:::smithi00132172-25:head [create,setxattr iZnWfebab`ptjURR\\f`psW]tpS`qpUaeWkY[Oqep (1330) in=40b,setxattr f[mZ_qQmcZejQ`fbdeic_tkPUaTYcnm[WprjVmc] (12488) in=40b,setxattr tkZgQ]S_pdlhZnqrVaZRa`oQQr_USmUfhado^dXT (1121) in=40b,setxattr srWjTqXhcth`hfZmQY_fg]UqNjWofa^dfgOV`sYP (455) in=40b,setxattr ^_mdiRroj_TtQ^gcNj`XtkW[ooqY`PWXqYmcOk_n (138) in=40b,setxattr jQlajdtcalhkcfYaakZfNb[^_TjkfcS`Vrk]q_d[ (806) in=40b,setxattr ZfYgkVcrsXcnpchgfcbldNrUTasnQNmfZW]Nbh\\o (561) in=40b,setxattr _hkYOqjPNhmhUlSNktfeQpsqSWjO\\fbf[fpTlZiO (1227) in=40b,setxattr ]mdV^Xdc\\sViXdkXNU[js_Qhh[b[sbfnZVXkdids (11788) in=40b,setxattr tb^mbOrNhfrhtce^g (455) in=17b,setxattr koTkVgrfU\\m`StnZ`Rirjhbhg`PhVqm^a^Xgaaqn (35) in=40b,setxattr aUtZPUPXPnQmT]\\\\n\\lZpnXakdNkTldVQoQhPUf] (559) in=40b,setxattr mrhR]^U`Tteo[Topmql`SNma\\pqfk`if]OppnYt` (770) in=40b,setxattr Oetk]RrnWeXbVctVtVrhXf_RfTdelqX^lWp[sTkp (1226) in=40b,setxattr WtedXoOd`VqnbksOqQkatk`gdTnW^lXbUtrlYgkl (3576) in=40b,setxattr dddOWN\\j\\XXg^fr[rZcskeVX]jOQ^ZmoWepk[]\\a (17824) in=40b,setxattr TeRO^djPgNZiemQ_aRXgP[]jgUZa\\^\\iUgesaqaN (208) in=40b,setxattr teqm]mUobZbWjcdQ_jTa[f[k[Sd\\lh]n\\osigqkN (7436) in=40b,setxattr dphaol[XkeitbkomWYaQctWfataUhbSiNntkWo`P (1893) in=40b,setxattr oQsnhO]ZklSVYQ[WUpaelY[p[Qie^UepOt`Z[q (455) in=38b] snapc 7=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:46:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:46:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.3261739630000005, 2024-09-15T03:46:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012365956000000001, 2024-09-15T03:46:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 243 2024-09-15T03:46:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:46:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:46:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168917+0000", 2024-09-15T03:46:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.725e-06 2024-09-15T03:46:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169030+0000", 2024-09-15T03:46:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113035 2024-09-15T03:46:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169047+0000", 2024-09-15T03:46:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6701000000000002e-05 2024-09-15T03:46:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169054+0000", 2024-09-15T03:46:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1369999999999996e-06 2024-09-15T03:46:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169140+0000", 2024-09-15T03:46:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6069999999999994e-05 2024-09-15T03:46:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169270+0000", 2024-09-15T03:46:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130071 2024-09-15T03:46:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.170661+0000", 2024-09-15T03:46:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001390666 2024-09-15T03:46:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171751+0000", 2024-09-15T03:46:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010899079999999999 2024-09-15T03:46:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171812+0000", 2024-09-15T03:46:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0797000000000001e-05 2024-09-15T03:46:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172279+0000", 2024-09-15T03:46:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046704500000000003 2024-09-15T03:46:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172299+0000", 2024-09-15T03:46:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0387999999999999e-05 2024-09-15T03:46:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.176110+0000", 2024-09-15T03:46:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038112340000000001 2024-09-15T03:46:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.181282+0000", 2024-09-15T03:46:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051711789999999997 2024-09-15T03:46:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 39/28 ECSubWrite(tid=42, reqid=client.4521.0:255, at_version=39'19, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:46:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.170801+0000", 2024-09-15T03:46:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.3242886089999999, 2024-09-15T03:46:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088143879999999994, 2024-09-15T03:46:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.170801+0000", 2024-09-15T03:46:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.170801+0000", 2024-09-15T03:46:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.170802+0000", 2024-09-15T03:46:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2099999999999995e-07 2024-09-15T03:46:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.170827+0000", 2024-09-15T03:46:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4902e-05 2024-09-15T03:46:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.170843+0000", 2024-09-15T03:46:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6568000000000001e-05 2024-09-15T03:46:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.170850+0000", 2024-09-15T03:46:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2010000000000002e-06 2024-09-15T03:46:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171020+0000", 2024-09-15T03:46:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169965 2024-09-15T03:46:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171031+0000", 2024-09-15T03:46:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0368e-05 2024-09-15T03:46:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171158+0000", 2024-09-15T03:46:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012757999999999999 2024-09-15T03:46:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171194+0000", 2024-09-15T03:46:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5339000000000002e-05 2024-09-15T03:46:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171259+0000", 2024-09-15T03:46:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5628000000000003e-05 2024-09-15T03:46:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171280+0000", 2024-09-15T03:46:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0268e-05 2024-09-15T03:46:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179615+0000", 2024-09-15T03:46:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0083357480000000005 2024-09-15T03:46:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 39/28 ECSubWrite(tid=43, reqid=client.4521.0:256, at_version=39'20, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:46:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.171322+0000", 2024-09-15T03:46:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.3237676189999998, 2024-09-15T03:46:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084811539999999994, 2024-09-15T03:46:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171322+0000", 2024-09-15T03:46:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171322+0000", 2024-09-15T03:46:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171322+0000", 2024-09-15T03:46:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1300000000000001e-07 2024-09-15T03:46:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171340+0000", 2024-09-15T03:46:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7978e-05 2024-09-15T03:46:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171352+0000", 2024-09-15T03:46:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1552000000000001e-05 2024-09-15T03:46:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171359+0000", 2024-09-15T03:46:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8639999999999998e-06 2024-09-15T03:46:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171415+0000", 2024-09-15T03:46:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6022e-05 2024-09-15T03:46:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171423+0000", 2024-09-15T03:46:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3869999999999995e-06 2024-09-15T03:46:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171587+0000", 2024-09-15T03:46:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000164267 2024-09-15T03:46:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171620+0000", 2024-09-15T03:46:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.222e-05 2024-09-15T03:46:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171658+0000", 2024-09-15T03:46:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8461999999999999e-05 2024-09-15T03:46:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171671+0000", 2024-09-15T03:46:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2944e-05 2024-09-15T03:46:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179803+0000", 2024-09-15T03:46:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0081320449999999992 2024-09-15T03:46:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 39/31 ECSubWrite(tid=43, reqid=client.4521.0:252, at_version=39'27, trim_to=0'0, roll_forward_to=39'25))", 2024-09-15T03:46:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.178401+0000", 2024-09-15T03:46:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.3166881069999992, 2024-09-15T03:46:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0072727959999999998, 2024-09-15T03:46:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.178401+0000", 2024-09-15T03:46:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.178401+0000", 2024-09-15T03:46:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.178402+0000", 2024-09-15T03:46:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9999999999999997e-07 2024-09-15T03:46:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.178465+0000", 2024-09-15T03:46:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2642000000000001e-05 2024-09-15T03:46:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.178477+0000", 2024-09-15T03:46:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2247e-05 2024-09-15T03:46:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.178483+0000", 2024-09-15T03:46:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5019999999999998e-06 2024-09-15T03:46:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:46:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7321999999999998e-05 2024-09-15T03:46:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.178554+0000", 2024-09-15T03:46:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2867000000000001e-05 2024-09-15T03:46:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.178871+0000", 2024-09-15T03:46:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031787100000000001 2024-09-15T03:46:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.178883+0000", 2024-09-15T03:46:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1653e-05 2024-09-15T03:46:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.178980+0000", 2024-09-15T03:46:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6788999999999998e-05 2024-09-15T03:46:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.178993+0000", 2024-09-15T03:46:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3137e-05 2024-09-15T03:46:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.185674+0000", 2024-09-15T03:46:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066811659999999997 2024-09-15T03:46:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:258 3.3s0 3:c9b663a2:::smithi00132172-7:head [delete] snapc 7=[6,5,3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e40)", 2024-09-15T03:46:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:47.146116+0000", 2024-09-15T03:46:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.3489731450000004, 2024-09-15T03:46:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096428380000000008, 2024-09-15T03:46:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 258 2024-09-15T03:46:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146116+0000", 2024-09-15T03:46:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146116+0000", 2024-09-15T03:46:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146119+0000", 2024-09-15T03:46:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.227e-06 2024-09-15T03:46:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146127+0000", 2024-09-15T03:46:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1529999999999997e-06 2024-09-15T03:46:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146146+0000", 2024-09-15T03:46:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9318999999999999e-05 2024-09-15T03:46:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146154+0000", 2024-09-15T03:46:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7200000000000006e-06 2024-09-15T03:46:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146233+0000", 2024-09-15T03:46:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8725000000000003e-05 2024-09-15T03:46:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:46:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146244+0000", 2024-09-15T03:46:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1725e-05 2024-09-15T03:46:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.149207+0000", 2024-09-15T03:46:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029627999999999998 2024-09-15T03:46:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.149326+0000", 2024-09-15T03:46:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011869199999999999 2024-09-15T03:46:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.150266+0000", 2024-09-15T03:46:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00094048500000000002 2024-09-15T03:46:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.150626+0000", 2024-09-15T03:46:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035946699999999998 2024-09-15T03:46:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.150751+0000", 2024-09-15T03:46:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012544800000000001 2024-09-15T03:46:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.151656+0000", 2024-09-15T03:46:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00090447499999999996 2024-09-15T03:46:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.153489+0000", 2024-09-15T03:46:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001833115 2024-09-15T03:46:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.155171+0000", 2024-09-15T03:46:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001682455 2024-09-15T03:46:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.155759+0000", 2024-09-15T03:46:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00058803200000000005 2024-09-15T03:46:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.626 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:54.626 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:54.626 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:54.627 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:46:54.628 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:46:54.628 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:54.626+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:46:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:46:54.728 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:54.726+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:46:54.769 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:54.770 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:46:54.770 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:46:54.770 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:54.779 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:46:54.828 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:54.827+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:46:54.828 INFO:tasks.rados.rados.0.smithi001.stdout:121: done (1 left) 2024-09-15T03:46:54.829 INFO:tasks.rados.rados.0.smithi001.stdout:124: done (0 left) 2024-09-15T03:46:54.829 INFO:tasks.rados.rados.0.smithi001.stdout:125: snap_create 2024-09-15T03:46:54.882 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:54.882 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:46:54.882 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:46:54.882 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:46:54.883 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:54.891 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:46:54.930 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:54.928+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:46:55.031 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:55.029+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:55.033 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:46:55.033 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:46:55.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:46:55.033 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:46:55.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.033 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs3 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108456+0000", 2024-09-15T03:46:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.923011131999999, 2024-09-15T03:46:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015075257, 2024-09-15T03:46:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108456+0000", 2024-09-15T03:46:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108456+0000", 2024-09-15T03:46:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108457+0000", 2024-09-15T03:46:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9999999999999997e-07 2024-09-15T03:46:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108477+0000", 2024-09-15T03:46:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0344e-05 2024-09-15T03:46:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108497+0000", 2024-09-15T03:46:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0143999999999999e-05 2024-09-15T03:46:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108504+0000", 2024-09-15T03:46:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8499999999999996e-06 2024-09-15T03:46:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108554+0000", 2024-09-15T03:46:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9567999999999997e-05 2024-09-15T03:46:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108568+0000", 2024-09-15T03:46:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4326e-05 2024-09-15T03:46:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108654+0000", 2024-09-15T03:46:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6193999999999995e-05 2024-09-15T03:46:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108663+0000", 2024-09-15T03:46:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7479999999999999e-06 2024-09-15T03:46:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111234+0000", 2024-09-15T03:46:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025709079999999998 2024-09-15T03:46:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111259+0000", 2024-09-15T03:46:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4797999999999999e-05 2024-09-15T03:46:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123531+0000", 2024-09-15T03:46:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012272777 2024-09-15T03:46:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as2 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:46:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.922782478, 2024-09-15T03:46:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0149108, 2024-09-15T03:46:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:46:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:46:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:46:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1699999999999999e-07 2024-09-15T03:46:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108710+0000", 2024-09-15T03:46:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5032999999999999e-05 2024-09-15T03:46:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108725+0000", 2024-09-15T03:46:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4788e-05 2024-09-15T03:46:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108733+0000", 2024-09-15T03:46:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4610000000000008e-06 2024-09-15T03:46:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108778+0000", 2024-09-15T03:46:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4224999999999998e-05 2024-09-15T03:46:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108800+0000", 2024-09-15T03:46:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2189999999999999e-05 2024-09-15T03:46:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108884+0000", 2024-09-15T03:46:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4569999999999998e-05 2024-09-15T03:46:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111237+0000", 2024-09-15T03:46:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002352313 2024-09-15T03:46:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114139+0000", 2024-09-15T03:46:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002902057 2024-09-15T03:46:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114173+0000", 2024-09-15T03:46:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4461000000000001e-05 2024-09-15T03:46:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123596+0000", 2024-09-15T03:46:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094222850000000007 2024-09-15T03:46:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs3 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:46:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.9227232, 2024-09-15T03:46:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014886043, 2024-09-15T03:46:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:46:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:46:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:46:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5100000000000001e-07 2024-09-15T03:46:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108759+0000", 2024-09-15T03:46:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4945e-05 2024-09-15T03:46:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108768+0000", 2024-09-15T03:46:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2309999999999996e-06 2024-09-15T03:46:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108795+0000", 2024-09-15T03:46:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7829000000000001e-05 2024-09-15T03:46:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108841+0000", 2024-09-15T03:46:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5330999999999997e-05 2024-09-15T03:46:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108850+0000", 2024-09-15T03:46:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9959999999999999e-06 2024-09-15T03:46:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108940+0000", 2024-09-15T03:46:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9823999999999999e-05 2024-09-15T03:46:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111237+0000", 2024-09-15T03:46:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002297927 2024-09-15T03:46:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114139+0000", 2024-09-15T03:46:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029020029999999998 2024-09-15T03:46:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114212+0000", 2024-09-15T03:46:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2177999999999999e-05 2024-09-15T03:46:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123630+0000", 2024-09-15T03:46:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094184279999999995 2024-09-15T03:46:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as2 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.109397+0000", 2024-09-15T03:46:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.922070142000001, 2024-09-15T03:46:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014322056, 2024-09-15T03:46:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109397+0000", 2024-09-15T03:46:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109397+0000", 2024-09-15T03:46:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109398+0000", 2024-09-15T03:46:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1600000000000001e-07 2024-09-15T03:46:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109410+0000", 2024-09-15T03:46:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2294e-05 2024-09-15T03:46:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109420+0000", 2024-09-15T03:46:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.764e-06 2024-09-15T03:46:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109426+0000", 2024-09-15T03:46:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9309999999999996e-06 2024-09-15T03:46:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109457+0000", 2024-09-15T03:46:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1047000000000003e-05 2024-09-15T03:46:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109462+0000", 2024-09-15T03:46:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1499999999999998e-06 2024-09-15T03:46:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109513+0000", 2024-09-15T03:46:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0722e-05 2024-09-15T03:46:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111238+0000", 2024-09-15T03:46:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017253380000000001 2024-09-15T03:46:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114140+0000", 2024-09-15T03:46:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029017370000000002 2024-09-15T03:46:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114201+0000", 2024-09-15T03:46:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1047e-05 2024-09-15T03:46:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123719+0000", 2024-09-15T03:46:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095184099999999997 2024-09-15T03:46:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs3 33/31 ECSubWrite(tid=27, reqid=client.4521.0:189, at_version=0'0, trim_to=0'0, roll_forward_to=33'10))", 2024-09-15T03:46:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.114638+0000", 2024-09-15T03:46:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.916828945000001, 2024-09-15T03:46:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090975689999999998, 2024-09-15T03:46:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114638+0000", 2024-09-15T03:46:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114638+0000", 2024-09-15T03:46:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114639+0000", 2024-09-15T03:46:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7899999999999998e-07 2024-09-15T03:46:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114648+0000", 2024-09-15T03:46:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.969e-06 2024-09-15T03:46:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114663+0000", 2024-09-15T03:46:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4813e-05 2024-09-15T03:46:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114669+0000", 2024-09-15T03:46:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9710000000000001e-06 2024-09-15T03:46:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114728+0000", 2024-09-15T03:46:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9794999999999999e-05 2024-09-15T03:46:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114737+0000", 2024-09-15T03:46:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3639999999999999e-06 2024-09-15T03:46:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114862+0000", 2024-09-15T03:46:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124826 2024-09-15T03:46:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114881+0000", 2024-09-15T03:46:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8901000000000001e-05 2024-09-15T03:46:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114920+0000", 2024-09-15T03:46:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9258000000000003e-05 2024-09-15T03:46:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114936+0000", 2024-09-15T03:46:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5665999999999999e-05 2024-09-15T03:46:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123736+0000", 2024-09-15T03:46:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088003270000000001 2024-09-15T03:46:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as2 33/28 ECSubWrite(tid=28, reqid=client.4521.0:186, at_version=0'0, trim_to=0'0, roll_forward_to=33'8))", 2024-09-15T03:46:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.114840+0000", 2024-09-15T03:46:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.916627409, 2024-09-15T03:46:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090295170000000008, 2024-09-15T03:46:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114840+0000", 2024-09-15T03:46:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114840+0000", 2024-09-15T03:46:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114840+0000", 2024-09-15T03:46:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6899999999999997e-07 2024-09-15T03:46:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114850+0000", 2024-09-15T03:46:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0023e-05 2024-09-15T03:46:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114862+0000", 2024-09-15T03:46:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1615e-05 2024-09-15T03:46:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114868+0000", 2024-09-15T03:46:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3570000000000003e-06 2024-09-15T03:46:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114917+0000", 2024-09-15T03:46:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8838999999999998e-05 2024-09-15T03:46:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114926+0000", 2024-09-15T03:46:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7390000000000005e-06 2024-09-15T03:46:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.115039+0000", 2024-09-15T03:46:55.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113114 2024-09-15T03:46:55.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.115048+0000", 2024-09-15T03:46:55.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5080000000000005e-06 2024-09-15T03:46:55.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.115077+0000", 2024-09-15T03:46:55.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.97e-05 2024-09-15T03:46:55.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.115104+0000", 2024-09-15T03:46:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6197999999999999e-05 2024-09-15T03:46:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123869+0000", 2024-09-15T03:46:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087657550000000001 2024-09-15T03:46:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:193 3.bs0 3:df8b18a3:::smithi00132172-17:head [rollback 3] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:46:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.112181+0000", 2024-09-15T03:46:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.919286132, 2024-09-15T03:46:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010302426999999999, 2024-09-15T03:46:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 193 2024-09-15T03:46:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112181+0000", 2024-09-15T03:46:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112181+0000", 2024-09-15T03:46:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112182+0000", 2024-09-15T03:46:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6000000000000004e-07 2024-09-15T03:46:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112185+0000", 2024-09-15T03:46:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.892e-06 2024-09-15T03:46:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112192+0000", 2024-09-15T03:46:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8110000000000003e-06 2024-09-15T03:46:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112197+0000", 2024-09-15T03:46:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3640000000000001e-06 2024-09-15T03:46:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.113873+0000", 2024-09-15T03:46:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016760760000000001 2024-09-15T03:46:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.113974+0000", 2024-09-15T03:46:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101303 2024-09-15T03:46:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114575+0000", 2024-09-15T03:46:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00060097600000000003 2024-09-15T03:46:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114707+0000", 2024-09-15T03:46:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000131543 2024-09-15T03:46:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114723+0000", 2024-09-15T03:46:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6449000000000001e-05 2024-09-15T03:46:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.114771+0000", 2024-09-15T03:46:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8312000000000001e-05 2024-09-15T03:46:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.115309+0000", 2024-09-15T03:46:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053803300000000004 2024-09-15T03:46:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.115740+0000", 2024-09-15T03:46:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043093100000000001 2024-09-15T03:46:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.122484+0000", 2024-09-15T03:46:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0067431770000000004 2024-09-15T03:46:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:194 3.bs0 3:df8b18a3:::smithi00132172-17:head [append 0~0] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:46:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.112221+0000", 2024-09-15T03:46:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.919246103000001, 2024-09-15T03:46:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013410515, 2024-09-15T03:46:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 194 2024-09-15T03:46:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112221+0000", 2024-09-15T03:46:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112221+0000", 2024-09-15T03:46:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112222+0000", 2024-09-15T03:46:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:46:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112226+0000", 2024-09-15T03:46:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6840000000000004e-06 2024-09-15T03:46:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112232+0000", 2024-09-15T03:46:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9660000000000001e-06 2024-09-15T03:46:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112236+0000", 2024-09-15T03:46:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2740000000000001e-06 2024-09-15T03:46:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116143+0000", 2024-09-15T03:46:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039065549999999999 2024-09-15T03:46:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.116257+0000", 2024-09-15T03:46:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113655 2024-09-15T03:46:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.122980+0000", 2024-09-15T03:46:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0067229860000000002 2024-09-15T03:46:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123101+0000", 2024-09-15T03:46:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121143 2024-09-15T03:46:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123117+0000", 2024-09-15T03:46:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6279000000000001e-05 2024-09-15T03:46:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123185+0000", 2024-09-15T03:46:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7613000000000005e-05 2024-09-15T03:46:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124310+0000", 2024-09-15T03:46:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011254679999999999 2024-09-15T03:46:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125591+0000", 2024-09-15T03:46:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012803669999999999 2024-09-15T03:46:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125632+0000", 2024-09-15T03:46:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1115000000000002e-05 2024-09-15T03:46:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:197 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [delete] snapc 3=[3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e34)", 2024-09-15T03:46:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:46:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.919201965999999, 2024-09-15T03:46:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021934704999999999, 2024-09-15T03:46:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197 2024-09-15T03:46:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:46:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:46:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112266+0000", 2024-09-15T03:46:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5000000000000002e-07 2024-09-15T03:46:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112269+0000", 2024-09-15T03:46:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.066e-06 2024-09-15T03:46:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112276+0000", 2024-09-15T03:46:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-09-15T03:46:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112281+0000", 2024-09-15T03:46:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3970000000000004e-06 2024-09-15T03:46:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123281+0000", 2024-09-15T03:46:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011000575 2024-09-15T03:46:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123395+0000", 2024-09-15T03:46:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113398 2024-09-15T03:46:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123963+0000", 2024-09-15T03:46:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000568101 2024-09-15T03:46:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124220+0000", 2024-09-15T03:46:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025704900000000003 2024-09-15T03:46:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124229+0000", 2024-09-15T03:46:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4860000000000003e-06 2024-09-15T03:46:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124277+0000", 2024-09-15T03:46:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7574000000000002e-05 2024-09-15T03:46:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124312+0000", 2024-09-15T03:46:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5256000000000003e-05 2024-09-15T03:46:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125716+0000", 2024-09-15T03:46:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014043720000000001 2024-09-15T03:46:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.134200+0000", 2024-09-15T03:46:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084835120000000003 2024-09-15T03:46:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es4 34/28 ECSubWrite(tid=33, reqid=client.4521.0:195, at_version=34'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-15T03:46:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:46:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.913374607, 2024-09-15T03:46:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015907482000000001, 2024-09-15T03:46:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:46:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:46:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:46:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9400000000000004e-07 2024-09-15T03:46:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118299+0000", 2024-09-15T03:46:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000205677 2024-09-15T03:46:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118315+0000", 2024-09-15T03:46:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5429000000000001e-05 2024-09-15T03:46:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118322+0000", 2024-09-15T03:46:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8290000000000008e-06 2024-09-15T03:46:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118416+0000", 2024-09-15T03:46:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3140000000000006e-05 2024-09-15T03:46:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118431+0000", 2024-09-15T03:46:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5648e-05 2024-09-15T03:46:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118678+0000", 2024-09-15T03:46:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024701200000000001 2024-09-15T03:46:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118691+0000", 2024-09-15T03:46:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2653000000000001e-05 2024-09-15T03:46:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.119431+0000", 2024-09-15T03:46:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00074060399999999996 2024-09-15T03:46:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124280+0000", 2024-09-15T03:46:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0048484410000000002 2024-09-15T03:46:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.134000+0000", 2024-09-15T03:46:55.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0097201549999999994 2024-09-15T03:46:55.101 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.101 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:55.101 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es4 34/28 ECSubWrite(tid=34, reqid=client.4521.0:195, at_version=0'0, trim_to=0'0, roll_forward_to=34'16))", 2024-09-15T03:46:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.124887+0000", 2024-09-15T03:46:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.906579955, 2024-09-15T03:46:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092852760000000003, 2024-09-15T03:46:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124887+0000", 2024-09-15T03:46:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124887+0000", 2024-09-15T03:46:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:55.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124888+0000", 2024-09-15T03:46:55.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2600000000000001e-07 2024-09-15T03:46:55.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:55.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124899+0000", 2024-09-15T03:46:55.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1256e-05 2024-09-15T03:46:55.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:55.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124910+0000", 2024-09-15T03:46:55.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0986e-05 2024-09-15T03:46:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124917+0000", 2024-09-15T03:46:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.297e-06 2024-09-15T03:46:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124985+0000", 2024-09-15T03:46:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7838999999999999e-05 2024-09-15T03:46:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124999+0000", 2024-09-15T03:46:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4201e-05 2024-09-15T03:46:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125095+0000", 2024-09-15T03:46:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5797000000000005e-05 2024-09-15T03:46:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125107+0000", 2024-09-15T03:46:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2194999999999999e-05 2024-09-15T03:46:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125145+0000", 2024-09-15T03:46:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8355000000000001e-05 2024-09-15T03:46:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125189+0000", 2024-09-15T03:46:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3741e-05 2024-09-15T03:46:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:55.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.134173+0000", 2024-09-15T03:46:55.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089836829999999993 2024-09-15T03:46:55.110 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.110 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:55.110 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:220 3.bs0 3:d99ddedc:::smithi00132172-44:head [copy-from ver 22] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:46:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:46:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.9038222959999995, 2024-09-15T03:46:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024273429999999999, 2024-09-15T03:46:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 220 2024-09-15T03:46:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:55.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:46:55.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:55.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:46:55.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:55.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127647+0000", 2024-09-15T03:46:55.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1799999999999999e-06 2024-09-15T03:46:55.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:55.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127656+0000", 2024-09-15T03:46:55.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3759999999999997e-06 2024-09-15T03:46:55.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:55.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127674+0000", 2024-09-15T03:46:55.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7374e-05 2024-09-15T03:46:55.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127680+0000", 2024-09-15T03:46:55.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6949999999999997e-06 2024-09-15T03:46:55.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127736+0000", 2024-09-15T03:46:55.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6011999999999998e-05 2024-09-15T03:46:55.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:46:55.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127744+0000", 2024-09-15T03:46:55.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2870000000000004e-06 2024-09-15T03:46:55.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130338+0000", 2024-09-15T03:46:55.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025940249999999998 2024-09-15T03:46:55.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:55.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130423+0000", 2024-09-15T03:46:55.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5303000000000006e-05 2024-09-15T03:46:55.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.147840+0000", 2024-09-15T03:46:55.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017416580000000001 2024-09-15T03:46:55.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148318+0000", 2024-09-15T03:46:55.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047822899999999999 2024-09-15T03:46:55.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148329+0000", 2024-09-15T03:46:55.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0611000000000001e-05 2024-09-15T03:46:55.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148488+0000", 2024-09-15T03:46:55.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015938399999999999 2024-09-15T03:46:55.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148870+0000", 2024-09-15T03:46:55.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000382405 2024-09-15T03:46:55.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:55.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.151092+0000", 2024-09-15T03:46:55.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022212529999999999 2024-09-15T03:46:55.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:55.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.151918+0000", 2024-09-15T03:46:55.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082671600000000002 2024-09-15T03:46:55.118 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.118 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:55.118 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.119 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:221 3.bs0 3:d99ddedc:::smithi00132172-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:46:55.119 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:46:55.119 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.9037781410000001, 2024-09-15T03:46:55.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021194474000000001, 2024-09-15T03:46:55.119 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:55.119 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:46:55.119 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:55.119 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:55.119 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:55.119 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 221 2024-09-15T03:46:55.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.120 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:55.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:55.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:46:55.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:55.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:46:55.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:55.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127690+0000", 2024-09-15T03:46:55.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:46:55.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:55.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127692+0000", 2024-09-15T03:46:55.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6589999999999999e-06 2024-09-15T03:46:55.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:55.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127700+0000", 2024-09-15T03:46:55.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4769999999999997e-06 2024-09-15T03:46:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127704+0000", 2024-09-15T03:46:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3819999999999997e-06 2024-09-15T03:46:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127805+0000", 2024-09-15T03:46:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100639 2024-09-15T03:46:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127809+0000", 2024-09-15T03:46:55.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8870000000000002e-06 2024-09-15T03:46:55.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130527+0000", 2024-09-15T03:46:55.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002718579 2024-09-15T03:46:55.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:46:55.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130558+0000", 2024-09-15T03:46:55.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0827999999999997e-05 2024-09-15T03:46:55.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148546+0000", 2024-09-15T03:46:55.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017988356 2024-09-15T03:46:55.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:55.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148683+0000", 2024-09-15T03:46:55.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013650899999999999 2024-09-15T03:46:55.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:55.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148884+0000", 2024-09-15T03:46:55.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020061499999999999 2024-09-15T03:46:55.125 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.126 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:55.126 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.126 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:252 3.bs0 3:d6f79b7b:::smithi00132172-29:head [copy-from ver 6] snapc 7=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:46:55.126 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.169146+0000", 2024-09-15T03:46:55.126 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.8623208719999997, 2024-09-15T03:46:55.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016731068000000002, 2024-09-15T03:46:55.126 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:55.126 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:55.126 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:55.127 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:55.127 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:55.127 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 252 2024-09-15T03:46:55.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.127 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:55.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169146+0000", 2024-09-15T03:46:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169146+0000", 2024-09-15T03:46:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169147+0000", 2024-09-15T03:46:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1400000000000001e-07 2024-09-15T03:46:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169149+0000", 2024-09-15T03:46:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0279999999999999e-06 2024-09-15T03:46:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:55.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169152+0000", 2024-09-15T03:46:55.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0800000000000002e-06 2024-09-15T03:46:55.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169154+0000", 2024-09-15T03:46:55.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4679999999999999e-06 2024-09-15T03:46:55.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.174427+0000", 2024-09-15T03:46:55.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052724349999999998 2024-09-15T03:46:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.174543+0000", 2024-09-15T03:46:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116803 2024-09-15T03:46:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.178297+0000", 2024-09-15T03:46:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037531539999999999 2024-09-15T03:46:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179112+0000", 2024-09-15T03:46:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081548600000000003 2024-09-15T03:46:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179151+0000", 2024-09-15T03:46:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9158e-05 2024-09-15T03:46:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179272+0000", 2024-09-15T03:46:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120802 2024-09-15T03:46:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179531+0000", 2024-09-15T03:46:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000259351 2024-09-15T03:46:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:46:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179723+0000", 2024-09-15T03:46:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000191447 2024-09-15T03:46:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.185877+0000", 2024-09-15T03:46:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061546420000000001 2024-09-15T03:46:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:253 3.bs0 3:d6f79b7b:::smithi00132172-29:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:46:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.169161+0000", 2024-09-15T03:46:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.8623065089999997, 2024-09-15T03:46:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010325136, 2024-09-15T03:46:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:46:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:46:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:46:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 253 2024-09-15T03:46:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169161+0000", 2024-09-15T03:46:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169161+0000", 2024-09-15T03:46:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169161+0000", 2024-09-15T03:46:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5899999999999998e-07 2024-09-15T03:46:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169162+0000", 2024-09-15T03:46:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1880000000000001e-06 2024-09-15T03:46:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169166+0000", 2024-09-15T03:46:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3629999999999998e-06 2024-09-15T03:46:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169168+0000", 2024-09-15T03:46:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3690000000000001e-06 2024-09-15T03:46:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.174740+0000", 2024-09-15T03:46:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055717470000000002 2024-09-15T03:46:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:46:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.174814+0000", 2024-09-15T03:46:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3965000000000001e-05 2024-09-15T03:46:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179305+0000", 2024-09-15T03:46:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0044917120000000001 2024-09-15T03:46:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:46:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179410+0000", 2024-09-15T03:46:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104734 2024-09-15T03:46:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179486+0000", 2024-09-15T03:46:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5798999999999997e-05 2024-09-15T03:46:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.3s3 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:46:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.171321+0000", 2024-09-15T03:46:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.8601465019999992, 2024-09-15T03:46:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0091209930000000008, 2024-09-15T03:46:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171321+0000", 2024-09-15T03:46:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171321+0000", 2024-09-15T03:46:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171322+0000", 2024-09-15T03:46:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.034e-06 2024-09-15T03:46:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171433+0000", 2024-09-15T03:46:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111079 2024-09-15T03:46:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171446+0000", 2024-09-15T03:46:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2802e-05 2024-09-15T03:46:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171453+0000", 2024-09-15T03:46:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9070000000000001e-06 2024-09-15T03:46:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171709+0000", 2024-09-15T03:46:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000256056 2024-09-15T03:46:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171717+0000", 2024-09-15T03:46:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9340000000000006e-06 2024-09-15T03:46:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172017+0000", 2024-09-15T03:46:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030057899999999998 2024-09-15T03:46:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172260+0000", 2024-09-15T03:46:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024294599999999999 2024-09-15T03:46:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172354+0000", 2024-09-15T03:46:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3660999999999997e-05 2024-09-15T03:46:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172385+0000", 2024-09-15T03:46:55.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1511999999999999e-05 2024-09-15T03:46:55.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:55.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.180442+0000", 2024-09-15T03:46:55.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0080564829999999997 2024-09-15T03:46:55.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.146 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:55.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.147 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.6s1 40/31 ECSubWrite(tid=61, reqid=client.4521.0:267, at_version=40'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:55.147 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:47.199701+0000", 2024-09-15T03:46:55.147 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.8317657949999999, 2024-09-15T03:46:55.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012707423000000001, 2024-09-15T03:46:55.147 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:55.148 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.148 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:55.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:55.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.199701+0000", 2024-09-15T03:46:55.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:55.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.199701+0000", 2024-09-15T03:46:55.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.199702+0000", 2024-09-15T03:46:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9299999999999997e-07 2024-09-15T03:46:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.199722+0000", 2024-09-15T03:46:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9417999999999999e-05 2024-09-15T03:46:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.199734+0000", 2024-09-15T03:46:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2621e-05 2024-09-15T03:46:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.199751+0000", 2024-09-15T03:46:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7223000000000001e-05 2024-09-15T03:46:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.199816+0000", 2024-09-15T03:46:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4974000000000005e-05 2024-09-15T03:46:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.199826+0000", 2024-09-15T03:46:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0074999999999999e-05 2024-09-15T03:46:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.199908+0000", 2024-09-15T03:46:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1561000000000001e-05 2024-09-15T03:46:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.199951+0000", 2024-09-15T03:46:55.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3055e-05 2024-09-15T03:46:55.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.199998+0000", 2024-09-15T03:46:55.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6795000000000002e-05 2024-09-15T03:46:55.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.200016+0000", 2024-09-15T03:46:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7689e-05 2024-09-15T03:46:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.212409+0000", 2024-09-15T03:46:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012393319 2024-09-15T03:46:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.6s1 40/31 ECSubWrite(tid=62, reqid=client.4521.0:268, at_version=40'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:47.200573+0000", 2024-09-15T03:46:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.8308942320000003, 2024-09-15T03:46:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012006539, 2024-09-15T03:46:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.200573+0000", 2024-09-15T03:46:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.200573+0000", 2024-09-15T03:46:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.200574+0000", 2024-09-15T03:46:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0399999999999995e-07 2024-09-15T03:46:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.200593+0000", 2024-09-15T03:46:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9165999999999999e-05 2024-09-15T03:46:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.200604+0000", 2024-09-15T03:46:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1610999999999999e-05 2024-09-15T03:46:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.200610+0000", 2024-09-15T03:46:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0390000000000001e-06 2024-09-15T03:46:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201000+0000", 2024-09-15T03:46:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038947999999999999 2024-09-15T03:46:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201009+0000", 2024-09-15T03:46:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.516e-06 2024-09-15T03:46:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201125+0000", 2024-09-15T03:46:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115114 2024-09-15T03:46:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201363+0000", 2024-09-15T03:46:55.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023833400000000001 2024-09-15T03:46:55.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203349+0000", 2024-09-15T03:46:55.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019857479999999999 2024-09-15T03:46:55.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203369+0000", 2024-09-15T03:46:55.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0404999999999999e-05 2024-09-15T03:46:55.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:55.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.212580+0000", 2024-09-15T03:46:55.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092104219999999994 2024-09-15T03:46:55.160 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.160 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:55.160 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.161 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.3s3 40/31 ECSubWrite(tid=50, reqid=client.4521.0:269, at_version=40'22, trim_to=0'0, roll_forward_to=40'21))", 2024-09-15T03:46:55.161 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:47.201224+0000", 2024-09-15T03:46:55.161 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.8302434219999997, 2024-09-15T03:46:55.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011822386000000001, 2024-09-15T03:46:55.161 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:55.161 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.161 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:55.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:55.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201224+0000", 2024-09-15T03:46:55.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201224+0000", 2024-09-15T03:46:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201225+0000", 2024-09-15T03:46:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7599999999999996e-07 2024-09-15T03:46:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201263+0000", 2024-09-15T03:46:55.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8495000000000003e-05 2024-09-15T03:46:55.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:55.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201281+0000", 2024-09-15T03:46:55.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8396000000000001e-05 2024-09-15T03:46:55.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201289+0000", 2024-09-15T03:46:55.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2019999999999998e-06 2024-09-15T03:46:55.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201342+0000", 2024-09-15T03:46:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3779999999999997e-05 2024-09-15T03:46:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201352+0000", 2024-09-15T03:46:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6500000000000008e-06 2024-09-15T03:46:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.201461+0000", 2024-09-15T03:46:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109146 2024-09-15T03:46:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203352+0000", 2024-09-15T03:46:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018905770000000001 2024-09-15T03:46:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203412+0000", 2024-09-15T03:46:55.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9734000000000001e-05 2024-09-15T03:46:55.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203445+0000", 2024-09-15T03:46:55.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3247999999999997e-05 2024-09-15T03:46:55.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:55.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.213046+0000", 2024-09-15T03:46:55.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096013820000000003 2024-09-15T03:46:55.168 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.168 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:55.168 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.169 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es4 40/28 ECSubWrite(tid=62, reqid=client.4521.0:261, at_version=0'0, trim_to=0'0, roll_forward_to=40'38))", 2024-09-15T03:46:55.169 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:47.203794+0000", 2024-09-15T03:46:55.169 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.8276735100000003, 2024-09-15T03:46:55.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088409040000000001, 2024-09-15T03:46:55.169 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:46:55.169 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.170 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:46:55.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:46:55.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203794+0000", 2024-09-15T03:46:55.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:46:55.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203794+0000", 2024-09-15T03:46:55.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:46:55.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:46:55.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203795+0000", 2024-09-15T03:46:55.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5999999999999991e-07 2024-09-15T03:46:55.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:46:55.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203806+0000", 2024-09-15T03:46:55.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0981e-05 2024-09-15T03:46:55.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:46:55.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203824+0000", 2024-09-15T03:46:55.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8391000000000001e-05 2024-09-15T03:46:55.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203832+0000", 2024-09-15T03:46:55.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8310000000000001e-06 2024-09-15T03:46:55.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:46:55.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203907+0000", 2024-09-15T03:46:55.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5160999999999994e-05 2024-09-15T03:46:55.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.203917+0000", 2024-09-15T03:46:55.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.91e-06 2024-09-15T03:46:55.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.204019+0000", 2024-09-15T03:46:55.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101928 2024-09-15T03:46:55.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.204034+0000", 2024-09-15T03:46:55.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5078e-05 2024-09-15T03:46:55.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.204127+0000", 2024-09-15T03:46:55.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2918999999999996e-05 2024-09-15T03:46:55.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.204217+0000", 2024-09-15T03:46:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9738e-05 2024-09-15T03:46:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:46:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:46:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:46:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.212635+0000", 2024-09-15T03:46:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084180069999999999 2024-09-15T03:46:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.178 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:55.178 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.178 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:46:55.178 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:46:55.178 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:46:55.179 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-15T03:46:55.179 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T03:46:55.179 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-15T03:46:55.179 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-15T03:46:55.179 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-15T03:46:55.180 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-09-15T03:46:55.182 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:46:55.182 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:55.179+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:46:55.280 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:55.278+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:55.291 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:55.291 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:55.291 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:46:55.291 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:55.293 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:46:55.293 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-15T03:46:55.293 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:46:55.300 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:46:55.381 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:55.379+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:46:55.398 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:55.398 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:55.398 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:46:55.398 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:46:55.399 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:55.407 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:46:55.482 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:55.481+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:46:55.505 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:55.505 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:46:55.505 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:46:55.505 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:46:55.505 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.505 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:46:55.506 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:46:55.506 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 37.600806665, 2024-09-15T03:46:55.506 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10988437500000001, 2024-09-15T03:46:55.506 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:55.506 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:55.506 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:55.506 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:46:55.506 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:46:55.506 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1 2024-09-15T03:46:55.506 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.506 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:55.506 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.507 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:55.507 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:46:55.507 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.507 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.507 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.507 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:55.507 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:46:55.507 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.507 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.507 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.507 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:55.507 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902355+0000", 2024-09-15T03:46:55.507 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.429e-06 2024-09-15T03:46:55.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.508 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.508 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:55.508 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902362+0000", 2024-09-15T03:46:55.508 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7290000000000003e-06 2024-09-15T03:46:55.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.508 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.508 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:55.508 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902373+0000", 2024-09-15T03:46:55.508 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0533999999999999e-05 2024-09-15T03:46:55.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.508 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.508 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902379+0000", 2024-09-15T03:46:55.509 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4350000000000002e-06 2024-09-15T03:46:55.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.509 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.509 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902428+0000", 2024-09-15T03:46:55.509 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8571e-05 2024-09-15T03:46:55.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.509 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.509 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:46:55.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902434+0000", 2024-09-15T03:46:55.509 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5610000000000004e-06 2024-09-15T03:46:55.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.510 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011307+0000", 2024-09-15T03:46:55.510 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.108872306 2024-09-15T03:46:55.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.510 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:55.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011448+0000", 2024-09-15T03:46:55.510 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014137199999999999 2024-09-15T03:46:55.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.510 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:46:55.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011569+0000", 2024-09-15T03:46:55.510 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120925 2024-09-15T03:46:55.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.511 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.511 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.511 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011695+0000", 2024-09-15T03:46:55.511 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012586800000000001 2024-09-15T03:46:55.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.511 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.511 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.511 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011703+0000", 2024-09-15T03:46:55.511 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4190000000000002e-06 2024-09-15T03:46:55.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.511 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.511 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.512 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011739+0000", 2024-09-15T03:46:55.512 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5474e-05 2024-09-15T03:46:55.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.512 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.512 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:46:55.512 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011940+0000", 2024-09-15T03:46:55.512 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020108699999999999 2024-09-15T03:46:55.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.512 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.512 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:46:55.512 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012179+0000", 2024-09-15T03:46:55.512 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023950299999999999 2024-09-15T03:46:55.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.513 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.513 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:55.513 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012215+0000", 2024-09-15T03:46:55.513 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5332e-05 2024-09-15T03:46:55.513 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.513 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.513 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:55.513 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012238+0000", 2024-09-15T03:46:55.513 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3830000000000001e-05 2024-09-15T03:46:55.513 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.513 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:55.513 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.513 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.514 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.514 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4263.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2024-09-15T03:46:55.514 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:46:55.514 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.459409979, 2024-09-15T03:46:55.514 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019495788999999999, 2024-09-15T03:46:55.514 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:55.514 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.514 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:55.514 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4263", 2024-09-15T03:46:55.514 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/646873808", 2024-09-15T03:46:55.514 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2 2024-09-15T03:46:55.514 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.514 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:55.515 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.515 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:55.515 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:46:55.515 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.515 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.515 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.515 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:55.515 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:46:55.515 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.515 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.515 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.515 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:55.515 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043752+0000", 2024-09-15T03:46:55.515 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.815e-06 2024-09-15T03:46:55.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.516 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:55.516 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043757+0000", 2024-09-15T03:46:55.516 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4240000000000003e-06 2024-09-15T03:46:55.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.516 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:55.516 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043766+0000", 2024-09-15T03:46:55.516 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8629999999999997e-06 2024-09-15T03:46:55.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.516 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043773+0000", 2024-09-15T03:46:55.517 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0099999999999998e-06 2024-09-15T03:46:55.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.517 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.517 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043830+0000", 2024-09-15T03:46:55.517 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7089999999999997e-05 2024-09-15T03:46:55.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.517 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.517 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:46:55.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043837+0000", 2024-09-15T03:46:55.517 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.08e-06 2024-09-15T03:46:55.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.518 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.518 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.518 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063103+0000", 2024-09-15T03:46:55.518 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019265619000000001 2024-09-15T03:46:55.518 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.518 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.518 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:55.518 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063246+0000", 2024-09-15T03:46:55.518 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000143888 2024-09-15T03:46:55.518 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.518 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:55.518 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.518 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.519 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.519 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.fs5 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:55.519 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:46:55.519 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.395562073000001, 2024-09-15T03:46:55.519 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098369180000000001, 2024-09-15T03:46:55.519 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:55.519 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.519 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:55.519 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.519 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:55.519 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:46:55.519 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.520 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:55.520 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:46:55.520 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.520 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:55.520 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:46:55.520 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4499999999999998e-07 2024-09-15T03:46:55.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.520 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:55.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107630+0000", 2024-09-15T03:46:55.521 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0877999999999998e-05 2024-09-15T03:46:55.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.521 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.521 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:55.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107642+0000", 2024-09-15T03:46:55.521 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2567e-05 2024-09-15T03:46:55.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.521 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.521 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107648+0000", 2024-09-15T03:46:55.521 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8690000000000001e-06 2024-09-15T03:46:55.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.522 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107894+0000", 2024-09-15T03:46:55.522 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024540400000000001 2024-09-15T03:46:55.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.522 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107900+0000", 2024-09-15T03:46:55.522 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5710000000000003e-06 2024-09-15T03:46:55.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.522 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107968+0000", 2024-09-15T03:46:55.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7957e-05 2024-09-15T03:46:55.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.523 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108274+0000", 2024-09-15T03:46:55.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030591700000000001 2024-09-15T03:46:55.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.523 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108504+0000", 2024-09-15T03:46:55.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022952199999999999 2024-09-15T03:46:55.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.524 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.524 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109438+0000", 2024-09-15T03:46:55.524 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00093439100000000002 2024-09-15T03:46:55.524 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.524 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.524 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:55.524 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117435+0000", 2024-09-15T03:46:55.524 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0079974969999999992 2024-09-15T03:46:55.524 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.524 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:55.524 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.524 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.525 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.525 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:55.525 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:46:55.525 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.395504820999999, 2024-09-15T03:46:55.525 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084894540000000004, 2024-09-15T03:46:55.525 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:55.525 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.525 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:55.525 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.526 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:55.526 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:46:55.526 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.526 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.526 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.526 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:55.527 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:46:55.527 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.527 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.527 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.527 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:55.527 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:46:55.527 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4700000000000002e-07 2024-09-15T03:46:55.527 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.527 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.528 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:55.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107670+0000", 2024-09-15T03:46:55.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3892e-05 2024-09-15T03:46:55.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.528 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.528 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:55.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107678+0000", 2024-09-15T03:46:55.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9589999999999995e-06 2024-09-15T03:46:55.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.528 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.528 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107683+0000", 2024-09-15T03:46:55.529 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7199999999999997e-06 2024-09-15T03:46:55.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.529 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.529 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.529 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107768+0000", 2024-09-15T03:46:55.529 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4968000000000004e-05 2024-09-15T03:46:55.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.529 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.529 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.529 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107776+0000", 2024-09-15T03:46:55.529 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9109999999999993e-06 2024-09-15T03:46:55.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.529 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.530 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.530 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107858+0000", 2024-09-15T03:46:55.530 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2349999999999996e-05 2024-09-15T03:46:55.530 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.530 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.530 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.530 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108273+0000", 2024-09-15T03:46:55.530 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00041525600000000002 2024-09-15T03:46:55.530 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.530 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.530 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.530 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108503+0000", 2024-09-15T03:46:55.530 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022990599999999999 2024-09-15T03:46:55.530 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.531 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.531 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.531 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108705+0000", 2024-09-15T03:46:55.531 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020173000000000001 2024-09-15T03:46:55.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.531 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.531 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:55.531 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.116145+0000", 2024-09-15T03:46:55.531 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0074404149999999997 2024-09-15T03:46:55.531 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.531 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:55.531 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.532 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.fs5 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:55.532 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.109087+0000", 2024-09-15T03:46:55.532 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.394073827, 2024-09-15T03:46:55.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084235809999999994, 2024-09-15T03:46:55.532 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:55.532 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.532 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:55.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.532 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:55.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109087+0000", 2024-09-15T03:46:55.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:55.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109087+0000", 2024-09-15T03:46:55.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:55.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109087+0000", 2024-09-15T03:46:55.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7599999999999997e-07 2024-09-15T03:46:55.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:55.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109107+0000", 2024-09-15T03:46:55.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9511999999999999e-05 2024-09-15T03:46:55.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.534 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:55.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109127+0000", 2024-09-15T03:46:55.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0268e-05 2024-09-15T03:46:55.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.534 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109136+0000", 2024-09-15T03:46:55.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1330000000000005e-06 2024-09-15T03:46:55.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.534 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.535 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109594+0000", 2024-09-15T03:46:55.535 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045802100000000002 2024-09-15T03:46:55.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.535 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.535 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.535 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109602+0000", 2024-09-15T03:46:55.535 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7020000000000001e-06 2024-09-15T03:46:55.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.535 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.535 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.535 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109678+0000", 2024-09-15T03:46:55.535 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5428999999999999e-05 2024-09-15T03:46:55.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.536 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109699+0000", 2024-09-15T03:46:55.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1759999999999998e-05 2024-09-15T03:46:55.536 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.536 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109728+0000", 2024-09-15T03:46:55.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8886000000000001e-05 2024-09-15T03:46:55.536 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.536 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109736+0000", 2024-09-15T03:46:55.537 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6310000000000003e-06 2024-09-15T03:46:55.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.537 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:55.537 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117510+0000", 2024-09-15T03:46:55.537 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.007774563 2024-09-15T03:46:55.537 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.537 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:55.537 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.537 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:55.537 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.109145+0000", 2024-09-15T03:46:55.538 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.394015918999999, 2024-09-15T03:46:55.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075088480000000003, 2024-09-15T03:46:55.538 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:55.538 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.538 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:55.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:55.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109145+0000", 2024-09-15T03:46:55.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:55.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109145+0000", 2024-09-15T03:46:55.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.539 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:55.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109145+0000", 2024-09-15T03:46:55.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7399999999999999e-07 2024-09-15T03:46:55.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.539 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:55.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109160+0000", 2024-09-15T03:46:55.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4544e-05 2024-09-15T03:46:55.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:55.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109168+0000", 2024-09-15T03:46:55.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9200000000000004e-06 2024-09-15T03:46:55.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109172+0000", 2024-09-15T03:46:55.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.498e-06 2024-09-15T03:46:55.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109221+0000", 2024-09-15T03:46:55.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9348000000000003e-05 2024-09-15T03:46:55.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.541 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109230+0000", 2024-09-15T03:46:55.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2020000000000004e-06 2024-09-15T03:46:55.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.541 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109314+0000", 2024-09-15T03:46:55.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4445999999999998e-05 2024-09-15T03:46:55.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.541 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109570+0000", 2024-09-15T03:46:55.542 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00025616300000000001 2024-09-15T03:46:55.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.542 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109642+0000", 2024-09-15T03:46:55.542 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1600999999999994e-05 2024-09-15T03:46:55.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.542 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109653+0000", 2024-09-15T03:46:55.542 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1048e-05 2024-09-15T03:46:55.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.543 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:55.543 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.116654+0000", 2024-09-15T03:46:55.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.007000704 2024-09-15T03:46:55.543 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.543 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:55.543 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.543 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.543 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.543 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es5 34/28 ECSubWrite(tid=33, reqid=client.4521.0:195, at_version=34'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-15T03:46:55.543 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.118140+0000", 2024-09-15T03:46:55.543 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.385020525, 2024-09-15T03:46:55.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069410629999999999, 2024-09-15T03:46:55.543 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:55.544 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.544 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:55.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:55.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118140+0000", 2024-09-15T03:46:55.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:55.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118140+0000", 2024-09-15T03:46:55.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.545 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:55.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118140+0000", 2024-09-15T03:46:55.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.29e-07 2024-09-15T03:46:55.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.545 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.545 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:55.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118354+0000", 2024-09-15T03:46:55.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000213291 2024-09-15T03:46:55.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.545 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.545 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:55.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118363+0000", 2024-09-15T03:46:55.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4739999999999994e-06 2024-09-15T03:46:55.546 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.546 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.546 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118368+0000", 2024-09-15T03:46:55.546 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2499999999999997e-06 2024-09-15T03:46:55.546 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.546 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.546 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118418+0000", 2024-09-15T03:46:55.546 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0166e-05 2024-09-15T03:46:55.546 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.546 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.547 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118427+0000", 2024-09-15T03:46:55.547 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0459999999999994e-06 2024-09-15T03:46:55.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.547 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.547 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.547 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118648+0000", 2024-09-15T03:46:55.547 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022084599999999999 2024-09-15T03:46:55.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.547 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.547 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.547 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118656+0000", 2024-09-15T03:46:55.547 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9899999999999997e-06 2024-09-15T03:46:55.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.548 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118725+0000", 2024-09-15T03:46:55.548 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8569e-05 2024-09-15T03:46:55.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.548 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118742+0000", 2024-09-15T03:46:55.548 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6776999999999999e-05 2024-09-15T03:46:55.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:55.549 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125081+0000", 2024-09-15T03:46:55.549 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0063395250000000004 2024-09-15T03:46:55.549 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.549 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:55.549 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.549 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.549 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 34/31 ECSubWrite(tid=34, reqid=client.4521.0:197, at_version=34'21, trim_to=0'0, roll_forward_to=34'18))", 2024-09-15T03:46:55.549 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.124009+0000", 2024-09-15T03:46:55.549 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.379151476000001, 2024-09-15T03:46:55.549 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0066902589999999996, 2024-09-15T03:46:55.549 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:55.549 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.550 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:55.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.550 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:55.550 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124009+0000", 2024-09-15T03:46:55.550 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.550 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:55.550 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124009+0000", 2024-09-15T03:46:55.550 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:55.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124010+0000", 2024-09-15T03:46:55.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.63e-07 2024-09-15T03:46:55.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:55.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124031+0000", 2024-09-15T03:46:55.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1043000000000002e-05 2024-09-15T03:46:55.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:55.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124043+0000", 2024-09-15T03:46:55.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2500999999999999e-05 2024-09-15T03:46:55.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124049+0000", 2024-09-15T03:46:55.552 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.331e-06 2024-09-15T03:46:55.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124090+0000", 2024-09-15T03:46:55.552 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0692999999999998e-05 2024-09-15T03:46:55.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.553 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124097+0000", 2024-09-15T03:46:55.553 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.216e-06 2024-09-15T03:46:55.553 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.553 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.553 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.553 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124657+0000", 2024-09-15T03:46:55.553 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00055963300000000003 2024-09-15T03:46:55.553 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.553 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.553 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.553 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124664+0000", 2024-09-15T03:46:55.553 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0310000000000001e-06 2024-09-15T03:46:55.553 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124694+0000", 2024-09-15T03:46:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0383999999999999e-05 2024-09-15T03:46:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124707+0000", 2024-09-15T03:46:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2098e-05 2024-09-15T03:46:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.130699+0000", 2024-09-15T03:46:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0059928660000000003 2024-09-15T03:46:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/31 ECSubWrite(tid=38, reqid=client.4521.0:223, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:46:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:46:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.355953641999999, 2024-09-15T03:46:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010504493, 2024-09-15T03:46:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.556 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:55.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:46:55.556 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.556 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.556 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:55.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:46:55.556 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.556 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.556 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:55.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:46:55.556 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4999999999999999e-07 2024-09-15T03:46:55.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:55.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147648+0000", 2024-09-15T03:46:55.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00044030399999999998 2024-09-15T03:46:55.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:55.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147660+0000", 2024-09-15T03:46:55.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2517000000000001e-05 2024-09-15T03:46:55.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147667+0000", 2024-09-15T03:46:55.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.737e-06 2024-09-15T03:46:55.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147713+0000", 2024-09-15T03:46:55.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6405e-05 2024-09-15T03:46:55.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147719+0000", 2024-09-15T03:46:55.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0580000000000003e-06 2024-09-15T03:46:55.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147905+0000", 2024-09-15T03:46:55.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018574600000000001 2024-09-15T03:46:55.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147917+0000", 2024-09-15T03:46:55.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1749999999999999e-05 2024-09-15T03:46:55.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148022+0000", 2024-09-15T03:46:55.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104863 2024-09-15T03:46:55.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.560 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.560 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148037+0000", 2024-09-15T03:46:55.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5302999999999999e-05 2024-09-15T03:46:55.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.560 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.560 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:55.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157711+0000", 2024-09-15T03:46:55.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0096745600000000005 2024-09-15T03:46:55.560 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.561 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:55.561 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.561 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/31 ECSubWrite(tid=39, reqid=client.4521.0:224, at_version=38'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:46:55.561 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.149886+0000", 2024-09-15T03:46:55.561 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.353274215000001, 2024-09-15T03:46:55.561 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.008019155, 2024-09-15T03:46:55.561 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:55.561 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.561 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:55.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.561 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:55.562 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149886+0000", 2024-09-15T03:46:55.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.562 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:55.562 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149886+0000", 2024-09-15T03:46:55.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.562 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:55.562 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149887+0000", 2024-09-15T03:46:55.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3399999999999999e-07 2024-09-15T03:46:55.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:55.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150075+0000", 2024-09-15T03:46:55.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018829200000000001 2024-09-15T03:46:55.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:55.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150089+0000", 2024-09-15T03:46:55.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3886000000000001e-05 2024-09-15T03:46:55.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150096+0000", 2024-09-15T03:46:55.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7429999999999996e-06 2024-09-15T03:46:55.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150137+0000", 2024-09-15T03:46:55.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1170000000000001e-05 2024-09-15T03:46:55.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150146+0000", 2024-09-15T03:46:55.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8610000000000004e-06 2024-09-15T03:46:55.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150218+0000", 2024-09-15T03:46:55.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2588000000000004e-05 2024-09-15T03:46:55.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150224+0000", 2024-09-15T03:46:55.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9390000000000002e-06 2024-09-15T03:46:55.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150308+0000", 2024-09-15T03:46:55.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3222999999999999e-05 2024-09-15T03:46:55.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.150318+0000", 2024-09-15T03:46:55.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0407e-05 2024-09-15T03:46:55.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:55.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157906+0000", 2024-09-15T03:46:55.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075875120000000002 2024-09-15T03:46:55.566 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.566 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:55.566 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.567 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.567 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 38/31 ECSubWrite(tid=37, reqid=client.4521.0:220, at_version=0'0, trim_to=0'0, roll_forward_to=38'23))", 2024-09-15T03:46:55.567 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.151298+0000", 2024-09-15T03:46:55.567 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.351862646000001, 2024-09-15T03:46:55.567 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0066381979999999997, 2024-09-15T03:46:55.567 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:55.567 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.567 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:55.567 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.567 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:55.567 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151298+0000", 2024-09-15T03:46:55.567 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.568 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:55.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151298+0000", 2024-09-15T03:46:55.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.568 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:55.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151298+0000", 2024-09-15T03:46:55.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5400000000000002e-07 2024-09-15T03:46:55.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.568 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:55.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151306+0000", 2024-09-15T03:46:55.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5689999999999999e-06 2024-09-15T03:46:55.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.569 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:55.569 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151315+0000", 2024-09-15T03:46:55.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4239999999999999e-06 2024-09-15T03:46:55.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.569 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.569 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151321+0000", 2024-09-15T03:46:55.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.118e-06 2024-09-15T03:46:55.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151364+0000", 2024-09-15T03:46:55.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3509e-05 2024-09-15T03:46:55.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151372+0000", 2024-09-15T03:46:55.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2239999999999995e-06 2024-09-15T03:46:55.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151461+0000", 2024-09-15T03:46:55.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8721000000000007e-05 2024-09-15T03:46:55.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151468+0000", 2024-09-15T03:46:55.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0330000000000003e-06 2024-09-15T03:46:55.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.153509+0000", 2024-09-15T03:46:55.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020411549999999998 2024-09-15T03:46:55.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.153529+0000", 2024-09-15T03:46:55.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0072e-05 2024-09-15T03:46:55.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:55.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157936+0000", 2024-09-15T03:46:55.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044069189999999996 2024-09-15T03:46:55.572 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.572 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:55.572 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.572 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/31 ECSubWrite(tid=40, reqid=client.4521.0:225, at_version=38'10, trim_to=0'0, roll_forward_to=38'8))", 2024-09-15T03:46:55.573 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.151586+0000", 2024-09-15T03:46:55.573 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.351574367, 2024-09-15T03:46:55.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064316060000000003, 2024-09-15T03:46:55.573 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:55.573 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.573 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:55.573 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.573 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:55.573 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151586+0000", 2024-09-15T03:46:55.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.573 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.573 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:55.573 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151586+0000", 2024-09-15T03:46:55.574 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.574 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.574 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:55.574 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151587+0000", 2024-09-15T03:46:55.574 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6400000000000003e-07 2024-09-15T03:46:55.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.574 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.574 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:55.574 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151600+0000", 2024-09-15T03:46:55.574 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3662999999999999e-05 2024-09-15T03:46:55.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.574 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.574 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:55.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151609+0000", 2024-09-15T03:46:55.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9120000000000004e-06 2024-09-15T03:46:55.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.575 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151619+0000", 2024-09-15T03:46:55.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0107e-05 2024-09-15T03:46:55.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.575 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151654+0000", 2024-09-15T03:46:55.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4295000000000003e-05 2024-09-15T03:46:55.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151662+0000", 2024-09-15T03:46:55.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6959999999999994e-06 2024-09-15T03:46:55.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151761+0000", 2024-09-15T03:46:55.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8356000000000005e-05 2024-09-15T03:46:55.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.153516+0000", 2024-09-15T03:46:55.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001754986 2024-09-15T03:46:55.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.577 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.153546+0000", 2024-09-15T03:46:55.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0028000000000001e-05 2024-09-15T03:46:55.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.577 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.153557+0000", 2024-09-15T03:46:55.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1561e-05 2024-09-15T03:46:55.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.578 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:55.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158018+0000", 2024-09-15T03:46:55.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.004460538 2024-09-15T03:46:55.578 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.578 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:55.578 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.578 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es5 38/28 ECSubWrite(tid=45, reqid=client.4521.0:227, at_version=38'24, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:46:55.578 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.155440+0000", 2024-09-15T03:46:55.578 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.34772031, 2024-09-15T03:46:55.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0059398020000000001, 2024-09-15T03:46:55.578 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:55.578 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.579 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:55.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:55.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155440+0000", 2024-09-15T03:46:55.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:55.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155440+0000", 2024-09-15T03:46:55.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:55.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155441+0000", 2024-09-15T03:46:55.580 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.077e-06 2024-09-15T03:46:55.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.580 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:55.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155981+0000", 2024-09-15T03:46:55.580 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00053971599999999998 2024-09-15T03:46:55.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.580 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:55.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155994+0000", 2024-09-15T03:46:55.580 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3339000000000001e-05 2024-09-15T03:46:55.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156000+0000", 2024-09-15T03:46:55.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9529999999999996e-06 2024-09-15T03:46:55.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156048+0000", 2024-09-15T03:46:55.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7981e-05 2024-09-15T03:46:55.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156054+0000", 2024-09-15T03:46:55.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5169999999999999e-06 2024-09-15T03:46:55.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.582 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157320+0000", 2024-09-15T03:46:55.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00126636 2024-09-15T03:46:55.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.582 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157331+0000", 2024-09-15T03:46:55.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0577000000000001e-05 2024-09-15T03:46:55.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.582 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.583 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.583 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157434+0000", 2024-09-15T03:46:55.583 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010285000000000001 2024-09-15T03:46:55.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.583 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.583 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157449+0000", 2024-09-15T03:46:55.583 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5545000000000001e-05 2024-09-15T03:46:55.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.583 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:55.583 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.161380+0000", 2024-09-15T03:46:55.583 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0039308870000000001 2024-09-15T03:46:55.584 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.584 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:55.584 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.584 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.584 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 39/28 ECSubWrite(tid=42, reqid=client.4521.0:255, at_version=39'19, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:46:55.584 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.170708+0000", 2024-09-15T03:46:55.584 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.3324528549999997, 2024-09-15T03:46:55.584 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0093655969999999998, 2024-09-15T03:46:55.584 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:55.584 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.584 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:55.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:55.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170708+0000", 2024-09-15T03:46:55.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:55.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170708+0000", 2024-09-15T03:46:55.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:55.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170708+0000", 2024-09-15T03:46:55.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8899999999999999e-07 2024-09-15T03:46:55.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.586 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:55.586 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170728+0000", 2024-09-15T03:46:55.586 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9184000000000001e-05 2024-09-15T03:46:55.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.586 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:55.586 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170741+0000", 2024-09-15T03:46:55.586 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3696999999999999e-05 2024-09-15T03:46:55.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.586 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170747+0000", 2024-09-15T03:46:55.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1569999999999997e-06 2024-09-15T03:46:55.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.587 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.587 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170803+0000", 2024-09-15T03:46:55.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5974000000000003e-05 2024-09-15T03:46:55.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.587 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.587 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170811+0000", 2024-09-15T03:46:55.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8259999999999993e-06 2024-09-15T03:46:55.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.588 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170902+0000", 2024-09-15T03:46:55.588 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1116e-05 2024-09-15T03:46:55.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.588 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170909+0000", 2024-09-15T03:46:55.588 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3489999999999997e-06 2024-09-15T03:46:55.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.588 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173637+0000", 2024-09-15T03:46:55.588 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027282270000000002 2024-09-15T03:46:55.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.589 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173649+0000", 2024-09-15T03:46:55.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2561000000000001e-05 2024-09-15T03:46:55.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.589 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:55.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180073+0000", 2024-09-15T03:46:55.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064239170000000003 2024-09-15T03:46:55.589 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.589 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:55.589 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.590 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.590 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:46:55.590 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:46:55.590 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.3323365900000006, 2024-09-15T03:46:55.590 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011878227, 2024-09-15T03:46:55.590 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:55.590 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.590 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:55.590 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.590 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:55.590 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:46:55.590 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:55.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:46:55.591 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:55.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170825+0000", 2024-09-15T03:46:55.591 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9599999999999999e-07 2024-09-15T03:46:55.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:55.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171100+0000", 2024-09-15T03:46:55.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027548500000000001 2024-09-15T03:46:55.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.592 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:55.592 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171112+0000", 2024-09-15T03:46:55.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2152e-05 2024-09-15T03:46:55.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.592 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.592 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171119+0000", 2024-09-15T03:46:55.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5050000000000004e-06 2024-09-15T03:46:55.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.593 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171162+0000", 2024-09-15T03:46:55.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2908000000000003e-05 2024-09-15T03:46:55.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.593 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.593 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171169+0000", 2024-09-15T03:46:55.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9739999999999996e-06 2024-09-15T03:46:55.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.593 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.593 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171398+0000", 2024-09-15T03:46:55.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022954799999999999 2024-09-15T03:46:55.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173640+0000", 2024-09-15T03:46:55.594 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022417000000000001 2024-09-15T03:46:55.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173705+0000", 2024-09-15T03:46:55.594 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4907999999999996e-05 2024-09-15T03:46:55.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173742+0000", 2024-09-15T03:46:55.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6940999999999997e-05 2024-09-15T03:46:55.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:55.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.182702+0000", 2024-09-15T03:46:55.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0089604100000000002 2024-09-15T03:46:55.595 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.595 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:55.595 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.595 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 39/28 ECSubWrite(tid=43, reqid=client.4521.0:256, at_version=39'20, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:46:55.596 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.171196+0000", 2024-09-15T03:46:55.596 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.3319645050000002, 2024-09-15T03:46:55.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0090205690000000009, 2024-09-15T03:46:55.596 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:55.596 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.596 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:55.596 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.596 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:55.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171196+0000", 2024-09-15T03:46:55.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.596 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.596 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:55.597 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171196+0000", 2024-09-15T03:46:55.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.597 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.597 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:55.597 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171197+0000", 2024-09-15T03:46:55.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3e-07 2024-09-15T03:46:55.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.597 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.597 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:55.597 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171218+0000", 2024-09-15T03:46:55.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0775e-05 2024-09-15T03:46:55.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.598 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.598 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:55.598 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171241+0000", 2024-09-15T03:46:55.598 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3538999999999999e-05 2024-09-15T03:46:55.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.598 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.598 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.598 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171248+0000", 2024-09-15T03:46:55.598 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6220000000000003e-06 2024-09-15T03:46:55.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.598 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.598 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.598 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171291+0000", 2024-09-15T03:46:55.598 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3115000000000003e-05 2024-09-15T03:46:55.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.599 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.599 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.599 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171301+0000", 2024-09-15T03:46:55.599 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.662e-06 2024-09-15T03:46:55.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.599 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.599 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.599 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171395+0000", 2024-09-15T03:46:55.599 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4368000000000002e-05 2024-09-15T03:46:55.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.599 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.599 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.600 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173639+0000", 2024-09-15T03:46:55.600 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022444040000000002 2024-09-15T03:46:55.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.600 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.600 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.600 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173704+0000", 2024-09-15T03:46:55.600 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5031999999999997e-05 2024-09-15T03:46:55.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.600 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.600 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.600 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173718+0000", 2024-09-15T03:46:55.600 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3159000000000001e-05 2024-09-15T03:46:55.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.601 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.601 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:55.601 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180217+0000", 2024-09-15T03:46:55.601 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064991629999999996 2024-09-15T03:46:55.601 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.601 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:55.601 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.601 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.601 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 39/28 ECSubWrite(tid=44, reqid=client.4521.0:256, at_version=0'0, trim_to=0'0, roll_forward_to=39'20))", 2024-09-15T03:46:55.601 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.174176+0000", 2024-09-15T03:46:55.601 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.3289842850000007, 2024-09-15T03:46:55.601 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0061969080000000001, 2024-09-15T03:46:55.602 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:55.602 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.602 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:55.602 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.602 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:55.602 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174176+0000", 2024-09-15T03:46:55.602 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.602 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.602 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.602 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:55.602 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174176+0000", 2024-09-15T03:46:55.602 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.602 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.602 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.603 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:55.604 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174177+0000", 2024-09-15T03:46:55.604 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0999999999999998e-07 2024-09-15T03:46:55.604 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.604 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.604 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:55.604 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174187+0000", 2024-09-15T03:46:55.605 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0077e-05 2024-09-15T03:46:55.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.605 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.605 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:55.605 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174201+0000", 2024-09-15T03:46:55.605 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3499999999999999e-05 2024-09-15T03:46:55.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.605 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.605 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.605 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174207+0000", 2024-09-15T03:46:55.605 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7360000000000004e-06 2024-09-15T03:46:55.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.605 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.605 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.606 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174253+0000", 2024-09-15T03:46:55.606 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5223999999999999e-05 2024-09-15T03:46:55.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.606 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.606 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.606 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174262+0000", 2024-09-15T03:46:55.606 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8369999999999992e-06 2024-09-15T03:46:55.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.606 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.606 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.606 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174370+0000", 2024-09-15T03:46:55.606 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000107611 2024-09-15T03:46:55.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.607 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.607 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174377+0000", 2024-09-15T03:46:55.607 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.759e-06 2024-09-15T03:46:55.607 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.607 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.607 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174406+0000", 2024-09-15T03:46:55.607 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8816e-05 2024-09-15T03:46:55.607 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.607 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.607 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.608 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174413+0000", 2024-09-15T03:46:55.608 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6669999999999996e-06 2024-09-15T03:46:55.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.608 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.608 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:55.608 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180373+0000", 2024-09-15T03:46:55.608 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0059599709999999997 2024-09-15T03:46:55.608 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.608 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:55.608 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.608 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.609 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 39/31 ECSubWrite(tid=44, reqid=client.4521.0:243, at_version=0'0, trim_to=0'0, roll_forward_to=39'17))", 2024-09-15T03:46:55.609 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.176341+0000", 2024-09-15T03:46:55.609 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.3268196440000004, 2024-09-15T03:46:55.609 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006411043, 2024-09-15T03:46:55.609 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:55.609 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.609 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:55.609 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.609 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:55.609 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176341+0000", 2024-09-15T03:46:55.609 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.609 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.610 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:55.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176341+0000", 2024-09-15T03:46:55.610 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.610 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.610 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.610 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:55.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176342+0000", 2024-09-15T03:46:55.610 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4000000000000001e-07 2024-09-15T03:46:55.610 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.610 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.610 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:55.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176356+0000", 2024-09-15T03:46:55.610 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4756999999999999e-05 2024-09-15T03:46:55.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.611 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.611 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:55.611 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176364+0000", 2024-09-15T03:46:55.611 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6939999999999995e-06 2024-09-15T03:46:55.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.611 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.611 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.611 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176371+0000", 2024-09-15T03:46:55.611 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6959999999999999e-06 2024-09-15T03:46:55.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.611 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.611 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.612 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176404+0000", 2024-09-15T03:46:55.612 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2972000000000002e-05 2024-09-15T03:46:55.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.612 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.612 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.612 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176413+0000", 2024-09-15T03:46:55.612 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2499999999999995e-06 2024-09-15T03:46:55.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.612 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.612 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.612 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176496+0000", 2024-09-15T03:46:55.612 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3287000000000004e-05 2024-09-15T03:46:55.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.613 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.613 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176524+0000", 2024-09-15T03:46:55.613 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7659999999999999e-05 2024-09-15T03:46:55.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.613 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.613 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176556+0000", 2024-09-15T03:46:55.613 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1687999999999999e-05 2024-09-15T03:46:55.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.613 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.613 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176573+0000", 2024-09-15T03:46:55.614 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7680999999999999e-05 2024-09-15T03:46:55.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.614 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.614 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:55.614 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.182752+0000", 2024-09-15T03:46:55.614 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0061787179999999997 2024-09-15T03:46:55.614 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.614 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:55.614 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.614 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.614 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 39/31 ECSubWrite(tid=43, reqid=client.4521.0:252, at_version=39'27, trim_to=0'0, roll_forward_to=39'25))", 2024-09-15T03:46:55.614 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:46:55.614 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.3246192719999996, 2024-09-15T03:46:55.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011286757, 2024-09-15T03:46:55.615 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:55.615 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.615 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:55.615 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.615 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:55.615 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:46:55.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.615 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.615 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:55.615 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:46:55.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:55.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178542+0000", 2024-09-15T03:46:55.616 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9400000000000004e-07 2024-09-15T03:46:55.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:55.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178754+0000", 2024-09-15T03:46:55.616 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021228800000000001 2024-09-15T03:46:55.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:55.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178765+0000", 2024-09-15T03:46:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0176e-05 2024-09-15T03:46:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178769+0000", 2024-09-15T03:46:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1359999999999999e-06 2024-09-15T03:46:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178810+0000", 2024-09-15T03:46:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0825999999999998e-05 2024-09-15T03:46:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178820+0000", 2024-09-15T03:46:55.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0237999999999999e-05 2024-09-15T03:46:55.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.618 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179033+0000", 2024-09-15T03:46:55.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021361500000000001 2024-09-15T03:46:55.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.618 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179040+0000", 2024-09-15T03:46:55.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0509999999999999e-06 2024-09-15T03:46:55.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179102+0000", 2024-09-15T03:46:55.620 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1977000000000006e-05 2024-09-15T03:46:55.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179114+0000", 2024-09-15T03:46:55.620 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1223000000000001e-05 2024-09-15T03:46:55.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:55.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.189828+0000", 2024-09-15T03:46:55.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010714433000000001 2024-09-15T03:46:55.621 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.621 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:55.621 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.621 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 39/31 ECSubWrite(tid=44, reqid=client.4521.0:252, at_version=0'0, trim_to=0'0, roll_forward_to=39'27))", 2024-09-15T03:46:55.621 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.179948+0000", 2024-09-15T03:46:55.621 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.3232121729999999, 2024-09-15T03:46:55.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010068739, 2024-09-15T03:46:55.621 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:55.621 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.621 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:55.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:55.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179948+0000", 2024-09-15T03:46:55.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:55.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179948+0000", 2024-09-15T03:46:55.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:55.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179949+0000", 2024-09-15T03:46:55.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6199999999999998e-07 2024-09-15T03:46:55.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:55.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179960+0000", 2024-09-15T03:46:55.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1216000000000001e-05 2024-09-15T03:46:55.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:55.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179970+0000", 2024-09-15T03:46:55.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8980000000000008e-06 2024-09-15T03:46:55.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179975+0000", 2024-09-15T03:46:55.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5650000000000003e-06 2024-09-15T03:46:55.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180004+0000", 2024-09-15T03:46:55.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9224999999999999e-05 2024-09-15T03:46:55.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180009+0000", 2024-09-15T03:46:55.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9250000000000003e-06 2024-09-15T03:46:55.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180061+0000", 2024-09-15T03:46:55.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1758000000000001e-05 2024-09-15T03:46:55.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180067+0000", 2024-09-15T03:46:55.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3269999999999997e-06 2024-09-15T03:46:55.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180101+0000", 2024-09-15T03:46:55.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3701000000000001e-05 2024-09-15T03:46:55.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180112+0000", 2024-09-15T03:46:55.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1287000000000001e-05 2024-09-15T03:46:55.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:55.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.190017+0000", 2024-09-15T03:46:55.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0099052749999999998 2024-09-15T03:46:55.626 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.626 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:55.626 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.626 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.627 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:55.627 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:55.628 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:46:55.627+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:46:55.629 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:46:55.727 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:55.727 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:55.727 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:46:55.727 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:55.728 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:55.727+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:55.735 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:46:55.829 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:55.828+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:46:55.831 INFO:tasks.rados.rados.0.smithi001.stdout:125: done (0 left) 2024-09-15T03:46:55.831 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 50 current snap is 12 2024-09-15T03:46:55.831 INFO:tasks.rados.rados.0.smithi001.stdout:126: seq_num 72 ranges {835584=442368,1277952=720896,1998848=49152} 2024-09-15T03:46:55.837 INFO:tasks.rados.rados.0.smithi001.stdout:126: writing smithi00132172-50 from 835584 to 1277952 tid 1 2024-09-15T03:46:55.842 INFO:tasks.rados.rados.0.smithi001.stdout:126: writing smithi00132172-50 from 1277952 to 1998848 tid 2 2024-09-15T03:46:55.843 INFO:tasks.rados.rados.0.smithi001.stdout:126: writing smithi00132172-50 from 1998848 to 2048000 tid 3 2024-09-15T03:46:55.843 INFO:tasks.rados.rados.0.smithi001.stdout:127: delete oid 16 current snap is 12 2024-09-15T03:46:55.846 INFO:tasks.rados.rados.0.smithi001.stdout:126: finishing write tid 1 to smithi00132172-50 2024-09-15T03:46:55.847 INFO:tasks.rados.rados.0.smithi001.stdout:127: done (1 left) 2024-09-15T03:46:55.848 INFO:tasks.rados.rados.0.smithi001.stdout:128: read oid 45 snap -1 2024-09-15T03:46:55.848 INFO:tasks.rados.rados.0.smithi001.stdout:128: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-15T03:46:55.848 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 30 current snap is 12 2024-09-15T03:46:55.848 INFO:tasks.rados.rados.0.smithi001.stdout:129: seq_num 74 ranges {0=475136,475136=622592,1097728=147456} 2024-09-15T03:46:55.851 INFO:tasks.rados.rados.0.smithi001.stdout:129: writing smithi00132172-30 from 0 to 475136 tid 1 2024-09-15T03:46:55.855 INFO:tasks.rados.rados.0.smithi001.stdout:129: writing smithi00132172-30 from 475136 to 1097728 tid 2 2024-09-15T03:46:55.856 INFO:tasks.rados.rados.0.smithi001.stdout:129: writing smithi00132172-30 from 1097728 to 1245184 tid 3 2024-09-15T03:46:55.856 INFO:tasks.rados.rados.0.smithi001.stdout:130: rollback oid 43 current snap is 12 2024-09-15T03:46:55.857 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 43 to 11 2024-09-15T03:46:55.857 INFO:tasks.rados.rados.0.smithi001.stdout:126: finishing write tid 2 to smithi00132172-50 2024-09-15T03:46:55.857 INFO:tasks.rados.rados.0.smithi001.stdout:126: finishing write tid 3 to smithi00132172-50 2024-09-15T03:46:55.857 INFO:tasks.rados.rados.0.smithi001.stdout:131: delete oid 33 current snap is 12 2024-09-15T03:46:55.857 INFO:tasks.rados.rados.0.smithi001.stdout:126: finishing write tid 4 to smithi00132172-50 2024-09-15T03:46:55.857 INFO:tasks.rados.rados.0.smithi001.stdout:128: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-15T03:46:55.869 INFO:tasks.rados.rados.0.smithi001.stdout:129: finishing write tid 1 to smithi00132172-30 2024-09-15T03:46:55.869 INFO:tasks.rados.rados.0.smithi001.stdout:129: finishing write tid 2 to smithi00132172-30 2024-09-15T03:46:55.869 INFO:tasks.rados.rados.0.smithi001.stdout:129: finishing write tid 3 to smithi00132172-30 2024-09-15T03:46:55.869 INFO:tasks.rados.rados.0.smithi001.stdout:129: finishing write tid 4 to smithi00132172-30 2024-09-15T03:46:55.869 INFO:tasks.rados.rados.0.smithi001.stdout:129: finishing write tid 5 to smithi00132172-30 2024-09-15T03:46:55.869 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 30 v 16 (ObjNum 74 snap 12 seq_num 74) dirty exists 2024-09-15T03:46:55.869 INFO:tasks.rados.rados.0.smithi001.stdout:129: left oid 30 (ObjNum 74 snap 12 seq_num 74) 2024-09-15T03:46:55.869 INFO:tasks.rados.rados.0.smithi001.stdout:128: done (4 left) 2024-09-15T03:46:55.869 INFO:tasks.rados.rados.0.smithi001.stdout:129: done (3 left) 2024-09-15T03:46:55.869 INFO:tasks.rados.rados.0.smithi001.stdout:131: done (2 left) 2024-09-15T03:46:55.870 INFO:tasks.rados.rados.0.smithi001.stdout:132: delete oid 17 current snap is 12 2024-09-15T03:46:55.870 INFO:tasks.rados.rados.0.smithi001.stdout:132: done (2 left) 2024-09-15T03:46:55.871 INFO:tasks.rados.rados.0.smithi001.stdout:133: snap_remove snap 8 2024-09-15T03:46:55.878 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:55.879 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:55.879 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:46:55.879 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:46:55.879 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:55.887 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:46:55.931 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:55.930+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:55.986 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:55.986 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:46:55.986 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:46:55.986 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:46:55.986 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.986 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:175 3.cs0 3:3164c049:::smithi00132172-34:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e32)", 2024-09-15T03:46:55.986 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:39.553220+0000", 2024-09-15T03:46:55.986 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.430884980999998, 2024-09-15T03:46:55.986 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0052725100000000002, 2024-09-15T03:46:55.987 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:55.987 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:55.987 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:55.987 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:55.987 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:55.987 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 175 2024-09-15T03:46:55.987 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.987 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:55.987 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.987 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:55.987 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553220+0000", 2024-09-15T03:46:55.987 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.987 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.988 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.988 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:55.988 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553220+0000", 2024-09-15T03:46:55.988 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.988 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.988 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:55.988 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553221+0000", 2024-09-15T03:46:55.988 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0860000000000001e-06 2024-09-15T03:46:55.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.988 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.988 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:55.988 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553226+0000", 2024-09-15T03:46:55.989 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1630000000000004e-06 2024-09-15T03:46:55.989 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.989 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.989 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:55.989 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553239+0000", 2024-09-15T03:46:55.989 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2187e-05 2024-09-15T03:46:55.989 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.989 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.989 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.989 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553244+0000", 2024-09-15T03:46:55.989 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8960000000000002e-06 2024-09-15T03:46:55.989 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.990 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.990 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.990 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553276+0000", 2024-09-15T03:46:55.990 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2237000000000003e-05 2024-09-15T03:46:55.990 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.990 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.991 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:55.991 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553345+0000", 2024-09-15T03:46:55.991 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9618999999999999e-05 2024-09-15T03:46:55.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.991 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.992 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.992 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553695+0000", 2024-09-15T03:46:55.992 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034979 2024-09-15T03:46:55.993 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.993 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.993 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.993 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553780+0000", 2024-09-15T03:46:55.993 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4633000000000001e-05 2024-09-15T03:46:55.993 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.993 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.993 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:55.993 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553787+0000", 2024-09-15T03:46:55.993 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2409999999999998e-06 2024-09-15T03:46:55.993 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.993 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.993 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:55.994 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553828+0000", 2024-09-15T03:46:55.994 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0564e-05 2024-09-15T03:46:55.994 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.994 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.994 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:55.994 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.553841+0000", 2024-09-15T03:46:55.994 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3249e-05 2024-09-15T03:46:55.994 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.994 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.994 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:55.995 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.554266+0000", 2024-09-15T03:46:55.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00042530299999999998 2024-09-15T03:46:55.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.995 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:55.995 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:39.558493+0000", 2024-09-15T03:46:55.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042265419999999998 2024-09-15T03:46:55.995 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.995 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:55.995 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:55.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.995 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:55.996 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.108773+0000", 2024-09-15T03:46:55.996 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.875332362, 2024-09-15T03:46:55.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0047381410000000004, 2024-09-15T03:46:55.996 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:55.996 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:55.996 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:55.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.996 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:55.996 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108773+0000", 2024-09-15T03:46:55.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.996 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.996 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:55.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108773+0000", 2024-09-15T03:46:55.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:55.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:55.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108774+0000", 2024-09-15T03:46:55.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8800000000000002e-07 2024-09-15T03:46:55.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:55.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108801+0000", 2024-09-15T03:46:55.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7339999999999999e-05 2024-09-15T03:46:55.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.998 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:55.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108816+0000", 2024-09-15T03:46:55.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5359999999999999e-05 2024-09-15T03:46:55.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.998 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:55.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108823+0000", 2024-09-15T03:46:55.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0060000000000003e-06 2024-09-15T03:46:55.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.999 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:55.999 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108930+0000", 2024-09-15T03:46:55.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000106569 2024-09-15T03:46:55.999 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.999 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:55.999 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108940+0000", 2024-09-15T03:46:55.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.679e-06 2024-09-15T03:46:55.999 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:55.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:55.999 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:55.999 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109030+0000", 2024-09-15T03:46:55.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0618999999999994e-05 2024-09-15T03:46:56.000 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.000 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.000 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109041+0000", 2024-09-15T03:46:56.000 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1011e-05 2024-09-15T03:46:56.000 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.000 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.000 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109073+0000", 2024-09-15T03:46:56.000 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1427999999999998e-05 2024-09-15T03:46:56.000 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.000 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109088+0000", 2024-09-15T03:46:56.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4956e-05 2024-09-15T03:46:56.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.113511+0000", 2024-09-15T03:46:56.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044233850000000002 2024-09-15T03:46:56.001 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.001 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.001 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.001 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:46:56.002 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.109181+0000", 2024-09-15T03:46:56.002 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.874924463999999, 2024-09-15T03:46:56.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044348879999999997, 2024-09-15T03:46:56.002 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.002 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.002 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109181+0000", 2024-09-15T03:46:56.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109181+0000", 2024-09-15T03:46:56.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.003 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.003 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109181+0000", 2024-09-15T03:46:56.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0900000000000001e-07 2024-09-15T03:46:56.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.003 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.003 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109197+0000", 2024-09-15T03:46:56.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5150999999999999e-05 2024-09-15T03:46:56.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.004 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.004 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109211+0000", 2024-09-15T03:46:56.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3998999999999999e-05 2024-09-15T03:46:56.004 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.004 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.004 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109217+0000", 2024-09-15T03:46:56.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.686e-06 2024-09-15T03:46:56.004 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.004 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.004 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109254+0000", 2024-09-15T03:46:56.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6424999999999998e-05 2024-09-15T03:46:56.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109262+0000", 2024-09-15T03:46:56.005 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8230000000000001e-06 2024-09-15T03:46:56.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109350+0000", 2024-09-15T03:46:56.005 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7120999999999995e-05 2024-09-15T03:46:56.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109358+0000", 2024-09-15T03:46:56.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3399999999999998e-06 2024-09-15T03:46:56.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109388+0000", 2024-09-15T03:46:56.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9807000000000001e-05 2024-09-15T03:46:56.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109398+0000", 2024-09-15T03:46:56.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0744000000000001e-05 2024-09-15T03:46:56.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.113616+0000", 2024-09-15T03:46:56.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042171830000000002 2024-09-15T03:46:56.007 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.007 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.007 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.007 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es7 34/28 ECSubWrite(tid=33, reqid=client.4521.0:195, at_version=34'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-15T03:46:56.007 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.118211+0000", 2024-09-15T03:46:56.007 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.865893871000001, 2024-09-15T03:46:56.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0070162009999999997, 2024-09-15T03:46:56.008 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.008 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.008 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.008 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118211+0000", 2024-09-15T03:46:56.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.008 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118211+0000", 2024-09-15T03:46:56.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118212+0000", 2024-09-15T03:46:56.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5599999999999999e-07 2024-09-15T03:46:56.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118422+0000", 2024-09-15T03:46:56.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021081 2024-09-15T03:46:56.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.012 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.012 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118433+0000", 2024-09-15T03:46:56.012 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0529e-05 2024-09-15T03:46:56.012 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.012 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.012 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.013 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118438+0000", 2024-09-15T03:46:56.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8210000000000001e-06 2024-09-15T03:46:56.013 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.013 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.013 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118479+0000", 2024-09-15T03:46:56.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1493999999999999e-05 2024-09-15T03:46:56.013 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.013 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.014 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118490+0000", 2024-09-15T03:46:56.014 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0550000000000001e-05 2024-09-15T03:46:56.014 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.014 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.014 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.014 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118700+0000", 2024-09-15T03:46:56.014 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021063500000000001 2024-09-15T03:46:56.014 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.014 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.014 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.014 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118711+0000", 2024-09-15T03:46:56.014 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0849000000000001e-05 2024-09-15T03:46:56.015 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.015 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.015 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.015 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118776+0000", 2024-09-15T03:46:56.015 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4702999999999994e-05 2024-09-15T03:46:56.015 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.015 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.015 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.015 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118797+0000", 2024-09-15T03:46:56.015 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1265999999999999e-05 2024-09-15T03:46:56.015 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.015 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.016 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.016 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125228+0000", 2024-09-15T03:46:56.016 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064303880000000004 2024-09-15T03:46:56.016 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.016 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.016 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.016 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.016 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.016 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s3 39/31 ECSubWrite(tid=44, reqid=osd.1.0:44, at_version=39'12, trim_to=0'0, roll_forward_to=38'10))", 2024-09-15T03:46:56.016 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.150872+0000", 2024-09-15T03:46:56.016 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.8332334340000003, 2024-09-15T03:46:56.016 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075168539999999999, 2024-09-15T03:46:56.016 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.017 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.017 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.017 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150872+0000", 2024-09-15T03:46:56.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.017 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.017 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150872+0000", 2024-09-15T03:46:56.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.017 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150872+0000", 2024-09-15T03:46:56.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3300000000000002e-07 2024-09-15T03:46:56.018 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150909+0000", 2024-09-15T03:46:56.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6294000000000002e-05 2024-09-15T03:46:56.018 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150919+0000", 2024-09-15T03:46:56.019 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9690000000000006e-06 2024-09-15T03:46:56.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.019 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150924+0000", 2024-09-15T03:46:56.019 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7710000000000003e-06 2024-09-15T03:46:56.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.019 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150958+0000", 2024-09-15T03:46:56.019 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3531999999999997e-05 2024-09-15T03:46:56.020 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.020 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.020 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.020 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150970+0000", 2024-09-15T03:46:56.020 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1831000000000001e-05 2024-09-15T03:46:56.020 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.020 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.020 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.020 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152463+0000", 2024-09-15T03:46:56.020 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0014934239999999999 2024-09-15T03:46:56.020 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.020 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.021 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152471+0000", 2024-09-15T03:46:56.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2830000000000004e-06 2024-09-15T03:46:56.021 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.021 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152507+0000", 2024-09-15T03:46:56.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6739999999999997e-05 2024-09-15T03:46:56.021 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.021 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152517+0000", 2024-09-15T03:46:56.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8570000000000007e-06 2024-09-15T03:46:56.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.022 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.022 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.158389+0000", 2024-09-15T03:46:56.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058715199999999999 2024-09-15T03:46:56.022 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.022 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.022 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.022 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.0s2 39/31 ECSubWrite(tid=45, reqid=client.4521.0:241, at_version=39'17, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:46:56.023 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.150931+0000", 2024-09-15T03:46:56.023 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.8331743150000008, 2024-09-15T03:46:56.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0072693200000000001, 2024-09-15T03:46:56.023 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.023 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.023 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.023 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.023 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.023 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150931+0000", 2024-09-15T03:46:56.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.023 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.023 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150931+0000", 2024-09-15T03:46:56.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150931+0000", 2024-09-15T03:46:56.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3699999999999999e-07 2024-09-15T03:46:56.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150942+0000", 2024-09-15T03:46:56.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1053000000000001e-05 2024-09-15T03:46:56.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150946+0000", 2024-09-15T03:46:56.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0890000000000002e-06 2024-09-15T03:46:56.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150949+0000", 2024-09-15T03:46:56.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5270000000000001e-06 2024-09-15T03:46:56.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.026 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150983+0000", 2024-09-15T03:46:56.026 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4258000000000003e-05 2024-09-15T03:46:56.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.026 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.026 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150994+0000", 2024-09-15T03:46:56.026 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0893999999999999e-05 2024-09-15T03:46:56.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.026 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.027 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.151057+0000", 2024-09-15T03:46:56.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3478000000000004e-05 2024-09-15T03:46:56.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.027 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.027 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.151063+0000", 2024-09-15T03:46:56.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7880000000000003e-06 2024-09-15T03:46:56.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.027 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.027 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152305+0000", 2024-09-15T03:46:56.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012418850000000001 2024-09-15T03:46:56.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.028 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.028 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152335+0000", 2024-09-15T03:46:56.028 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0049e-05 2024-09-15T03:46:56.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.028 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.028 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.158200+0000", 2024-09-15T03:46:56.028 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058650619999999999 2024-09-15T03:46:56.028 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.028 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.028 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.029 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.0s2 39/31 ECSubWrite(tid=46, reqid=client.4521.0:242, at_version=39'18, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:46:56.029 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.150954+0000", 2024-09-15T03:46:56.029 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.8331508129999996, 2024-09-15T03:46:56.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.007342459, 2024-09-15T03:46:56.029 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.029 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.029 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.029 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.029 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150954+0000", 2024-09-15T03:46:56.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.029 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150954+0000", 2024-09-15T03:46:56.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.030 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150955+0000", 2024-09-15T03:46:56.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2700000000000001e-07 2024-09-15T03:46:56.030 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150964+0000", 2024-09-15T03:46:56.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2820000000000002e-06 2024-09-15T03:46:56.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150970+0000", 2024-09-15T03:46:56.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0700000000000003e-06 2024-09-15T03:46:56.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150973+0000", 2024-09-15T03:46:56.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.576e-06 2024-09-15T03:46:56.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.151082+0000", 2024-09-15T03:46:56.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000109261 2024-09-15T03:46:56.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.151088+0000", 2024-09-15T03:46:56.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2480000000000001e-06 2024-09-15T03:46:56.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.151145+0000", 2024-09-15T03:46:56.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6886999999999999e-05 2024-09-15T03:46:56.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152307+0000", 2024-09-15T03:46:56.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0011623180000000001 2024-09-15T03:46:56.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.033 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152336+0000", 2024-09-15T03:46:56.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8677999999999999e-05 2024-09-15T03:46:56.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.033 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152365+0000", 2024-09-15T03:46:56.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9170999999999999e-05 2024-09-15T03:46:56.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.033 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.034 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.158297+0000", 2024-09-15T03:46:56.034 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0059317409999999999 2024-09-15T03:46:56.034 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.034 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.034 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.034 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.034 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.034 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 39/28 ECSubWrite(tid=42, reqid=client.4521.0:255, at_version=39'19, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:46:56.034 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.170724+0000", 2024-09-15T03:46:56.034 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.8133812260000006, 2024-09-15T03:46:56.034 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0086604270000000001, 2024-09-15T03:46:56.034 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.034 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.034 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.035 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.035 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.035 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170724+0000", 2024-09-15T03:46:56.035 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.035 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.035 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.035 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.035 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170724+0000", 2024-09-15T03:46:56.035 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.035 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.035 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.035 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.035 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170725+0000", 2024-09-15T03:46:56.036 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8999999999999995e-07 2024-09-15T03:46:56.036 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.036 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.036 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.036 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170742+0000", 2024-09-15T03:46:56.036 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7181e-05 2024-09-15T03:46:56.036 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.036 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.036 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.036 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170757+0000", 2024-09-15T03:46:56.036 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4693e-05 2024-09-15T03:46:56.036 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.036 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.036 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.037 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170763+0000", 2024-09-15T03:46:56.037 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4509999999999997e-06 2024-09-15T03:46:56.037 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.037 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170807+0000", 2024-09-15T03:46:56.037 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3574e-05 2024-09-15T03:46:56.037 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.037 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170815+0000", 2024-09-15T03:46:56.037 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3459999999999994e-06 2024-09-15T03:46:56.037 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.038 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.038 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.038 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170900+0000", 2024-09-15T03:46:56.038 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5436e-05 2024-09-15T03:46:56.038 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.038 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.038 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.038 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170906+0000", 2024-09-15T03:46:56.038 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9190000000000004e-06 2024-09-15T03:46:56.038 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.038 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.038 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.038 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173681+0000", 2024-09-15T03:46:56.038 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00277437 2024-09-15T03:46:56.039 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.039 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.039 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.039 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173695+0000", 2024-09-15T03:46:56.039 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4446e-05 2024-09-15T03:46:56.039 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.039 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.039 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.039 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179384+0000", 2024-09-15T03:46:56.039 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0056892210000000004 2024-09-15T03:46:56.039 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.039 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.039 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.040 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.040 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.040 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:46:56.040 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:46:56.040 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.8130033910000005, 2024-09-15T03:46:56.040 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012183955999999999, 2024-09-15T03:46:56.040 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.040 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.040 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.040 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.040 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.040 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:46:56.040 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.040 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.041 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.041 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.041 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:46:56.041 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.041 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.041 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.041 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.041 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:46:56.041 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.17e-07 2024-09-15T03:46:56.041 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.041 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.041 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.041 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171454+0000", 2024-09-15T03:46:56.042 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00035226700000000002 2024-09-15T03:46:56.042 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.042 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.042 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.042 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171465+0000", 2024-09-15T03:46:56.042 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0943e-05 2024-09-15T03:46:56.042 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.042 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.042 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.042 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171471+0000", 2024-09-15T03:46:56.042 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7400000000000001e-06 2024-09-15T03:46:56.042 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.042 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.042 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.043 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171508+0000", 2024-09-15T03:46:56.043 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7147999999999997e-05 2024-09-15T03:46:56.043 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.043 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.043 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.043 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171517+0000", 2024-09-15T03:46:56.043 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2560000000000008e-06 2024-09-15T03:46:56.043 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.043 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.043 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.043 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171716+0000", 2024-09-15T03:46:56.043 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019899699999999999 2024-09-15T03:46:56.043 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.043 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.044 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.044 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173686+0000", 2024-09-15T03:46:56.044 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019696829999999999 2024-09-15T03:46:56.044 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.044 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.044 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.044 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173777+0000", 2024-09-15T03:46:56.044 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0637999999999995e-05 2024-09-15T03:46:56.044 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.044 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.044 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.044 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173814+0000", 2024-09-15T03:46:56.045 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7499000000000001e-05 2024-09-15T03:46:56.045 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.045 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.045 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.045 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.183286+0000", 2024-09-15T03:46:56.045 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094716680000000008 2024-09-15T03:46:56.045 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.045 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.045 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.045 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.045 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.045 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 39/28 ECSubWrite(tid=43, reqid=client.4521.0:256, at_version=39'20, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:46:56.045 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.171211+0000", 2024-09-15T03:46:56.045 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.8128942979999998, 2024-09-15T03:46:56.046 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082752069999999997, 2024-09-15T03:46:56.046 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.046 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.046 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.046 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.046 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.046 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171211+0000", 2024-09-15T03:46:56.046 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.046 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.046 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.047 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.047 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171211+0000", 2024-09-15T03:46:56.047 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.047 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.047 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.047 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.047 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171211+0000", 2024-09-15T03:46:56.047 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.39e-07 2024-09-15T03:46:56.047 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.047 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.047 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.047 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171228+0000", 2024-09-15T03:46:56.047 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6254999999999999e-05 2024-09-15T03:46:56.047 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.048 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.048 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.048 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171241+0000", 2024-09-15T03:46:56.048 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3482999999999999e-05 2024-09-15T03:46:56.049 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.049 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.049 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.049 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171247+0000", 2024-09-15T03:46:56.049 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.088e-06 2024-09-15T03:46:56.049 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.049 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.049 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.049 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171292+0000", 2024-09-15T03:46:56.049 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5062000000000003e-05 2024-09-15T03:46:56.049 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.049 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.049 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.050 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171302+0000", 2024-09-15T03:46:56.050 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.302e-06 2024-09-15T03:46:56.050 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.050 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.050 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.050 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171382+0000", 2024-09-15T03:46:56.050 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0717999999999995e-05 2024-09-15T03:46:56.050 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.050 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.050 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.050 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173686+0000", 2024-09-15T03:46:56.050 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023032320000000001 2024-09-15T03:46:56.050 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.050 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.051 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.051 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173776+0000", 2024-09-15T03:46:56.051 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0302000000000004e-05 2024-09-15T03:46:56.051 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.051 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.051 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.051 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173805+0000", 2024-09-15T03:46:56.051 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8979000000000002e-05 2024-09-15T03:46:56.051 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.051 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.051 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.051 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179486+0000", 2024-09-15T03:46:56.051 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0056813469999999998 2024-09-15T03:46:56.051 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.052 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.052 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.052 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.052 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.052 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 39/28 ECSubWrite(tid=44, reqid=client.4521.0:256, at_version=0'0, trim_to=0'0, roll_forward_to=39'20))", 2024-09-15T03:46:56.052 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.174185+0000", 2024-09-15T03:46:56.052 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.809920193, 2024-09-15T03:46:56.052 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0054108059999999998, 2024-09-15T03:46:56.052 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.052 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.052 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.052 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.052 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.053 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174185+0000", 2024-09-15T03:46:56.053 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.053 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.053 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.053 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.053 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174185+0000", 2024-09-15T03:46:56.053 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.053 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.053 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.053 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.053 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174186+0000", 2024-09-15T03:46:56.053 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5099999999999999e-07 2024-09-15T03:46:56.053 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.053 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.054 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.054 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174197+0000", 2024-09-15T03:46:56.054 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0823e-05 2024-09-15T03:46:56.054 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.054 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.054 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.054 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174212+0000", 2024-09-15T03:46:56.054 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5268e-05 2024-09-15T03:46:56.054 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.054 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.054 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.054 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174218+0000", 2024-09-15T03:46:56.054 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9580000000000004e-06 2024-09-15T03:46:56.055 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.055 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.055 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.055 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174273+0000", 2024-09-15T03:46:56.055 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4982999999999999e-05 2024-09-15T03:46:56.055 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.055 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.055 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.055 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174281+0000", 2024-09-15T03:46:56.055 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0309999999999998e-06 2024-09-15T03:46:56.055 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.055 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.055 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.055 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174386+0000", 2024-09-15T03:46:56.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000105215 2024-09-15T03:46:56.056 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.056 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.056 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.056 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174392+0000", 2024-09-15T03:46:56.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2500000000000003e-06 2024-09-15T03:46:56.056 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.056 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.056 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.056 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174418+0000", 2024-09-15T03:46:56.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5259999999999999e-05 2024-09-15T03:46:56.056 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.056 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.056 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.057 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.174428+0000", 2024-09-15T03:46:56.057 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0047999999999999e-05 2024-09-15T03:46:56.057 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.057 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.057 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.057 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179596+0000", 2024-09-15T03:46:56.057 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0051682189999999999 2024-09-15T03:46:56.057 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.057 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.057 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.057 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.057 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.057 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 39/31 ECSubWrite(tid=44, reqid=client.4521.0:243, at_version=0'0, trim_to=0'0, roll_forward_to=39'17))", 2024-09-15T03:46:56.058 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.176404+0000", 2024-09-15T03:46:56.058 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.8077015010000004, 2024-09-15T03:46:56.058 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069470829999999997, 2024-09-15T03:46:56.058 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.058 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.058 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.058 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.058 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.058 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176404+0000", 2024-09-15T03:46:56.058 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.058 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.058 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.058 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.058 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176404+0000", 2024-09-15T03:46:56.059 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.059 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.059 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.059 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.059 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176404+0000", 2024-09-15T03:46:56.059 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.54e-07 2024-09-15T03:46:56.059 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.059 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.059 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.059 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176417+0000", 2024-09-15T03:46:56.059 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2785999999999999e-05 2024-09-15T03:46:56.059 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.059 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.059 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.060 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176430+0000", 2024-09-15T03:46:56.060 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2622999999999999e-05 2024-09-15T03:46:56.060 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.060 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.060 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.060 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176436+0000", 2024-09-15T03:46:56.060 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3570000000000003e-06 2024-09-15T03:46:56.060 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.060 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.060 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.060 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176479+0000", 2024-09-15T03:46:56.060 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2939999999999999e-05 2024-09-15T03:46:56.060 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.060 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.061 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.061 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176487+0000", 2024-09-15T03:46:56.061 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2979999999999992e-06 2024-09-15T03:46:56.061 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.061 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.061 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.061 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176580+0000", 2024-09-15T03:46:56.061 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2349999999999995e-05 2024-09-15T03:46:56.061 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.061 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.061 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.061 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176588+0000", 2024-09-15T03:46:56.061 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2360000000000004e-06 2024-09-15T03:46:56.061 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.062 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.062 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176618+0000", 2024-09-15T03:46:56.062 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9459999999999999e-05 2024-09-15T03:46:56.062 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.062 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.062 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176629+0000", 2024-09-15T03:46:56.062 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1382999999999999e-05 2024-09-15T03:46:56.062 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.062 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.062 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.183351+0000", 2024-09-15T03:46:56.062 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0067218959999999998 2024-09-15T03:46:56.063 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.063 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.063 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.063 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.063 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.063 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:300 3.cs0 3:3164c049:::smithi00132172-34:head [append 0~802816] snapc c=[c,b,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-15T03:46:56.063 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:50.162456+0000", 2024-09-15T03:46:56.063 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.8216493749999998, 2024-09-15T03:46:56.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0057491620000000004, 2024-09-15T03:46:56.063 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.063 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:56.063 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:56.063 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:56.063 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:56.064 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 300 2024-09-15T03:46:56.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.064 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.064 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.064 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.064 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.162456+0000", 2024-09-15T03:46:56.064 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.064 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.064 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.064 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.162456+0000", 2024-09-15T03:46:56.064 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.064 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.065 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.065 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.162458+0000", 2024-09-15T03:46:56.065 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7710000000000001e-06 2024-09-15T03:46:56.065 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.065 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.065 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.065 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.163778+0000", 2024-09-15T03:46:56.065 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013201 2024-09-15T03:46:56.065 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.065 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.065 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.065 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.163798+0000", 2024-09-15T03:46:56.065 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0669000000000001e-05 2024-09-15T03:46:56.065 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.066 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.066 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.066 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.163806+0000", 2024-09-15T03:46:56.066 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8259999999999993e-06 2024-09-15T03:46:56.066 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.066 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.066 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.066 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.163851+0000", 2024-09-15T03:46:56.066 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4449000000000001e-05 2024-09-15T03:46:56.066 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.066 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.066 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:56.066 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.163921+0000", 2024-09-15T03:46:56.066 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0483999999999998e-05 2024-09-15T03:46:56.067 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.067 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.067 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.067 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.164900+0000", 2024-09-15T03:46:56.067 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0009785970000000001 2024-09-15T03:46:56.067 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.067 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.067 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.067 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.165000+0000", 2024-09-15T03:46:56.067 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9877999999999995e-05 2024-09-15T03:46:56.067 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.067 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.068 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.068 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.165014+0000", 2024-09-15T03:46:56.068 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.412e-05 2024-09-15T03:46:56.068 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.068 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.068 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.068 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.165142+0000", 2024-09-15T03:46:56.068 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000128622 2024-09-15T03:46:56.068 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.068 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.068 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.068 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.165179+0000", 2024-09-15T03:46:56.068 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6752999999999999e-05 2024-09-15T03:46:56.068 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.069 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.069 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:56.069 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.168155+0000", 2024-09-15T03:46:56.069 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002976027 2024-09-15T03:46:56.069 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.069 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.069 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.069 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:50.168205+0000", 2024-09-15T03:46:56.069 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9866e-05 2024-09-15T03:46:56.069 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.069 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.069 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.069 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.069 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.070 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 49/31 ECSubWrite(tid=80, reqid=client.4521.0:320, at_version=49'15, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:46:56.070 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.841201+0000", 2024-09-15T03:46:56.070 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.142904107, 2024-09-15T03:46:56.070 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058584550000000003, 2024-09-15T03:46:56.070 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.070 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.070 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.070 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.070 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.070 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.841201+0000", 2024-09-15T03:46:56.070 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.070 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.070 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.070 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.071 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.841201+0000", 2024-09-15T03:46:56.071 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.071 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.071 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.071 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.071 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.841201+0000", 2024-09-15T03:46:56.071 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3099999999999999e-07 2024-09-15T03:46:56.071 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.071 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.071 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.071 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.841429+0000", 2024-09-15T03:46:56.071 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022737899999999999 2024-09-15T03:46:56.071 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.071 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.072 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.072 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.841444+0000", 2024-09-15T03:46:56.072 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5339999999999999e-05 2024-09-15T03:46:56.072 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.072 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.072 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.072 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.841454+0000", 2024-09-15T03:46:56.072 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0200999999999999e-05 2024-09-15T03:46:56.072 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.072 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.072 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.072 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.841514+0000", 2024-09-15T03:46:56.072 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9715e-05 2024-09-15T03:46:56.072 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.073 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.073 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.073 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.841526+0000", 2024-09-15T03:46:56.073 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1834000000000001e-05 2024-09-15T03:46:56.073 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.073 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.073 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.073 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.841847+0000", 2024-09-15T03:46:56.073 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00032141 2024-09-15T03:46:56.073 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.073 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.073 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.073 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.842137+0000", 2024-09-15T03:46:56.073 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00028981100000000001 2024-09-15T03:46:56.074 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.074 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.074 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.074 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.842212+0000", 2024-09-15T03:46:56.074 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4672999999999995e-05 2024-09-15T03:46:56.074 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.074 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.074 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.074 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.842230+0000", 2024-09-15T03:46:56.074 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7873999999999999e-05 2024-09-15T03:46:56.074 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.074 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.074 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.074 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.847060+0000", 2024-09-15T03:46:56.075 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.004829987 2024-09-15T03:46:56.075 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.075 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.075 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.075 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.075 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.075 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.9s4 49/31 ECSubWrite(tid=30, reqid=client.4521.0:325, at_version=49'12, trim_to=0'0, roll_forward_to=40'10))", 2024-09-15T03:46:56.075 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.845212+0000", 2024-09-15T03:46:56.075 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.13889368499999999, 2024-09-15T03:46:56.075 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0051511609999999996, 2024-09-15T03:46:56.075 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.075 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.075 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.075 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.076 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.076 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.845212+0000", 2024-09-15T03:46:56.076 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.076 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.076 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.076 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.076 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.845212+0000", 2024-09-15T03:46:56.076 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.076 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.076 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.076 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.076 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.845212+0000", 2024-09-15T03:46:56.076 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7599999999999997e-07 2024-09-15T03:46:56.076 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.077 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.077 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.077 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.845239+0000", 2024-09-15T03:46:56.077 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6815e-05 2024-09-15T03:46:56.077 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.077 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.077 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.077 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.845251+0000", 2024-09-15T03:46:56.077 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2079000000000001e-05 2024-09-15T03:46:56.077 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.077 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.077 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.077 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.845258+0000", 2024-09-15T03:46:56.077 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.64e-06 2024-09-15T03:46:56.078 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.078 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.078 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.078 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.845326+0000", 2024-09-15T03:46:56.078 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7957e-05 2024-09-15T03:46:56.078 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.078 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.078 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.078 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.845337+0000", 2024-09-15T03:46:56.078 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1997000000000001e-05 2024-09-15T03:46:56.078 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.078 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.078 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.078 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.845575+0000", 2024-09-15T03:46:56.079 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023760799999999999 2024-09-15T03:46:56.079 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.079 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.079 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.079 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.845741+0000", 2024-09-15T03:46:56.079 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016634199999999999 2024-09-15T03:46:56.079 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.079 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.079 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.079 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.846742+0000", 2024-09-15T03:46:56.079 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010006609999999999 2024-09-15T03:46:56.079 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.079 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.079 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.080 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.846761+0000", 2024-09-15T03:46:56.080 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9134e-05 2024-09-15T03:46:56.080 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.080 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.080 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.080 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.850363+0000", 2024-09-15T03:46:56.080 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0036014520000000002 2024-09-15T03:46:56.080 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.080 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.080 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.080 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.080 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.080 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:337 3.cs0 3:390b2f04:::smithi00132172-33:head [delete] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e49)", 2024-09-15T03:46:56.081 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.856819+0000", 2024-09-15T03:46:56.081 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.12728615700000001, 2024-09-15T03:46:56.081 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0077070849999999998, 2024-09-15T03:46:56.081 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.081 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:56.081 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:56.081 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:56.081 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:56.081 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 337 2024-09-15T03:46:56.081 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.081 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.081 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.081 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.081 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856819+0000", 2024-09-15T03:46:56.082 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.082 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.082 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.082 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.082 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856819+0000", 2024-09-15T03:46:56.082 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.082 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.082 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.082 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.082 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856821+0000", 2024-09-15T03:46:56.082 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6700000000000001e-06 2024-09-15T03:46:56.082 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.082 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.082 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.083 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856827+0000", 2024-09-15T03:46:56.083 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8080000000000001e-06 2024-09-15T03:46:56.083 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.083 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.083 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.083 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856843+0000", 2024-09-15T03:46:56.083 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6430999999999999e-05 2024-09-15T03:46:56.083 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.083 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.083 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.083 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:46:56.083 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7920000000000004e-06 2024-09-15T03:46:56.083 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.083 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.084 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856889+0000", 2024-09-15T03:46:56.084 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8825000000000002e-05 2024-09-15T03:46:56.084 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.084 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:56.084 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856960+0000", 2024-09-15T03:46:56.084 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1470999999999994e-05 2024-09-15T03:46:56.084 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.084 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.084 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857667+0000", 2024-09-15T03:46:56.084 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00070671199999999997 2024-09-15T03:46:56.085 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.085 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.085 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.085 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857896+0000", 2024-09-15T03:46:56.085 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000229621 2024-09-15T03:46:56.085 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.085 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.085 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.085 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857912+0000", 2024-09-15T03:46:56.085 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5400000000000002e-05 2024-09-15T03:46:56.085 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.085 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.085 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.086 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857949+0000", 2024-09-15T03:46:56.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7239e-05 2024-09-15T03:46:56.086 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.086 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.086 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.086 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858108+0000", 2024-09-15T03:46:56.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015922399999999999 2024-09-15T03:46:56.086 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.086 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.086 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:56.086 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859023+0000", 2024-09-15T03:46:56.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00091487899999999995 2024-09-15T03:46:56.086 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.086 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.087 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.087 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.864526+0000", 2024-09-15T03:46:56.087 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0055030130000000002 2024-09-15T03:46:56.087 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.087 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.087 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.087 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.087 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.087 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 49/31 ECSubWrite(tid=87, reqid=client.4521.0:335, at_version=49'21, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:46:56.087 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.858786+0000", 2024-09-15T03:46:56.087 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.12531952199999999, 2024-09-15T03:46:56.087 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0085806300000000005, 2024-09-15T03:46:56.087 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.087 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.088 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.088 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.088 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.088 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858786+0000", 2024-09-15T03:46:56.088 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.088 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.088 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.088 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.088 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858786+0000", 2024-09-15T03:46:56.088 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.088 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.088 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.088 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.088 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858786+0000", 2024-09-15T03:46:56.089 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3099999999999997e-07 2024-09-15T03:46:56.089 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.089 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.089 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.089 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858809+0000", 2024-09-15T03:46:56.089 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2331999999999999e-05 2024-09-15T03:46:56.089 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.089 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.089 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.089 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858819+0000", 2024-09-15T03:46:56.089 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0468e-05 2024-09-15T03:46:56.089 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.089 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.089 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.090 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858828+0000", 2024-09-15T03:46:56.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0499999999999997e-06 2024-09-15T03:46:56.090 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.090 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.090 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858864+0000", 2024-09-15T03:46:56.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6072000000000003e-05 2024-09-15T03:46:56.090 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.090 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.090 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858870+0000", 2024-09-15T03:46:56.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.896e-06 2024-09-15T03:46:56.090 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858938+0000", 2024-09-15T03:46:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7657000000000006e-05 2024-09-15T03:46:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858946+0000", 2024-09-15T03:46:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0609999999999996e-06 2024-09-15T03:46:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861287+0000", 2024-09-15T03:46:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023413779999999999 2024-09-15T03:46:56.092 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.092 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.092 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861301+0000", 2024-09-15T03:46:56.092 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3626e-05 2024-09-15T03:46:56.092 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.092 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.092 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.867366+0000", 2024-09-15T03:46:56.092 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0060654589999999996 2024-09-15T03:46:56.092 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.092 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.092 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.092 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.093 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 49/31 ECSubWrite(tid=88, reqid=client.4521.0:336, at_version=49'22, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:46:56.093 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.859239+0000", 2024-09-15T03:46:56.093 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.12486655000000001, 2024-09-15T03:46:56.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082420800000000006, 2024-09-15T03:46:56.093 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.093 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.093 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859239+0000", 2024-09-15T03:46:56.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.094 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859239+0000", 2024-09-15T03:46:56.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.094 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859241+0000", 2024-09-15T03:46:56.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4880000000000001e-06 2024-09-15T03:46:56.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.094 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859256+0000", 2024-09-15T03:46:56.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5119999999999999e-05 2024-09-15T03:46:56.095 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.095 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.095 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.095 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859265+0000", 2024-09-15T03:46:56.095 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1500000000000005e-06 2024-09-15T03:46:56.095 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.095 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.095 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.095 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859274+0000", 2024-09-15T03:46:56.095 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2060000000000007e-06 2024-09-15T03:46:56.095 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.095 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.095 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.095 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859314+0000", 2024-09-15T03:46:56.096 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0250000000000003e-05 2024-09-15T03:46:56.096 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.096 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.096 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859324+0000", 2024-09-15T03:46:56.096 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0513000000000001e-05 2024-09-15T03:46:56.096 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.096 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.096 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859418+0000", 2024-09-15T03:46:56.096 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3397e-05 2024-09-15T03:46:56.096 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.096 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.097 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861290+0000", 2024-09-15T03:46:56.097 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0018723920000000001 2024-09-15T03:46:56.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.097 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.097 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861353+0000", 2024-09-15T03:46:56.097 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2706000000000006e-05 2024-09-15T03:46:56.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.097 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.097 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861365+0000", 2024-09-15T03:46:56.097 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2316e-05 2024-09-15T03:46:56.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.098 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.098 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.867481+0000", 2024-09-15T03:46:56.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0061155419999999999 2024-09-15T03:46:56.098 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.098 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.098 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.098 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.098 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.098 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.9s4 49/31 ECSubWrite(tid=35, reqid=client.4521.0:331, at_version=49'15, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:46:56.098 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.860386+0000", 2024-09-15T03:46:56.098 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.123719334, 2024-09-15T03:46:56.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0070832780000000001, 2024-09-15T03:46:56.098 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.099 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.099 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.099 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860386+0000", 2024-09-15T03:46:56.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.099 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.099 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860386+0000", 2024-09-15T03:46:56.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.099 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.100 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860386+0000", 2024-09-15T03:46:56.100 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1199999999999999e-07 2024-09-15T03:46:56.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.100 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.100 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860431+0000", 2024-09-15T03:46:56.100 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5210999999999998e-05 2024-09-15T03:46:56.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.100 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.100 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860438+0000", 2024-09-15T03:46:56.100 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3659999999999997e-06 2024-09-15T03:46:56.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860441+0000", 2024-09-15T03:46:56.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4709999999999999e-06 2024-09-15T03:46:56.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.101 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860476+0000", 2024-09-15T03:46:56.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4699000000000001e-05 2024-09-15T03:46:56.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.101 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860485+0000", 2024-09-15T03:46:56.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4190000000000008e-06 2024-09-15T03:46:56.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860578+0000", 2024-09-15T03:46:56.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2390000000000001e-05 2024-09-15T03:46:56.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861291+0000", 2024-09-15T03:46:56.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00071288899999999997 2024-09-15T03:46:56.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861354+0000", 2024-09-15T03:46:56.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3004999999999997e-05 2024-09-15T03:46:56.103 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.103 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.103 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.103 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861400+0000", 2024-09-15T03:46:56.103 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6711000000000001e-05 2024-09-15T03:46:56.103 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.103 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.103 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.103 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.867469+0000", 2024-09-15T03:46:56.103 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006068805 2024-09-15T03:46:56.103 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.103 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.103 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.103 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.104 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.9s4 49/31 ECSubWrite(tid=36, reqid=client.4521.0:332, at_version=49'16, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:46:56.104 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.861449+0000", 2024-09-15T03:46:56.104 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.122656364, 2024-09-15T03:46:56.104 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0060935980000000004, 2024-09-15T03:46:56.104 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.104 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.104 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.104 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.104 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861449+0000", 2024-09-15T03:46:56.104 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.104 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861449+0000", 2024-09-15T03:46:56.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.105 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861449+0000", 2024-09-15T03:46:56.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9599999999999999e-07 2024-09-15T03:46:56.105 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861466+0000", 2024-09-15T03:46:56.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6908000000000001e-05 2024-09-15T03:46:56.105 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.106 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.106 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.106 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861478+0000", 2024-09-15T03:46:56.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2099000000000001e-05 2024-09-15T03:46:56.106 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.106 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.106 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.106 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861485+0000", 2024-09-15T03:46:56.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2779999999999999e-06 2024-09-15T03:46:56.106 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.106 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.106 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.106 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861531+0000", 2024-09-15T03:46:56.107 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6326000000000003e-05 2024-09-15T03:46:56.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861541+0000", 2024-09-15T03:46:56.107 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9539999999999999e-06 2024-09-15T03:46:56.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861626+0000", 2024-09-15T03:46:56.107 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4926000000000006e-05 2024-09-15T03:46:56.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.108 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861633+0000", 2024-09-15T03:46:56.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4220000000000003e-06 2024-09-15T03:46:56.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.108 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.108 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861690+0000", 2024-09-15T03:46:56.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7076e-05 2024-09-15T03:46:56.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.108 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.108 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861706+0000", 2024-09-15T03:46:56.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.556e-05 2024-09-15T03:46:56.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.109 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.109 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.109 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.867542+0000", 2024-09-15T03:46:56.109 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058364530000000001 2024-09-15T03:46:56.109 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.109 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.109 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.109 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.109 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.109 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:56.111 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:46:56.111 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:56.110+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:56.198 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T03:46:56.243 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:56.241+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:46:56.248 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:56.248 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:56.248 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:46:56.248 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:56.256 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:46:56.312 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:56.310+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:46:56.354 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:56.354 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:56.354 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:46:56.355 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:46:56.355 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:56.362 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:46:56.414 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:56.413+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:56.460 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:56.460 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:46:56.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:46:56.460 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [] 2024-09-15T03:46:56.460 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:56.469 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:46:56.554 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:56.553+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:56.606 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:56.607 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:56.607 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:46:56.607 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:56.615 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:46:56.616 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:56.614+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:56.630 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:46:56.715 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:56.714+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:46:56.748 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:46:56.748 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:46:56.752 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:56.753 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:46:56.753 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:46:56.753 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:46:56.753 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:56.761 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:46:56.817 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:56.816+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:56.833 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-15T03:46:56.834 INFO:tasks.rados.rados.0.smithi001.stdout:133: done (2 left) 2024-09-15T03:46:56.834 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 27 current snap is 12 2024-09-15T03:46:56.834 INFO:tasks.rados.rados.0.smithi001.stdout:134: seq_num 77 ranges {2473984=524288,2998272=524288,3522560=245760} 2024-09-15T03:46:56.850 INFO:tasks.rados.rados.0.smithi001.stdout:134: writing smithi00132172-27 from 2473984 to 2998272 tid 1 2024-09-15T03:46:56.853 INFO:tasks.rados.rados.0.smithi001.stdout:134: writing smithi00132172-27 from 2998272 to 3522560 tid 2 2024-09-15T03:46:56.854 INFO:tasks.rados.rados.0.smithi001.stdout:134: writing smithi00132172-27 from 3522560 to 3768320 tid 3 2024-09-15T03:46:56.855 INFO:tasks.rados.rados.0.smithi001.stdout:135: snap_create 2024-09-15T03:46:56.863 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:46:56.863 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:46:56.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:46:56.863 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:46:56.863 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.863 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:185 3.as0 3:5515acd6:::smithi00132172-13:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:56.863 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:46:56.863 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.756396338999998, 2024-09-15T03:46:56.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01268643, 2024-09-15T03:46:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 185 2024-09-15T03:46:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:46:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:46:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104710+0000", 2024-09-15T03:46:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0200000000000003e-07 2024-09-15T03:46:56.866 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.866 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.866 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.866 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104713+0000", 2024-09-15T03:46:56.866 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2339999999999999e-06 2024-09-15T03:46:56.866 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.866 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.866 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.866 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104718+0000", 2024-09-15T03:46:56.866 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3449999999999996e-06 2024-09-15T03:46:56.866 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.866 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.867 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.867 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104722+0000", 2024-09-15T03:46:56.867 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8399999999999997e-06 2024-09-15T03:46:56.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.867 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.867 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.867 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104849+0000", 2024-09-15T03:46:56.867 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012709599999999999 2024-09-15T03:46:56.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.867 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.867 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:56.868 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104856+0000", 2024-09-15T03:46:56.868 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.652e-06 2024-09-15T03:46:56.868 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.868 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.868 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.868 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106851+0000", 2024-09-15T03:46:56.868 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019954840000000001 2024-09-15T03:46:56.868 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.868 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.868 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:56.868 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106935+0000", 2024-09-15T03:46:56.869 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4095e-05 2024-09-15T03:46:56.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.869 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.869 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.869 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107453+0000", 2024-09-15T03:46:56.869 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00051729499999999997 2024-09-15T03:46:56.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.869 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.869 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.869 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107554+0000", 2024-09-15T03:46:56.869 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000101251 2024-09-15T03:46:56.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.870 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.870 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.870 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108104+0000", 2024-09-15T03:46:56.870 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00054995100000000002 2024-09-15T03:46:56.870 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.870 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.870 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.870 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108503+0000", 2024-09-15T03:46:56.870 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039953500000000002 2024-09-15T03:46:56.870 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.870 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.870 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.870 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109405+0000", 2024-09-15T03:46:56.870 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00090170099999999998 2024-09-15T03:46:56.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.871 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.871 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:56.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114365+0000", 2024-09-15T03:46:56.871 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049602120000000003 2024-09-15T03:46:56.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.871 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.872 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.872 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117396+0000", 2024-09-15T03:46:56.872 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030303370000000001 2024-09-15T03:46:56.872 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.872 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.872 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.872 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.872 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:186 3.as0 3:5515acd6:::smithi00132172-13:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:56.872 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:46:56.872 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.756356548999999, 2024-09-15T03:46:56.872 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012817485999999999, 2024-09-15T03:46:56.872 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.873 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:56.873 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:56.873 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:56.873 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:56.873 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 186 2024-09-15T03:46:56.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.873 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.873 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.873 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:46:56.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.873 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:46:56.874 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.874 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104750+0000", 2024-09-15T03:46:56.874 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1500000000000004e-07 2024-09-15T03:46:56.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.874 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104754+0000", 2024-09-15T03:46:56.874 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.946e-06 2024-09-15T03:46:56.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.875 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.875 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104769+0000", 2024-09-15T03:46:56.875 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.507e-05 2024-09-15T03:46:56.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.875 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.875 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104776+0000", 2024-09-15T03:46:56.875 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7849999999999992e-06 2024-09-15T03:46:56.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.875 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.875 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104888+0000", 2024-09-15T03:46:56.875 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011164500000000001 2024-09-15T03:46:56.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.876 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:56.876 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104894+0000", 2024-09-15T03:46:56.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.942e-06 2024-09-15T03:46:56.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.876 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.876 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108172+0000", 2024-09-15T03:46:56.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032781939999999999 2024-09-15T03:46:56.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.876 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:56.876 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108257+0000", 2024-09-15T03:46:56.877 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4488999999999997e-05 2024-09-15T03:46:56.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.877 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.877 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.877 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108732+0000", 2024-09-15T03:46:56.877 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047547399999999998 2024-09-15T03:46:56.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.877 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.877 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.877 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108823+0000", 2024-09-15T03:46:56.877 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1106000000000005e-05 2024-09-15T03:46:56.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.877 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.877 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108961+0000", 2024-09-15T03:46:56.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013735099999999999 2024-09-15T03:46:56.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.878 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109065+0000", 2024-09-15T03:46:56.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104715 2024-09-15T03:46:56.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.878 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109406+0000", 2024-09-15T03:46:56.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034062300000000002 2024-09-15T03:46:56.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.879 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.879 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:56.879 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114529+0000", 2024-09-15T03:46:56.879 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.005123219 2024-09-15T03:46:56.879 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.879 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.879 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.879 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117566+0000", 2024-09-15T03:46:56.879 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030372120000000001 2024-09-15T03:46:56.879 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.879 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.879 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.879 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.879 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.880 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:187 3.fs0 3:fe9c5513:::smithi00132172-47:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:56.880 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104782+0000", 2024-09-15T03:46:56.880 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.75632353, 2024-09-15T03:46:56.880 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069218609999999996, 2024-09-15T03:46:56.880 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.880 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:56.880 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:56.880 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:56.880 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:56.880 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 187 2024-09-15T03:46:56.880 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.880 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.880 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.880 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.881 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104782+0000", 2024-09-15T03:46:56.881 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.881 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.881 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.881 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.881 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104782+0000", 2024-09-15T03:46:56.881 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.881 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.881 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.881 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.881 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104782+0000", 2024-09-15T03:46:56.881 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8599999999999998e-07 2024-09-15T03:46:56.881 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.881 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.882 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.882 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104785+0000", 2024-09-15T03:46:56.882 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.649e-06 2024-09-15T03:46:56.882 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.882 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.882 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.882 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104800+0000", 2024-09-15T03:46:56.882 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5305e-05 2024-09-15T03:46:56.882 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.882 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.882 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.882 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104805+0000", 2024-09-15T03:46:56.882 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5569999999999997e-06 2024-09-15T03:46:56.882 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.883 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.883 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104854+0000", 2024-09-15T03:46:56.883 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9039999999999998e-05 2024-09-15T03:46:56.883 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.883 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-09-15T03:46:56.883 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104860+0000", 2024-09-15T03:46:56.883 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2979999999999997e-06 2024-09-15T03:46:56.883 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.883 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.883 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.105869+0000", 2024-09-15T03:46:56.883 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001009036 2024-09-15T03:46:56.884 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.884 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.884 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:56.884 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.105954+0000", 2024-09-15T03:46:56.884 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4357000000000006e-05 2024-09-15T03:46:56.884 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.884 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.884 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.884 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106626+0000", 2024-09-15T03:46:56.884 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00067191099999999997 2024-09-15T03:46:56.884 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.884 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.884 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.885 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106786+0000", 2024-09-15T03:46:56.885 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000160469 2024-09-15T03:46:56.885 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.885 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.885 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.885 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106788+0000", 2024-09-15T03:46:56.885 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9630000000000001e-06 2024-09-15T03:46:56.885 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.885 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.885 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.885 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106846+0000", 2024-09-15T03:46:56.885 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7719000000000001e-05 2024-09-15T03:46:56.885 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.885 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.886 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.886 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108372+0000", 2024-09-15T03:46:56.886 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0015264499999999999 2024-09-15T03:46:56.886 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.886 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.886 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:56.886 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.111475+0000", 2024-09-15T03:46:56.886 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031028459999999998 2024-09-15T03:46:56.886 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.886 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.886 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.886 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.111704+0000", 2024-09-15T03:46:56.886 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022877499999999999 2024-09-15T03:46:56.886 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.887 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.887 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.887 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.887 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.887 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:188 3.fs0 3:fe9c5513:::smithi00132172-47:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:56.887 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:46:56.887 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.756273852, 2024-09-15T03:46:56.887 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012637061, 2024-09-15T03:46:56.887 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.887 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:56.887 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:56.887 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:56.887 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:56.887 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 188 2024-09-15T03:46:56.888 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.888 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.888 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.888 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.888 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:46:56.888 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.888 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.888 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.888 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.888 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:46:56.888 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.888 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.888 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.888 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.889 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:46:56.889 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2799999999999996e-07 2024-09-15T03:46:56.889 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.889 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.889 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.889 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104835+0000", 2024-09-15T03:46:56.889 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7190000000000001e-06 2024-09-15T03:46:56.889 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.889 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.889 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.889 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104842+0000", 2024-09-15T03:46:56.889 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4580000000000004e-06 2024-09-15T03:46:56.889 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.889 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.890 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.890 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104846+0000", 2024-09-15T03:46:56.890 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1250000000000003e-06 2024-09-15T03:46:56.890 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.890 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.890 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.890 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104884+0000", 2024-09-15T03:46:56.890 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7075e-05 2024-09-15T03:46:56.890 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.890 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.890 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:56.890 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104888+0000", 2024-09-15T03:46:56.890 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8520000000000003e-06 2024-09-15T03:46:56.890 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.891 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106860+0000", 2024-09-15T03:46:56.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019719440000000002 2024-09-15T03:46:56.891 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:56.891 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106938+0000", 2024-09-15T03:46:56.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7421000000000003e-05 2024-09-15T03:46:56.891 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.891 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107368+0000", 2024-09-15T03:46:56.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043042999999999998 2024-09-15T03:46:56.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107460+0000", 2024-09-15T03:46:56.892 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1392999999999997e-05 2024-09-15T03:46:56.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107473+0000", 2024-09-15T03:46:56.892 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3702e-05 2024-09-15T03:46:56.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.893 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107504+0000", 2024-09-15T03:46:56.893 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1000000000000001e-05 2024-09-15T03:46:56.893 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.893 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.893 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.893 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108376+0000", 2024-09-15T03:46:56.893 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00087162000000000003 2024-09-15T03:46:56.893 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.893 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.893 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:56.893 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.111657+0000", 2024-09-15T03:46:56.893 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032815589999999999 2024-09-15T03:46:56.893 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.893 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.894 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.894 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117469+0000", 2024-09-15T03:46:56.894 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058112349999999997 2024-09-15T03:46:56.894 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.894 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.894 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.894 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.894 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.894 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:189 3.fs0 3:fe9c5513:::smithi00132172-47:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:56.894 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:46:56.894 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.756254666, 2024-09-15T03:46:56.894 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012731009999999999, 2024-09-15T03:46:56.894 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.894 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:56.895 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:56.895 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:56.895 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:56.895 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 189 2024-09-15T03:46:56.895 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.895 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.895 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.895 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.895 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:46:56.895 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.895 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.895 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.895 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.895 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:46:56.896 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.896 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.896 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.896 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.896 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:46:56.896 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7899999999999999e-07 2024-09-15T03:46:56.896 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.896 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.896 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.896 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104853+0000", 2024-09-15T03:46:56.896 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.933e-06 2024-09-15T03:46:56.896 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.896 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.897 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.897 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104859+0000", 2024-09-15T03:46:56.897 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.592e-06 2024-09-15T03:46:56.897 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.897 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.897 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.897 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104863+0000", 2024-09-15T03:46:56.897 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4000000000000002e-06 2024-09-15T03:46:56.897 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.897 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.897 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.897 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104910+0000", 2024-09-15T03:46:56.897 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6678000000000003e-05 2024-09-15T03:46:56.897 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.898 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.898 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:56.898 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104916+0000", 2024-09-15T03:46:56.898 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1079999999999998e-06 2024-09-15T03:46:56.898 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.898 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.898 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.898 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107550+0000", 2024-09-15T03:46:56.898 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026344110000000001 2024-09-15T03:46:56.898 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.899 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:56.899 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107624+0000", 2024-09-15T03:46:56.899 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4134000000000005e-05 2024-09-15T03:46:56.899 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.899 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.899 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107988+0000", 2024-09-15T03:46:56.899 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036353600000000002 2024-09-15T03:46:56.899 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.900 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.900 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.900 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108066+0000", 2024-09-15T03:46:56.900 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8047999999999995e-05 2024-09-15T03:46:56.900 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.900 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.900 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.900 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108105+0000", 2024-09-15T03:46:56.900 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8868000000000001e-05 2024-09-15T03:46:56.900 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.900 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.900 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.900 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108504+0000", 2024-09-15T03:46:56.900 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039916600000000001 2024-09-15T03:46:56.901 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.901 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.901 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.901 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108955+0000", 2024-09-15T03:46:56.901 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045126200000000002 2024-09-15T03:46:56.901 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.901 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.901 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:56.901 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114394+0000", 2024-09-15T03:46:56.901 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0054382750000000002 2024-09-15T03:46:56.901 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.901 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.901 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.902 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117582+0000", 2024-09-15T03:46:56.902 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031882199999999999 2024-09-15T03:46:56.902 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.902 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.902 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.902 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.902 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.902 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es3 34/28 ECSubWrite(tid=33, reqid=client.4521.0:195, at_version=34'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-15T03:46:56.902 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.118213+0000", 2024-09-15T03:46:56.902 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.742892441, 2024-09-15T03:46:56.902 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006662701, 2024-09-15T03:46:56.902 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.902 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.902 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.903 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.903 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.903 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118213+0000", 2024-09-15T03:46:56.903 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.903 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.903 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.903 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.903 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118213+0000", 2024-09-15T03:46:56.903 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.903 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.903 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.903 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.903 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118213+0000", 2024-09-15T03:46:56.903 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6300000000000001e-07 2024-09-15T03:46:56.904 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.904 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.904 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.904 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118447+0000", 2024-09-15T03:46:56.904 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000234016 2024-09-15T03:46:56.904 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.904 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.904 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.904 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118465+0000", 2024-09-15T03:46:56.904 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7978e-05 2024-09-15T03:46:56.904 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.904 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.904 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.905 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118475+0000", 2024-09-15T03:46:56.905 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3179999999999996e-06 2024-09-15T03:46:56.905 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.905 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.905 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.905 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118524+0000", 2024-09-15T03:46:56.905 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9069000000000001e-05 2024-09-15T03:46:56.905 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.905 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.905 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.905 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118534+0000", 2024-09-15T03:46:56.905 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0222e-05 2024-09-15T03:46:56.905 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.905 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.906 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.906 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118727+0000", 2024-09-15T03:46:56.906 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000193124 2024-09-15T03:46:56.906 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.906 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.906 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.906 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118734+0000", 2024-09-15T03:46:56.906 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2080000000000003e-06 2024-09-15T03:46:56.906 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.906 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.906 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.906 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118804+0000", 2024-09-15T03:46:56.906 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9720000000000003e-05 2024-09-15T03:46:56.906 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.907 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.907 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.907 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118820+0000", 2024-09-15T03:46:56.907 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5705999999999998e-05 2024-09-15T03:46:56.907 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.907 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.907 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.907 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124876+0000", 2024-09-15T03:46:56.907 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006056077 2024-09-15T03:46:56.907 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.907 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.907 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.907 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.907 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.908 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 34/31 ECSubWrite(tid=34, reqid=client.4521.0:197, at_version=34'21, trim_to=0'0, roll_forward_to=34'18))", 2024-09-15T03:46:56.908 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.123980+0000", 2024-09-15T03:46:56.908 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.737125612, 2024-09-15T03:46:56.908 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0066547489999999997, 2024-09-15T03:46:56.908 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.908 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.908 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.908 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.908 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.908 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123980+0000", 2024-09-15T03:46:56.908 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.908 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.908 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.909 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.909 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123980+0000", 2024-09-15T03:46:56.909 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.909 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.909 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.909 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.909 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123980+0000", 2024-09-15T03:46:56.909 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8800000000000003e-07 2024-09-15T03:46:56.909 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.909 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.909 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.909 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.123996+0000", 2024-09-15T03:46:56.909 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5838e-05 2024-09-15T03:46:56.909 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.910 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.910 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.910 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124005+0000", 2024-09-15T03:46:56.910 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1249999999999999e-06 2024-09-15T03:46:56.910 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.910 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.910 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.910 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124010+0000", 2024-09-15T03:46:56.910 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2640000000000002e-06 2024-09-15T03:46:56.910 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.910 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.910 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.910 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124045+0000", 2024-09-15T03:46:56.910 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5145000000000003e-05 2024-09-15T03:46:56.911 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.911 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.911 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.911 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124054+0000", 2024-09-15T03:46:56.911 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8389999999999996e-06 2024-09-15T03:46:56.911 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.911 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.911 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.911 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124696+0000", 2024-09-15T03:46:56.911 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00064247499999999999 2024-09-15T03:46:56.911 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.911 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.911 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.912 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124703+0000", 2024-09-15T03:46:56.912 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.905e-06 2024-09-15T03:46:56.912 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.912 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.912 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.912 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124736+0000", 2024-09-15T03:46:56.912 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2966999999999998e-05 2024-09-15T03:46:56.912 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.912 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.912 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.912 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.124758+0000", 2024-09-15T03:46:56.912 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2569999999999999e-05 2024-09-15T03:46:56.912 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.913 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.913 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.913 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.130635+0000", 2024-09-15T03:46:56.913 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058761330000000004 2024-09-15T03:46:56.913 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.913 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.913 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.913 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.913 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.913 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 38/31 ECSubWrite(tid=36, reqid=client.4521.0:220, at_version=38'23, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:46:56.913 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.148035+0000", 2024-09-15T03:46:56.913 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.71307056, 2024-09-15T03:46:56.913 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010074534, 2024-09-15T03:46:56.913 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.914 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.914 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.914 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.914 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.914 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148035+0000", 2024-09-15T03:46:56.914 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.914 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.914 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.914 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.914 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148035+0000", 2024-09-15T03:46:56.914 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.914 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.914 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.915 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.915 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148036+0000", 2024-09-15T03:46:56.915 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3939999999999999e-06 2024-09-15T03:46:56.915 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.915 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.915 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.915 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148681+0000", 2024-09-15T03:46:56.915 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00064493899999999997 2024-09-15T03:46:56.915 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.915 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.915 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.915 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148695+0000", 2024-09-15T03:46:56.915 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4078e-05 2024-09-15T03:46:56.915 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.916 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.916 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.916 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148701+0000", 2024-09-15T03:46:56.916 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7230000000000001e-06 2024-09-15T03:46:56.916 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.916 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.916 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.916 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148742+0000", 2024-09-15T03:46:56.916 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0781000000000001e-05 2024-09-15T03:46:56.916 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.916 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.916 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.916 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148749+0000", 2024-09-15T03:46:56.916 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6109999999999996e-06 2024-09-15T03:46:56.917 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.917 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.917 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.917 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149077+0000", 2024-09-15T03:46:56.917 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000327597 2024-09-15T03:46:56.917 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.917 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.917 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.917 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149087+0000", 2024-09-15T03:46:56.917 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4739999999999994e-06 2024-09-15T03:46:56.917 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.917 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.917 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.918 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149230+0000", 2024-09-15T03:46:56.918 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014398300000000001 2024-09-15T03:46:56.918 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.918 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.918 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.918 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149247+0000", 2024-09-15T03:46:56.918 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6201e-05 2024-09-15T03:46:56.918 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.918 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.918 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.918 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158109+0000", 2024-09-15T03:46:56.918 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0088627529999999993 2024-09-15T03:46:56.918 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.918 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.918 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.919 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.919 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.919 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 38/31 ECSubWrite(tid=37, reqid=client.4521.0:220, at_version=0'0, trim_to=0'0, roll_forward_to=38'23))", 2024-09-15T03:46:56.919 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.151308+0000", 2024-09-15T03:46:56.919 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.709797314999999, 2024-09-15T03:46:56.919 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069978310000000004, 2024-09-15T03:46:56.919 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.919 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.919 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.919 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.919 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.919 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151308+0000", 2024-09-15T03:46:56.919 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.920 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.920 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.920 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.920 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151308+0000", 2024-09-15T03:46:56.920 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.920 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.920 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.920 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.920 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151309+0000", 2024-09-15T03:46:56.920 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2200000000000003e-07 2024-09-15T03:46:56.920 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.920 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.920 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.920 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151318+0000", 2024-09-15T03:46:56.921 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7250000000000003e-06 2024-09-15T03:46:56.921 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.921 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.921 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.921 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151330+0000", 2024-09-15T03:46:56.921 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2425e-05 2024-09-15T03:46:56.921 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.921 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.921 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.921 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151337+0000", 2024-09-15T03:46:56.921 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6139999999999997e-06 2024-09-15T03:46:56.921 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.921 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.922 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.922 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151372+0000", 2024-09-15T03:46:56.922 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5683999999999999e-05 2024-09-15T03:46:56.922 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.922 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.922 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.922 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151378+0000", 2024-09-15T03:46:56.922 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7880000000000003e-06 2024-09-15T03:46:56.922 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.922 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.922 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.922 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151438+0000", 2024-09-15T03:46:56.922 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9679999999999998e-05 2024-09-15T03:46:56.922 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.923 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.923 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.923 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151446+0000", 2024-09-15T03:46:56.923 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.909e-06 2024-09-15T03:46:56.923 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.923 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.923 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.923 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151481+0000", 2024-09-15T03:46:56.923 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5005000000000001e-05 2024-09-15T03:46:56.923 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.923 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.923 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.923 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151492+0000", 2024-09-15T03:46:56.923 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1133e-05 2024-09-15T03:46:56.924 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.924 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.924 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.924 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158306+0000", 2024-09-15T03:46:56.924 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0068140459999999998 2024-09-15T03:46:56.924 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.924 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.924 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.924 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.924 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.924 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:231 3.fs0 3:f40d4749:::smithi00132172-35:head [copy-from ver 6] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:46:56.924 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.155669+0000", 2024-09-15T03:46:56.924 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.705436860000001, 2024-09-15T03:46:56.925 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010370569, 2024-09-15T03:46:56.925 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.925 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:56.925 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:56.925 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:56.925 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:56.925 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 231 2024-09-15T03:46:56.925 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.925 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.925 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.925 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.925 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155669+0000", 2024-09-15T03:46:56.925 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.926 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.926 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155669+0000", 2024-09-15T03:46:56.926 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.926 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.926 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155671+0000", 2024-09-15T03:46:56.926 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0829999999999998e-06 2024-09-15T03:46:56.926 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.926 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155676+0000", 2024-09-15T03:46:56.926 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6210000000000001e-06 2024-09-15T03:46:56.927 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.927 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.927 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.927 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155690+0000", 2024-09-15T03:46:56.927 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4166e-05 2024-09-15T03:46:56.927 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.927 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.927 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.927 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155698+0000", 2024-09-15T03:46:56.927 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5560000000000002e-06 2024-09-15T03:46:56.927 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.927 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.927 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.928 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155748+0000", 2024-09-15T03:46:56.928 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9851000000000001e-05 2024-09-15T03:46:56.928 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.928 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.928 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:56.928 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155872+0000", 2024-09-15T03:46:56.928 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000124091 2024-09-15T03:46:56.928 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.928 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.928 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.928 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163470+0000", 2024-09-15T03:46:56.928 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075976730000000001 2024-09-15T03:46:56.928 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.928 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.929 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.929 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163801+0000", 2024-09-15T03:46:56.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033157999999999999 2024-09-15T03:46:56.929 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.929 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.929 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.929 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163814+0000", 2024-09-15T03:46:56.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2659e-05 2024-09-15T03:46:56.929 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.929 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.929 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.929 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163935+0000", 2024-09-15T03:46:56.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000121155 2024-09-15T03:46:56.930 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.930 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.930 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.930 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.164131+0000", 2024-09-15T03:46:56.930 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019600499999999999 2024-09-15T03:46:56.930 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.930 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.930 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:46:56.930 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.165998+0000", 2024-09-15T03:46:56.930 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001866882 2024-09-15T03:46:56.930 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.930 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.930 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.930 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.166039+0000", 2024-09-15T03:46:56.931 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1247e-05 2024-09-15T03:46:56.931 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.931 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.931 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.931 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.931 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.931 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:232 3.fs0 3:f40d4749:::smithi00132172-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:46:56.931 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:46:56.931 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.705399570999999, 2024-09-15T03:46:56.931 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084205730000000006, 2024-09-15T03:46:56.931 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.931 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:46:56.931 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:46:56.931 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:46:56.932 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:56.932 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 232 2024-09-15T03:46:56.932 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.932 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.932 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.932 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.932 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:46:56.932 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.932 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.932 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.932 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.932 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:46:56.932 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.932 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.933 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.933 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.933 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:46:56.933 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7399999999999999e-07 2024-09-15T03:46:56.933 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.933 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.933 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.933 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155708+0000", 2024-09-15T03:46:56.933 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6500000000000001e-06 2024-09-15T03:46:56.933 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.933 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.933 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.933 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155713+0000", 2024-09-15T03:46:56.933 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2519999999999999e-06 2024-09-15T03:46:56.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.934 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.934 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155716+0000", 2024-09-15T03:46:56.934 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4430000000000002e-06 2024-09-15T03:46:56.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.934 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.934 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156060+0000", 2024-09-15T03:46:56.934 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034440000000000002 2024-09-15T03:46:56.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.934 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2024-09-15T03:46:56.934 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156112+0000", 2024-09-15T03:46:56.935 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1928000000000002e-05 2024-09-15T03:46:56.935 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.935 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.935 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.935 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163929+0000", 2024-09-15T03:46:56.935 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0078168999999999999 2024-09-15T03:46:56.935 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.935 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.935 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:46:56.935 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.164027+0000", 2024-09-15T03:46:56.935 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8109999999999994e-05 2024-09-15T03:46:56.935 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.935 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.935 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.936 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.164126+0000", 2024-09-15T03:46:56.936 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9516000000000001e-05 2024-09-15T03:46:56.936 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.936 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.936 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.936 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.936 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.936 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 39/31 ECSubWrite(tid=43, reqid=client.4521.0:252, at_version=39'27, trim_to=0'0, roll_forward_to=39'25))", 2024-09-15T03:46:56.936 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:46:56.936 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.682759878000001, 2024-09-15T03:46:56.936 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082900309999999998, 2024-09-15T03:46:56.936 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.936 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.937 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.937 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.937 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.937 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:46:56.937 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.937 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.937 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.937 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.937 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:46:56.937 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.937 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.937 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.937 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.938 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:46:56.938 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6199999999999998e-07 2024-09-15T03:46:56.938 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.938 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.938 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178642+0000", 2024-09-15T03:46:56.938 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029559399999999999 2024-09-15T03:46:56.938 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.938 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.938 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178653+0000", 2024-09-15T03:46:56.938 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0865e-05 2024-09-15T03:46:56.938 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.939 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.939 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178658+0000", 2024-09-15T03:46:56.939 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3720000000000004e-06 2024-09-15T03:46:56.939 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.939 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.939 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.939 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178695+0000", 2024-09-15T03:46:56.939 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6708999999999997e-05 2024-09-15T03:46:56.939 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.939 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.939 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.939 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178701+0000", 2024-09-15T03:46:56.939 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7989999999999996e-06 2024-09-15T03:46:56.939 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.939 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.940 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.940 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178926+0000", 2024-09-15T03:46:56.940 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022405299999999999 2024-09-15T03:46:56.940 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.940 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.940 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.940 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178935+0000", 2024-09-15T03:46:56.940 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9649999999999997e-06 2024-09-15T03:46:56.940 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.940 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.940 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.940 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179230+0000", 2024-09-15T03:46:56.940 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029551000000000002 2024-09-15T03:46:56.941 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.941 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.941 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.941 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179247+0000", 2024-09-15T03:46:56.941 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6685999999999999e-05 2024-09-15T03:46:56.941 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.941 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.941 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.941 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.186636+0000", 2024-09-15T03:46:56.941 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0073889159999999997 2024-09-15T03:46:56.941 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.941 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.941 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.942 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.942 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.942 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 39/31 ECSubWrite(tid=44, reqid=client.4521.0:252, at_version=0'0, trim_to=0'0, roll_forward_to=39'27))", 2024-09-15T03:46:56.942 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.180059+0000", 2024-09-15T03:46:56.942 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.681046615, 2024-09-15T03:46:56.942 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00676445, 2024-09-15T03:46:56.942 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.942 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.942 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.942 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.942 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.942 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180059+0000", 2024-09-15T03:46:56.942 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.942 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.943 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.943 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.943 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180059+0000", 2024-09-15T03:46:56.943 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.943 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.943 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.943 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.943 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180059+0000", 2024-09-15T03:46:56.943 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:46:56.943 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.943 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.943 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.943 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180070+0000", 2024-09-15T03:46:56.943 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0407e-05 2024-09-15T03:46:56.944 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.944 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.944 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.944 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180083+0000", 2024-09-15T03:46:56.944 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3341e-05 2024-09-15T03:46:56.944 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.944 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.944 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.944 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180089+0000", 2024-09-15T03:46:56.944 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8000000000000004e-06 2024-09-15T03:46:56.944 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.944 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.944 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.944 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180126+0000", 2024-09-15T03:46:56.945 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7151999999999999e-05 2024-09-15T03:46:56.945 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.945 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.945 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.945 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180136+0000", 2024-09-15T03:46:56.945 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8349999999999999e-06 2024-09-15T03:46:56.945 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.945 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.945 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.945 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180200+0000", 2024-09-15T03:46:56.945 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3612e-05 2024-09-15T03:46:56.945 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.945 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.946 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.946 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180206+0000", 2024-09-15T03:46:56.946 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3749999999999999e-06 2024-09-15T03:46:56.946 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.946 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.946 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.946 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180235+0000", 2024-09-15T03:46:56.946 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9068999999999999e-05 2024-09-15T03:46:56.946 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.946 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.946 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.946 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180244+0000", 2024-09-15T03:46:56.946 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1030000000000008e-06 2024-09-15T03:46:56.946 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.947 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.947 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.947 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.186823+0000", 2024-09-15T03:46:56.947 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0065792130000000004 2024-09-15T03:46:56.947 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.947 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.947 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.947 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.947 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.947 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 49/31 ECSubWrite(tid=86, reqid=client.4521.0:334, at_version=49'20, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:46:56.947 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.857479+0000", 2024-09-15T03:46:56.947 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.0036261719999999, 2024-09-15T03:46:56.947 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069722819999999998, 2024-09-15T03:46:56.947 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.948 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.948 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.948 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.948 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.948 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857479+0000", 2024-09-15T03:46:56.948 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.948 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.948 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.948 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.948 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857479+0000", 2024-09-15T03:46:56.948 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.948 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.948 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.948 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.949 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857480+0000", 2024-09-15T03:46:56.949 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.85e-07 2024-09-15T03:46:56.949 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.949 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.949 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.949 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857500+0000", 2024-09-15T03:46:56.949 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0006000000000001e-05 2024-09-15T03:46:56.949 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.949 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.949 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.949 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857515+0000", 2024-09-15T03:46:56.949 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5255999999999999e-05 2024-09-15T03:46:56.949 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.950 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.950 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.950 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857521+0000", 2024-09-15T03:46:56.950 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0430000000000004e-06 2024-09-15T03:46:56.950 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.950 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.950 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.950 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857573+0000", 2024-09-15T03:46:56.950 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1992e-05 2024-09-15T03:46:56.950 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.950 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.950 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.950 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857582+0000", 2024-09-15T03:46:56.950 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0070000000000003e-06 2024-09-15T03:46:56.951 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.951 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.951 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.951 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857854+0000", 2024-09-15T03:46:56.951 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027109999999999998 2024-09-15T03:46:56.951 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.951 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.951 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.951 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857867+0000", 2024-09-15T03:46:56.951 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3552e-05 2024-09-15T03:46:56.951 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.951 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.951 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.951 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857910+0000", 2024-09-15T03:46:56.952 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3343000000000002e-05 2024-09-15T03:46:56.952 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.952 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.952 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.952 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857930+0000", 2024-09-15T03:46:56.952 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9267e-05 2024-09-15T03:46:56.952 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.952 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.952 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.952 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.864452+0000", 2024-09-15T03:46:56.952 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0065218309999999996 2024-09-15T03:46:56.952 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.952 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.953 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.953 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.953 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.953 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.cs2 49/28 ECSubWrite(tid=15, reqid=client.4521.0:337, at_version=49'10, trim_to=0'0, roll_forward_to=43'8))", 2024-09-15T03:46:56.953 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.858095+0000", 2024-09-15T03:46:56.953 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.003010175, 2024-09-15T03:46:56.953 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006658611, 2024-09-15T03:46:56.953 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.953 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.953 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.953 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.953 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.953 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858095+0000", 2024-09-15T03:46:56.953 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.954 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.954 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.954 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.954 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858095+0000", 2024-09-15T03:46:56.954 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.954 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.954 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.954 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.954 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858096+0000", 2024-09-15T03:46:56.954 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5499999999999999e-07 2024-09-15T03:46:56.954 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.954 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.954 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.954 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858118+0000", 2024-09-15T03:46:56.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2921999999999998e-05 2024-09-15T03:46:56.955 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.955 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.955 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.955 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858126+0000", 2024-09-15T03:46:56.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7670000000000003e-06 2024-09-15T03:46:56.955 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.955 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.955 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.955 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858131+0000", 2024-09-15T03:46:56.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3950000000000003e-06 2024-09-15T03:46:56.955 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.955 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.955 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.956 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858186+0000", 2024-09-15T03:46:56.956 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5782999999999998e-05 2024-09-15T03:46:56.956 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.956 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.956 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.956 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858197+0000", 2024-09-15T03:46:56.956 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0431999999999999e-05 2024-09-15T03:46:56.956 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.956 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.956 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.956 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858458+0000", 2024-09-15T03:46:56.956 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00026135299999999998 2024-09-15T03:46:56.956 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.957 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.957 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.957 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858467+0000", 2024-09-15T03:46:56.957 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7280000000000001e-06 2024-09-15T03:46:56.957 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.957 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.957 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.957 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858497+0000", 2024-09-15T03:46:56.957 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0090000000000002e-05 2024-09-15T03:46:56.957 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.957 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.957 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.958 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858509+0000", 2024-09-15T03:46:56.958 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1983e-05 2024-09-15T03:46:56.958 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.958 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.958 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.958 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.864754+0000", 2024-09-15T03:46:56.958 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0062449030000000004 2024-09-15T03:46:56.958 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.958 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.958 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.958 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.958 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.958 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 49/31 ECSubWrite(tid=87, reqid=client.4521.0:335, at_version=49'21, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:46:56.959 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.858534+0000", 2024-09-15T03:46:56.959 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.002571742, 2024-09-15T03:46:56.959 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094628430000000003, 2024-09-15T03:46:56.959 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.959 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.959 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.959 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.959 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.959 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858534+0000", 2024-09-15T03:46:56.959 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.959 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.959 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.959 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.959 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858534+0000", 2024-09-15T03:46:56.960 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.960 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.960 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.960 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.960 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858534+0000", 2024-09-15T03:46:56.960 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2300000000000001e-07 2024-09-15T03:46:56.960 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.960 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.960 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.960 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858553+0000", 2024-09-15T03:46:56.960 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8221000000000001e-05 2024-09-15T03:46:56.960 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.960 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.961 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858564+0000", 2024-09-15T03:46:56.961 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1471000000000001e-05 2024-09-15T03:46:56.961 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.961 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.961 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858570+0000", 2024-09-15T03:46:56.961 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7640000000000002e-06 2024-09-15T03:46:56.961 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.961 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.961 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858621+0000", 2024-09-15T03:46:56.961 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0943999999999999e-05 2024-09-15T03:46:56.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858626+0000", 2024-09-15T03:46:56.962 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4949999999999999e-06 2024-09-15T03:46:56.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858715+0000", 2024-09-15T03:46:56.962 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9061000000000007e-05 2024-09-15T03:46:56.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858722+0000", 2024-09-15T03:46:56.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9689999999999997e-06 2024-09-15T03:46:56.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858748+0000", 2024-09-15T03:46:56.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6179000000000001e-05 2024-09-15T03:46:56.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858761+0000", 2024-09-15T03:46:56.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2159e-05 2024-09-15T03:46:56.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.867997+0000", 2024-09-15T03:46:56.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092359569999999995 2024-09-15T03:46:56.964 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.964 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.964 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.964 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.cs2 49/28 ECSubWrite(tid=16, reqid=client.4521.0:337, at_version=0'0, trim_to=0'0, roll_forward_to=49'10))", 2024-09-15T03:46:56.964 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.859201+0000", 2024-09-15T03:46:56.964 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.001904028, 2024-09-15T03:46:56.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.007966585, 2024-09-15T03:46:56.964 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.964 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.964 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.965 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.965 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859201+0000", 2024-09-15T03:46:56.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.965 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.965 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.965 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859201+0000", 2024-09-15T03:46:56.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.965 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.965 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.965 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859202+0000", 2024-09-15T03:46:56.966 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6899999999999997e-07 2024-09-15T03:46:56.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.966 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.966 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.966 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859211+0000", 2024-09-15T03:46:56.966 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0799999999999995e-06 2024-09-15T03:46:56.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.966 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.966 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.966 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859224+0000", 2024-09-15T03:46:56.966 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2510999999999999e-05 2024-09-15T03:46:56.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.966 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.966 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.967 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859230+0000", 2024-09-15T03:46:56.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1569999999999997e-06 2024-09-15T03:46:56.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.967 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.967 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859282+0000", 2024-09-15T03:46:56.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2126000000000002e-05 2024-09-15T03:46:56.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.967 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.967 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859291+0000", 2024-09-15T03:46:56.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3000000000000007e-06 2024-09-15T03:46:56.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.968 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859375+0000", 2024-09-15T03:46:56.968 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3321999999999999e-05 2024-09-15T03:46:56.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.968 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859383+0000", 2024-09-15T03:46:56.968 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0430000000000008e-06 2024-09-15T03:46:56.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.968 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859413+0000", 2024-09-15T03:46:56.968 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0510000000000001e-05 2024-09-15T03:46:56.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.969 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.969 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.969 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859423+0000", 2024-09-15T03:46:56.969 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0365e-05 2024-09-15T03:46:56.969 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.969 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.969 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.969 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.867168+0000", 2024-09-15T03:46:56.969 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0077446019999999997 2024-09-15T03:46:56.969 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.969 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.969 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.969 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.969 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.970 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 49/31 ECSubWrite(tid=88, reqid=client.4521.0:336, at_version=49'22, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:46:56.970 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:46:56.970 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.0018441, 2024-09-15T03:46:56.970 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0088255839999999992, 2024-09-15T03:46:56.970 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.970 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.970 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.970 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.970 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.970 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:46:56.970 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.970 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.970 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.971 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.971 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:46:56.971 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.971 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.971 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.971 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.971 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859262+0000", 2024-09-15T03:46:56.971 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2399999999999999e-07 2024-09-15T03:46:56.971 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.971 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.971 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.971 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859275+0000", 2024-09-15T03:46:56.971 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3395e-05 2024-09-15T03:46:56.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.972 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859285+0000", 2024-09-15T03:46:56.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8959999999999998e-06 2024-09-15T03:46:56.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.972 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859289+0000", 2024-09-15T03:46:56.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2549999999999999e-06 2024-09-15T03:46:56.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.973 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.973 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.973 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859327+0000", 2024-09-15T03:46:56.973 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7753000000000003e-05 2024-09-15T03:46:56.973 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.973 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.973 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.973 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859335+0000", 2024-09-15T03:46:56.973 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3000000000000002e-06 2024-09-15T03:46:56.973 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.973 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.973 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.973 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859391+0000", 2024-09-15T03:46:56.973 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6085000000000003e-05 2024-09-15T03:46:56.974 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.974 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.974 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.974 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859415+0000", 2024-09-15T03:46:56.974 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3295999999999999e-05 2024-09-15T03:46:56.974 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.974 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.974 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.974 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859432+0000", 2024-09-15T03:46:56.974 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7087000000000001e-05 2024-09-15T03:46:56.974 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.974 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.974 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.974 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859442+0000", 2024-09-15T03:46:56.975 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0339e-05 2024-09-15T03:46:56.975 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.975 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.975 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.868087+0000", 2024-09-15T03:46:56.975 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0086447539999999993 2024-09-15T03:46:56.975 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.975 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.975 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.975 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.975 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.9s7 49/31 ECSubWrite(tid=35, reqid=client.4521.0:331, at_version=49'15, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:46:56.975 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.860608+0000", 2024-09-15T03:46:56.975 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.000497931, 2024-09-15T03:46:56.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0074920869999999997, 2024-09-15T03:46:56.976 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.976 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.976 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860608+0000", 2024-09-15T03:46:56.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860608+0000", 2024-09-15T03:46:56.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.977 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.977 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.977 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.977 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860608+0000", 2024-09-15T03:46:56.977 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9299999999999998e-07 2024-09-15T03:46:56.977 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.977 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.977 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.977 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860651+0000", 2024-09-15T03:46:56.977 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2463000000000003e-05 2024-09-15T03:46:56.977 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.977 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.977 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.977 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860664+0000", 2024-09-15T03:46:56.978 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2948999999999999e-05 2024-09-15T03:46:56.978 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.978 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860670+0000", 2024-09-15T03:46:56.978 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8689999999999998e-06 2024-09-15T03:46:56.978 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.978 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860709+0000", 2024-09-15T03:46:56.978 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8339e-05 2024-09-15T03:46:56.978 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860714+0000", 2024-09-15T03:46:56.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4029999999999998e-06 2024-09-15T03:46:56.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860775+0000", 2024-09-15T03:46:56.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0988e-05 2024-09-15T03:46:56.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860781+0000", 2024-09-15T03:46:56.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9660000000000001e-06 2024-09-15T03:46:56.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860836+0000", 2024-09-15T03:46:56.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4409e-05 2024-09-15T03:46:56.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860847+0000", 2024-09-15T03:46:56.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1061e-05 2024-09-15T03:46:56.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.868100+0000", 2024-09-15T03:46:56.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0072530470000000003 2024-09-15T03:46:56.981 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.981 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.981 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.981 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.981 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.981 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.9s7 49/31 ECSubWrite(tid=36, reqid=client.4521.0:332, at_version=49'16, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:46:56.981 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.861724+0000", 2024-09-15T03:46:56.981 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.99938151600000003, 2024-09-15T03:46:56.981 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064560060000000002, 2024-09-15T03:46:56.981 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:46:56.981 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:46:56.981 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:46:56.981 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.981 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:46:56.982 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861724+0000", 2024-09-15T03:46:56.982 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.982 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:46:56.982 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861724+0000", 2024-09-15T03:46:56.982 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:46:56.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.982 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:46:56.982 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861725+0000", 2024-09-15T03:46:56.982 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5400000000000001e-07 2024-09-15T03:46:56.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:46:56.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861739+0000", 2024-09-15T03:46:56.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4616e-05 2024-09-15T03:46:56.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:46:56.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861754+0000", 2024-09-15T03:46:56.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4354e-05 2024-09-15T03:46:56.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:46:56.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861763+0000", 2024-09-15T03:46:56.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4560000000000006e-06 2024-09-15T03:46:56.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.984 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:46:56.984 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861807+0000", 2024-09-15T03:46:56.984 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4186e-05 2024-09-15T03:46:56.984 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.984 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:46:56.984 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861816+0000", 2024-09-15T03:46:56.984 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5040000000000002e-06 2024-09-15T03:46:56.984 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.984 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:56.984 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861911+0000", 2024-09-15T03:46:56.984 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5095999999999999e-05 2024-09-15T03:46:56.985 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.985 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.985 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:56.985 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861918+0000", 2024-09-15T03:46:56.985 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5519999999999999e-06 2024-09-15T03:46:56.985 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.985 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.985 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:56.985 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.864012+0000", 2024-09-15T03:46:56.985 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002094055 2024-09-15T03:46:56.985 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.985 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.985 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:46:56.985 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.864024+0000", 2024-09-15T03:46:56.986 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1894e-05 2024-09-15T03:46:56.986 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:46:56.986 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:46:56.986 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:46:56.986 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.868180+0000", 2024-09-15T03:46:56.986 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.004155639 2024-09-15T03:46:56.986 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.986 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.986 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.986 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:46:56.986 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:46:56.986 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:46:56.988 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:46:56.988 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:56.986+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:46:57.089 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:57.088+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:57.090 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:57.090 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:57.098 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:46:57.189 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:57.189+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:46:57.199 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:57.199 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:57.208 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:46:57.291 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:57.289+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:46:57.308 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:57.308 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:57.317 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:46:57.401 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:57.399+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:57.460 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:46:57.460 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-15T03:46:57.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.460 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=80, reqid=client.4521.0:320, at_version=49'15, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:46:57.460 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.840885+0000", 2024-09-15T03:46:57.461 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.6169139210000001, 2024-09-15T03:46:57.461 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6169371529999998, 2024-09-15T03:46:57.461 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.461 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.461 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.461 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.461 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.840885+0000", 2024-09-15T03:46:57.461 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.840885+0000", 2024-09-15T03:46:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.840885+0000", 2024-09-15T03:46:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1100000000000002e-07 2024-09-15T03:46:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841181+0000", 2024-09-15T03:46:57.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029618199999999998 2024-09-15T03:46:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841195+0000", 2024-09-15T03:46:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3424000000000001e-05 2024-09-15T03:46:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841204+0000", 2024-09-15T03:46:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0170000000000002e-06 2024-09-15T03:46:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841274+0000", 2024-09-15T03:46:57.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0561000000000004e-05 2024-09-15T03:46:57.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841289+0000", 2024-09-15T03:46:57.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4127e-05 2024-09-15T03:46:57.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841717+0000", 2024-09-15T03:46:57.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00042811800000000001 2024-09-15T03:46:57.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.465 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.465 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841745+0000", 2024-09-15T03:46:57.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8297e-05 2024-09-15T03:46:57.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841852+0000", 2024-09-15T03:46:57.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010660199999999999 2024-09-15T03:46:57.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.842419+0000", 2024-09-15T03:46:57.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00056784499999999998 2024-09-15T03:46:57.466 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.466 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.467 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.467 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=30, reqid=client.4521.0:325, at_version=49'12, trim_to=0'0, roll_forward_to=40'10))", 2024-09-15T03:46:57.467 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.844912+0000", 2024-09-15T03:46:57.467 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.612887339, 2024-09-15T03:46:57.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6129666729999999, 2024-09-15T03:46:57.467 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.467 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.468 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.468 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.468 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844912+0000", 2024-09-15T03:46:57.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.468 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844912+0000", 2024-09-15T03:46:57.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844912+0000", 2024-09-15T03:46:57.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.84e-07 2024-09-15T03:46:57.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844937+0000", 2024-09-15T03:46:57.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4729e-05 2024-09-15T03:46:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844957+0000", 2024-09-15T03:46:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0285999999999999e-05 2024-09-15T03:46:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844965+0000", 2024-09-15T03:46:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5709999999999998e-06 2024-09-15T03:46:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845086+0000", 2024-09-15T03:46:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000120856 2024-09-15T03:46:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845121+0000", 2024-09-15T03:46:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4292999999999998e-05 2024-09-15T03:46:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845492+0000", 2024-09-15T03:46:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00037167400000000001 2024-09-15T03:46:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845529+0000", 2024-09-15T03:46:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7092000000000003e-05 2024-09-15T03:46:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845591+0000", 2024-09-15T03:46:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1824999999999998e-05 2024-09-15T03:46:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845612+0000", 2024-09-15T03:46:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.084e-05 2024-09-15T03:46:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=81, reqid=client.4521.0:321, at_version=49'16, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:46:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.845113+0000", 2024-09-15T03:46:57.473 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.612685932, 2024-09-15T03:46:57.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6128101539999999, 2024-09-15T03:46:57.474 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.474 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.474 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845113+0000", 2024-09-15T03:46:57.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845113+0000", 2024-09-15T03:46:57.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845114+0000", 2024-09-15T03:46:57.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0210000000000001e-06 2024-09-15T03:46:57.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845422+0000", 2024-09-15T03:46:57.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00030762999999999998 2024-09-15T03:46:57.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845436+0000", 2024-09-15T03:46:57.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3998999999999999e-05 2024-09-15T03:46:57.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845445+0000", 2024-09-15T03:46:57.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5799999999999998e-06 2024-09-15T03:46:57.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845538+0000", 2024-09-15T03:46:57.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2739999999999996e-05 2024-09-15T03:46:57.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845548+0000", 2024-09-15T03:46:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0548e-05 2024-09-15T03:46:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845803+0000", 2024-09-15T03:46:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00025495599999999998 2024-09-15T03:46:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845998+0000", 2024-09-15T03:46:57.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019432799999999999 2024-09-15T03:46:57.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846122+0000", 2024-09-15T03:46:57.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012434600000000001 2024-09-15T03:46:57.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846178+0000", 2024-09-15T03:46:57.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5943999999999998e-05 2024-09-15T03:46:57.479 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.479 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.479 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.480 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=82, reqid=client.4521.0:322, at_version=49'17, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:46:57.480 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.846151+0000", 2024-09-15T03:46:57.480 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.6116479099999999, 2024-09-15T03:46:57.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.611816715, 2024-09-15T03:46:57.480 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.480 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.481 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.481 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846151+0000", 2024-09-15T03:46:57.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846151+0000", 2024-09-15T03:46:57.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846152+0000", 2024-09-15T03:46:57.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0800000000000004e-07 2024-09-15T03:46:57.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846238+0000", 2024-09-15T03:46:57.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5866999999999996e-05 2024-09-15T03:46:57.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846255+0000", 2024-09-15T03:46:57.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6954999999999999e-05 2024-09-15T03:46:57.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846262+0000", 2024-09-15T03:46:57.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7680000000000008e-06 2024-09-15T03:46:57.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846325+0000", 2024-09-15T03:46:57.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2191000000000001e-05 2024-09-15T03:46:57.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846335+0000", 2024-09-15T03:46:57.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0767e-05 2024-09-15T03:46:57.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846446+0000", 2024-09-15T03:46:57.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110522 2024-09-15T03:46:57.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846458+0000", 2024-09-15T03:46:57.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2023e-05 2024-09-15T03:46:57.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846508+0000", 2024-09-15T03:46:57.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0479000000000002e-05 2024-09-15T03:46:57.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846526+0000", 2024-09-15T03:46:57.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7836e-05 2024-09-15T03:46:57.491 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.491 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.491 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.492 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=31, reqid=client.4521.0:325, at_version=0'0, trim_to=0'0, roll_forward_to=49'12))", 2024-09-15T03:46:57.492 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.847407+0000", 2024-09-15T03:46:57.492 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.610391707, 2024-09-15T03:46:57.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6106043059999999, 2024-09-15T03:46:57.492 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.493 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.493 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.493 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.493 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847407+0000", 2024-09-15T03:46:57.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847407+0000", 2024-09-15T03:46:57.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847408+0000", 2024-09-15T03:46:57.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0070000000000001e-06 2024-09-15T03:46:57.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847421+0000", 2024-09-15T03:46:57.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3203999999999999e-05 2024-09-15T03:46:57.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847443+0000", 2024-09-15T03:46:57.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1539000000000002e-05 2024-09-15T03:46:57.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847452+0000", 2024-09-15T03:46:57.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6570000000000003e-06 2024-09-15T03:46:57.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847534+0000", 2024-09-15T03:46:57.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2309000000000001e-05 2024-09-15T03:46:57.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847544+0000", 2024-09-15T03:46:57.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0482000000000001e-05 2024-09-15T03:46:57.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847645+0000", 2024-09-15T03:46:57.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010021700000000001 2024-09-15T03:46:57.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847665+0000", 2024-09-15T03:46:57.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0271999999999998e-05 2024-09-15T03:46:57.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847708+0000", 2024-09-15T03:46:57.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2828999999999999e-05 2024-09-15T03:46:57.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847724+0000", 2024-09-15T03:46:57.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6334e-05 2024-09-15T03:46:57.500 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=83, reqid=client.4521.0:323, at_version=49'18, trim_to=0'0, roll_forward_to=49'15))", 2024-09-15T03:46:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.847691+0000", 2024-09-15T03:46:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.6101078599999998, 2024-09-15T03:46:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.610363639, 2024-09-15T03:46:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847691+0000", 2024-09-15T03:46:57.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847691+0000", 2024-09-15T03:46:57.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847692+0000", 2024-09-15T03:46:57.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9899999999999999e-07 2024-09-15T03:46:57.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847715+0000", 2024-09-15T03:46:57.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2818000000000001e-05 2024-09-15T03:46:57.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847731+0000", 2024-09-15T03:46:57.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6401000000000001e-05 2024-09-15T03:46:57.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847738+0000", 2024-09-15T03:46:57.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3699999999999997e-06 2024-09-15T03:46:57.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847791+0000", 2024-09-15T03:46:57.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2584999999999999e-05 2024-09-15T03:46:57.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847801+0000", 2024-09-15T03:46:57.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0295999999999999e-05 2024-09-15T03:46:57.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847931+0000", 2024-09-15T03:46:57.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0001295 2024-09-15T03:46:57.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847946+0000", 2024-09-15T03:46:57.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4732000000000001e-05 2024-09-15T03:46:57.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847989+0000", 2024-09-15T03:46:57.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3625000000000003e-05 2024-09-15T03:46:57.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.848004+0000", 2024-09-15T03:46:57.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4385e-05 2024-09-15T03:46:57.508 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.508 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.508 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.508 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=84, reqid=client.4521.0:323, at_version=0'0, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:46:57.508 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.849295+0000", 2024-09-15T03:46:57.509 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.6085037799999999, 2024-09-15T03:46:57.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.608803983, 2024-09-15T03:46:57.509 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.509 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.509 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849295+0000", 2024-09-15T03:46:57.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849295+0000", 2024-09-15T03:46:57.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849296+0000", 2024-09-15T03:46:57.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6300000000000004e-07 2024-09-15T03:46:57.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849308+0000", 2024-09-15T03:46:57.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2261e-05 2024-09-15T03:46:57.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849325+0000", 2024-09-15T03:46:57.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6588000000000001e-05 2024-09-15T03:46:57.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849332+0000", 2024-09-15T03:46:57.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.464e-06 2024-09-15T03:46:57.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849379+0000", 2024-09-15T03:46:57.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6498000000000001e-05 2024-09-15T03:46:57.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849391+0000", 2024-09-15T03:46:57.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1926000000000001e-05 2024-09-15T03:46:57.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849497+0000", 2024-09-15T03:46:57.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010634200000000001 2024-09-15T03:46:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849516+0000", 2024-09-15T03:46:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8995999999999999e-05 2024-09-15T03:46:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849565+0000", 2024-09-15T03:46:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8735e-05 2024-09-15T03:46:57.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849577+0000", 2024-09-15T03:46:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2585999999999999e-05 2024-09-15T03:46:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpRead(3.7s5 49/31 ECSubRead(tid=85, to_read={3:e40d1694:::smithi00132172-50:head=0,4096,0}, subchunks={3:e40d1694:::smithi00132172-50:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:46:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.849814+0000", 2024-09-15T03:46:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.607984466, 2024-09-15T03:46:57.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6083332179999998, 2024-09-15T03:46:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849814+0000", 2024-09-15T03:46:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849814+0000", 2024-09-15T03:46:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849815+0000", 2024-09-15T03:46:57.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6499999999999999e-07 2024-09-15T03:46:57.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849826+0000", 2024-09-15T03:46:57.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0635000000000001e-05 2024-09-15T03:46:57.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.850154+0000", 2024-09-15T03:46:57.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00032866900000000002 2024-09-15T03:46:57.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.850169+0000", 2024-09-15T03:46:57.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5051e-05 2024-09-15T03:46:57.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.850226+0000", 2024-09-15T03:46:57.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6926999999999999e-05 2024-09-15T03:46:57.521 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.521 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=32, reqid=client.4521.0:329, at_version=49'13, trim_to=0'0, roll_forward_to=49'12))", 2024-09-15T03:46:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.853281+0000", 2024-09-15T03:46:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.60451798, 2024-09-15T03:46:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.604895221, 2024-09-15T03:46:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853281+0000", 2024-09-15T03:46:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853281+0000", 2024-09-15T03:46:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853282+0000", 2024-09-15T03:46:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.347e-06 2024-09-15T03:46:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853401+0000", 2024-09-15T03:46:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118711 2024-09-15T03:46:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853441+0000", 2024-09-15T03:46:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0059999999999999e-05 2024-09-15T03:46:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853451+0000", 2024-09-15T03:46:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5850000000000006e-06 2024-09-15T03:46:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853617+0000", 2024-09-15T03:46:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016655899999999999 2024-09-15T03:46:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853632+0000", 2024-09-15T03:46:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4605e-05 2024-09-15T03:46:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853842+0000", 2024-09-15T03:46:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000210523 2024-09-15T03:46:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853871+0000", 2024-09-15T03:46:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8441000000000001e-05 2024-09-15T03:46:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854188+0000", 2024-09-15T03:46:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031700800000000003 2024-09-15T03:46:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854209+0000", 2024-09-15T03:46:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0795999999999999e-05 2024-09-15T03:46:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=33, reqid=client.4521.0:329, at_version=0'0, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:46:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.854592+0000", 2024-09-15T03:46:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.6032069689999999, 2024-09-15T03:46:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.603632599, 2024-09-15T03:46:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854592+0000", 2024-09-15T03:46:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854592+0000", 2024-09-15T03:46:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854593+0000", 2024-09-15T03:46:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1200000000000002e-07 2024-09-15T03:46:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854605+0000", 2024-09-15T03:46:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1946999999999999e-05 2024-09-15T03:46:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854624+0000", 2024-09-15T03:46:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9502000000000001e-05 2024-09-15T03:46:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854631+0000", 2024-09-15T03:46:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2490000000000004e-06 2024-09-15T03:46:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854683+0000", 2024-09-15T03:46:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1643e-05 2024-09-15T03:46:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854693+0000", 2024-09-15T03:46:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8199999999999992e-06 2024-09-15T03:46:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854768+0000", 2024-09-15T03:46:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5104999999999993e-05 2024-09-15T03:46:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854830+0000", 2024-09-15T03:46:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2191000000000001e-05 2024-09-15T03:46:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854934+0000", 2024-09-15T03:46:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010424099999999999 2024-09-15T03:46:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854952+0000", 2024-09-15T03:46:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7521e-05 2024-09-15T03:46:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=34, reqid=client.4521.0:330, at_version=49'14, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:46:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.857132+0000", 2024-09-15T03:46:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.6006665879999999, 2024-09-15T03:46:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.601137502, 2024-09-15T03:46:57.535 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.535 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.535 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857132+0000", 2024-09-15T03:46:57.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857132+0000", 2024-09-15T03:46:57.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857133+0000", 2024-09-15T03:46:57.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.61e-07 2024-09-15T03:46:57.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857223+0000", 2024-09-15T03:46:57.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0476999999999994e-05 2024-09-15T03:46:57.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857245+0000", 2024-09-15T03:46:57.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1415000000000001e-05 2024-09-15T03:46:57.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857254+0000", 2024-09-15T03:46:57.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.329e-06 2024-09-15T03:46:57.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857312+0000", 2024-09-15T03:46:57.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7865000000000002e-05 2024-09-15T03:46:57.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857323+0000", 2024-09-15T03:46:57.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0655999999999999e-05 2024-09-15T03:46:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857817+0000", 2024-09-15T03:46:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00049391300000000001 2024-09-15T03:46:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857946+0000", 2024-09-15T03:46:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012926699999999999 2024-09-15T03:46:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858076+0000", 2024-09-15T03:46:57.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012957600000000001 2024-09-15T03:46:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858094+0000", 2024-09-15T03:46:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8715e-05 2024-09-15T03:46:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=86, reqid=client.4521.0:334, at_version=49'20, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:46:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.857463+0000", 2024-09-15T03:46:57.539 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.6003362700000001, 2024-09-15T03:46:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.600855159, 2024-09-15T03:46:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "queued for pg", 2024-09-15T03:46:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857463+0000", 2024-09-15T03:46:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857463+0000", 2024-09-15T03:46:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857464+0000", 2024-09-15T03:46:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0210000000000001e-06 2024-09-15T03:46:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857489+0000", 2024-09-15T03:46:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5500999999999999e-05 2024-09-15T03:46:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857508+0000", 2024-09-15T03:46:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9025000000000002e-05 2024-09-15T03:46:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857515+0000", 2024-09-15T03:46:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9759999999999998e-06 2024-09-15T03:46:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.cs3 49/28 ECSubWrite(tid=15, reqid=client.4521.0:337, at_version=49'10, trim_to=0'0, roll_forward_to=43'8))", 2024-09-15T03:46:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.857712+0000", 2024-09-15T03:46:57.542 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.600086951, 2024-09-15T03:46:57.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6006315789999999, 2024-09-15T03:46:57.543 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.543 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.543 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857712+0000", 2024-09-15T03:46:57.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857712+0000", 2024-09-15T03:46:57.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857712+0000", 2024-09-15T03:46:57.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8000000000000002e-07 2024-09-15T03:46:57.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857755+0000", 2024-09-15T03:46:57.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2960999999999997e-05 2024-09-15T03:46:57.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857773+0000", 2024-09-15T03:46:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7635999999999999e-05 2024-09-15T03:46:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857780+0000", 2024-09-15T03:46:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.024e-06 2024-09-15T03:46:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857883+0000", 2024-09-15T03:46:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103224 2024-09-15T03:46:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857893+0000", 2024-09-15T03:46:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0265000000000001e-05 2024-09-15T03:46:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858180+0000", 2024-09-15T03:46:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000286602 2024-09-15T03:46:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858193+0000", 2024-09-15T03:46:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3342e-05 2024-09-15T03:46:57.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858233+0000", 2024-09-15T03:46:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9811999999999998e-05 2024-09-15T03:46:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858253+0000", 2024-09-15T03:46:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0259e-05 2024-09-15T03:46:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=87, reqid=client.4521.0:335, at_version=49'21, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:46:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.858481+0000", 2024-09-15T03:46:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.5993178019999998, 2024-09-15T03:46:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5999054990000001, 2024-09-15T03:46:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "queued for pg", 2024-09-15T03:46:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858481+0000", 2024-09-15T03:46:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858481+0000", 2024-09-15T03:46:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858482+0000", 2024-09-15T03:46:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.09e-07 2024-09-15T03:46:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858505+0000", 2024-09-15T03:46:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3133e-05 2024-09-15T03:46:57.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858524+0000", 2024-09-15T03:46:57.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8706e-05 2024-09-15T03:46:57.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858532+0000", 2024-09-15T03:46:57.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5539999999999998e-06 2024-09-15T03:46:57.550 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.550 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.550 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.551 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.cs3 49/28 ECSubWrite(tid=16, reqid=client.4521.0:337, at_version=0'0, trim_to=0'0, roll_forward_to=49'10))", 2024-09-15T03:46:57.551 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.859219+0000", 2024-09-15T03:46:57.551 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.5985798309999999, 2024-09-15T03:46:57.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5991924960000001, 2024-09-15T03:46:57.551 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.551 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.551 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859219+0000", 2024-09-15T03:46:57.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859219+0000", 2024-09-15T03:46:57.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859220+0000", 2024-09-15T03:46:57.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3300000000000001e-07 2024-09-15T03:46:57.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859233+0000", 2024-09-15T03:46:57.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2781e-05 2024-09-15T03:46:57.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859251+0000", 2024-09-15T03:46:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8770000000000002e-05 2024-09-15T03:46:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859259+0000", 2024-09-15T03:46:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7309999999999993e-06 2024-09-15T03:46:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859342+0000", 2024-09-15T03:46:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2909000000000002e-05 2024-09-15T03:46:57.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859357+0000", 2024-09-15T03:46:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5245000000000001e-05 2024-09-15T03:46:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859453+0000", 2024-09-15T03:46:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5878999999999995e-05 2024-09-15T03:46:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859475+0000", 2024-09-15T03:46:57.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1591999999999999e-05 2024-09-15T03:46:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859518+0000", 2024-09-15T03:46:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3021e-05 2024-09-15T03:46:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859536+0000", 2024-09-15T03:46:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8420999999999999e-05 2024-09-15T03:46:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=88, reqid=client.4521.0:336, at_version=49'22, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:46:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.859709+0000", 2024-09-15T03:46:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.598090054, 2024-09-15T03:46:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.598745796, 2024-09-15T03:46:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "queued for pg", 2024-09-15T03:46:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859709+0000", 2024-09-15T03:46:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859709+0000", 2024-09-15T03:46:57.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859710+0000", 2024-09-15T03:46:57.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.052e-06 2024-09-15T03:46:57.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859729+0000", 2024-09-15T03:46:57.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9409999999999999e-05 2024-09-15T03:46:57.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859759+0000", 2024-09-15T03:46:57.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9998e-05 2024-09-15T03:46:57.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859767+0000", 2024-09-15T03:46:57.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.14e-06 2024-09-15T03:46:57.558 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.558 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.558 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.559 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=35, reqid=client.4521.0:331, at_version=49'15, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:46:57.559 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.860778+0000", 2024-09-15T03:46:57.559 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.5970211590000001, 2024-09-15T03:46:57.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5977019719999999, 2024-09-15T03:46:57.559 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.559 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.559 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860778+0000", 2024-09-15T03:46:57.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860778+0000", 2024-09-15T03:46:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860779+0000", 2024-09-15T03:46:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0109999999999999e-06 2024-09-15T03:46:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860922+0000", 2024-09-15T03:46:57.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014357399999999999 2024-09-15T03:46:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860944+0000", 2024-09-15T03:46:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1424e-05 2024-09-15T03:46:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860951+0000", 2024-09-15T03:46:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5610000000000001e-06 2024-09-15T03:46:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861022+0000", 2024-09-15T03:46:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1126999999999998e-05 2024-09-15T03:46:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861034+0000", 2024-09-15T03:46:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1810999999999999e-05 2024-09-15T03:46:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861164+0000", 2024-09-15T03:46:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000129727 2024-09-15T03:46:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861196+0000", 2024-09-15T03:46:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2240000000000003e-05 2024-09-15T03:46:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861259+0000", 2024-09-15T03:46:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2454999999999997e-05 2024-09-15T03:46:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861274+0000", 2024-09-15T03:46:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5588000000000001e-05 2024-09-15T03:46:57.563 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=36, reqid=client.4521.0:332, at_version=49'16, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:46:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.861537+0000", 2024-09-15T03:46:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.5962615959999999, 2024-09-15T03:46:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.596985954, 2024-09-15T03:46:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861537+0000", 2024-09-15T03:46:57.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861537+0000", 2024-09-15T03:46:57.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861538+0000", 2024-09-15T03:46:57.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7000000000000003e-07 2024-09-15T03:46:57.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861571+0000", 2024-09-15T03:46:57.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2493000000000002e-05 2024-09-15T03:46:57.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861588+0000", 2024-09-15T03:46:57.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7076e-05 2024-09-15T03:46:57.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861595+0000", 2024-09-15T03:46:57.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3440000000000003e-06 2024-09-15T03:46:57.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861665+0000", 2024-09-15T03:46:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9814999999999995e-05 2024-09-15T03:46:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861674+0000", 2024-09-15T03:46:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9740000000000008e-06 2024-09-15T03:46:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861767+0000", 2024-09-15T03:46:57.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3572999999999993e-05 2024-09-15T03:46:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861778+0000", 2024-09-15T03:46:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1083e-05 2024-09-15T03:46:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861888+0000", 2024-09-15T03:46:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010937700000000001 2024-09-15T03:46:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861906+0000", 2024-09-15T03:46:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.863e-05 2024-09-15T03:46:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=37, reqid=client.4521.0:332, at_version=0'0, trim_to=0'0, roll_forward_to=49'16))", 2024-09-15T03:46:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.864457+0000", 2024-09-15T03:46:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.5933419070000001, 2024-09-15T03:46:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.594109765, 2024-09-15T03:46:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864457+0000", 2024-09-15T03:46:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864457+0000", 2024-09-15T03:46:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864458+0000", 2024-09-15T03:46:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.029e-06 2024-09-15T03:46:57.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864472+0000", 2024-09-15T03:46:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4287000000000001e-05 2024-09-15T03:46:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864504+0000", 2024-09-15T03:46:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1435e-05 2024-09-15T03:46:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864511+0000", 2024-09-15T03:46:57.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4900000000000003e-06 2024-09-15T03:46:57.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864583+0000", 2024-09-15T03:46:57.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1828000000000004e-05 2024-09-15T03:46:57.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864593+0000", 2024-09-15T03:46:57.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0427e-05 2024-09-15T03:46:57.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864701+0000", 2024-09-15T03:46:57.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000107969 2024-09-15T03:46:57.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864721+0000", 2024-09-15T03:46:57.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9422000000000001e-05 2024-09-15T03:46:57.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864798+0000", 2024-09-15T03:46:57.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6731999999999996e-05 2024-09-15T03:46:57.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864815+0000", 2024-09-15T03:46:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7286999999999999e-05 2024-09-15T03:46:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.bs4 49/31 ECSubWrite(tid=75, reqid=client.4521.0:338, at_version=49'58, trim_to=0'0, roll_forward_to=47'56))", 2024-09-15T03:46:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.870036+0000", 2024-09-15T03:46:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.5877632830000001, 2024-09-15T03:46:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5885746439999999, 2024-09-15T03:46:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.574 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870036+0000", 2024-09-15T03:46:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870036+0000", 2024-09-15T03:46:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870036+0000", 2024-09-15T03:46:57.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.61e-07 2024-09-15T03:46:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870051+0000", 2024-09-15T03:46:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5074e-05 2024-09-15T03:46:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870061+0000", 2024-09-15T03:46:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.066e-05 2024-09-15T03:46:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870067+0000", 2024-09-15T03:46:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1580000000000004e-06 2024-09-15T03:46:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870121+0000", 2024-09-15T03:46:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4605999999999999e-05 2024-09-15T03:46:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870134+0000", 2024-09-15T03:46:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2748999999999999e-05 2024-09-15T03:46:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870450+0000", 2024-09-15T03:46:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031624399999999998 2024-09-15T03:46:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870459+0000", 2024-09-15T03:46:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0499999999999997e-06 2024-09-15T03:46:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870496+0000", 2024-09-15T03:46:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6548999999999999e-05 2024-09-15T03:46:57.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870509+0000", 2024-09-15T03:46:57.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3216e-05 2024-09-15T03:46:57.579 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.579 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.579 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.579 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.bs4 49/31 ECSubWrite(tid=76, reqid=client.4521.0:338, at_version=0'0, trim_to=0'0, roll_forward_to=49'58))", 2024-09-15T03:46:57.579 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.871323+0000", 2024-09-15T03:46:57.579 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 1.586476056, 2024-09-15T03:46:57.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.587330989, 2024-09-15T03:46:57.579 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.580 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.580 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.580 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.580 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.871323+0000", 2024-09-15T03:46:57.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.580 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.580 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.871323+0000", 2024-09-15T03:46:57.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.580 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.871324+0000", 2024-09-15T03:46:57.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9999999999999996e-07 2024-09-15T03:46:57.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.581 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.871333+0000", 2024-09-15T03:46:57.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7939999999999997e-06 2024-09-15T03:46:57.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.581 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.871350+0000", 2024-09-15T03:46:57.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6979000000000001e-05 2024-09-15T03:46:57.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.871357+0000", 2024-09-15T03:46:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1169999999999998e-06 2024-09-15T03:46:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.871404+0000", 2024-09-15T03:46:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6542999999999998e-05 2024-09-15T03:46:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.871414+0000", 2024-09-15T03:46:57.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6709999999999994e-06 2024-09-15T03:46:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.871523+0000", 2024-09-15T03:46:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010942699999999999 2024-09-15T03:46:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.871547+0000", 2024-09-15T03:46:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3788999999999999e-05 2024-09-15T03:46:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.871595+0000", 2024-09-15T03:46:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8004999999999999e-05 2024-09-15T03:46:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.871614+0000", 2024-09-15T03:46:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8729999999999999e-05 2024-09-15T03:46:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:340 3.5s0 3.708bb8f5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:46:57.584 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:56.851127+0000", 2024-09-15T03:46:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.60667217200000001, 2024-09-15T03:46:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60757220499999998, 2024-09-15T03:46:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "op must wait for map", 2024-09-15T03:46:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 340 2024-09-15T03:46:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.851127+0000", 2024-09-15T03:46:57.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.851127+0000", 2024-09-15T03:46:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.851129+0000", 2024-09-15T03:46:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.328e-06 2024-09-15T03:46:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.587 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.851491+0000", 2024-09-15T03:46:57.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00036243100000000003 2024-09-15T03:46:57.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.587 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.587 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.851512+0000", 2024-09-15T03:46:57.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0169999999999998e-05 2024-09-15T03:46:57.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.587 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.587 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.851519+0000", 2024-09-15T03:46:57.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3289999999999996e-06 2024-09-15T03:46:57.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.851578+0000", 2024-09-15T03:46:57.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8987000000000003e-05 2024-09-15T03:46:57.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-15T03:46:57.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.851587+0000", 2024-09-15T03:46:57.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4069999999999999e-06 2024-09-15T03:46:57.588 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.588 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.588 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.588 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:341 3.5s0 3.708bb8f5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:46:57.589 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:56.853966+0000", 2024-09-15T03:46:57.589 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.60383244300000005, 2024-09-15T03:46:57.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60477077700000004, 2024-09-15T03:46:57.589 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.589 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting_for_map not empty", 2024-09-15T03:46:57.589 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:57.589 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:57.589 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:57.589 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 341 2024-09-15T03:46:57.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.589 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.853966+0000", 2024-09-15T03:46:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.853966+0000", 2024-09-15T03:46:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.853967+0000", 2024-09-15T03:46:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9899999999999999e-07 2024-09-15T03:46:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.854184+0000", 2024-09-15T03:46:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000216939 2024-09-15T03:46:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.854196+0000", 2024-09-15T03:46:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1581e-05 2024-09-15T03:46:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.854200+0000", 2024-09-15T03:46:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3379999999999998e-06 2024-09-15T03:46:57.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.854232+0000", 2024-09-15T03:46:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2097000000000001e-05 2024-09-15T03:46:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.854236+0000", 2024-09-15T03:46:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3100000000000002e-06 2024-09-15T03:46:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:342 3.5s0 3.708bb8f5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:46:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:56.855305+0000", 2024-09-15T03:46:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.60249404200000001, 2024-09-15T03:46:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60346660799999996, 2024-09-15T03:46:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting_for_map not empty", 2024-09-15T03:46:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 342 2024-09-15T03:46:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855305+0000", 2024-09-15T03:46:57.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.594 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855305+0000", 2024-09-15T03:46:57.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.594 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855306+0000", 2024-09-15T03:46:57.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4600000000000003e-07 2024-09-15T03:46:57.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855440+0000", 2024-09-15T03:46:57.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013463700000000001 2024-09-15T03:46:57.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855451+0000", 2024-09-15T03:46:57.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0258000000000001e-05 2024-09-15T03:46:57.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855455+0000", 2024-09-15T03:46:57.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3619999999999999e-06 2024-09-15T03:46:57.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855487+0000", 2024-09-15T03:46:57.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1767999999999998e-05 2024-09-15T03:46:57.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:57.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855491+0000", 2024-09-15T03:46:57.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1010000000000002e-06 2024-09-15T03:46:57.596 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.596 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.596 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:343 3.5s0 3.708bb8f5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:46:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:56.855461+0000", 2024-09-15T03:46:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.60233810899999995, 2024-09-15T03:46:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60335085799999999, 2024-09-15T03:46:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting_for_map not empty", 2024-09-15T03:46:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 343 2024-09-15T03:46:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855461+0000", 2024-09-15T03:46:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855461+0000", 2024-09-15T03:46:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855461+0000", 2024-09-15T03:46:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8800000000000003e-07 2024-09-15T03:46:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855465+0000", 2024-09-15T03:46:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0810000000000004e-06 2024-09-15T03:46:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855473+0000", 2024-09-15T03:46:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2280000000000001e-06 2024-09-15T03:46:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855477+0000", 2024-09-15T03:46:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1030000000000004e-06 2024-09-15T03:46:57.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855512+0000", 2024-09-15T03:46:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5828000000000003e-05 2024-09-15T03:46:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855519+0000", 2024-09-15T03:46:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0260000000000002e-06 2024-09-15T03:46:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.601 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:344 3.5s0 3.708bb8f5 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:46:57.601 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:56.855482+0000", 2024-09-15T03:46:57.601 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.602316501, 2024-09-15T03:46:57.601 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.60336328500000003, 2024-09-15T03:46:57.601 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.601 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "waiting_for_map not empty", 2024-09-15T03:46:57.601 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:57.601 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:57.601 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:57.601 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 344 2024-09-15T03:46:57.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.601 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855482+0000", 2024-09-15T03:46:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855482+0000", 2024-09-15T03:46:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855483+0000", 2024-09-15T03:46:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6699999999999999e-07 2024-09-15T03:46:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855485+0000", 2024-09-15T03:46:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0020000000000001e-06 2024-09-15T03:46:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855491+0000", 2024-09-15T03:46:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6900000000000003e-06 2024-09-15T03:46:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855496+0000", 2024-09-15T03:46:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0969999999999999e-06 2024-09-15T03:46:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855532+0000", 2024-09-15T03:46:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.608e-05 2024-09-15T03:46:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:56.855535+0000", 2024-09-15T03:46:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1829999999999998e-06 2024-09-15T03:46:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:46:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 26 2024-09-15T03:46:57.605 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:46:57.606 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:46:57.606 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:57.605+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:46:57.707 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:46:57.706+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:46:57.748 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:46:57.749 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:46:57.749 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-15T03:46:57.749 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-15T03:46:57.749 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:46:57.757 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:46:57.808 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:57.806+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:46:57.838 INFO:tasks.rados.rados.0.smithi001.stdout:135: done (3 left) 2024-09-15T03:46:57.838 INFO:tasks.rados.rados.0.smithi001.stdout:136: rollback oid 5 current snap is 13 2024-09-15T03:46:57.838 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 5 to 7 2024-09-15T03:46:57.838 INFO:tasks.rados.rados.0.smithi001.stdout:137: rollback oid 25 current snap is 13 2024-09-15T03:46:57.839 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 25 to 12 2024-09-15T03:46:57.839 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 11 current snap is 13 2024-09-15T03:46:57.839 INFO:tasks.rados.rados.0.smithi001.stdout:138: seq_num 78 ranges {180224=557056,737280=671744,1409024=98304} 2024-09-15T03:46:57.842 INFO:tasks.rados.rados.0.smithi001.stdout:138: writing smithi00132172-11 from 180224 to 737280 tid 1 2024-09-15T03:46:57.847 INFO:tasks.rados.rados.0.smithi001.stdout:138: writing smithi00132172-11 from 737280 to 1409024 tid 2 2024-09-15T03:46:57.848 INFO:tasks.rados.rados.0.smithi001.stdout:138: writing smithi00132172-11 from 1409024 to 1507328 tid 3 2024-09-15T03:46:57.849 INFO:tasks.rados.rados.0.smithi001.stdout:139: read oid 44 snap -1 2024-09-15T03:46:57.849 INFO:tasks.rados.rados.0.smithi001.stdout:139: expect (ObjNum 71 snap 9 seq_num 71) 2024-09-15T03:46:57.849 INFO:tasks.rados.rados.0.smithi001.stdout:140: snap_create 2024-09-15T03:46:57.864 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:168 3.0s0 3:08ea3aba:::smithi00132172-6:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:39.553158+0000", 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 18.309335054999998, 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.005608231, 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:57.865 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 168 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.553158+0000", 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.553158+0000", 2024-09-15T03:46:57.866 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.553160+0000", 2024-09-15T03:46:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.269e-06 2024-09-15T03:46:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.553247+0000", 2024-09-15T03:46:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7536999999999999e-05 2024-09-15T03:46:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.867 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.553258+0000", 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0686000000000001e-05 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.553263+0000", 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4060000000000004e-06 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.553562+0000", 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029855700000000002 2024-09-15T03:46:57.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.553674+0000", 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112234 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.554872+0000", 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001197851 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.870 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.554973+0000", 2024-09-15T03:46:57.870 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100917 2024-09-15T03:46:57.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.870 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.555011+0000", 2024-09-15T03:46:57.870 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7922e-05 2024-09-15T03:46:57.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.555091+0000", 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9929e-05 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.555974+0000", 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00088307700000000002 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.871 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.558695+0000", 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0027207500000000001 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.558767+0000", 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2095999999999995e-05 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.872 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:169 3.0s0 3:08ea3aba:::smithi00132172-6:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-15T03:46:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:39.553275+0000", 2024-09-15T03:46:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 18.309218753, 2024-09-15T03:46:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0056151480000000004, 2024-09-15T03:46:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:57.873 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 169 2024-09-15T03:46:57.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.874 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.874 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.874 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.553275+0000", 2024-09-15T03:46:57.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.553275+0000", 2024-09-15T03:46:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.553275+0000", 2024-09-15T03:46:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8400000000000004e-07 2024-09-15T03:46:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.875 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.553279+0000", 2024-09-15T03:46:57.876 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1509999999999997e-06 2024-09-15T03:46:57.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.876 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.876 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.553287+0000", 2024-09-15T03:46:57.876 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.515e-06 2024-09-15T03:46:57.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.876 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.876 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.553291+0000", 2024-09-15T03:46:57.876 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3039999999999998e-06 2024-09-15T03:46:57.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.877 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.877 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.555078+0000", 2024-09-15T03:46:57.877 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001786852 2024-09-15T03:46:57.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.877 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:57.877 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.555163+0000", 2024-09-15T03:46:57.877 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5291999999999996e-05 2024-09-15T03:46:57.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.555605+0000", 2024-09-15T03:46:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00044189099999999997 2024-09-15T03:46:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.555817+0000", 2024-09-15T03:46:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021159600000000001 2024-09-15T03:46:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.878 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.555833+0000", 2024-09-15T03:46:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5954999999999999e-05 2024-09-15T03:46:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.555878+0000", 2024-09-15T03:46:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5102000000000003e-05 2024-09-15T03:46:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.556239+0000", 2024-09-15T03:46:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00036072600000000002 2024-09-15T03:46:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.879 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:46:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.558837+0000", 2024-09-15T03:46:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0025981929999999999 2024-09-15T03:46:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.558890+0000", 2024-09-15T03:46:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2986999999999999e-05 2024-09-15T03:46:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.880 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:170 3.0s0 3:08ea3aba:::smithi00132172-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-15T03:46:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:39.553296+0000", 2024-09-15T03:46:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 18.309197562000001, 2024-09-15T03:46:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0078702040000000004, 2024-09-15T03:46:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:46:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:57.881 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 170 2024-09-15T03:46:57.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.882 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.882 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.882 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.553296+0000", 2024-09-15T03:46:57.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.882 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.882 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.553296+0000", 2024-09-15T03:46:57.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.882 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.883 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.553296+0000", 2024-09-15T03:46:57.883 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6899999999999998e-07 2024-09-15T03:46:57.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.883 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.883 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.553300+0000", 2024-09-15T03:46:57.883 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.213e-06 2024-09-15T03:46:57.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.883 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.883 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.553306+0000", 2024-09-15T03:46:57.883 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8229999999999997e-06 2024-09-15T03:46:57.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.884 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.884 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.553310+0000", 2024-09-15T03:46:57.884 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0559999999999998e-06 2024-09-15T03:46:57.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.884 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.884 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.556448+0000", 2024-09-15T03:46:57.884 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003137305 2024-09-15T03:46:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for rw locks", 2024-09-15T03:46:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.556529+0000", 2024-09-15T03:46:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1092000000000002e-05 2024-09-15T03:46:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.885 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.559618+0000", 2024-09-15T03:46:57.886 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0030895089999999998 2024-09-15T03:46:57.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.887 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:57.887 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.559733+0000", 2024-09-15T03:46:57.887 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114978 2024-09-15T03:46:57.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.888 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:57.888 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.561166+0000", 2024-09-15T03:46:57.888 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001432759 2024-09-15T03:46:57.889 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.889 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.889 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.889 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:181 3.0s0 3:02f07a0d:::smithi00132172-42:head [read 0~704512,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:57.890 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:40.104625+0000", 2024-09-15T03:46:57.890 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.757868468000002, 2024-09-15T03:46:57.890 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0058489229999999998, 2024-09-15T03:46:57.890 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.890 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:46:57.891 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:57.891 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:57.891 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:57.891 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 181 2024-09-15T03:46:57.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.891 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.892 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.892 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104625+0000", 2024-09-15T03:46:57.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.894 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.894 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104625+0000", 2024-09-15T03:46:57.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.895 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.895 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104627+0000", 2024-09-15T03:46:57.895 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0090000000000002e-06 2024-09-15T03:46:57.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.895 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.896 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.896 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104648+0000", 2024-09-15T03:46:57.896 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1339e-05 2024-09-15T03:46:57.896 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.896 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.897 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104673+0000", 2024-09-15T03:46:57.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4811999999999999e-05 2024-09-15T03:46:57.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.897 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104683+0000", 2024-09-15T03:46:57.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8980000000000008e-06 2024-09-15T03:46:57.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104754+0000", 2024-09-15T03:46:57.899 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0438000000000006e-05 2024-09-15T03:46:57.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.899 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-15T03:46:57.899 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104762+0000", 2024-09-15T03:46:57.900 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8030000000000003e-06 2024-09-15T03:46:57.900 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.900 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.900 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.900 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.106110+0000", 2024-09-15T03:46:57.901 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0013476880000000001 2024-09-15T03:46:57.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.901 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:57.901 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.106217+0000", 2024-09-15T03:46:57.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000106659 2024-09-15T03:46:57.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.903 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:57.903 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.110474+0000", 2024-09-15T03:46:57.903 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0042572770000000003 2024-09-15T03:46:57.904 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.904 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.904 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.905 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:182 3.0s0 3:02f07a0d:::smithi00132172-42:head [read 0~704512,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:57.905 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:40.104697+0000", 2024-09-15T03:46:57.905 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.757796076999998, 2024-09-15T03:46:57.905 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0072319460000000004, 2024-09-15T03:46:57.905 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.905 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:46:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 182 2024-09-15T03:46:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104697+0000", 2024-09-15T03:46:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.907 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.907 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104697+0000", 2024-09-15T03:46:57.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.907 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.907 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104698+0000", 2024-09-15T03:46:57.907 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1500000000000004e-07 2024-09-15T03:46:57.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.907 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.908 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104705+0000", 2024-09-15T03:46:57.908 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7329999999999997e-06 2024-09-15T03:46:57.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.908 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.908 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104718+0000", 2024-09-15T03:46:57.908 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2974e-05 2024-09-15T03:46:57.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.908 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.908 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104722+0000", 2024-09-15T03:46:57.908 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9060000000000004e-06 2024-09-15T03:46:57.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104834+0000", 2024-09-15T03:46:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011285699999999999 2024-09-15T03:46:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104839+0000", 2024-09-15T03:46:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4859999999999999e-06 2024-09-15T03:46:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.106598+0000", 2024-09-15T03:46:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0017586679999999999 2024-09-15T03:46:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.106677+0000", 2024-09-15T03:46:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9585999999999993e-05 2024-09-15T03:46:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:57.910 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.111929+0000", 2024-09-15T03:46:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0052521210000000002 2024-09-15T03:46:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:183 3.0s0 3:02f07a0d:::smithi00132172-42:head [read 0~704512,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:46:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:40.104737+0000", 2024-09-15T03:46:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.75775634, 2024-09-15T03:46:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0078220000000000008, 2024-09-15T03:46:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:46:57.911 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 183 2024-09-15T03:46:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104737+0000", 2024-09-15T03:46:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.912 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.913 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104737+0000", 2024-09-15T03:46:57.913 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.913 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.913 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104737+0000", 2024-09-15T03:46:57.913 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2000000000000001e-07 2024-09-15T03:46:57.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.914 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.914 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104740+0000", 2024-09-15T03:46:57.914 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5289999999999998e-06 2024-09-15T03:46:57.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.914 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.914 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104747+0000", 2024-09-15T03:46:57.914 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9160000000000004e-06 2024-09-15T03:46:57.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.914 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.914 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104751+0000", 2024-09-15T03:46:57.915 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0400000000000003e-06 2024-09-15T03:46:57.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.915 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.915 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104871+0000", 2024-09-15T03:46:57.915 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000120386 2024-09-15T03:46:57.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.915 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:57.915 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.104878+0000", 2024-09-15T03:46:57.915 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9020000000000002e-06 2024-09-15T03:46:57.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107899+0000", 2024-09-15T03:46:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0030209360000000001 2024-09-15T03:46:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108167+0000", 2024-09-15T03:46:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026736499999999999 2024-09-15T03:46:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:57.916 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.112559+0000", 2024-09-15T03:46:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0043926060000000003 2024-09-15T03:46:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs1 33/31 ECSubWrite(tid=22, reqid=client.4521.0:187, at_version=33'8, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:40.107718+0000", 2024-09-15T03:46:57.917 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.754775500000001, 2024-09-15T03:46:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0059233910000000001, 2024-09-15T03:46:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107718+0000", 2024-09-15T03:46:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107718+0000", 2024-09-15T03:46:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107719+0000", 2024-09-15T03:46:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0890000000000001e-06 2024-09-15T03:46:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107748+0000", 2024-09-15T03:46:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8571000000000001e-05 2024-09-15T03:46:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.919 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107759+0000", 2024-09-15T03:46:57.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1413999999999999e-05 2024-09-15T03:46:57.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107768+0000", 2024-09-15T03:46:57.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5830000000000007e-06 2024-09-15T03:46:57.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107835+0000", 2024-09-15T03:46:57.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7219999999999997e-05 2024-09-15T03:46:57.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107846+0000", 2024-09-15T03:46:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1021e-05 2024-09-15T03:46:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.109984+0000", 2024-09-15T03:46:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002138315 2024-09-15T03:46:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.110187+0000", 2024-09-15T03:46:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020236700000000001 2024-09-15T03:46:57.921 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.922 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.110240+0000", 2024-09-15T03:46:57.922 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3012e-05 2024-09-15T03:46:57.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.922 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.110573+0000", 2024-09-15T03:46:57.922 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000333195 2024-09-15T03:46:57.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:57.923 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.113641+0000", 2024-09-15T03:46:57.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0030686039999999999 2024-09-15T03:46:57.923 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.923 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.923 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.923 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs1 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:57.923 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:40.108530+0000", 2024-09-15T03:46:57.924 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.753963462000002, 2024-09-15T03:46:57.924 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.006985041, 2024-09-15T03:46:57.924 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.924 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.924 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.924 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.924 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108530+0000", 2024-09-15T03:46:57.924 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108530+0000", 2024-09-15T03:46:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108531+0000", 2024-09-15T03:46:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5000000000000002e-07 2024-09-15T03:46:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108561+0000", 2024-09-15T03:46:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9876999999999999e-05 2024-09-15T03:46:57.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108571+0000", 2024-09-15T03:46:57.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0871e-05 2024-09-15T03:46:57.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108578+0000", 2024-09-15T03:46:57.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9229999999999996e-06 2024-09-15T03:46:57.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.110040+0000", 2024-09-15T03:46:57.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014617549999999999 2024-09-15T03:46:57.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.927 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.110049+0000", 2024-09-15T03:46:57.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9209999999999998e-06 2024-09-15T03:46:57.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.927 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.110155+0000", 2024-09-15T03:46:57.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000106427 2024-09-15T03:46:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.110188+0000", 2024-09-15T03:46:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2397999999999997e-05 2024-09-15T03:46:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.110240+0000", 2024-09-15T03:46:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2376999999999997e-05 2024-09-15T03:46:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.928 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.110590+0000", 2024-09-15T03:46:57.929 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00035007499999999998 2024-09-15T03:46:57.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.929 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:57.929 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.115515+0000", 2024-09-15T03:46:57.929 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.004924767 2024-09-15T03:46:57.929 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.929 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.929 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.929 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs1 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:46:57.929 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:40.109343+0000", 2024-09-15T03:46:57.930 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.753150037000001, 2024-09-15T03:46:57.930 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.006289635, 2024-09-15T03:46:57.930 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.930 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.930 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.930 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.930 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.109343+0000", 2024-09-15T03:46:57.930 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.930 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.930 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.109343+0000", 2024-09-15T03:46:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.109344+0000", 2024-09-15T03:46:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5699999999999998e-07 2024-09-15T03:46:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.109361+0000", 2024-09-15T03:46:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7526000000000001e-05 2024-09-15T03:46:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.931 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.109378+0000", 2024-09-15T03:46:57.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6393999999999999e-05 2024-09-15T03:46:57.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.109385+0000", 2024-09-15T03:46:57.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1459999999999999e-06 2024-09-15T03:46:57.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.933 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.110312+0000", 2024-09-15T03:46:57.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00092661000000000002 2024-09-15T03:46:57.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.933 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.110322+0000", 2024-09-15T03:46:57.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0603e-05 2024-09-15T03:46:57.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.110499+0000", 2024-09-15T03:46:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017644899999999999 2024-09-15T03:46:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.110532+0000", 2024-09-15T03:46:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3086999999999997e-05 2024-09-15T03:46:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.934 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.110573+0000", 2024-09-15T03:46:57.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0976999999999997e-05 2024-09-15T03:46:57.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.110613+0000", 2024-09-15T03:46:57.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0738999999999997e-05 2024-09-15T03:46:57.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:57.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.115633+0000", 2024-09-15T03:46:57.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0050196470000000003 2024-09-15T03:46:57.935 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.935 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.936 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.936 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:223 3.5s0 3:af1dd10e:::smithi00132172-27:head [append 0~688128] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:46:57.936 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:45.144659+0000", 2024-09-15T03:46:57.936 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.717834096000001, 2024-09-15T03:46:57.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0058305220000000003, 2024-09-15T03:46:57.936 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.936 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:57.936 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:57.936 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:57.937 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:57.937 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 223 2024-09-15T03:46:57.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.937 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.144659+0000", 2024-09-15T03:46:57.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.938 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.144659+0000", 2024-09-15T03:46:57.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.938 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.144662+0000", 2024-09-15T03:46:57.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.21e-06 2024-09-15T03:46:57.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.144919+0000", 2024-09-15T03:46:57.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00025739699999999998 2024-09-15T03:46:57.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.144942+0000", 2024-09-15T03:46:57.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3110999999999999e-05 2024-09-15T03:46:57.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.144950+0000", 2024-09-15T03:46:57.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2959999999999999e-06 2024-09-15T03:46:57.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.145018+0000", 2024-09-15T03:46:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7121999999999999e-05 2024-09-15T03:46:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.145147+0000", 2024-09-15T03:46:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012974899999999999 2024-09-15T03:46:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.146839+0000", 2024-09-15T03:46:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016916189999999999 2024-09-15T03:46:57.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.941 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.147154+0000", 2024-09-15T03:46:57.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031516900000000002 2024-09-15T03:46:57.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.941 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.147171+0000", 2024-09-15T03:46:57.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6826000000000001e-05 2024-09-15T03:46:57.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.147450+0000", 2024-09-15T03:46:57.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00027873699999999998 2024-09-15T03:46:57.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.942 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.147535+0000", 2024-09-15T03:46:57.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5092000000000005e-05 2024-09-15T03:46:57.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.942 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:46:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.149968+0000", 2024-09-15T03:46:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024329709999999999 2024-09-15T03:46:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.150490+0000", 2024-09-15T03:46:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00052222300000000004 2024-09-15T03:46:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:224 3.5s0 3:af1dd10e:::smithi00132172-27:head [append 0~376832] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:46:57.943 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:45.147385+0000", 2024-09-15T03:46:57.944 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.715108935, 2024-09-15T03:46:57.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0050911389999999997, 2024-09-15T03:46:57.944 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.944 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:57.944 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:57.944 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:57.944 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:57.944 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 224 2024-09-15T03:46:57.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.944 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.944 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.147385+0000", 2024-09-15T03:46:57.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.147385+0000", 2024-09-15T03:46:57.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.147386+0000", 2024-09-15T03:46:57.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9300000000000002e-06 2024-09-15T03:46:57.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148347+0000", 2024-09-15T03:46:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00096087099999999997 2024-09-15T03:46:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148365+0000", 2024-09-15T03:46:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7697e-05 2024-09-15T03:46:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148372+0000", 2024-09-15T03:46:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4379999999999998e-06 2024-09-15T03:46:57.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.947 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.947 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148411+0000", 2024-09-15T03:46:57.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8661999999999997e-05 2024-09-15T03:46:57.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.947 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:57.947 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148584+0000", 2024-09-15T03:46:57.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017262700000000001 2024-09-15T03:46:57.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.947 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.947 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.149565+0000", 2024-09-15T03:46:57.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00098099899999999998 2024-09-15T03:46:57.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.948 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.948 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.149671+0000", 2024-09-15T03:46:57.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000106578 2024-09-15T03:46:57.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.948 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.948 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.149918+0000", 2024-09-15T03:46:57.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024690699999999999 2024-09-15T03:46:57.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.150252+0000", 2024-09-15T03:46:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00033355100000000002 2024-09-15T03:46:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.151018+0000", 2024-09-15T03:46:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00076570100000000003 2024-09-15T03:46:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:46:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.152409+0000", 2024-09-15T03:46:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0013910190000000001 2024-09-15T03:46:57.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.950 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:57.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.152476+0000", 2024-09-15T03:46:57.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7158999999999999e-05 2024-09-15T03:46:57.950 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.950 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.950 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.950 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.bs4 38/31 ECSubWrite(tid=36, reqid=client.4521.0:220, at_version=38'23, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:46:57.950 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:45.148091+0000", 2024-09-15T03:46:57.950 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.714402445999999, 2024-09-15T03:46:57.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0049662680000000002, 2024-09-15T03:46:57.951 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.951 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.951 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148091+0000", 2024-09-15T03:46:57.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148091+0000", 2024-09-15T03:46:57.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148092+0000", 2024-09-15T03:46:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5199999999999996e-07 2024-09-15T03:46:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148911+0000", 2024-09-15T03:46:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00081892100000000002 2024-09-15T03:46:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148927+0000", 2024-09-15T03:46:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6606999999999999e-05 2024-09-15T03:46:57.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148934+0000", 2024-09-15T03:46:57.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7340000000000002e-06 2024-09-15T03:46:57.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.149001+0000", 2024-09-15T03:46:57.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6901000000000002e-05 2024-09-15T03:46:57.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.149011+0000", 2024-09-15T03:46:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0594999999999999e-05 2024-09-15T03:46:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.149541+0000", 2024-09-15T03:46:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00052918300000000002 2024-09-15T03:46:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.149916+0000", 2024-09-15T03:46:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00037554599999999999 2024-09-15T03:46:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.150251+0000", 2024-09-15T03:46:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00033451800000000001 2024-09-15T03:46:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.150283+0000", 2024-09-15T03:46:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2088000000000002e-05 2024-09-15T03:46:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.153057+0000", 2024-09-15T03:46:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0027745230000000001 2024-09-15T03:46:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:225 3.5s0 3:af1dd10e:::smithi00132172-27:head [setxattr _header (56) in=7b] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:46:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:45.148381+0000", 2024-09-15T03:46:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.714112459000001, 2024-09-15T03:46:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0054497779999999997, 2024-09-15T03:46:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 225 2024-09-15T03:46:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148381+0000", 2024-09-15T03:46:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148381+0000", 2024-09-15T03:46:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148382+0000", 2024-09-15T03:46:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.088e-06 2024-09-15T03:46:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148387+0000", 2024-09-15T03:46:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9930000000000003e-06 2024-09-15T03:46:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148395+0000", 2024-09-15T03:46:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7279999999999995e-06 2024-09-15T03:46:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.959 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.959 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148400+0000", 2024-09-15T03:46:57.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8330000000000001e-06 2024-09-15T03:46:57.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.959 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.959 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.150097+0000", 2024-09-15T03:46:57.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016969280000000001 2024-09-15T03:46:57.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.959 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:57.959 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.150202+0000", 2024-09-15T03:46:57.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010492099999999999 2024-09-15T03:46:57.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.960 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.960 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.150630+0000", 2024-09-15T03:46:57.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00042846699999999998 2024-09-15T03:46:57.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.960 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.960 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.150759+0000", 2024-09-15T03:46:57.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012856799999999999 2024-09-15T03:46:57.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.960 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.961 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.150778+0000", 2024-09-15T03:46:57.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9607e-05 2024-09-15T03:46:57.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.961 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.150817+0000", 2024-09-15T03:46:57.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8531000000000001e-05 2024-09-15T03:46:57.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.961 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.151048+0000", 2024-09-15T03:46:57.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000231368 2024-09-15T03:46:57.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.962 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:46:57.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.153760+0000", 2024-09-15T03:46:57.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0027118620000000002 2024-09-15T03:46:57.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.962 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:57.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.153831+0000", 2024-09-15T03:46:57.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0883999999999994e-05 2024-09-15T03:46:57.962 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.962 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.962 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.963 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:226 3.5s0 3:af1dd10e:::smithi00132172-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:46:57.963 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:45.148407+0000", 2024-09-15T03:46:57.963 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.714086844000001, 2024-09-15T03:46:57.963 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011483769, 2024-09-15T03:46:57.963 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.963 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:46:57.963 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:57.963 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:57.963 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:57.963 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 226 2024-09-15T03:46:57.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.963 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.964 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.964 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148407+0000", 2024-09-15T03:46:57.964 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.964 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.964 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148407+0000", 2024-09-15T03:46:57.964 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.964 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.964 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148407+0000", 2024-09-15T03:46:57.964 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9400000000000001e-07 2024-09-15T03:46:57.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148409+0000", 2024-09-15T03:46:57.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0339999999999999e-06 2024-09-15T03:46:57.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148416+0000", 2024-09-15T03:46:57.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5549999999999999e-06 2024-09-15T03:46:57.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148420+0000", 2024-09-15T03:46:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9040000000000002e-06 2024-09-15T03:46:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.150901+0000", 2024-09-15T03:46:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024816199999999999 2024-09-15T03:46:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for rw locks", 2024-09-15T03:46:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.151005+0000", 2024-09-15T03:46:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103738 2024-09-15T03:46:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.154147+0000", 2024-09-15T03:46:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0031425379999999998 2024-09-15T03:46:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.154233+0000", 2024-09-15T03:46:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5418000000000001e-05 2024-09-15T03:46:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.159890+0000", 2024-09-15T03:46:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.005657568 2024-09-15T03:46:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:240 3.0s0 3:08ea3aba:::smithi00132172-6:head [append 0~0] snapc 7=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:46:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:46.141508+0000", 2024-09-15T03:46:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.720985497999999, 2024-09-15T03:46:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012496024999999999, 2024-09-15T03:46:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 240 2024-09-15T03:46:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.141508+0000", 2024-09-15T03:46:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.141508+0000", 2024-09-15T03:46:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.141510+0000", 2024-09-15T03:46:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4779999999999998e-06 2024-09-15T03:46:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.141518+0000", 2024-09-15T03:46:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6690000000000006e-06 2024-09-15T03:46:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.141536+0000", 2024-09-15T03:46:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7509000000000001e-05 2024-09-15T03:46:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.141543+0000", 2024-09-15T03:46:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8320000000000006e-06 2024-09-15T03:46:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.141609+0000", 2024-09-15T03:46:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5389999999999996e-05 2024-09-15T03:46:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-15T03:46:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.141618+0000", 2024-09-15T03:46:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1390000000000001e-06 2024-09-15T03:46:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.146609+0000", 2024-09-15T03:46:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0049914770000000002 2024-09-15T03:46:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.146797+0000", 2024-09-15T03:46:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000187763 2024-09-15T03:46:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.147841+0000", 2024-09-15T03:46:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010435570000000001 2024-09-15T03:46:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.148375+0000", 2024-09-15T03:46:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00053440900000000001 2024-09-15T03:46:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149071+0000", 2024-09-15T03:46:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000696151 2024-09-15T03:46:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149443+0000", 2024-09-15T03:46:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00037159399999999999 2024-09-15T03:46:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150872+0000", 2024-09-15T03:46:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001428601 2024-09-15T03:46:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:46:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153615+0000", 2024-09-15T03:46:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0027435820000000001 2024-09-15T03:46:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.154004+0000", 2024-09-15T03:46:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00038887399999999999 2024-09-15T03:46:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:241 3.0s0 3:08ea3aba:::smithi00132172-6:head [rollback 2] snapc 7=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:46:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:46.141660+0000", 2024-09-15T03:46:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.720833749000001, 2024-09-15T03:46:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.016475725, 2024-09-15T03:46:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 241 2024-09-15T03:46:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.977 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.977 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.977 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.141660+0000", 2024-09-15T03:46:57.977 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.977 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.977 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.141660+0000", 2024-09-15T03:46:57.977 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.977 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.978 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.141661+0000", 2024-09-15T03:46:57.978 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7599999999999996e-07 2024-09-15T03:46:57.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.978 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.978 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.141665+0000", 2024-09-15T03:46:57.978 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8740000000000003e-06 2024-09-15T03:46:57.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.978 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.978 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.141674+0000", 2024-09-15T03:46:57.978 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1099999999999992e-06 2024-09-15T03:46:57.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.141681+0000", 2024-09-15T03:46:57.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.596e-06 2024-09-15T03:46:57.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.141808+0000", 2024-09-15T03:46:57.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012678400000000001 2024-09-15T03:46:57.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:46:57.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.141814+0000", 2024-09-15T03:46:57.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8649999999999998e-06 2024-09-15T03:46:57.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.980 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.980 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.148441+0000", 2024-09-15T03:46:57.980 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0066269129999999999 2024-09-15T03:46:57.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.980 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:57.980 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.148540+0000", 2024-09-15T03:46:57.980 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9073000000000005e-05 2024-09-15T03:46:57.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.980 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.981 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149323+0000", 2024-09-15T03:46:57.981 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00078302999999999999 2024-09-15T03:46:57.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.981 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.981 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149419+0000", 2024-09-15T03:46:57.981 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6576999999999994e-05 2024-09-15T03:46:57.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.981 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.981 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149449+0000", 2024-09-15T03:46:57.981 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9958999999999999e-05 2024-09-15T03:46:57.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149505+0000", 2024-09-15T03:46:57.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5235999999999998e-05 2024-09-15T03:46:57.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150922+0000", 2024-09-15T03:46:57.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014176779999999999 2024-09-15T03:46:57.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:46:57.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.155884+0000", 2024-09-15T03:46:57.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.004961788 2024-09-15T03:46:57.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.983 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:57.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.158135+0000", 2024-09-15T03:46:57.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022514660000000001 2024-09-15T03:46:57.983 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.983 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.983 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.983 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:242 3.0s0 3:08ea3aba:::smithi00132172-6:head [append 0~0] snapc 7=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:46:57.983 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:46.145632+0000", 2024-09-15T03:46:57.983 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.716861564, 2024-09-15T03:46:57.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012678794, 2024-09-15T03:46:57.983 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.984 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:57.984 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:57.984 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:57.984 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:57.984 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 242 2024-09-15T03:46:57.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.984 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.984 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.984 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.145632+0000", 2024-09-15T03:46:57.984 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.985 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.985 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.145632+0000", 2024-09-15T03:46:57.985 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.985 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.985 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.145634+0000", 2024-09-15T03:46:57.985 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.131e-06 2024-09-15T03:46:57.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.985 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.985 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.145640+0000", 2024-09-15T03:46:57.985 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8459999999999996e-06 2024-09-15T03:46:57.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.986 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.986 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.145657+0000", 2024-09-15T03:46:57.986 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7127999999999999e-05 2024-09-15T03:46:57.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.986 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.986 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.145665+0000", 2024-09-15T03:46:57.986 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7640000000000005e-06 2024-09-15T03:46:57.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.986 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.987 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149918+0000", 2024-09-15T03:46:57.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0042532569999999999 2024-09-15T03:46:57.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.987 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:57.987 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150015+0000", 2024-09-15T03:46:57.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6793000000000007e-05 2024-09-15T03:46:57.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.987 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.987 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150503+0000", 2024-09-15T03:46:57.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00048841300000000004 2024-09-15T03:46:57.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.988 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150659+0000", 2024-09-15T03:46:57.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015616100000000001 2024-09-15T03:46:57.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.988 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150676+0000", 2024-09-15T03:46:57.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7017e-05 2024-09-15T03:46:57.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.988 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150947+0000", 2024-09-15T03:46:57.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000270165 2024-09-15T03:46:57.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.989 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.989 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.151010+0000", 2024-09-15T03:46:57.989 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3320999999999999e-05 2024-09-15T03:46:57.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.989 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:46:57.989 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.156273+0000", 2024-09-15T03:46:57.989 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0052634609999999997 2024-09-15T03:46:57.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.989 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:57.989 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.158311+0000", 2024-09-15T03:46:57.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020373370000000002 2024-09-15T03:46:57.990 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.990 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.990 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.990 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.es2 39/28 ECSubWrite(tid=50, reqid=osd.2.0:50, at_version=39'28, trim_to=0'0, roll_forward_to=38'26))", 2024-09-15T03:46:57.990 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:46.149356+0000", 2024-09-15T03:46:57.990 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.713137875999999, 2024-09-15T03:46:57.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0051005920000000001, 2024-09-15T03:46:57.990 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.990 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.990 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149356+0000", 2024-09-15T03:46:57.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149356+0000", 2024-09-15T03:46:57.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149356+0000", 2024-09-15T03:46:57.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1600000000000001e-07 2024-09-15T03:46:57.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.992 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149399+0000", 2024-09-15T03:46:57.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3350000000000003e-05 2024-09-15T03:46:57.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.992 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149410+0000", 2024-09-15T03:46:57.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.079e-05 2024-09-15T03:46:57.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.992 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.993 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149418+0000", 2024-09-15T03:46:57.993 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3490000000000003e-06 2024-09-15T03:46:57.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.993 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.993 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149463+0000", 2024-09-15T03:46:57.993 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4919000000000001e-05 2024-09-15T03:46:57.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.993 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.994 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149471+0000", 2024-09-15T03:46:57.994 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1829999999999994e-06 2024-09-15T03:46:57.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.994 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:57.994 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149735+0000", 2024-09-15T03:46:57.994 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026458900000000002 2024-09-15T03:46:57.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.994 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:57.994 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149762+0000", 2024-09-15T03:46:57.994 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6806999999999999e-05 2024-09-15T03:46:57.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.995 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:57.995 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149805+0000", 2024-09-15T03:46:57.995 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2500999999999998e-05 2024-09-15T03:46:57.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.995 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:57.995 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149827+0000", 2024-09-15T03:46:57.995 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2297e-05 2024-09-15T03:46:57.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.995 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:57.995 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.154456+0000", 2024-09-15T03:46:57.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0046292909999999998 2024-09-15T03:46:57.996 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.996 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:57.996 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:57.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.996 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.bs4 39/31 ECSubWrite(tid=39, reqid=osd.7.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=39'25))", 2024-09-15T03:46:57.996 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:46.150801+0000", 2024-09-15T03:46:57.996 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.711692054, 2024-09-15T03:46:57.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.007585474, 2024-09-15T03:46:57.996 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:57.996 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:57.997 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:57.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:57.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150801+0000", 2024-09-15T03:46:57.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:57.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150801+0000", 2024-09-15T03:46:57.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:57.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:57.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150802+0000", 2024-09-15T03:46:57.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3000000000000002e-07 2024-09-15T03:46:57.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.998 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:57.998 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150810+0000", 2024-09-15T03:46:57.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4109999999999996e-06 2024-09-15T03:46:57.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.998 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:57.998 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150820+0000", 2024-09-15T03:46:57.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9299999999999998e-06 2024-09-15T03:46:57.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.999 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:57.999 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150825+0000", 2024-09-15T03:46:57.999 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8459999999999999e-06 2024-09-15T03:46:57.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.999 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:57.999 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.151793+0000", 2024-09-15T03:46:57.999 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00096842399999999996 2024-09-15T03:46:57.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:57.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:57.999 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:57.999 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.151804+0000", 2024-09-15T03:46:58.000 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0739e-05 2024-09-15T03:46:58.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.000 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:58.000 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.151935+0000", 2024-09-15T03:46:58.000 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000131328 2024-09-15T03:46:58.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.000 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:58.000 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.151963+0000", 2024-09-15T03:46:58.000 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7169e-05 2024-09-15T03:46:58.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:58.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.152006+0000", 2024-09-15T03:46:58.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3890000000000002e-05 2024-09-15T03:46:58.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:58.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.152586+0000", 2024-09-15T03:46:58.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00057960300000000004 2024-09-15T03:46:58.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:58.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.158387+0000", 2024-09-15T03:46:58.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0058008039999999997 2024-09-15T03:46:58.002 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.002 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.002 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.002 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.3s4 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:46:58.002 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:46.170750+0000", 2024-09-15T03:46:58.002 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 11.69174336, 2024-09-15T03:46:58.002 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0054717009999999998, 2024-09-15T03:46:58.002 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:58.003 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:58.003 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:58.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.003 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:58.003 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.170750+0000", 2024-09-15T03:46:58.003 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.003 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:58.003 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.170750+0000", 2024-09-15T03:46:58.004 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.004 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:58.004 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.170751+0000", 2024-09-15T03:46:58.004 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2130000000000001e-06 2024-09-15T03:46:58.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.004 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:58.004 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171114+0000", 2024-09-15T03:46:58.004 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000362832 2024-09-15T03:46:58.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:58.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171125+0000", 2024-09-15T03:46:58.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0621999999999999e-05 2024-09-15T03:46:58.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:58.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171131+0000", 2024-09-15T03:46:58.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.117e-06 2024-09-15T03:46:58.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171174+0000", 2024-09-15T03:46:58.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3247000000000001e-05 2024-09-15T03:46:58.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.006 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:58.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171183+0000", 2024-09-15T03:46:58.006 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1570000000000006e-06 2024-09-15T03:46:58.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.006 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:58.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171455+0000", 2024-09-15T03:46:58.006 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00027172399999999999 2024-09-15T03:46:58.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.006 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:58.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171468+0000", 2024-09-15T03:46:58.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3461e-05 2024-09-15T03:46:58.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.007 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:58.007 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171535+0000", 2024-09-15T03:46:58.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6045999999999998e-05 2024-09-15T03:46:58.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.007 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:58.007 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171670+0000", 2024-09-15T03:46:58.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013535399999999999 2024-09-15T03:46:58.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.007 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:58.008 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.176222+0000", 2024-09-15T03:46:58.008 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0045519280000000002 2024-09-15T03:46:58.008 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.008 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.008 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.008 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.008 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.008 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:46:58.009 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:46:58.010 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:46:58.009+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:46:58.110 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:58.109+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:46:58.154 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:46:58.154 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:46:58.154 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-15T03:46:58.154 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:46:58.162 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:46:58.211 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:58.209+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:46:58.265 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:46:58.265 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:46:58.265 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-15T03:46:58.265 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-15T03:46:58.265 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:46:58.272 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:46:58.312 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:46:58.311+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:46:58.412 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:46:58.410+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:46:58.415 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:46:58.415 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-15T03:46:58.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-15T03:46:58.415 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-15T03:46:58.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.416 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:72 3.es0 3:7c6e4a96:::smithi00132172-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-15T03:46:58.416 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:39.404190+0000", 2024-09-15T03:46:58.416 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 19.009013242000002, 2024-09-15T03:46:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0043226150000000001, 2024-09-15T03:46:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:46:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 72 2024-09-15T03:46:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.418 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:58.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.418 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:58.418 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.404190+0000", 2024-09-15T03:46:58.418 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.418 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:58.418 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.404190+0000", 2024-09-15T03:46:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.404190+0000", 2024-09-15T03:46:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9699999999999996e-07 2024-09-15T03:46:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.404192+0000", 2024-09-15T03:46:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.917e-06 2024-09-15T03:46:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.404198+0000", 2024-09-15T03:46:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0739999999999998e-06 2024-09-15T03:46:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.404202+0000", 2024-09-15T03:46:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8500000000000004e-06 2024-09-15T03:46:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.406300+0000", 2024-09-15T03:46:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020981770000000001 2024-09-15T03:46:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for rw locks", 2024-09-15T03:46:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.406381+0000", 2024-09-15T03:46:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0383000000000006e-05 2024-09-15T03:46:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.407550+0000", 2024-09-15T03:46:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001169132 2024-09-15T03:46:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.407636+0000", 2024-09-15T03:46:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6238999999999999e-05 2024-09-15T03:46:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.408512+0000", 2024-09-15T03:46:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00087624600000000001 2024-09-15T03:46:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:84 3.es0 3:79526987:::smithi00132172-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-15T03:46:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:39.417556+0000", 2024-09-15T03:46:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 18.995646829000002, 2024-09-15T03:46:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0043908860000000001, 2024-09-15T03:46:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:46:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 84 2024-09-15T03:46:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.417556+0000", 2024-09-15T03:46:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.425 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:58.425 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.417556+0000", 2024-09-15T03:46:58.425 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.425 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:58.425 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.417557+0000", 2024-09-15T03:46:58.425 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6499999999999999e-07 2024-09-15T03:46:58.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.425 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:58.426 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.417560+0000", 2024-09-15T03:46:58.426 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7019999999999999e-06 2024-09-15T03:46:58.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.426 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:58.426 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.417569+0000", 2024-09-15T03:46:58.426 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8200000000000003e-06 2024-09-15T03:46:58.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.426 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:58.427 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.417573+0000", 2024-09-15T03:46:58.427 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0690000000000003e-06 2024-09-15T03:46:58.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.427 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.427 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.419432+0000", 2024-09-15T03:46:58.427 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001858927 2024-09-15T03:46:58.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.427 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for rw locks", 2024-09-15T03:46:58.427 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.419487+0000", 2024-09-15T03:46:58.427 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5250000000000001e-05 2024-09-15T03:46:58.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.428 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.428 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.420733+0000", 2024-09-15T03:46:58.428 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001245674 2024-09-15T03:46:58.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.428 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:58.428 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.420815+0000", 2024-09-15T03:46:58.428 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1779000000000004e-05 2024-09-15T03:46:58.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.429 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:58.429 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.421947+0000", 2024-09-15T03:46:58.429 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0011322000000000001 2024-09-15T03:46:58.429 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.429 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.429 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.429 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:93 3.es0 3:7f0816e1:::smithi00132172-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-15T03:46:58.429 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:39.427856+0000", 2024-09-15T03:46:58.429 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 18.985347048000001, 2024-09-15T03:46:58.430 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0044151099999999999, 2024-09-15T03:46:58.430 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:58.430 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:46:58.430 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:58.430 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:58.430 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:58.430 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 93 2024-09-15T03:46:58.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.430 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:58.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.430 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:58.430 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.427856+0000", 2024-09-15T03:46:58.430 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.431 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:58.431 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.427856+0000", 2024-09-15T03:46:58.431 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.431 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:58.431 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.427856+0000", 2024-09-15T03:46:58.431 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6899999999999998e-07 2024-09-15T03:46:58.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.431 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:58.432 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.427858+0000", 2024-09-15T03:46:58.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.756e-06 2024-09-15T03:46:58.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.432 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:58.432 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.427878+0000", 2024-09-15T03:46:58.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9565999999999999e-05 2024-09-15T03:46:58.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.432 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:58.432 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.427882+0000", 2024-09-15T03:46:58.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1760000000000003e-06 2024-09-15T03:46:58.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.430119+0000", 2024-09-15T03:46:58.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022372099999999999 2024-09-15T03:46:58.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for rw locks", 2024-09-15T03:46:58.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.430207+0000", 2024-09-15T03:46:58.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7893999999999996e-05 2024-09-15T03:46:58.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.431335+0000", 2024-09-15T03:46:58.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0011275289999999999 2024-09-15T03:46:58.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:58.434 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.431409+0000", 2024-09-15T03:46:58.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4449999999999994e-05 2024-09-15T03:46:58.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:58.435 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:39.432271+0000", 2024-09-15T03:46:58.435 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00086216000000000005 2024-09-15T03:46:58.435 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.435 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.435 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.435 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.435 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:195 3.es0 3:7f0816e1:::smithi00132172-31:head [copy-from ver 4] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:46:58.435 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:41.112078+0000", 2024-09-15T03:46:58.435 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.301125356, 2024-09-15T03:46:58.435 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012535605, 2024-09-15T03:46:58.435 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:58.435 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 195 2024-09-15T03:46:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112078+0000", 2024-09-15T03:46:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:58.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112078+0000", 2024-09-15T03:46:58.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:58.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112080+0000", 2024-09-15T03:46:58.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4399999999999999e-06 2024-09-15T03:46:58.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112090+0000", 2024-09-15T03:46:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7310000000000005e-06 2024-09-15T03:46:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112107+0000", 2024-09-15T03:46:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7557000000000001e-05 2024-09-15T03:46:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112113+0000", 2024-09-15T03:46:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6910000000000002e-06 2024-09-15T03:46:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112176+0000", 2024-09-15T03:46:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3014000000000003e-05 2024-09-15T03:46:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112308+0000", 2024-09-15T03:46:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000131402 2024-09-15T03:46:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118088+0000", 2024-09-15T03:46:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0057801570000000002 2024-09-15T03:46:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118560+0000", 2024-09-15T03:46:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00047206199999999998 2024-09-15T03:46:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118573+0000", 2024-09-15T03:46:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3521e-05 2024-09-15T03:46:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:58.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118667+0000", 2024-09-15T03:46:58.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.323e-05 2024-09-15T03:46:58.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:58.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.119046+0000", 2024-09-15T03:46:58.442 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00037924699999999998 2024-09-15T03:46:58.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:46:58.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.124533+0000", 2024-09-15T03:46:58.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0054869849999999998 2024-09-15T03:46:58.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:58.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.124613+0000", 2024-09-15T03:46:58.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0568000000000005e-05 2024-09-15T03:46:58.443 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.444 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.444 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.445 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:196 3.es0 3:7f0816e1:::smithi00132172-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:46:58.445 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:41.112125+0000", 2024-09-15T03:46:58.445 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.30107774, 2024-09-15T03:46:58.445 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0068455030000000002, 2024-09-15T03:46:58.445 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:58.445 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:46:58.445 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:58.446 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:58.446 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:58.446 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 196 2024-09-15T03:46:58.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.446 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:58.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.447 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:58.447 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112125+0000", 2024-09-15T03:46:58.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.447 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:58.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112125+0000", 2024-09-15T03:46:58.448 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.448 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:58.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112126+0000", 2024-09-15T03:46:58.449 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4099999999999998e-07 2024-09-15T03:46:58.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.449 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:58.449 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112130+0000", 2024-09-15T03:46:58.449 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6909999999999999e-06 2024-09-15T03:46:58.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.449 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:58.450 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112138+0000", 2024-09-15T03:46:58.450 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6969999999999999e-06 2024-09-15T03:46:58.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.450 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:58.450 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112143+0000", 2024-09-15T03:46:58.450 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1620000000000001e-06 2024-09-15T03:46:58.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.450 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.450 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112506+0000", 2024-09-15T03:46:58.450 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00036344600000000002 2024-09-15T03:46:58.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for blocked object", 2024-09-15T03:46:58.451 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112569+0000", 2024-09-15T03:46:58.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3367000000000005e-05 2024-09-15T03:46:58.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.451 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118741+0000", 2024-09-15T03:46:58.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0061715700000000004 2024-09-15T03:46:58.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:58.451 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118852+0000", 2024-09-15T03:46:58.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011062500000000001 2024-09-15T03:46:58.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.452 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:58.452 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118971+0000", 2024-09-15T03:46:58.452 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011920400000000001 2024-09-15T03:46:58.452 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.452 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.452 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.452 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(osd.7.17:1 3.es0 3:7e6c26d0:::smithi00132172-14:head [assert-version v22,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:46:58.452 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:45.131370+0000", 2024-09-15T03:46:58.452 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 13.281832598999999, 2024-09-15T03:46:58.452 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0052243280000000003, 2024-09-15T03:46:58.453 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:58.453 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:46:58.453 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:58.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.453 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:58.453 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.131370+0000", 2024-09-15T03:46:58.453 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.453 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:58.453 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.131370+0000", 2024-09-15T03:46:58.454 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.455 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:58.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.131372+0000", 2024-09-15T03:46:58.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9420000000000002e-06 2024-09-15T03:46:58.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.455 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:58.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.131379+0000", 2024-09-15T03:46:58.456 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8959999999999997e-06 2024-09-15T03:46:58.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.456 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:58.456 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.131398+0000", 2024-09-15T03:46:58.456 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8366999999999999e-05 2024-09-15T03:46:58.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.457 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:58.457 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.131406+0000", 2024-09-15T03:46:58.457 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0800000000000006e-06 2024-09-15T03:46:58.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.457 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.461 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.131465+0000", 2024-09-15T03:46:58.461 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8924e-05 2024-09-15T03:46:58.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.461 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:58.461 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.131582+0000", 2024-09-15T03:46:58.461 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011693600000000001 2024-09-15T03:46:58.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.461 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:58.461 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.136595+0000", 2024-09-15T03:46:58.461 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.005013183 2024-09-15T03:46:58.462 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.462 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.462 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.462 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.bs5 38/31 ECSubWrite(tid=36, reqid=client.4521.0:220, at_version=38'23, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:46:58.462 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:45.147807+0000", 2024-09-15T03:46:58.462 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 13.265396020000001, 2024-09-15T03:46:58.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0046676189999999996, 2024-09-15T03:46:58.462 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:58.462 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:58.462 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:58.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:58.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.147807+0000", 2024-09-15T03:46:58.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:58.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.147807+0000", 2024-09-15T03:46:58.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:58.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.147809+0000", 2024-09-15T03:46:58.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7600000000000001e-06 2024-09-15T03:46:58.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:58.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148346+0000", 2024-09-15T03:46:58.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00053685099999999997 2024-09-15T03:46:58.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:58.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148372+0000", 2024-09-15T03:46:58.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6094000000000001e-05 2024-09-15T03:46:58.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:58.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148382+0000", 2024-09-15T03:46:58.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0008e-05 2024-09-15T03:46:58.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.465 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.465 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148438+0000", 2024-09-15T03:46:58.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6242000000000001e-05 2024-09-15T03:46:58.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.465 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:58.465 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148450+0000", 2024-09-15T03:46:58.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2189e-05 2024-09-15T03:46:58.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.465 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:58.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148853+0000", 2024-09-15T03:46:58.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00040301399999999999 2024-09-15T03:46:58.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:58.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148880+0000", 2024-09-15T03:46:58.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6375000000000001e-05 2024-09-15T03:46:58.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:58.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.149135+0000", 2024-09-15T03:46:58.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00025514599999999999 2024-09-15T03:46:58.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:58.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.149161+0000", 2024-09-15T03:46:58.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6427999999999998e-05 2024-09-15T03:46:58.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:58.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.152475+0000", 2024-09-15T03:46:58.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0033135119999999998 2024-09-15T03:46:58.467 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.467 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.467 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.468 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:227 3.es0 3:77a7da34:::smithi00132172-4:head [append 0~704512] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:46:58.468 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:45.152975+0000", 2024-09-15T03:46:58.468 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 13.260227793, 2024-09-15T03:46:58.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0058200999999999999, 2024-09-15T03:46:58.468 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:58.468 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:58.468 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:58.468 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:58.468 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:58.468 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 227 2024-09-15T03:46:58.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.468 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:58.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:58.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.152975+0000", 2024-09-15T03:46:58.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:58.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.152975+0000", 2024-09-15T03:46:58.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:58.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.152977+0000", 2024-09-15T03:46:58.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7799999999999999e-06 2024-09-15T03:46:58.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:58.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.153247+0000", 2024-09-15T03:46:58.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00027026799999999999 2024-09-15T03:46:58.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:58.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.153262+0000", 2024-09-15T03:46:58.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4966999999999999e-05 2024-09-15T03:46:58.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:58.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.153268+0000", 2024-09-15T03:46:58.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4970000000000001e-06 2024-09-15T03:46:58.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.153325+0000", 2024-09-15T03:46:58.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7383999999999998e-05 2024-09-15T03:46:58.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:58.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.153443+0000", 2024-09-15T03:46:58.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000117327 2024-09-15T03:46:58.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:58.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155145+0000", 2024-09-15T03:46:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0017023609999999999 2024-09-15T03:46:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155506+0000", 2024-09-15T03:46:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00036064300000000001 2024-09-15T03:46:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155529+0000", 2024-09-15T03:46:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3229000000000001e-05 2024-09-15T03:46:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155709+0000", 2024-09-15T03:46:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018021000000000001 2024-09-15T03:46:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.157102+0000", 2024-09-15T03:46:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001392534 2024-09-15T03:46:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:46:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158717+0000", 2024-09-15T03:46:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001615989 2024-09-15T03:46:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158795+0000", 2024-09-15T03:46:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7911000000000007e-05 2024-09-15T03:46:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:228 3.es0 3:77a7da34:::smithi00132172-4:head [append 0~376832] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:46:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:45.155394+0000", 2024-09-15T03:46:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 13.25780915, 2024-09-15T03:46:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0046305629999999999, 2024-09-15T03:46:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 228 2024-09-15T03:46:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155394+0000", 2024-09-15T03:46:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155394+0000", 2024-09-15T03:46:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155396+0000", 2024-09-15T03:46:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3070000000000001e-06 2024-09-15T03:46:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155562+0000", 2024-09-15T03:46:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016546599999999999 2024-09-15T03:46:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155578+0000", 2024-09-15T03:46:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6392000000000002e-05 2024-09-15T03:46:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155585+0000", 2024-09-15T03:46:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.267e-06 2024-09-15T03:46:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155783+0000", 2024-09-15T03:46:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000197784 2024-09-15T03:46:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155887+0000", 2024-09-15T03:46:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000104066 2024-09-15T03:46:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.156917+0000", 2024-09-15T03:46:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001029709 2024-09-15T03:46:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.157046+0000", 2024-09-15T03:46:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012913399999999999 2024-09-15T03:46:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.157343+0000", 2024-09-15T03:46:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000296483 2024-09-15T03:46:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.157619+0000", 2024-09-15T03:46:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000276562 2024-09-15T03:46:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158027+0000", 2024-09-15T03:46:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00040807799999999998 2024-09-15T03:46:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:46:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.159959+0000", 2024-09-15T03:46:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001931938 2024-09-15T03:46:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.160025+0000", 2024-09-15T03:46:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5376999999999994e-05 2024-09-15T03:46:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:229 3.es0 3:77a7da34:::smithi00132172-4:head [setxattr _header (55) in=7b] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:46:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:45.155618+0000", 2024-09-15T03:46:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 13.257585393999999, 2024-09-15T03:46:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0045676670000000001, 2024-09-15T03:46:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:46:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 229 2024-09-15T03:46:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155618+0000", 2024-09-15T03:46:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155618+0000", 2024-09-15T03:46:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155618+0000", 2024-09-15T03:46:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6499999999999998e-07 2024-09-15T03:46:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155623+0000", 2024-09-15T03:46:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7219999999999999e-06 2024-09-15T03:46:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155683+0000", 2024-09-15T03:46:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9769999999999999e-05 2024-09-15T03:46:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155690+0000", 2024-09-15T03:46:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4580000000000004e-06 2024-09-15T03:46:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.157123+0000", 2024-09-15T03:46:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014322429999999999 2024-09-15T03:46:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.157217+0000", 2024-09-15T03:46:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4394999999999993e-05 2024-09-15T03:46:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.157790+0000", 2024-09-15T03:46:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00057274400000000003 2024-09-15T03:46:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.157892+0000", 2024-09-15T03:46:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010196000000000001 2024-09-15T03:46:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.157903+0000", 2024-09-15T03:46:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0995999999999999e-05 2024-09-15T03:46:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.157956+0000", 2024-09-15T03:46:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2986999999999999e-05 2024-09-15T03:46:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158059+0000", 2024-09-15T03:46:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0001032 2024-09-15T03:46:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:46:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.160106+0000", 2024-09-15T03:46:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020472720000000002 2024-09-15T03:46:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.160185+0000", 2024-09-15T03:46:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9154999999999997e-05 2024-09-15T03:46:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:230 3.es0 3:77a7da34:::smithi00132172-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:46:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:45.155698+0000", 2024-09-15T03:46:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 13.257504938, 2024-09-15T03:46:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0062491680000000003, 2024-09-15T03:46:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:46:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:46:58.489 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:46:58.489 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:46:58.489 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 230 2024-09-15T03:46:58.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.489 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:58.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:58.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155698+0000", 2024-09-15T03:46:58.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:58.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155698+0000", 2024-09-15T03:46:58.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:58.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155699+0000", 2024-09-15T03:46:58.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2900000000000003e-07 2024-09-15T03:46:58.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:58.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155702+0000", 2024-09-15T03:46:58.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0809999999999998e-06 2024-09-15T03:46:58.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:58.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155708+0000", 2024-09-15T03:46:58.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0440000000000001e-06 2024-09-15T03:46:58.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:58.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155712+0000", 2024-09-15T03:46:58.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.827e-06 2024-09-15T03:46:58.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158184+0000", 2024-09-15T03:46:58.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024722260000000001 2024-09-15T03:46:58.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for rw locks", 2024-09-15T03:46:58.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158303+0000", 2024-09-15T03:46:58.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118845 2024-09-15T03:46:58.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.160521+0000", 2024-09-15T03:46:58.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022177239999999999 2024-09-15T03:46:58.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:46:58.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.160616+0000", 2024-09-15T03:46:58.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5948999999999999e-05 2024-09-15T03:46:58.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.493 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:58.493 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.161947+0000", 2024-09-15T03:46:58.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001330843 2024-09-15T03:46:58.493 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.493 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.493 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.493 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.0s1 39/31 ECSubWrite(tid=43, reqid=client.4521.0:240, at_version=39'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:46:58.493 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:46.149555+0000", 2024-09-15T03:46:58.493 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.263648257, 2024-09-15T03:46:58.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0056873840000000002, 2024-09-15T03:46:58.494 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:58.494 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:58.494 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:58.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:58.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149555+0000", 2024-09-15T03:46:58.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:58.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149555+0000", 2024-09-15T03:46:58.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149555+0000", 2024-09-15T03:46:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2300000000000002e-07 2024-09-15T03:46:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149613+0000", 2024-09-15T03:46:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7519000000000003e-05 2024-09-15T03:46:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149701+0000", 2024-09-15T03:46:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8067999999999998e-05 2024-09-15T03:46:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149778+0000", 2024-09-15T03:46:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7546000000000006e-05 2024-09-15T03:46:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149835+0000", 2024-09-15T03:46:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6489000000000001e-05 2024-09-15T03:46:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149845+0000", 2024-09-15T03:46:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0583e-05 2024-09-15T03:46:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150114+0000", 2024-09-15T03:46:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026860399999999998 2024-09-15T03:46:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150145+0000", 2024-09-15T03:46:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0617999999999997e-05 2024-09-15T03:46:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150194+0000", 2024-09-15T03:46:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9573999999999997e-05 2024-09-15T03:46:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.151015+0000", 2024-09-15T03:46:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00082130099999999998 2024-09-15T03:46:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.155242+0000", 2024-09-15T03:46:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0042267590000000001 2024-09-15T03:46:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.0s1 39/31 ECSubWrite(tid=45, reqid=client.4521.0:241, at_version=39'17, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:46:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:46.153423+0000", 2024-09-15T03:46:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.259780159, 2024-09-15T03:46:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0046692590000000003, 2024-09-15T03:46:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153423+0000", 2024-09-15T03:46:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153423+0000", 2024-09-15T03:46:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153423+0000", 2024-09-15T03:46:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6100000000000001e-07 2024-09-15T03:46:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153445+0000", 2024-09-15T03:46:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1447999999999999e-05 2024-09-15T03:46:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153456+0000", 2024-09-15T03:46:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1199000000000001e-05 2024-09-15T03:46:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153463+0000", 2024-09-15T03:46:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6089999999999997e-06 2024-09-15T03:46:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153597+0000", 2024-09-15T03:46:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013469799999999999 2024-09-15T03:46:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153609+0000", 2024-09-15T03:46:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1293e-05 2024-09-15T03:46:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153733+0000", 2024-09-15T03:46:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000124127 2024-09-15T03:46:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153776+0000", 2024-09-15T03:46:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2731999999999997e-05 2024-09-15T03:46:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:58.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153821+0000", 2024-09-15T03:46:58.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5250000000000002e-05 2024-09-15T03:46:58.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:58.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.155216+0000", 2024-09-15T03:46:58.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001395164 2024-09-15T03:46:58.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:58.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.158092+0000", 2024-09-15T03:46:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0028761780000000001 2024-09-15T03:46:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.0s1 39/31 ECSubWrite(tid=46, reqid=client.4521.0:242, at_version=39'18, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:46:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:46.153470+0000", 2024-09-15T03:46:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.259733497999999, 2024-09-15T03:46:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0047847699999999998, 2024-09-15T03:46:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153470+0000", 2024-09-15T03:46:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153470+0000", 2024-09-15T03:46:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153470+0000", 2024-09-15T03:46:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.34e-07 2024-09-15T03:46:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153491+0000", 2024-09-15T03:46:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0565999999999999e-05 2024-09-15T03:46:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153499+0000", 2024-09-15T03:46:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9850000000000006e-06 2024-09-15T03:46:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153504+0000", 2024-09-15T03:46:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2900000000000002e-06 2024-09-15T03:46:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153776+0000", 2024-09-15T03:46:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000271959 2024-09-15T03:46:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153784+0000", 2024-09-15T03:46:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3410000000000003e-06 2024-09-15T03:46:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.155184+0000", 2024-09-15T03:46:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014001910000000001 2024-09-15T03:46:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.155579+0000", 2024-09-15T03:46:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00039442399999999999 2024-09-15T03:46:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.155844+0000", 2024-09-15T03:46:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026551999999999999 2024-09-15T03:46:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.155874+0000", 2024-09-15T03:46:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9388999999999999e-05 2024-09-15T03:46:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.158254+0000", 2024-09-15T03:46:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002380671 2024-09-15T03:46:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.3s5 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:46:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:46.170728+0000", 2024-09-15T03:46:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.242475224, 2024-09-15T03:46:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.004627585, 2024-09-15T03:46:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.170728+0000", 2024-09-15T03:46:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.170728+0000", 2024-09-15T03:46:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.170729+0000", 2024-09-15T03:46:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2999999999999999e-07 2024-09-15T03:46:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:58.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171187+0000", 2024-09-15T03:46:58.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00045862099999999999 2024-09-15T03:46:58.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:58.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171205+0000", 2024-09-15T03:46:58.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8073000000000001e-05 2024-09-15T03:46:58.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:58.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171212+0000", 2024-09-15T03:46:58.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9190000000000002e-06 2024-09-15T03:46:58.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171305+0000", 2024-09-15T03:46:58.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2961000000000007e-05 2024-09-15T03:46:58.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:58.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171315+0000", 2024-09-15T03:46:58.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6900000000000004e-06 2024-09-15T03:46:58.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:58.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171560+0000", 2024-09-15T03:46:58.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024454499999999998 2024-09-15T03:46:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171582+0000", 2024-09-15T03:46:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2246999999999999e-05 2024-09-15T03:46:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171710+0000", 2024-09-15T03:46:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012822300000000001 2024-09-15T03:46:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171733+0000", 2024-09-15T03:46:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.264e-05 2024-09-15T03:46:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.175355+0000", 2024-09-15T03:46:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0036227360000000001 2024-09-15T03:46:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s4 49/31 ECSubWrite(tid=80, reqid=client.4521.0:320, at_version=49'15, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:46:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.841069+0000", 2024-09-15T03:46:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 2.572133794, 2024-09-15T03:46:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0054374860000000001, 2024-09-15T03:46:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841069+0000", 2024-09-15T03:46:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841069+0000", 2024-09-15T03:46:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841070+0000", 2024-09-15T03:46:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.27e-07 2024-09-15T03:46:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841549+0000", 2024-09-15T03:46:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000479166 2024-09-15T03:46:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841571+0000", 2024-09-15T03:46:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2061000000000001e-05 2024-09-15T03:46:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841579+0000", 2024-09-15T03:46:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9999999999999996e-06 2024-09-15T03:46:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841688+0000", 2024-09-15T03:46:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000109467 2024-09-15T03:46:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:58.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841700+0000", 2024-09-15T03:46:58.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1663e-05 2024-09-15T03:46:58.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:58.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.843886+0000", 2024-09-15T03:46:58.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021862169999999998 2024-09-15T03:46:58.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:58.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844020+0000", 2024-09-15T03:46:58.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013349499999999999 2024-09-15T03:46:58.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:58.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844124+0000", 2024-09-15T03:46:58.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010440100000000001 2024-09-15T03:46:58.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:58.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844156+0000", 2024-09-15T03:46:58.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1653999999999999e-05 2024-09-15T03:46:58.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:58.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846507+0000", 2024-09-15T03:46:58.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002351035 2024-09-15T03:46:58.522 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.522 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.522 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.522 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s1 49/31 ECSubWrite(tid=30, reqid=client.4521.0:325, at_version=49'12, trim_to=0'0, roll_forward_to=40'10))", 2024-09-15T03:46:58.522 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.844978+0000", 2024-09-15T03:46:58.522 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 2.5682251420000002, 2024-09-15T03:46:58.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0049714099999999999, 2024-09-15T03:46:58.522 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:58.522 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:58.523 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:58.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:58.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844978+0000", 2024-09-15T03:46:58.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:58.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844978+0000", 2024-09-15T03:46:58.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:58.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844978+0000", 2024-09-15T03:46:58.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5199999999999998e-07 2024-09-15T03:46:58.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:58.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845006+0000", 2024-09-15T03:46:58.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7492e-05 2024-09-15T03:46:58.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:58.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845024+0000", 2024-09-15T03:46:58.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8017e-05 2024-09-15T03:46:58.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:58.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845031+0000", 2024-09-15T03:46:58.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5830000000000001e-06 2024-09-15T03:46:58.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845277+0000", 2024-09-15T03:46:58.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024547499999999999 2024-09-15T03:46:58.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:58.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845286+0000", 2024-09-15T03:46:58.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5710000000000004e-06 2024-09-15T03:46:58.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:58.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846025+0000", 2024-09-15T03:46:58.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00073880099999999998 2024-09-15T03:46:58.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:58.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846059+0000", 2024-09-15T03:46:58.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4032999999999998e-05 2024-09-15T03:46:58.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:58.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846135+0000", 2024-09-15T03:46:58.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6045999999999997e-05 2024-09-15T03:46:58.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:58.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846367+0000", 2024-09-15T03:46:58.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023208899999999999 2024-09-15T03:46:58.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:58.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849949+0000", 2024-09-15T03:46:58.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0035819509999999999 2024-09-15T03:46:58.528 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.528 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.528 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.528 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.cs7 49/28 ECSubWrite(tid=15, reqid=client.4521.0:337, at_version=49'10, trim_to=0'0, roll_forward_to=43'8))", 2024-09-15T03:46:58.528 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.858004+0000", 2024-09-15T03:46:58.528 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 2.555199268, 2024-09-15T03:46:58.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0048927559999999998, 2024-09-15T03:46:58.528 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:58.528 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:58.528 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:58.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:58.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858004+0000", 2024-09-15T03:46:58.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:58.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858004+0000", 2024-09-15T03:46:58.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:58.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858004+0000", 2024-09-15T03:46:58.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2300000000000001e-07 2024-09-15T03:46:58.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:58.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858050+0000", 2024-09-15T03:46:58.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5488999999999998e-05 2024-09-15T03:46:58.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:58.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858069+0000", 2024-09-15T03:46:58.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9457999999999999e-05 2024-09-15T03:46:58.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858116+0000", 2024-09-15T03:46:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6613000000000002e-05 2024-09-15T03:46:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858284+0000", 2024-09-15T03:46:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016815499999999999 2024-09-15T03:46:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858293+0000", 2024-09-15T03:46:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3279999999999995e-06 2024-09-15T03:46:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858569+0000", 2024-09-15T03:46:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00027534100000000002 2024-09-15T03:46:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858602+0000", 2024-09-15T03:46:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3577000000000001e-05 2024-09-15T03:46:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858650+0000", 2024-09-15T03:46:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7808e-05 2024-09-15T03:46:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858670+0000", 2024-09-15T03:46:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0224000000000002e-05 2024-09-15T03:46:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.862897+0000", 2024-09-15T03:46:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00422624 2024-09-15T03:46:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s4 49/31 ECSubWrite(tid=87, reqid=client.4521.0:335, at_version=49'21, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:46:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.858523+0000", 2024-09-15T03:46:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 2.5546802099999999, 2024-09-15T03:46:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0045008510000000002, 2024-09-15T03:46:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858523+0000", 2024-09-15T03:46:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858523+0000", 2024-09-15T03:46:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858524+0000", 2024-09-15T03:46:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3200000000000004e-07 2024-09-15T03:46:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858548+0000", 2024-09-15T03:46:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3853e-05 2024-09-15T03:46:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858559+0000", 2024-09-15T03:46:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1501999999999999e-05 2024-09-15T03:46:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858566+0000", 2024-09-15T03:46:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8519999999999998e-06 2024-09-15T03:46:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858619+0000", 2024-09-15T03:46:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3059000000000001e-05 2024-09-15T03:46:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858628+0000", 2024-09-15T03:46:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1330000000000005e-06 2024-09-15T03:46:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858724+0000", 2024-09-15T03:46:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5452999999999996e-05 2024-09-15T03:46:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858736+0000", 2024-09-15T03:46:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2320999999999999e-05 2024-09-15T03:46:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859075+0000", 2024-09-15T03:46:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00033932800000000001 2024-09-15T03:46:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859117+0000", 2024-09-15T03:46:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2252999999999997e-05 2024-09-15T03:46:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.863024+0000", 2024-09-15T03:46:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0039062649999999999 2024-09-15T03:46:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.3s5 51/31 ECSubWrite(tid=67, reqid=client.4521.0:350, at_version=51'33, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:46:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:57.843550+0000", 2024-09-15T03:46:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.56965321599999996, 2024-09-15T03:46:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0044606899999999998, 2024-09-15T03:46:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:46:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:46:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:46:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:46:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:57.843550+0000", 2024-09-15T03:46:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:46:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:57.843550+0000", 2024-09-15T03:46:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:46:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:46:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:57.843551+0000", 2024-09-15T03:46:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.139e-06 2024-09-15T03:46:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:46:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:57.843576+0000", 2024-09-15T03:46:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4834e-05 2024-09-15T03:46:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:46:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:57.843593+0000", 2024-09-15T03:46:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7269e-05 2024-09-15T03:46:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:46:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:57.843601+0000", 2024-09-15T03:46:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2609999999999993e-06 2024-09-15T03:46:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:46:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:57.844128+0000", 2024-09-15T03:46:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00052642300000000004 2024-09-15T03:46:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:46:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:57.844138+0000", 2024-09-15T03:46:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0217000000000001e-05 2024-09-15T03:46:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:46:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:57.844243+0000", 2024-09-15T03:46:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000104831 2024-09-15T03:46:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:46:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:57.844253+0000", 2024-09-15T03:46:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0299000000000001e-05 2024-09-15T03:46:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:46:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:57.844295+0000", 2024-09-15T03:46:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1903000000000002e-05 2024-09-15T03:46:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:46:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:57.844314+0000", 2024-09-15T03:46:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8597000000000002e-05 2024-09-15T03:46:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:46:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:46:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:46:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:57.848011+0000", 2024-09-15T03:46:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0036969170000000001 2024-09-15T03:46:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:46:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:46:58.545 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:46:58.546 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:46:58.547 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:58.545+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:58.647 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:58.646+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:58.689 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:46:58.690 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:46:58.690 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-15T03:46:58.690 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:46:58.697 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:46:58.748 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:58.746+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:46:58.798 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:46:58.799 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:46:58.799 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-15T03:46:58.799 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-15T03:46:58.799 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:46:58.807 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:46:58.842 INFO:tasks.rados.rados.0.smithi001.stdout:140: done (7 left) 2024-09-15T03:46:58.842 INFO:tasks.rados.rados.0.smithi001.stdout:141: snap_create 2024-09-15T03:46:58.843 INFO:tasks.rados.rados.0.smithi001.stdout:137: finishing rollback tid 0 to smithi00132172-25 2024-09-15T03:46:58.843 INFO:tasks.rados.rados.0.smithi001.stdout:137: finishing rollback tid 1 to smithi00132172-25 2024-09-15T03:46:58.843 INFO:tasks.rados.rados.0.smithi001.stdout:137: finishing rollback tid 2 to smithi00132172-25 2024-09-15T03:46:58.844 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 25 v 34 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-15T03:46:58.844 INFO:tasks.rados.rados.0.smithi001.stdout:136: finishing rollback tid 0 to smithi00132172-5 2024-09-15T03:46:58.844 INFO:tasks.rados.rados.0.smithi001.stdout:136: finishing rollback tid 1 to smithi00132172-5 2024-09-15T03:46:58.844 INFO:tasks.rados.rados.0.smithi001.stdout:136: finishing rollback tid 2 to smithi00132172-5 2024-09-15T03:46:58.844 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 5 v 14 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-15T03:46:58.847 INFO:tasks.rados.rados.0.smithi001.stdout:126: finishing write tid 5 to smithi00132172-50 2024-09-15T03:46:58.847 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 50 v 18 (ObjNum 72 snap 12 seq_num 72) dirty exists 2024-09-15T03:46:58.847 INFO:tasks.rados.rados.0.smithi001.stdout:126: left oid 50 (ObjNum 72 snap 12 seq_num 72) 2024-09-15T03:46:58.849 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:58.847+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:46:58.850 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T03:46:58.858 INFO:tasks.rados.rados.0.smithi001.stdout:130: finishing rollback tid 0 to smithi00132172-43 2024-09-15T03:46:58.858 INFO:tasks.rados.rados.0.smithi001.stdout:130: finishing rollback tid 1 to smithi00132172-43 2024-09-15T03:46:58.858 INFO:tasks.rados.rados.0.smithi001.stdout:130: finishing rollback tid 2 to smithi00132172-43 2024-09-15T03:46:58.859 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 43 v 22 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-15T03:46:58.861 INFO:tasks.rados.rados.0.smithi001.stdout:138: finishing write tid 1 to smithi00132172-11 2024-09-15T03:46:58.861 INFO:tasks.rados.rados.0.smithi001.stdout:138: finishing write tid 2 to smithi00132172-11 2024-09-15T03:46:58.866 INFO:tasks.rados.rados.0.smithi001.stdout:134: finishing write tid 1 to smithi00132172-27 2024-09-15T03:46:58.866 INFO:tasks.rados.rados.0.smithi001.stdout:134: finishing write tid 2 to smithi00132172-27 2024-09-15T03:46:58.867 INFO:tasks.rados.rados.0.smithi001.stdout:134: finishing write tid 3 to smithi00132172-27 2024-09-15T03:46:58.867 INFO:tasks.rados.rados.0.smithi001.stdout:138: finishing write tid 3 to smithi00132172-11 2024-09-15T03:46:58.867 INFO:tasks.rados.rados.0.smithi001.stdout:138: finishing write tid 4 to smithi00132172-11 2024-09-15T03:46:58.869 INFO:tasks.rados.rados.0.smithi001.stdout:138: finishing write tid 5 to smithi00132172-11 2024-09-15T03:46:58.869 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 11 v 27 (ObjNum 78 snap 13 seq_num 78) dirty exists 2024-09-15T03:46:58.869 INFO:tasks.rados.rados.0.smithi001.stdout:138: left oid 11 (ObjNum 78 snap 13 seq_num 78) 2024-09-15T03:46:58.870 INFO:tasks.rados.rados.0.smithi001.stdout:134: finishing write tid 4 to smithi00132172-27 2024-09-15T03:46:58.871 INFO:tasks.rados.rados.0.smithi001.stdout:134: finishing write tid 5 to smithi00132172-27 2024-09-15T03:46:58.871 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 27 v 23 (ObjNum 77 snap 12 seq_num 77) dirty exists 2024-09-15T03:46:58.871 INFO:tasks.rados.rados.0.smithi001.stdout:134: left oid 27 (ObjNum 77 snap 12 seq_num 77) 2024-09-15T03:46:58.876 INFO:tasks.rados.rados.0.smithi001.stdout:139: expect (ObjNum 71 snap 9 seq_num 71) 2024-09-15T03:46:58.877 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:46:58.880 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: filestore_inject_stall = '' (not observed, change may require restart) osd_enable_op_tracker = '' 2024-09-15T03:46:58.893 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:46:58.893 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:46:58.921 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:46:58.922 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:46:58.927 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:58.927 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:58.937 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:46:58.937 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:46:58.937 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:46:58.952 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:46:58.952 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:46:58.980 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:46:58.980 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:46:58.981 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:46:58.980+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:46:58.995 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:46:58.995 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:46:59.010 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:46:59.010 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:46:59.025 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:46:59.025 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:46:59.039 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:46:59.040 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:46:59.048 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:59.049 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:59.050 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:46:59.049+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:46:59.055 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:46:59.055 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:46:59.058 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:46:59.150 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:46:59.149+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:46:59.200 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:59.200 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:59.208 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:46:59.252 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:46:59.251+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:46:59.308 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:59.308 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:59.317 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:46:59.360 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:59.359+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:46:59.417 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:59.417 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:59.426 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:46:59.467 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:59.465+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:46:59.555 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:46:59.553+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:46:59.566 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:59.566 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:59.575 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:46:59.656 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:46:59.655+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:46:59.716 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:59.716 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:59.724 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:46:59.768 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:46:59.766+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:46:59.825 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:59.825 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:59.833 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:46:59.849 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-15T03:46:59.850 INFO:tasks.rados.rados.0.smithi001.stdout:126: done (7 left) 2024-09-15T03:46:59.850 INFO:tasks.rados.rados.0.smithi001.stdout:130: done (6 left) 2024-09-15T03:46:59.850 INFO:tasks.rados.rados.0.smithi001.stdout:134: done (5 left) 2024-09-15T03:46:59.850 INFO:tasks.rados.rados.0.smithi001.stdout:136: done (4 left) 2024-09-15T03:46:59.850 INFO:tasks.rados.rados.0.smithi001.stdout:137: done (3 left) 2024-09-15T03:46:59.850 INFO:tasks.rados.rados.0.smithi001.stdout:138: done (2 left) 2024-09-15T03:46:59.850 INFO:tasks.rados.rados.0.smithi001.stdout:139: done (1 left) 2024-09-15T03:46:59.850 INFO:tasks.rados.rados.0.smithi001.stdout:141: done (0 left) 2024-09-15T03:46:59.850 INFO:tasks.rados.rados.0.smithi001.stdout:142: snap_create 2024-09-15T03:46:59.859 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:46:59.857+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:46:59.958 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:46:59.958+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:46:59.974 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:46:59.974 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:46:59.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:47:00.059 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:00.058+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:00.084 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:00.084 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:00.092 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:47:00.161 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:00.159+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:00.233 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:00.234 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:00.242 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:47:00.285 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:00.284+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:00.303 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-15T03:47:00.303 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T03:47:00.303 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-15T03:47:00.303 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-15T03:47:00.342 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:00.343 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:00.351 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:47:00.362 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:00.360+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:00.462 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:00.461+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:00.493 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:00.493 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:00.501 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:47:00.563 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:00.563+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:00.589 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:47:00.589 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:00.589 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 53, 2024-09-15T03:47:00.589 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "f27b540e-e5e3-4e95-afe8-c95ee66b0a62", 2024-09-15T03:47:00.589 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-09-15T03:46:09.843094+0000", 2024-09-15T03:47:00.590 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-09-15T03:46:59.841023+0000", 2024-09-15T03:47:00.590 INFO:teuthology.orchestra.run.smithi001.stdout: "last_up_change": "2024-09-15T03:46:15.871985+0000", 2024-09-15T03:47:00.590 INFO:teuthology.orchestra.run.smithi001.stdout: "last_in_change": "2024-09-15T03:46:38.089065+0000", 2024-09-15T03:47:00.590 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-15T03:47:00.590 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_num": 5806080, 2024-09-15T03:47:00.590 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_set": [ 2024-09-15T03:47:00.590 INFO:teuthology.orchestra.run.smithi001.stdout: "nodeep-scrub", 2024-09-15T03:47:00.590 INFO:teuthology.orchestra.run.smithi001.stdout: "noscrub", 2024-09-15T03:47:00.590 INFO:teuthology.orchestra.run.smithi001.stdout: "pglog_hardlimit", 2024-09-15T03:47:00.590 INFO:teuthology.orchestra.run.smithi001.stdout: "purged_snapdirs", 2024-09-15T03:47:00.590 INFO:teuthology.orchestra.run.smithi001.stdout: "recovery_deletes", 2024-09-15T03:47:00.590 INFO:teuthology.orchestra.run.smithi001.stdout: "sortbitwise" 2024-09-15T03:47:00.590 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:47:00.591 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_version": 5, 2024-09-15T03:47:00.591 INFO:teuthology.orchestra.run.smithi001.stdout: "full_ratio": 0.94999998807907104, 2024-09-15T03:47:00.591 INFO:teuthology.orchestra.run.smithi001.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-15T03:47:00.591 INFO:teuthology.orchestra.run.smithi001.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-15T03:47:00.591 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_snapshot": "", 2024-09-15T03:47:00.591 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_max": 3, 2024-09-15T03:47:00.591 INFO:teuthology.orchestra.run.smithi001.stdout: "max_osd": 12, 2024-09-15T03:47:00.591 INFO:teuthology.orchestra.run.smithi001.stdout: "require_min_compat_client": "luminous", 2024-09-15T03:47:00.591 INFO:teuthology.orchestra.run.smithi001.stdout: "min_compat_client": "luminous", 2024-09-15T03:47:00.591 INFO:teuthology.orchestra.run.smithi001.stdout: "require_osd_release": "quincy", 2024-09-15T03:47:00.591 INFO:teuthology.orchestra.run.smithi001.stdout: "pools": [ 2024-09-15T03:47:00.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.591 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 1, 2024-09-15T03:47:00.591 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": ".mgr", 2024-09-15T03:47:00.592 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-15T03:46:16.185530+0000", 2024-09-15T03:47:00.592 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-09-15T03:47:00.592 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool", 2024-09-15T03:47:00.592 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-15T03:47:00.592 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-15T03:47:00.592 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 1, 2024-09-15T03:47:00.592 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-15T03:47:00.592 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-15T03:47:00.592 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-15T03:47:00.592 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T03:47:00.592 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T03:47:00.592 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-15T03:47:00.593 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-15T03:47:00.593 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 1, 2024-09-15T03:47:00.593 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 1, 2024-09-15T03:47:00.593 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 1, 2024-09-15T03:47:00.593 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 1, 2024-09-15T03:47:00.593 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 1, 2024-09-15T03:47:00.593 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-15T03:47:00.593 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-15T03:47:00.593 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-15T03:47:00.593 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-15T03:47:00.593 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-15T03:47:00.593 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-15T03:47:00.593 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-15T03:47:00.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.594 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "20", 2024-09-15T03:47:00.594 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-15T03:47:00.594 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T03:47:00.594 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T03:47:00.594 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-15T03:47:00.594 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-15T03:47:00.594 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 0, 2024-09-15T03:47:00.594 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 0, 2024-09-15T03:47:00.595 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-15T03:47:00.595 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-15T03:47:00.595 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-15T03:47:00.595 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-15T03:47:00.595 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-15T03:47:00.595 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-15T03:47:00.595 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-15T03:47:00.595 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-15T03:47:00.595 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-15T03:47:00.595 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-15T03:47:00.595 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-15T03:47:00.595 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T03:47:00.595 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T03:47:00.596 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T03:47:00.596 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-15T03:47:00.596 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-15T03:47:00.596 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-15T03:47:00.596 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-15T03:47:00.596 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-15T03:47:00.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.596 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-15T03:47:00.596 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-15T03:47:00.596 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-15T03:47:00.596 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-15T03:47:00.596 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-15T03:47:00.596 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T03:47:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-15T03:47:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-15T03:47:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-15T03:47:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-15T03:47:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-15T03:47:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: "options": { 2024-09-15T03:47:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_max": 32, 2024-09-15T03:47:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_min": 1 2024-09-15T03:47:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-15T03:47:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": {} 2024-09-15T03:47:00.598 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.598 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 2, 2024-09-15T03:47:00.599 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": "rbd", 2024-09-15T03:47:00.599 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-15T03:46:22.284406+0000", 2024-09-15T03:47:00.599 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 8193, 2024-09-15T03:47:00.599 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-15T03:47:00.599 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-15T03:47:00.599 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-15T03:47:00.599 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 1, 2024-09-15T03:47:00.599 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-15T03:47:00.600 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-15T03:47:00.600 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-15T03:47:00.600 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T03:47:00.600 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T03:47:00.600 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-15T03:47:00.600 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-15T03:47:00.600 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 8, 2024-09-15T03:47:00.600 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 8, 2024-09-15T03:47:00.600 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 8, 2024-09-15T03:47:00.600 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 8, 2024-09-15T03:47:00.600 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 8, 2024-09-15T03:47:00.600 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-15T03:47:00.600 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-15T03:47:00.601 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-15T03:47:00.601 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-15T03:47:00.601 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-15T03:47:00.601 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-15T03:47:00.601 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-15T03:47:00.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.601 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "24", 2024-09-15T03:47:00.601 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-15T03:47:00.601 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T03:47:00.601 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T03:47:00.601 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-15T03:47:00.601 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-15T03:47:00.601 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 3, 2024-09-15T03:47:00.601 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 24, 2024-09-15T03:47:00.602 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-15T03:47:00.602 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-15T03:47:00.602 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-15T03:47:00.602 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-15T03:47:00.602 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-15T03:47:00.602 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-15T03:47:00.602 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-15T03:47:00.602 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-15T03:47:00.602 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-15T03:47:00.602 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-15T03:47:00.602 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-15T03:47:00.602 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T03:47:00.602 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T03:47:00.603 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T03:47:00.603 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-15T03:47:00.603 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-15T03:47:00.603 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-15T03:47:00.603 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-15T03:47:00.603 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-15T03:47:00.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.603 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-15T03:47:00.603 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-15T03:47:00.603 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-15T03:47:00.603 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-15T03:47:00.603 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-15T03:47:00.603 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T03:47:00.604 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-15T03:47:00.604 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-15T03:47:00.604 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-15T03:47:00.604 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-15T03:47:00.604 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-15T03:47:00.604 INFO:teuthology.orchestra.run.smithi001.stdout: "options": {}, 2024-09-15T03:47:00.604 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-15T03:47:00.604 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd": {} 2024-09-15T03:47:00.604 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.604 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 3, 2024-09-15T03:47:00.604 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": "unique_pool_0", 2024-09-15T03:47:00.604 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-15T03:46:34.105566+0000", 2024-09-15T03:47:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 8193, 2024-09-15T03:47:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-15T03:47:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 3, 2024-09-15T03:47:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 8, 2024-09-15T03:47:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 5, 2024-09-15T03:47:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 1, 2024-09-15T03:47:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-15T03:47:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-15T03:47:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T03:47:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T03:47:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-15T03:47:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-15T03:47:00.605 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 16, 2024-09-15T03:47:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 16, 2024-09-15T03:47:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 16, 2024-09-15T03:47:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 16, 2024-09-15T03:47:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 16, 2024-09-15T03:47:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-15T03:47:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-15T03:47:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-15T03:47:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-15T03:47:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-15T03:47:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-15T03:47:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-15T03:47:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "53", 2024-09-15T03:47:00.606 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-15T03:47:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T03:47:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T03:47:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-15T03:47:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-15T03:47:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 22, 2024-09-15T03:47:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 53, 2024-09-15T03:47:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-15T03:47:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-15T03:47:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-15T03:47:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-15T03:47:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-15T03:47:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-15T03:47:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-15T03:47:00.607 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-15T03:47:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-15T03:47:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-15T03:47:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-15T03:47:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T03:47:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T03:47:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T03:47:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-15T03:47:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-15T03:47:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "lrcprofile", 2024-09-15T03:47:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-15T03:47:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-15T03:47:00.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-15T03:47:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-15T03:47:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-15T03:47:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-15T03:47:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-15T03:47:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T03:47:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-15T03:47:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-15T03:47:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 16384, 2024-09-15T03:47:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-15T03:47:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-15T03:47:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "options": {}, 2024-09-15T03:47:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-15T03:47:00.609 INFO:teuthology.orchestra.run.smithi001.stdout: "rados": {} 2024-09-15T03:47:00.610 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.610 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.610 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:47:00.610 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-15T03:47:00.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.610 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 0, 2024-09-15T03:47:00.610 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "cd433a6a-c875-4e6e-b89c-938caecbae3a", 2024-09-15T03:47:00.610 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:00.610 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:00.610 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:00.610 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:00.610 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:00.610 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:00.610 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:00.611 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 0, 2024-09-15T03:47:00.611 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:00.611 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:00.611 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:00.611 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.611 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.611 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6800", 2024-09-15T03:47:00.611 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:00.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.611 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.611 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6801", 2024-09-15T03:47:00.611 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:00.612 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.612 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.612 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:00.612 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.612 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.612 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6802", 2024-09-15T03:47:00.612 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:00.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.612 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.612 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6803", 2024-09-15T03:47:00.613 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:00.613 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.613 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.613 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:00.613 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.613 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.613 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6806", 2024-09-15T03:47:00.614 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:00.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.614 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.614 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6807", 2024-09-15T03:47:00.614 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:00.614 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.614 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.614 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:00.614 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6804", 2024-09-15T03:47:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6805", 2024-09-15T03:47:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6801/28908", 2024-09-15T03:47:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6803/28908", 2024-09-15T03:47:00.615 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6807/28908", 2024-09-15T03:47:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6805/28908", 2024-09-15T03:47:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 1, 2024-09-15T03:47:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "789930f9-42bf-476b-9f09-a21bffe33a79", 2024-09-15T03:47:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:00.616 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 31, 2024-09-15T03:47:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6808", 2024-09-15T03:47:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:00.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6809", 2024-09-15T03:47:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6810", 2024-09-15T03:47:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:00.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6811", 2024-09-15T03:47:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6814", 2024-09-15T03:47:00.619 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:00.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.620 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.620 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6815", 2024-09-15T03:47:00.620 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:00.620 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.620 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.620 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:00.620 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.620 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6812", 2024-09-15T03:47:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6813", 2024-09-15T03:47:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6809/28913", 2024-09-15T03:47:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6811/28913", 2024-09-15T03:47:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6815/28913", 2024-09-15T03:47:00.621 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6813/28913", 2024-09-15T03:47:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 2, 2024-09-15T03:47:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "9a83fec5-0d63-461c-b516-289443b3882b", 2024-09-15T03:47:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:00.622 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 28, 2024-09-15T03:47:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6821", 2024-09-15T03:47:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:00.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.624 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.624 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6823", 2024-09-15T03:47:00.624 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:00.624 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.624 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.624 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:00.624 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.624 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.624 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6825", 2024-09-15T03:47:00.624 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:00.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.625 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.625 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6827", 2024-09-15T03:47:00.625 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:00.625 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.625 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.625 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:00.625 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.625 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.625 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6830", 2024-09-15T03:47:00.625 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:00.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6831", 2024-09-15T03:47:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6828", 2024-09-15T03:47:00.626 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6829", 2024-09-15T03:47:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6823/28911", 2024-09-15T03:47:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6827/28911", 2024-09-15T03:47:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6831/28911", 2024-09-15T03:47:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6829/28911", 2024-09-15T03:47:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:00.627 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:00.628 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:00.628 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.628 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 3, 2024-09-15T03:47:00.628 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "d944bf89-e183-496a-9ca3-7949741b4be4", 2024-09-15T03:47:00.628 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:00.628 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:00.628 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:00.628 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:00.628 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:00.628 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:00.628 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 31, 2024-09-15T03:47:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6816", 2024-09-15T03:47:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6817", 2024-09-15T03:47:00.629 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:00.630 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.630 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.630 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:00.630 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.630 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.630 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6818", 2024-09-15T03:47:00.630 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:00.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.630 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.630 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6819", 2024-09-15T03:47:00.630 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:00.631 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.631 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.631 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:00.631 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.631 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.631 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6824", 2024-09-15T03:47:00.631 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:00.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.631 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.631 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6826", 2024-09-15T03:47:00.632 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:00.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.632 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.632 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:00.632 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.632 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.632 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6820", 2024-09-15T03:47:00.632 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:00.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.632 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.632 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6822", 2024-09-15T03:47:00.633 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:00.633 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.633 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.633 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6817/28914", 2024-09-15T03:47:00.633 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6819/28914", 2024-09-15T03:47:00.633 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6826/28914", 2024-09-15T03:47:00.633 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6822/28914", 2024-09-15T03:47:00.633 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:00.633 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:00.633 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:00.633 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.634 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 4, 2024-09-15T03:47:00.634 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "15ec2dbd-d099-4aff-94b2-1bf9dbb00b56", 2024-09-15T03:47:00.634 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:00.634 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:00.634 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:00.634 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:00.634 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:00.634 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:00.634 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:00.634 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 46, 2024-09-15T03:47:00.634 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:00.634 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:00.634 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:00.635 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.635 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.635 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6803", 2024-09-15T03:47:00.635 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:00.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.635 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.635 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6805", 2024-09-15T03:47:00.635 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:00.635 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.635 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.635 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:00.636 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.636 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.636 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6807", 2024-09-15T03:47:00.636 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:00.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.636 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.636 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6809", 2024-09-15T03:47:00.636 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:00.636 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.636 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.637 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:00.637 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.637 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.637 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6814", 2024-09-15T03:47:00.637 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:00.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.637 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.637 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6815", 2024-09-15T03:47:00.637 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:00.637 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.637 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.638 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:00.638 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.638 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.638 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6811", 2024-09-15T03:47:00.638 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:00.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.638 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.638 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6813", 2024-09-15T03:47:00.638 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:00.638 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.638 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.639 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.67:6805/28468", 2024-09-15T03:47:00.639 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.67:6809/28468", 2024-09-15T03:47:00.639 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.67:6815/28468", 2024-09-15T03:47:00.639 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.67:6813/28468", 2024-09-15T03:47:00.639 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:00.639 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:00.639 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:00.639 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.639 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 5, 2024-09-15T03:47:00.639 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "1e45de07-746b-4612-9a52-d05509cae9a3", 2024-09-15T03:47:00.639 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:00.640 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:00.640 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:00.640 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:00.640 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:00.640 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:00.640 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:00.640 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 21, 2024-09-15T03:47:00.640 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:00.640 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:00.640 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:00.640 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.640 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.640 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6824", 2024-09-15T03:47:00.641 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:00.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.641 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.641 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6825", 2024-09-15T03:47:00.641 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:00.641 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.641 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.641 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:00.641 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.641 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.641 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6826", 2024-09-15T03:47:00.642 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:00.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.642 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.642 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6827", 2024-09-15T03:47:00.642 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:00.642 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.642 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.642 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:00.642 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.642 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.642 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6830", 2024-09-15T03:47:00.643 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:00.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.643 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.643 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6831", 2024-09-15T03:47:00.643 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:00.643 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.643 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.643 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:00.643 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.643 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.643 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6828", 2024-09-15T03:47:00.644 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:00.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.644 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.644 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6829", 2024-09-15T03:47:00.644 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:00.644 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.644 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.644 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.67:6825/28495", 2024-09-15T03:47:00.644 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.67:6827/28495", 2024-09-15T03:47:00.644 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.67:6831/28495", 2024-09-15T03:47:00.644 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.67:6829/28495", 2024-09-15T03:47:00.645 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:00.645 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:00.645 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:00.645 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.645 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 6, 2024-09-15T03:47:00.645 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "ccf785f8-751d-473e-b17c-3df9346be9f8", 2024-09-15T03:47:00.645 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:00.645 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:00.645 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:00.645 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:00.645 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:00.645 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:00.646 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:00.646 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 31, 2024-09-15T03:47:00.646 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:00.646 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:00.646 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:00.646 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.646 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.646 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6800", 2024-09-15T03:47:00.646 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:00.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.646 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.646 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6801", 2024-09-15T03:47:00.647 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:00.647 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.647 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.647 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:00.647 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.647 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.647 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6802", 2024-09-15T03:47:00.647 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:00.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.647 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.648 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6804", 2024-09-15T03:47:00.648 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:00.648 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.648 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.648 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:00.648 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.648 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.648 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6810", 2024-09-15T03:47:00.648 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:00.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.649 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.649 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6812", 2024-09-15T03:47:00.649 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:00.649 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.649 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.649 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:00.649 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.649 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.649 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6806", 2024-09-15T03:47:00.649 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:00.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.650 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.650 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6808", 2024-09-15T03:47:00.650 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:00.650 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.650 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.650 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.67:6801/28469", 2024-09-15T03:47:00.650 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.67:6804/28469", 2024-09-15T03:47:00.650 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.67:6812/28469", 2024-09-15T03:47:00.650 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.67:6808/28469", 2024-09-15T03:47:00.650 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:00.650 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:00.650 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:00.651 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.651 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 7, 2024-09-15T03:47:00.651 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "f9f59987-869d-42eb-9eb6-08c39e3afe6d", 2024-09-15T03:47:00.651 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:00.651 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:00.651 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:00.651 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:00.651 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:00.651 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:00.651 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:00.651 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 31, 2024-09-15T03:47:00.651 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:00.652 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:00.652 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:00.652 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.652 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.652 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6816", 2024-09-15T03:47:00.652 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:00.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.652 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.652 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6817", 2024-09-15T03:47:00.652 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:00.652 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.653 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.653 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:00.653 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.653 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.653 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6818", 2024-09-15T03:47:00.653 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:00.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.653 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.653 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6819", 2024-09-15T03:47:00.653 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:00.654 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.654 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.654 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:00.654 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.654 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.654 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6822", 2024-09-15T03:47:00.654 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:00.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.654 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.654 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6823", 2024-09-15T03:47:00.655 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:00.655 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.655 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.655 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:00.655 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.655 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.655 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6820", 2024-09-15T03:47:00.655 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:00.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.655 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.655 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6821", 2024-09-15T03:47:00.656 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:00.656 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.656 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.656 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.67:6817/28470", 2024-09-15T03:47:00.656 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.67:6819/28470", 2024-09-15T03:47:00.656 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.67:6823/28470", 2024-09-15T03:47:00.656 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.67:6821/28470", 2024-09-15T03:47:00.656 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:00.656 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:00.656 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:00.656 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.657 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 8, 2024-09-15T03:47:00.657 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "69ddfe52-11df-4608-9a10-e8907c4cd87f", 2024-09-15T03:47:00.657 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:00.657 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:00.657 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:00.657 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:00.657 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:00.657 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:00.657 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:00.657 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 31, 2024-09-15T03:47:00.657 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:00.657 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:00.657 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:00.658 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.658 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.658 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6808", 2024-09-15T03:47:00.658 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:00.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.658 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.658 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6809", 2024-09-15T03:47:00.658 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:00.658 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.659 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.659 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:00.659 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.659 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.659 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6810", 2024-09-15T03:47:00.659 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:00.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.659 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.659 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6811", 2024-09-15T03:47:00.659 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:00.660 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.660 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.660 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:00.660 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.660 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.660 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6814", 2024-09-15T03:47:00.660 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:00.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.660 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.660 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6815", 2024-09-15T03:47:00.660 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:00.661 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.661 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.661 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:00.661 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.661 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.661 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6812", 2024-09-15T03:47:00.661 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:00.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.661 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.661 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6813", 2024-09-15T03:47:00.662 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:00.662 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.662 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.662 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.190:6809/29472", 2024-09-15T03:47:00.662 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.190:6811/29472", 2024-09-15T03:47:00.662 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.190:6815/29472", 2024-09-15T03:47:00.662 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.190:6813/29472", 2024-09-15T03:47:00.662 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:00.662 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:00.662 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:00.663 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.663 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 9, 2024-09-15T03:47:00.663 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "f92e1609-2a1a-4814-9ac0-44a73373d229", 2024-09-15T03:47:00.663 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:00.663 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:00.663 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:00.663 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:00.663 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:00.663 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:00.663 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:00.664 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 28, 2024-09-15T03:47:00.664 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:00.664 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:00.664 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:00.664 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.664 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.664 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6800", 2024-09-15T03:47:00.664 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:00.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.664 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.664 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6801", 2024-09-15T03:47:00.665 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:00.665 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.665 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.665 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:00.665 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.665 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.665 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6802", 2024-09-15T03:47:00.665 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:00.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.665 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.665 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6803", 2024-09-15T03:47:00.666 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:00.666 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.666 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.666 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:00.666 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.666 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.666 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6806", 2024-09-15T03:47:00.666 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:00.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.666 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.666 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6807", 2024-09-15T03:47:00.667 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:00.667 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.667 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.667 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:00.667 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.667 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.667 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6804", 2024-09-15T03:47:00.667 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:00.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.667 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.668 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6805", 2024-09-15T03:47:00.668 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:00.668 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.668 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.668 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.190:6801/29473", 2024-09-15T03:47:00.668 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.190:6803/29473", 2024-09-15T03:47:00.668 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.190:6807/29473", 2024-09-15T03:47:00.668 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.190:6805/29473", 2024-09-15T03:47:00.668 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:00.668 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:00.668 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:00.668 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.669 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 10, 2024-09-15T03:47:00.669 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "5f2e8ca3-8c65-4595-8e82-27dc3768ab99", 2024-09-15T03:47:00.669 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:00.669 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 0, 2024-09-15T03:47:00.669 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-09-15T03:47:00.669 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:00.669 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:00.669 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:00.669 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:00.669 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 28, 2024-09-15T03:47:00.669 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:00.669 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:00.670 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:00.670 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.670 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.670 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6816", 2024-09-15T03:47:00.670 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:00.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.670 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.670 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6817", 2024-09-15T03:47:00.670 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:00.670 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.670 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.671 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:00.671 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.671 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.671 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6818", 2024-09-15T03:47:00.671 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:00.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.671 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.671 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6819", 2024-09-15T03:47:00.671 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:00.671 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.672 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.672 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:00.672 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.672 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.672 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6822", 2024-09-15T03:47:00.672 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:00.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.672 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.672 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6823", 2024-09-15T03:47:00.672 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:00.673 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.673 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.673 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:00.673 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.673 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.673 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6820", 2024-09-15T03:47:00.673 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:00.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.673 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.673 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6821", 2024-09-15T03:47:00.673 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:00.674 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.674 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.674 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.190:6817/29474", 2024-09-15T03:47:00.674 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.190:6819/29474", 2024-09-15T03:47:00.674 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.190:6823/29474", 2024-09-15T03:47:00.674 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.190:6821/29474", 2024-09-15T03:47:00.674 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:00.674 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:00.674 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:00.674 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.675 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 11, 2024-09-15T03:47:00.675 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "cedf7df4-f875-42f5-a8cf-13cb82f05cc2", 2024-09-15T03:47:00.675 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:00.675 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:00.675 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:00.675 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:00.675 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:00.675 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:00.675 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:00.675 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 31, 2024-09-15T03:47:00.675 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:00.675 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:00.675 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:00.675 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.676 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.676 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6824", 2024-09-15T03:47:00.676 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:00.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.676 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.676 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6825", 2024-09-15T03:47:00.676 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:00.676 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.676 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.676 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:00.676 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.677 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.677 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6826", 2024-09-15T03:47:00.677 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:00.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.677 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.677 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6827", 2024-09-15T03:47:00.677 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:00.677 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.677 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.677 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:00.678 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.678 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.678 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6830", 2024-09-15T03:47:00.678 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:00.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.678 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.678 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6831", 2024-09-15T03:47:00.678 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:00.678 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.678 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.678 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:00.679 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:00.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.679 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:00.679 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6828", 2024-09-15T03:47:00.679 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:00.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.679 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:00.679 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6829", 2024-09-15T03:47:00.679 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:00.679 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.679 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.679 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.190:6825/29475", 2024-09-15T03:47:00.680 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.190:6827/29475", 2024-09-15T03:47:00.680 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.190:6831/29475", 2024-09-15T03:47:00.680 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.190:6829/29475", 2024-09-15T03:47:00.680 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:00.680 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:00.680 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:00.680 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.680 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.680 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:47:00.680 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_xinfo": [ 2024-09-15T03:47:00.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.680 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 0, 2024-09-15T03:47:00.680 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:00.680 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:00.681 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:00.681 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:00.681 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:00.681 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.943165+0000", 2024-09-15T03:47:00.681 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:00.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.681 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 1, 2024-09-15T03:47:00.681 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:00.681 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:00.681 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:00.681 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:00.681 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:00.681 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.946049+0000", 2024-09-15T03:47:00.682 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:00.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.682 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 2, 2024-09-15T03:47:00.682 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:00.682 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:00.682 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:00.682 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:00.682 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:00.682 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.981045+0000", 2024-09-15T03:47:00.682 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:00.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.682 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 3, 2024-09-15T03:47:00.683 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:00.683 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:00.683 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:00.683 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:00.683 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:00.683 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:14.019932+0000", 2024-09-15T03:47:00.683 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:00.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.683 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 4, 2024-09-15T03:47:00.683 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:00.683 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:00.683 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:00.684 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:00.684 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:00.684 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.936197+0000", 2024-09-15T03:47:00.684 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:00.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.684 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 5, 2024-09-15T03:47:00.684 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:00.684 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:00.684 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:00.684 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:00.684 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:00.684 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.982429+0000", 2024-09-15T03:47:00.684 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:00.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.685 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 6, 2024-09-15T03:47:00.685 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:00.685 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:00.685 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:00.685 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:00.685 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:00.685 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.953833+0000", 2024-09-15T03:47:00.685 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:00.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.685 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 7, 2024-09-15T03:47:00.686 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:00.686 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:00.686 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:00.686 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:00.686 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:00.686 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.967611+0000", 2024-09-15T03:47:00.686 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:00.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.686 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 8, 2024-09-15T03:47:00.686 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:00.686 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:00.686 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:00.687 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:00.687 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:00.687 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.996438+0000", 2024-09-15T03:47:00.687 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:00.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.687 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 9, 2024-09-15T03:47:00.687 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:00.687 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:00.687 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:00.687 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:00.687 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:00.687 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.926618+0000", 2024-09-15T03:47:00.688 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:00.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.688 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 10, 2024-09-15T03:47:00.688 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:00.688 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:00.688 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:00.688 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:00.688 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 65536, 2024-09-15T03:47:00.688 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.993934+0000", 2024-09-15T03:47:00.688 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:00.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.689 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 11, 2024-09-15T03:47:00.689 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:00.689 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:00.689 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:00.689 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:00.689 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:00.689 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:14.022484+0000", 2024-09-15T03:47:00.689 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:00.689 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.689 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:47:00.689 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap": [], 2024-09-15T03:47:00.689 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap_items": [ 2024-09-15T03:47:00.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.689 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.6", 2024-09-15T03:47:00.690 INFO:teuthology.orchestra.run.smithi001.stdout: "mappings": [ 2024-09-15T03:47:00.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.690 INFO:teuthology.orchestra.run.smithi001.stdout: "from": 3, 2024-09-15T03:47:00.690 INFO:teuthology.orchestra.run.smithi001.stdout: "to": 5 2024-09-15T03:47:00.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.690 INFO:teuthology.orchestra.run.smithi001.stdout: "from": 1, 2024-09-15T03:47:00.690 INFO:teuthology.orchestra.run.smithi001.stdout: "to": 11 2024-09-15T03:47:00.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.690 INFO:teuthology.orchestra.run.smithi001.stdout: "from": 8, 2024-09-15T03:47:00.690 INFO:teuthology.orchestra.run.smithi001.stdout: "to": 9 2024-09-15T03:47:00.690 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.690 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.691 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.691 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:47:00.691 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_temp": [], 2024-09-15T03:47:00.691 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_temp": [], 2024-09-15T03:47:00.691 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklist": {}, 2024-09-15T03:47:00.691 INFO:teuthology.orchestra.run.smithi001.stdout: "range_blocklist": {}, 2024-09-15T03:47:00.691 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profiles": { 2024-09-15T03:47:00.691 INFO:teuthology.orchestra.run.smithi001.stdout: "default": { 2024-09-15T03:47:00.691 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-failure-domain": "osd", 2024-09-15T03:47:00.691 INFO:teuthology.orchestra.run.smithi001.stdout: "k": "2", 2024-09-15T03:47:00.691 INFO:teuthology.orchestra.run.smithi001.stdout: "m": "1", 2024-09-15T03:47:00.691 INFO:teuthology.orchestra.run.smithi001.stdout: "plugin": "jerasure", 2024-09-15T03:47:00.691 INFO:teuthology.orchestra.run.smithi001.stdout: "technique": "reed_sol_van" 2024-09-15T03:47:00.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.692 INFO:teuthology.orchestra.run.smithi001.stdout: "lrcprofile": { 2024-09-15T03:47:00.692 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-device-class": "", 2024-09-15T03:47:00.692 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-failure-domain": "osd", 2024-09-15T03:47:00.692 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-root": "default", 2024-09-15T03:47:00.692 INFO:teuthology.orchestra.run.smithi001.stdout: "k": "4", 2024-09-15T03:47:00.692 INFO:teuthology.orchestra.run.smithi001.stdout: "l": "3", 2024-09-15T03:47:00.692 INFO:teuthology.orchestra.run.smithi001.stdout: "m": "2", 2024-09-15T03:47:00.692 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "lrcprofile", 2024-09-15T03:47:00.692 INFO:teuthology.orchestra.run.smithi001.stdout: "plugin": "lrc" 2024-09-15T03:47:00.692 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.692 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps_queue": [ 2024-09-15T03:47:00.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.692 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 3, 2024-09-15T03:47:00.693 INFO:teuthology.orchestra.run.smithi001.stdout: "snaps": [ 2024-09-15T03:47:00.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.693 INFO:teuthology.orchestra.run.smithi001.stdout: "begin": 5, 2024-09-15T03:47:00.693 INFO:teuthology.orchestra.run.smithi001.stdout: "length": 1 2024-09-15T03:47:00.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.693 INFO:teuthology.orchestra.run.smithi001.stdout: "begin": 11, 2024-09-15T03:47:00.693 INFO:teuthology.orchestra.run.smithi001.stdout: "length": 1 2024-09-15T03:47:00.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:00.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:00.693 INFO:teuthology.orchestra.run.smithi001.stdout: "begin": 13, 2024-09-15T03:47:00.693 INFO:teuthology.orchestra.run.smithi001.stdout: "length": 1 2024-09-15T03:47:00.693 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.693 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:00.694 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.694 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:47:00.694 INFO:teuthology.orchestra.run.smithi001.stdout: "new_removed_snaps": [], 2024-09-15T03:47:00.694 INFO:teuthology.orchestra.run.smithi001.stdout: "new_purged_snaps": [], 2024-09-15T03:47:00.694 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_node_flags": {}, 2024-09-15T03:47:00.694 INFO:teuthology.orchestra.run.smithi001.stdout: "device_class_flags": {}, 2024-09-15T03:47:00.694 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": { 2024-09-15T03:47:00.694 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode_enabled": false, 2024-09-15T03:47:00.694 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_bucket_count": 0, 2024-09-15T03:47:00.694 INFO:teuthology.orchestra.run.smithi001.stdout: "degraded_stretch_mode": 0, 2024-09-15T03:47:00.694 INFO:teuthology.orchestra.run.smithi001.stdout: "recovering_stretch_mode": 0, 2024-09-15T03:47:00.694 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode_bucket": 0 2024-09-15T03:47:00.694 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:00.695 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:00.696 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:00.697 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:00.698 INFO:tasks.thrashosds.thrasher:j is {'epoch': 53, 'fsid': 'f27b540e-e5e3-4e95-afe8-c95ee66b0a62', 'created': '2024-09-15T03:46:09.843094+0000', 'modified': '2024-09-15T03:46:59.841023+0000', 'last_up_change': '2024-09-15T03:46:15.871985+0000', 'last_in_change': '2024-09-15T03:46:38.089065+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-15T03:46:16.185530+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-15T03:46:22.284406+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 24, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-15T03:46:34.105566+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 8, 'min_size': 5, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 22, 'snap_epoch': 53, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'lrcprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 16384, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'cd433a6a-c875-4e6e-b89c-938caecbae3a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6800', 'nonce': 28908}, {'type': 'v1', 'addr': '172.21.15.1:6801', 'nonce': 28908}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6802', 'nonce': 28908}, {'type': 'v1', 'addr': '172.21.15.1:6803', 'nonce': 28908}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6806', 'nonce': 28908}, {'type': 'v1', 'addr': '172.21.15.1:6807', 'nonce': 28908}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6804', 'nonce': 28908}, {'type': 'v1', 'addr': '172.21.15.1:6805', 'nonce': 28908}]}, 'public_addr': '172.21.15.1:6801/28908', 'cluster_addr': '172.21.15.1:6803/28908', 'heartbeat_back_addr': '172.21.15.1:6807/28908', 'heartbeat_front_addr': '172.21.15.1:6805/28908', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '789930f9-42bf-476b-9f09-a21bffe33a79', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6808', 'nonce': 28913}, {'type': 'v1', 'addr': '172.21.15.1:6809', 'nonce': 28913}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6810', 'nonce': 28913}, {'type': 'v1', 'addr': '172.21.15.1:6811', 'nonce': 28913}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6814', 'nonce': 28913}, {'type': 'v1', 'addr': '172.21.15.1:6815', 'nonce': 28913}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6812', 'nonce': 28913}, {'type': 'v1', 'addr': '172.21.15.1:6813', 'nonce': 28913}]}, 'public_addr': '172.21.15.1:6809/28913', 'cluster_addr': '172.21.15.1:6811/28913', 'heartbeat_back_addr': '172.21.15.1:6815/28913', 'heartbeat_front_addr': '172.21.15.1:6813/28913', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '9a83fec5-0d63-461c-b516-289443b3882b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 28, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6821', 'nonce': 28911}, {'type': 'v1', 'addr': '172.21.15.1:6823', 'nonce': 28911}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6825', 'nonce': 28911}, {'type': 'v1', 'addr': '172.21.15.1:6827', 'nonce': 28911}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6830', 'nonce': 28911}, {'type': 'v1', 'addr': '172.21.15.1:6831', 'nonce': 28911}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6828', 'nonce': 28911}, {'type': 'v1', 'addr': '172.21.15.1:6829', 'nonce': 28911}]}, 'public_addr': '172.21.15.1:6823/28911', 'cluster_addr': '172.21.15.1:6827/28911', 'heartbeat_back_addr': '172.21.15.1:6831/28911', 'heartbeat_front_addr': '172.21.15.1:6829/28911', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd944bf89-e183-496a-9ca3-7949741b4be4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6816', 'nonce': 28914}, {'type': 'v1', 'addr': '172.21.15.1:6817', 'nonce': 28914}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6818', 'nonce': 28914}, {'type': 'v1', 'addr': '172.21.15.1:6819', 'nonce': 28914}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6824', 'nonce': 28914}, {'type': 'v1', 'addr': '172.21.15.1:6826', 'nonce': 28914}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6820', 'nonce': 28914}, {'type': 'v1', 'addr': '172.21.15.1:6822', 'nonce': 28914}]}, 'public_addr': '172.21.15.1:6817/28914', 'cluster_addr': '172.21.15.1:6819/28914', 'heartbeat_back_addr': '172.21.15.1:6826/28914', 'heartbeat_front_addr': '172.21.15.1:6822/28914', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '15ec2dbd-d099-4aff-94b2-1bf9dbb00b56', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 46, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6803', 'nonce': 28468}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 28468}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6807', 'nonce': 28468}, {'type': 'v1', 'addr': '172.21.15.67:6809', 'nonce': 28468}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 28468}, {'type': 'v1', 'addr': '172.21.15.67:6815', 'nonce': 28468}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6811', 'nonce': 28468}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 28468}]}, 'public_addr': '172.21.15.67:6805/28468', 'cluster_addr': '172.21.15.67:6809/28468', 'heartbeat_back_addr': '172.21.15.67:6815/28468', 'heartbeat_front_addr': '172.21.15.67:6813/28468', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1e45de07-746b-4612-9a52-d05509cae9a3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6824', 'nonce': 28495}, {'type': 'v1', 'addr': '172.21.15.67:6825', 'nonce': 28495}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6826', 'nonce': 28495}, {'type': 'v1', 'addr': '172.21.15.67:6827', 'nonce': 28495}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6830', 'nonce': 28495}, {'type': 'v1', 'addr': '172.21.15.67:6831', 'nonce': 28495}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6828', 'nonce': 28495}, {'type': 'v1', 'addr': '172.21.15.67:6829', 'nonce': 28495}]}, 'public_addr': '172.21.15.67:6825/28495', 'cluster_addr': '172.21.15.67:6827/28495', 'heartbeat_back_addr': '172.21.15.67:6831/28495', 'heartbeat_front_addr': '172.21.15.67:6829/28495', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ccf785f8-751d-473e-b17c-3df9346be9f8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 28469}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 28469}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 28469}, {'type': 'v1', 'addr': '172.21.15.67:6804', 'nonce': 28469}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 28469}, {'type': 'v1', 'addr': '172.21.15.67:6812', 'nonce': 28469}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 28469}, {'type': 'v1', 'addr': '172.21.15.67:6808', 'nonce': 28469}]}, 'public_addr': '172.21.15.67:6801/28469', 'cluster_addr': '172.21.15.67:6804/28469', 'heartbeat_back_addr': '172.21.15.67:6812/28469', 'heartbeat_front_addr': '172.21.15.67:6808/28469', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f9f59987-869d-42eb-9eb6-08c39e3afe6d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 28470}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 28470}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6818', 'nonce': 28470}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 28470}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6822', 'nonce': 28470}, {'type': 'v1', 'addr': '172.21.15.67:6823', 'nonce': 28470}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 28470}, {'type': 'v1', 'addr': '172.21.15.67:6821', 'nonce': 28470}]}, 'public_addr': '172.21.15.67:6817/28470', 'cluster_addr': '172.21.15.67:6819/28470', 'heartbeat_back_addr': '172.21.15.67:6823/28470', 'heartbeat_front_addr': '172.21.15.67:6821/28470', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '69ddfe52-11df-4608-9a10-e8907c4cd87f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6808', 'nonce': 29472}, {'type': 'v1', 'addr': '172.21.15.190:6809', 'nonce': 29472}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6810', 'nonce': 29472}, {'type': 'v1', 'addr': '172.21.15.190:6811', 'nonce': 29472}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6814', 'nonce': 29472}, {'type': 'v1', 'addr': '172.21.15.190:6815', 'nonce': 29472}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6812', 'nonce': 29472}, {'type': 'v1', 'addr': '172.21.15.190:6813', 'nonce': 29472}]}, 'public_addr': '172.21.15.190:6809/29472', 'cluster_addr': '172.21.15.190:6811/29472', 'heartbeat_back_addr': '172.21.15.190:6815/29472', 'heartbeat_front_addr': '172.21.15.190:6813/29472', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'f92e1609-2a1a-4814-9ac0-44a73373d229', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 28, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6800', 'nonce': 29473}, {'type': 'v1', 'addr': '172.21.15.190:6801', 'nonce': 29473}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6802', 'nonce': 29473}, {'type': 'v1', 'addr': '172.21.15.190:6803', 'nonce': 29473}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6806', 'nonce': 29473}, {'type': 'v1', 'addr': '172.21.15.190:6807', 'nonce': 29473}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6804', 'nonce': 29473}, {'type': 'v1', 'addr': '172.21.15.190:6805', 'nonce': 29473}]}, 'public_addr': '172.21.15.190:6801/29473', 'cluster_addr': '172.21.15.190:6803/29473', 'heartbeat_back_addr': '172.21.15.190:6807/29473', 'heartbeat_front_addr': '172.21.15.190:6805/29473', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '5f2e8ca3-8c65-4595-8e82-27dc3768ab99', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 28, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6816', 'nonce': 29474}, {'type': 'v1', 'addr': '172.21.15.190:6817', 'nonce': 29474}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6818', 'nonce': 29474}, {'type': 'v1', 'addr': '172.21.15.190:6819', 'nonce': 29474}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6822', 'nonce': 29474}, {'type': 'v1', 'addr': '172.21.15.190:6823', 'nonce': 29474}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6820', 'nonce': 29474}, {'type': 'v1', 'addr': '172.21.15.190:6821', 'nonce': 29474}]}, 'public_addr': '172.21.15.190:6817/29474', 'cluster_addr': '172.21.15.190:6819/29474', 'heartbeat_back_addr': '172.21.15.190:6823/29474', 'heartbeat_front_addr': '172.21.15.190:6821/29474', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'cedf7df4-f875-42f5-a8cf-13cb82f05cc2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6824', 'nonce': 29475}, {'type': 'v1', 'addr': '172.21.15.190:6825', 'nonce': 29475}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6826', 'nonce': 29475}, {'type': 'v1', 'addr': '172.21.15.190:6827', 'nonce': 29475}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6830', 'nonce': 29475}, {'type': 'v1', 'addr': '172.21.15.190:6831', 'nonce': 29475}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6828', 'nonce': 29475}, {'type': 'v1', 'addr': '172.21.15.190:6829', 'nonce': 29475}]}, 'public_addr': '172.21.15.190:6825/29475', 'cluster_addr': '172.21.15.190:6827/29475', 'heartbeat_back_addr': '172.21.15.190:6831/29475', 'heartbeat_front_addr': '172.21.15.190:6829/29475', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.943165+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.946049+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.981045+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:14.019932+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.936197+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.982429+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.953833+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.967611+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.996438+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.926618+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.993934+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:14.022484+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.6', 'mappings': [{'from': 3, 'to': 5}, {'from': 1, 'to': 11}, {'from': 8, 'to': 9}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'lrcprofile': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'k': '4', 'l': '3', 'm': '2', 'name': 'lrcprofile', 'plugin': 'lrc'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 11, 'length': 1}, {'begin': 13, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-15T03:47:00.698 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 3.6 2024-09-15T03:47:00.698 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd rm-pg-upmap-items 3.6 2024-09-15T03:47:00.698 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:47:00.699 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:00.695+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:00.796 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:00.795+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:00.834 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:00.834 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:00.842 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:47:00.854 INFO:tasks.rados.rados.0.smithi001.stdout:142: done (0 left) 2024-09-15T03:47:00.854 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 45 current snap is 16 2024-09-15T03:47:00.854 INFO:tasks.rados.rados.0.smithi001.stdout:143: seq_num 79 ranges {360448=573440,933888=425984,1359872=344064} 2024-09-15T03:47:00.858 INFO:tasks.rados.rados.0.smithi001.stdout:143: writing smithi00132172-45 from 360448 to 933888 tid 1 2024-09-15T03:47:00.861 INFO:tasks.rados.rados.0.smithi001.stdout:143: writing smithi00132172-45 from 933888 to 1359872 tid 2 2024-09-15T03:47:00.863 INFO:tasks.rados.rados.0.smithi001.stdout:143: writing smithi00132172-45 from 1359872 to 1703936 tid 3 2024-09-15T03:47:00.864 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 10 current snap is 16 2024-09-15T03:47:00.864 INFO:tasks.rados.rados.0.smithi001.stdout:144: seq_num 80 ranges {163840=589824,753664=573440,1327104=196608} 2024-09-15T03:47:00.868 INFO:tasks.rados.rados.0.smithi001.stdout:144: writing smithi00132172-10 from 163840 to 753664 tid 1 2024-09-15T03:47:00.872 INFO:tasks.rados.rados.0.smithi001.stdout:144: writing smithi00132172-10 from 753664 to 1327104 tid 2 2024-09-15T03:47:00.873 INFO:tasks.rados.rados.0.smithi001.stdout:144: writing smithi00132172-10 from 1327104 to 1523712 tid 3 2024-09-15T03:47:00.874 INFO:tasks.rados.rados.0.smithi001.stdout:145: setattr oid 20 current snap is 16 2024-09-15T03:47:00.874 INFO:tasks.rados.rados.0.smithi001.stdout:143: finishing write tid 1 to smithi00132172-45 2024-09-15T03:47:00.874 INFO:tasks.rados.rados.0.smithi001.stdout:143: finishing write tid 2 to smithi00132172-45 2024-09-15T03:47:00.874 INFO:tasks.rados.rados.0.smithi001.stdout:143: finishing write tid 3 to smithi00132172-45 2024-09-15T03:47:00.874 INFO:tasks.rados.rados.0.smithi001.stdout:143: finishing write tid 4 to smithi00132172-45 2024-09-15T03:47:00.874 INFO:tasks.rados.rados.0.smithi001.stdout:143: finishing write tid 5 to smithi00132172-45 2024-09-15T03:47:00.874 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 45 v 20 (ObjNum 79 snap 16 seq_num 79) dirty exists 2024-09-15T03:47:00.874 INFO:tasks.rados.rados.0.smithi001.stdout:143: left oid 45 (ObjNum 79 snap 16 seq_num 79) 2024-09-15T03:47:00.874 INFO:tasks.rados.rados.0.smithi001.stdout:144: finishing write tid 1 to smithi00132172-10 2024-09-15T03:47:00.875 INFO:tasks.rados.rados.0.smithi001.stdout:143: done (2 left) 2024-09-15T03:47:00.875 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 14 current snap is 16 2024-09-15T03:47:00.875 INFO:tasks.rados.rados.0.smithi001.stdout:146: seq_num 81 ranges {1245184=606208,1851392=720896,2572288=49152} 2024-09-15T03:47:00.885 INFO:tasks.rados.rados.0.smithi001.stdout:146: writing smithi00132172-14 from 1245184 to 1851392 tid 1 2024-09-15T03:47:00.890 INFO:tasks.rados.rados.0.smithi001.stdout:146: writing smithi00132172-14 from 1851392 to 2572288 tid 2 2024-09-15T03:47:00.890 INFO:tasks.rados.rados.0.smithi001.stdout:146: writing smithi00132172-14 from 2572288 to 2621440 tid 3 2024-09-15T03:47:00.890 INFO:tasks.rados.rados.0.smithi001.stdout:147: snap_create 2024-09-15T03:47:00.890 INFO:tasks.rados.rados.0.smithi001.stdout:144: finishing write tid 2 to smithi00132172-10 2024-09-15T03:47:00.890 INFO:tasks.rados.rados.0.smithi001.stdout:144: finishing write tid 3 to smithi00132172-10 2024-09-15T03:47:00.890 INFO:tasks.rados.rados.0.smithi001.stdout:144: finishing write tid 4 to smithi00132172-10 2024-09-15T03:47:00.890 INFO:tasks.rados.rados.0.smithi001.stdout:144: finishing write tid 5 to smithi00132172-10 2024-09-15T03:47:00.890 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 10 v 39 (ObjNum 80 snap 16 seq_num 80) dirty exists 2024-09-15T03:47:00.890 INFO:tasks.rados.rados.0.smithi001.stdout:144: left oid 10 (ObjNum 80 snap 16 seq_num 80) 2024-09-15T03:47:00.890 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-15T03:47:00.891 INFO:tasks.rados.rados.0.smithi001.stdout:146: finishing write tid 1 to smithi00132172-14 2024-09-15T03:47:00.897 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:00.895+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:00.898 INFO:tasks.rados.rados.0.smithi001.stdout:146: finishing write tid 2 to smithi00132172-14 2024-09-15T03:47:00.898 INFO:tasks.rados.rados.0.smithi001.stdout:146: finishing write tid 3 to smithi00132172-14 2024-09-15T03:47:00.899 INFO:tasks.rados.rados.0.smithi001.stdout:146: finishing write tid 4 to smithi00132172-14 2024-09-15T03:47:00.899 INFO:tasks.rados.rados.0.smithi001.stdout:146: finishing write tid 5 to smithi00132172-14 2024-09-15T03:47:00.899 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 14 v 51 (ObjNum 81 snap 16 seq_num 81) dirty exists 2024-09-15T03:47:00.899 INFO:tasks.rados.rados.0.smithi001.stdout:146: left oid 14 (ObjNum 81 snap 16 seq_num 81) 2024-09-15T03:47:00.980 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:00.980 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:00.989 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:47:01.033 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:01.032+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:01.066 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:47:01.087 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:01.087 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:01.095 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:47:01.112 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:01.110+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:01.183 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:01.183 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:01.197 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:01.197 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:01.199 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:01.198+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:01.211 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:01.212 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:01.232 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:01.232 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:01.241 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:47:01.301 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:01.300+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:01.338 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:01.338 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:01.347 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:47:01.401 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:01.400+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:01.426 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:01.426 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:01.441 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:01.441 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:01.444 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:01.444 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:01.453 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:47:01.456 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:01.456 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:01.471 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:01.471 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:01.486 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:01.487 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:01.501 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:01.502 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:01.500+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:01.515 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:01.515 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:01.531 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:47:01.531 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:47:01.546 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:01.546 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:01.552 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:01.552 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:01.552 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:01.552 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:01.552 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:01.561 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:47:01.603 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:01.602+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:01.657 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:01.658 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:01.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:01.658 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [] 2024-09-15T03:47:01.658 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:01.666 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:47:01.704 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:01.703+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:01.762 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:01.762 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:01.762 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:01.762 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:01.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:47:01.814 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:01.814+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:01.857 INFO:tasks.rados.rados.0.smithi001.stdout:144: done (3 left) 2024-09-15T03:47:01.858 INFO:tasks.rados.rados.0.smithi001.stdout:145: done (2 left) 2024-09-15T03:47:01.858 INFO:tasks.rados.rados.0.smithi001.stdout:146: done (1 left) 2024-09-15T03:47:01.858 INFO:tasks.rados.rados.0.smithi001.stdout:147: done (0 left) 2024-09-15T03:47:01.858 INFO:tasks.rados.rados.0.smithi001.stdout:148: read oid 11 snap 9 2024-09-15T03:47:01.858 INFO:tasks.rados.rados.0.smithi001.stdout:148: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-15T03:47:01.858 INFO:tasks.rados.rados.0.smithi001.stdout:149: snap_create 2024-09-15T03:47:01.858 INFO:teuthology.orchestra.run.smithi001.stderr:clear 3.6 pg_upmap_items mapping 2024-09-15T03:47:01.863 INFO:tasks.rados.rados.0.smithi001.stdout:148: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-15T03:47:01.866 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T03:47:01.869 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:01.869 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:01.869 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:01.869 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:01.869 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:01.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:47:01.919 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:01.918+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:02.006 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:02.004+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:02.017 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:02.017 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:02.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:02.017 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:47:02.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.017 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:185 3.as0 3:5515acd6:::smithi00132172-13:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:47:02.017 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:47:02.018 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.910334419000002, 2024-09-15T03:47:02.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01268643, 2024-09-15T03:47:02.018 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:02.018 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:02.018 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:02.018 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:02.018 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:02.018 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 185 2024-09-15T03:47:02.018 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.019 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:02.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:02.019 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:47:02.019 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:02.019 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:47:02.019 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:02.019 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104710+0000", 2024-09-15T03:47:02.020 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0200000000000003e-07 2024-09-15T03:47:02.020 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.020 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.020 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:02.020 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104713+0000", 2024-09-15T03:47:02.020 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2339999999999999e-06 2024-09-15T03:47:02.020 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.020 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.020 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:02.020 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104718+0000", 2024-09-15T03:47:02.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3449999999999996e-06 2024-09-15T03:47:02.021 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.021 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104722+0000", 2024-09-15T03:47:02.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8399999999999997e-06 2024-09-15T03:47:02.021 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.021 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104849+0000", 2024-09-15T03:47:02.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012709599999999999 2024-09-15T03:47:02.021 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.022 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:02.022 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104856+0000", 2024-09-15T03:47:02.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.652e-06 2024-09-15T03:47:02.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.022 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.022 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106851+0000", 2024-09-15T03:47:02.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019954840000000001 2024-09-15T03:47:02.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.022 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:02.022 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106935+0000", 2024-09-15T03:47:02.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4095e-05 2024-09-15T03:47:02.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.023 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.023 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.023 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107453+0000", 2024-09-15T03:47:02.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00051729499999999997 2024-09-15T03:47:02.023 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.023 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.023 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.023 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107554+0000", 2024-09-15T03:47:02.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000101251 2024-09-15T03:47:02.023 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.023 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.023 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.023 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108104+0000", 2024-09-15T03:47:02.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00054995100000000002 2024-09-15T03:47:02.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108503+0000", 2024-09-15T03:47:02.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039953500000000002 2024-09-15T03:47:02.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109405+0000", 2024-09-15T03:47:02.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00090170099999999998 2024-09-15T03:47:02.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:02.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114365+0000", 2024-09-15T03:47:02.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049602120000000003 2024-09-15T03:47:02.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:02.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117396+0000", 2024-09-15T03:47:02.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030303370000000001 2024-09-15T03:47:02.025 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.026 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:02.026 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.026 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:186 3.as0 3:5515acd6:::smithi00132172-13:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:47:02.027 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:47:02.027 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.910294628999999, 2024-09-15T03:47:02.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012817485999999999, 2024-09-15T03:47:02.027 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:02.027 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:02.027 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:02.027 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:02.027 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:02.027 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 186 2024-09-15T03:47:02.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.027 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:02.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.027 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:02.028 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:47:02.028 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.028 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:02.028 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:47:02.028 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.028 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:02.028 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104750+0000", 2024-09-15T03:47:02.028 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1500000000000004e-07 2024-09-15T03:47:02.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.029 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:02.029 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104754+0000", 2024-09-15T03:47:02.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.946e-06 2024-09-15T03:47:02.029 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.029 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:02.029 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104769+0000", 2024-09-15T03:47:02.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.507e-05 2024-09-15T03:47:02.029 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.029 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.029 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104776+0000", 2024-09-15T03:47:02.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7849999999999992e-06 2024-09-15T03:47:02.030 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104888+0000", 2024-09-15T03:47:02.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011164500000000001 2024-09-15T03:47:02.030 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:02.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104894+0000", 2024-09-15T03:47:02.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.942e-06 2024-09-15T03:47:02.030 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108172+0000", 2024-09-15T03:47:02.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032781939999999999 2024-09-15T03:47:02.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:02.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108257+0000", 2024-09-15T03:47:02.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4488999999999997e-05 2024-09-15T03:47:02.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108732+0000", 2024-09-15T03:47:02.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047547399999999998 2024-09-15T03:47:02.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108823+0000", 2024-09-15T03:47:02.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1106000000000005e-05 2024-09-15T03:47:02.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108961+0000", 2024-09-15T03:47:02.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013735099999999999 2024-09-15T03:47:02.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109065+0000", 2024-09-15T03:47:02.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104715 2024-09-15T03:47:02.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.033 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109406+0000", 2024-09-15T03:47:02.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034062300000000002 2024-09-15T03:47:02.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.033 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:02.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114529+0000", 2024-09-15T03:47:02.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.005123219 2024-09-15T03:47:02.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.034 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.034 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:02.034 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117566+0000", 2024-09-15T03:47:02.034 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030372120000000001 2024-09-15T03:47:02.034 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.034 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:02.034 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.034 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.034 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.034 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:188 3.fs0 3:fe9c5513:::smithi00132172-47:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:47:02.034 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:47:02.034 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.910211931999999, 2024-09-15T03:47:02.034 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012637061, 2024-09-15T03:47:02.034 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:02.035 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:02.035 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:02.035 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:02.035 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:02.035 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 188 2024-09-15T03:47:02.035 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.035 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:02.035 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.035 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:02.035 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:47:02.035 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.035 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.036 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.036 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:02.036 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:47:02.036 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.036 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.036 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.036 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:02.036 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:47:02.036 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2799999999999996e-07 2024-09-15T03:47:02.036 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.036 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.036 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:02.037 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104835+0000", 2024-09-15T03:47:02.037 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7190000000000001e-06 2024-09-15T03:47:02.037 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:02.037 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104842+0000", 2024-09-15T03:47:02.037 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4580000000000004e-06 2024-09-15T03:47:02.037 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.037 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104846+0000", 2024-09-15T03:47:02.037 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1250000000000003e-06 2024-09-15T03:47:02.037 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.038 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.038 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104884+0000", 2024-09-15T03:47:02.038 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7075e-05 2024-09-15T03:47:02.038 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.038 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.038 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:02.038 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104888+0000", 2024-09-15T03:47:02.038 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8520000000000003e-06 2024-09-15T03:47:02.038 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.038 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.038 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.038 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106860+0000", 2024-09-15T03:47:02.038 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019719440000000002 2024-09-15T03:47:02.038 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.039 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.039 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:02.039 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106938+0000", 2024-09-15T03:47:02.039 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7421000000000003e-05 2024-09-15T03:47:02.039 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.039 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.039 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.039 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107368+0000", 2024-09-15T03:47:02.039 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043042999999999998 2024-09-15T03:47:02.039 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.039 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.039 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.039 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107460+0000", 2024-09-15T03:47:02.039 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1392999999999997e-05 2024-09-15T03:47:02.040 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.040 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.040 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.040 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107473+0000", 2024-09-15T03:47:02.040 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3702e-05 2024-09-15T03:47:02.040 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.040 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.040 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.040 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107504+0000", 2024-09-15T03:47:02.040 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1000000000000001e-05 2024-09-15T03:47:02.040 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.040 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.040 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.041 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108376+0000", 2024-09-15T03:47:02.041 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00087162000000000003 2024-09-15T03:47:02.041 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.041 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.041 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:02.041 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.111657+0000", 2024-09-15T03:47:02.041 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032815589999999999 2024-09-15T03:47:02.041 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.041 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.041 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:02.041 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117469+0000", 2024-09-15T03:47:02.041 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058112349999999997 2024-09-15T03:47:02.041 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.041 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:02.042 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.042 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.042 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.042 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:189 3.fs0 3:fe9c5513:::smithi00132172-47:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:47:02.042 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:47:02.042 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.910192746, 2024-09-15T03:47:02.042 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012731009999999999, 2024-09-15T03:47:02.042 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:02.042 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:02.042 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:02.042 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:02.042 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:02.042 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 189 2024-09-15T03:47:02.042 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.043 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:02.043 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.043 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:02.043 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:47:02.043 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.043 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.043 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.043 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:02.043 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:47:02.043 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.043 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.043 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.043 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:02.043 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:47:02.044 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7899999999999999e-07 2024-09-15T03:47:02.044 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.044 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.044 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:02.044 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104853+0000", 2024-09-15T03:47:02.044 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.933e-06 2024-09-15T03:47:02.044 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.044 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.044 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:02.044 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104859+0000", 2024-09-15T03:47:02.044 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.592e-06 2024-09-15T03:47:02.044 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.044 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.045 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.045 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104863+0000", 2024-09-15T03:47:02.045 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4000000000000002e-06 2024-09-15T03:47:02.045 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.045 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.045 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.045 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104910+0000", 2024-09-15T03:47:02.045 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6678000000000003e-05 2024-09-15T03:47:02.045 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.045 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.045 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:02.045 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104916+0000", 2024-09-15T03:47:02.045 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1079999999999998e-06 2024-09-15T03:47:02.045 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.045 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.046 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.046 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107550+0000", 2024-09-15T03:47:02.046 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026344110000000001 2024-09-15T03:47:02.046 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.046 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.046 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:02.046 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107624+0000", 2024-09-15T03:47:02.046 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4134000000000005e-05 2024-09-15T03:47:02.046 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.046 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.046 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.046 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107988+0000", 2024-09-15T03:47:02.046 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036353600000000002 2024-09-15T03:47:02.046 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.047 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.047 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.047 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108066+0000", 2024-09-15T03:47:02.047 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8047999999999995e-05 2024-09-15T03:47:02.047 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.047 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.047 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.047 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108105+0000", 2024-09-15T03:47:02.047 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8868000000000001e-05 2024-09-15T03:47:02.047 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.047 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.047 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.047 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108504+0000", 2024-09-15T03:47:02.047 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039916600000000001 2024-09-15T03:47:02.048 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.048 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.048 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.048 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108955+0000", 2024-09-15T03:47:02.048 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045126200000000002 2024-09-15T03:47:02.048 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.048 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.048 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:02.048 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114394+0000", 2024-09-15T03:47:02.048 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0054382750000000002 2024-09-15T03:47:02.048 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.048 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.048 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:02.049 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117582+0000", 2024-09-15T03:47:02.049 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031882199999999999 2024-09-15T03:47:02.049 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.049 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:02.049 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.049 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.049 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.049 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 38/31 ECSubWrite(tid=36, reqid=client.4521.0:220, at_version=38'23, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:47:02.049 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.148035+0000", 2024-09-15T03:47:02.049 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.867008640000002, 2024-09-15T03:47:02.049 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010074534, 2024-09-15T03:47:02.049 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:02.049 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.049 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:02.050 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.050 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:02.050 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148035+0000", 2024-09-15T03:47:02.050 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.050 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.050 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.050 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:02.050 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148035+0000", 2024-09-15T03:47:02.050 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.050 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.050 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.050 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:02.050 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148036+0000", 2024-09-15T03:47:02.050 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3939999999999999e-06 2024-09-15T03:47:02.051 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.051 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.051 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:02.051 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148681+0000", 2024-09-15T03:47:02.051 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00064493899999999997 2024-09-15T03:47:02.051 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.051 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.051 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:02.051 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148695+0000", 2024-09-15T03:47:02.051 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4078e-05 2024-09-15T03:47:02.051 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.051 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.051 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.052 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148701+0000", 2024-09-15T03:47:02.052 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7230000000000001e-06 2024-09-15T03:47:02.052 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.052 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.052 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.052 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148742+0000", 2024-09-15T03:47:02.052 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0781000000000001e-05 2024-09-15T03:47:02.052 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.052 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.052 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.052 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148749+0000", 2024-09-15T03:47:02.052 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6109999999999996e-06 2024-09-15T03:47:02.052 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.052 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.053 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.053 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149077+0000", 2024-09-15T03:47:02.053 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000327597 2024-09-15T03:47:02.053 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.053 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.053 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.053 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149087+0000", 2024-09-15T03:47:02.053 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4739999999999994e-06 2024-09-15T03:47:02.053 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.053 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.053 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.053 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149230+0000", 2024-09-15T03:47:02.053 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014398300000000001 2024-09-15T03:47:02.053 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.054 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.054 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.054 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149247+0000", 2024-09-15T03:47:02.054 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6201e-05 2024-09-15T03:47:02.054 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.054 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.054 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:02.054 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158109+0000", 2024-09-15T03:47:02.054 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0088627529999999993 2024-09-15T03:47:02.054 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.054 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:02.054 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.054 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.054 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.055 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 38/31 ECSubWrite(tid=37, reqid=client.4521.0:220, at_version=0'0, trim_to=0'0, roll_forward_to=38'23))", 2024-09-15T03:47:02.055 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.151308+0000", 2024-09-15T03:47:02.055 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.863735394999999, 2024-09-15T03:47:02.055 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069978310000000004, 2024-09-15T03:47:02.055 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:02.055 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.055 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:02.055 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.055 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:02.055 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151308+0000", 2024-09-15T03:47:02.055 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.055 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.055 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.055 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:02.056 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151308+0000", 2024-09-15T03:47:02.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.056 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.056 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.056 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:02.056 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151309+0000", 2024-09-15T03:47:02.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2200000000000003e-07 2024-09-15T03:47:02.056 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.056 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.056 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:02.056 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151318+0000", 2024-09-15T03:47:02.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7250000000000003e-06 2024-09-15T03:47:02.056 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.057 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.057 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:02.057 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151330+0000", 2024-09-15T03:47:02.057 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2425e-05 2024-09-15T03:47:02.057 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.057 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.057 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.057 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151337+0000", 2024-09-15T03:47:02.057 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6139999999999997e-06 2024-09-15T03:47:02.057 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.057 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.057 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.057 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151372+0000", 2024-09-15T03:47:02.057 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5683999999999999e-05 2024-09-15T03:47:02.058 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.058 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.058 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.058 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151378+0000", 2024-09-15T03:47:02.058 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7880000000000003e-06 2024-09-15T03:47:02.058 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.058 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.058 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.058 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151438+0000", 2024-09-15T03:47:02.058 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9679999999999998e-05 2024-09-15T03:47:02.058 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.058 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.058 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.058 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151446+0000", 2024-09-15T03:47:02.059 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.909e-06 2024-09-15T03:47:02.059 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.059 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.059 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.059 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151481+0000", 2024-09-15T03:47:02.059 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5005000000000001e-05 2024-09-15T03:47:02.059 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.059 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.059 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.059 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.151492+0000", 2024-09-15T03:47:02.059 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1133e-05 2024-09-15T03:47:02.059 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.059 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.059 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:02.060 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158306+0000", 2024-09-15T03:47:02.060 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0068140459999999998 2024-09-15T03:47:02.060 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.060 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:02.060 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.060 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.060 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.060 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:231 3.fs0 3:f40d4749:::smithi00132172-35:head [copy-from ver 6] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:02.060 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.155669+0000", 2024-09-15T03:47:02.060 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.859374939999999, 2024-09-15T03:47:02.060 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010370569, 2024-09-15T03:47:02.060 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:02.060 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:02.060 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:02.061 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:02.061 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:02.061 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 231 2024-09-15T03:47:02.061 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.061 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:02.061 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.061 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:02.061 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155669+0000", 2024-09-15T03:47:02.061 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.061 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.061 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.061 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:02.061 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155669+0000", 2024-09-15T03:47:02.061 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.062 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.062 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:02.062 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155671+0000", 2024-09-15T03:47:02.062 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0829999999999998e-06 2024-09-15T03:47:02.062 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.062 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:02.062 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155676+0000", 2024-09-15T03:47:02.062 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6210000000000001e-06 2024-09-15T03:47:02.062 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.062 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:02.063 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155690+0000", 2024-09-15T03:47:02.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4166e-05 2024-09-15T03:47:02.063 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.063 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.063 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.063 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155698+0000", 2024-09-15T03:47:02.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5560000000000002e-06 2024-09-15T03:47:02.063 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.063 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.063 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.063 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155748+0000", 2024-09-15T03:47:02.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9851000000000001e-05 2024-09-15T03:47:02.063 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.063 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.063 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:02.064 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155872+0000", 2024-09-15T03:47:02.064 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000124091 2024-09-15T03:47:02.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.064 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.064 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.064 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163470+0000", 2024-09-15T03:47:02.064 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075976730000000001 2024-09-15T03:47:02.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.064 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.064 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.064 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163801+0000", 2024-09-15T03:47:02.064 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033157999999999999 2024-09-15T03:47:02.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.065 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.065 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.065 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163814+0000", 2024-09-15T03:47:02.065 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2659e-05 2024-09-15T03:47:02.065 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.065 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.065 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.065 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163935+0000", 2024-09-15T03:47:02.065 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000121155 2024-09-15T03:47:02.065 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.065 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.065 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.065 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.164131+0000", 2024-09-15T03:47:02.065 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019600499999999999 2024-09-15T03:47:02.066 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.066 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.066 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:02.066 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.165998+0000", 2024-09-15T03:47:02.066 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001866882 2024-09-15T03:47:02.066 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.066 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.066 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:02.066 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.166039+0000", 2024-09-15T03:47:02.066 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1247e-05 2024-09-15T03:47:02.066 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.066 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:02.066 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.066 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.067 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.067 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:232 3.fs0 3:f40d4749:::smithi00132172-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:02.067 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:47:02.067 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.859337651000001, 2024-09-15T03:47:02.067 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084205730000000006, 2024-09-15T03:47:02.067 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:02.067 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:02.067 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:02.067 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:02.067 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:02.067 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 232 2024-09-15T03:47:02.067 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.067 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:02.067 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.068 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:02.068 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:47:02.068 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.068 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.068 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.068 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:02.068 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:47:02.068 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.068 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.068 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.068 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:02.068 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:47:02.068 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7399999999999999e-07 2024-09-15T03:47:02.069 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.069 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.069 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:02.069 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155708+0000", 2024-09-15T03:47:02.069 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6500000000000001e-06 2024-09-15T03:47:02.069 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.069 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.069 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:02.069 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155713+0000", 2024-09-15T03:47:02.069 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2519999999999999e-06 2024-09-15T03:47:02.069 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.069 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.069 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.069 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155716+0000", 2024-09-15T03:47:02.070 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4430000000000002e-06 2024-09-15T03:47:02.070 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.070 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.070 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.070 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156060+0000", 2024-09-15T03:47:02.070 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034440000000000002 2024-09-15T03:47:02.070 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.070 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.070 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:02.070 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156112+0000", 2024-09-15T03:47:02.070 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1928000000000002e-05 2024-09-15T03:47:02.070 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.070 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.071 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.071 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163929+0000", 2024-09-15T03:47:02.071 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0078168999999999999 2024-09-15T03:47:02.071 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.071 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.071 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:02.071 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.164027+0000", 2024-09-15T03:47:02.071 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8109999999999994e-05 2024-09-15T03:47:02.071 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.071 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.071 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:02.071 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.164126+0000", 2024-09-15T03:47:02.071 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9516000000000001e-05 2024-09-15T03:47:02.071 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.072 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:02.072 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.072 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.072 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.072 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 39/31 ECSubWrite(tid=43, reqid=client.4521.0:252, at_version=39'27, trim_to=0'0, roll_forward_to=39'25))", 2024-09-15T03:47:02.072 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:47:02.072 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.836697958, 2024-09-15T03:47:02.072 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082900309999999998, 2024-09-15T03:47:02.072 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:02.072 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.072 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:02.072 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.072 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:02.073 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:47:02.073 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.073 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.073 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.073 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:02.073 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:47:02.073 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.073 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.073 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.073 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:02.073 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:47:02.073 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6199999999999998e-07 2024-09-15T03:47:02.073 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.073 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.074 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:02.074 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178642+0000", 2024-09-15T03:47:02.074 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029559399999999999 2024-09-15T03:47:02.074 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.074 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.074 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:02.074 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178653+0000", 2024-09-15T03:47:02.074 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0865e-05 2024-09-15T03:47:02.074 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.074 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.074 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.074 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178658+0000", 2024-09-15T03:47:02.075 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3720000000000004e-06 2024-09-15T03:47:02.075 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.075 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.075 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.075 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178695+0000", 2024-09-15T03:47:02.075 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6708999999999997e-05 2024-09-15T03:47:02.075 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.075 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.075 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.075 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178701+0000", 2024-09-15T03:47:02.075 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7989999999999996e-06 2024-09-15T03:47:02.075 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.075 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.075 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.076 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178926+0000", 2024-09-15T03:47:02.076 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022405299999999999 2024-09-15T03:47:02.076 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.076 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.076 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.076 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178935+0000", 2024-09-15T03:47:02.076 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9649999999999997e-06 2024-09-15T03:47:02.076 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.076 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.076 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.076 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179230+0000", 2024-09-15T03:47:02.076 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029551000000000002 2024-09-15T03:47:02.076 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.077 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.077 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.077 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179247+0000", 2024-09-15T03:47:02.077 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6685999999999999e-05 2024-09-15T03:47:02.077 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.077 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.077 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:02.077 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.186636+0000", 2024-09-15T03:47:02.077 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0073889159999999997 2024-09-15T03:47:02.077 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.077 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:02.077 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.077 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 49/31 ECSubWrite(tid=87, reqid=client.4521.0:335, at_version=49'21, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.858534+0000", 2024-09-15T03:47:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.1565098220000003, 2024-09-15T03:47:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094628430000000003, 2024-09-15T03:47:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858534+0000", 2024-09-15T03:47:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858534+0000", 2024-09-15T03:47:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858534+0000", 2024-09-15T03:47:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2300000000000001e-07 2024-09-15T03:47:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858553+0000", 2024-09-15T03:47:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8221000000000001e-05 2024-09-15T03:47:02.080 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.080 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.080 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:02.080 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858564+0000", 2024-09-15T03:47:02.080 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1471000000000001e-05 2024-09-15T03:47:02.080 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.080 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.080 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.080 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858570+0000", 2024-09-15T03:47:02.080 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7640000000000002e-06 2024-09-15T03:47:02.080 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.080 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.080 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.081 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858621+0000", 2024-09-15T03:47:02.081 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0943999999999999e-05 2024-09-15T03:47:02.081 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.081 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.081 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.081 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858626+0000", 2024-09-15T03:47:02.081 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4949999999999999e-06 2024-09-15T03:47:02.081 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.081 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.081 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.081 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858715+0000", 2024-09-15T03:47:02.081 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9061000000000007e-05 2024-09-15T03:47:02.081 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.081 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.082 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.082 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858722+0000", 2024-09-15T03:47:02.082 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9689999999999997e-06 2024-09-15T03:47:02.082 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.082 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.082 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.082 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858748+0000", 2024-09-15T03:47:02.082 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6179000000000001e-05 2024-09-15T03:47:02.082 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.082 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.082 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.082 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858761+0000", 2024-09-15T03:47:02.082 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2159e-05 2024-09-15T03:47:02.083 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.083 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.083 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:02.083 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.867997+0000", 2024-09-15T03:47:02.083 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092359569999999995 2024-09-15T03:47:02.083 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.083 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:02.083 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.083 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.083 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.083 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.cs2 49/28 ECSubWrite(tid=16, reqid=client.4521.0:337, at_version=0'0, trim_to=0'0, roll_forward_to=49'10))", 2024-09-15T03:47:02.083 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.859201+0000", 2024-09-15T03:47:02.083 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.1558421079999999, 2024-09-15T03:47:02.083 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.007966585, 2024-09-15T03:47:02.084 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:02.084 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.084 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:02.084 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:02.084 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859201+0000", 2024-09-15T03:47:02.084 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.084 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.084 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:02.084 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859201+0000", 2024-09-15T03:47:02.084 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.085 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.085 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.085 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:02.085 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859202+0000", 2024-09-15T03:47:02.085 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6899999999999997e-07 2024-09-15T03:47:02.085 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.085 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.085 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:02.085 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859211+0000", 2024-09-15T03:47:02.085 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0799999999999995e-06 2024-09-15T03:47:02.085 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.085 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.085 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:02.086 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859224+0000", 2024-09-15T03:47:02.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2510999999999999e-05 2024-09-15T03:47:02.086 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.086 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.086 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.086 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859230+0000", 2024-09-15T03:47:02.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1569999999999997e-06 2024-09-15T03:47:02.086 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.086 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.086 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.086 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859282+0000", 2024-09-15T03:47:02.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2126000000000002e-05 2024-09-15T03:47:02.086 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.086 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.087 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.087 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859291+0000", 2024-09-15T03:47:02.087 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3000000000000007e-06 2024-09-15T03:47:02.087 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.087 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.087 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.087 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859375+0000", 2024-09-15T03:47:02.087 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3321999999999999e-05 2024-09-15T03:47:02.087 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.087 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.087 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.087 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859383+0000", 2024-09-15T03:47:02.087 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0430000000000008e-06 2024-09-15T03:47:02.087 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.088 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.088 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.088 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859413+0000", 2024-09-15T03:47:02.088 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0510000000000001e-05 2024-09-15T03:47:02.088 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.088 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.088 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.088 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859423+0000", 2024-09-15T03:47:02.088 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0365e-05 2024-09-15T03:47:02.088 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.088 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.088 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:02.088 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.867168+0000", 2024-09-15T03:47:02.088 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0077446019999999997 2024-09-15T03:47:02.089 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.089 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:02.089 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.089 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.089 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.089 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 49/31 ECSubWrite(tid=88, reqid=client.4521.0:336, at_version=49'22, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:02.089 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:47:02.089 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.1557821800000001, 2024-09-15T03:47:02.089 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0088255839999999992, 2024-09-15T03:47:02.089 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:02.089 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.089 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:02.089 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.090 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:02.090 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:47:02.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.090 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.090 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:02.090 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:47:02.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.090 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.090 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:02.090 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859262+0000", 2024-09-15T03:47:02.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2399999999999999e-07 2024-09-15T03:47:02.090 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.091 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.091 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:02.091 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859275+0000", 2024-09-15T03:47:02.091 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3395e-05 2024-09-15T03:47:02.091 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.091 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.091 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:02.091 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859285+0000", 2024-09-15T03:47:02.091 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8959999999999998e-06 2024-09-15T03:47:02.091 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.091 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.091 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.091 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859289+0000", 2024-09-15T03:47:02.092 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2549999999999999e-06 2024-09-15T03:47:02.092 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.092 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.092 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859327+0000", 2024-09-15T03:47:02.092 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7753000000000003e-05 2024-09-15T03:47:02.092 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.092 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.092 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859335+0000", 2024-09-15T03:47:02.092 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3000000000000002e-06 2024-09-15T03:47:02.092 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859391+0000", 2024-09-15T03:47:02.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6085000000000003e-05 2024-09-15T03:47:02.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859415+0000", 2024-09-15T03:47:02.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3295999999999999e-05 2024-09-15T03:47:02.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859432+0000", 2024-09-15T03:47:02.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7087000000000001e-05 2024-09-15T03:47:02.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.094 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859442+0000", 2024-09-15T03:47:02.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0339e-05 2024-09-15T03:47:02.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.094 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:02.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.868087+0000", 2024-09-15T03:47:02.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0086447539999999993 2024-09-15T03:47:02.094 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.094 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:02.094 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.095 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.095 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.095 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.9s7 49/31 ECSubWrite(tid=35, reqid=client.4521.0:331, at_version=49'15, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:47:02.095 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.860608+0000", 2024-09-15T03:47:02.095 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.1544360109999996, 2024-09-15T03:47:02.095 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0074920869999999997, 2024-09-15T03:47:02.095 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:02.095 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.095 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:02.095 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.095 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:02.095 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860608+0000", 2024-09-15T03:47:02.095 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.095 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.096 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:02.096 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860608+0000", 2024-09-15T03:47:02.096 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.096 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.096 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:02.096 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860608+0000", 2024-09-15T03:47:02.096 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9299999999999998e-07 2024-09-15T03:47:02.096 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.096 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:02.096 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860651+0000", 2024-09-15T03:47:02.097 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2463000000000003e-05 2024-09-15T03:47:02.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.097 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:02.097 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860664+0000", 2024-09-15T03:47:02.097 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2948999999999999e-05 2024-09-15T03:47:02.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.097 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.097 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860670+0000", 2024-09-15T03:47:02.097 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8689999999999998e-06 2024-09-15T03:47:02.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.098 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.098 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860709+0000", 2024-09-15T03:47:02.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8339e-05 2024-09-15T03:47:02.098 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.098 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.098 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.098 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860714+0000", 2024-09-15T03:47:02.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4029999999999998e-06 2024-09-15T03:47:02.098 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.098 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.098 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.098 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860775+0000", 2024-09-15T03:47:02.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0988e-05 2024-09-15T03:47:02.098 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.099 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860781+0000", 2024-09-15T03:47:02.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9660000000000001e-06 2024-09-15T03:47:02.099 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.099 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860836+0000", 2024-09-15T03:47:02.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4409e-05 2024-09-15T03:47:02.099 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.099 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860847+0000", 2024-09-15T03:47:02.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1061e-05 2024-09-15T03:47:02.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.100 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:02.100 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.868100+0000", 2024-09-15T03:47:02.100 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0072530470000000003 2024-09-15T03:47:02.100 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.100 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:02.100 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.100 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s6 51/31 ECSubWrite(tid=90, reqid=client.4521.0:346, at_version=51'12, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:02.100 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.842187+0000", 2024-09-15T03:47:02.100 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.1728561209999997, 2024-09-15T03:47:02.100 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082230800000000007, 2024-09-15T03:47:02.101 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:02.101 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.101 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:02.101 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:02.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842187+0000", 2024-09-15T03:47:02.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.101 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:02.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842187+0000", 2024-09-15T03:47:02.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:02.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842188+0000", 2024-09-15T03:47:02.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2899999999999999e-07 2024-09-15T03:47:02.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:02.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842218+0000", 2024-09-15T03:47:02.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0216999999999999e-05 2024-09-15T03:47:02.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:02.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842233+0000", 2024-09-15T03:47:02.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5444999999999999e-05 2024-09-15T03:47:02.103 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.103 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.103 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.103 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842242+0000", 2024-09-15T03:47:02.103 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6090000000000001e-06 2024-09-15T03:47:02.103 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.103 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.103 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.103 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842310+0000", 2024-09-15T03:47:02.103 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7986999999999998e-05 2024-09-15T03:47:02.103 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.103 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.103 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.104 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842322+0000", 2024-09-15T03:47:02.104 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2160999999999999e-05 2024-09-15T03:47:02.104 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.104 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.104 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842617+0000", 2024-09-15T03:47:02.104 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029521700000000002 2024-09-15T03:47:02.104 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.104 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.104 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842640+0000", 2024-09-15T03:47:02.104 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2433999999999999e-05 2024-09-15T03:47:02.104 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842682+0000", 2024-09-15T03:47:02.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1826000000000002e-05 2024-09-15T03:47:02.105 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842842+0000", 2024-09-15T03:47:02.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016048500000000001 2024-09-15T03:47:02.105 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:02.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850411+0000", 2024-09-15T03:47:02.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075683699999999996 2024-09-15T03:47:02.106 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.106 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:02.106 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.106 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.106 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.106 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s6 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:02.106 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.846035+0000", 2024-09-15T03:47:02.106 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.1690089529999996, 2024-09-15T03:47:02.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010491004, 2024-09-15T03:47:02.106 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:02.106 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.106 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:02.106 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:02.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846035+0000", 2024-09-15T03:47:02.107 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:02.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846035+0000", 2024-09-15T03:47:02.107 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:02.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846035+0000", 2024-09-15T03:47:02.107 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8599999999999997e-07 2024-09-15T03:47:02.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.108 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:02.108 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846068+0000", 2024-09-15T03:47:02.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2138000000000003e-05 2024-09-15T03:47:02.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.108 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:02.108 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846081+0000", 2024-09-15T03:47:02.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3127e-05 2024-09-15T03:47:02.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.108 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.108 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846092+0000", 2024-09-15T03:47:02.109 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1634000000000001e-05 2024-09-15T03:47:02.109 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.109 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.109 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.109 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846136+0000", 2024-09-15T03:47:02.109 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3999999999999999e-05 2024-09-15T03:47:02.109 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.109 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.109 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.109 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846148+0000", 2024-09-15T03:47:02.109 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1939e-05 2024-09-15T03:47:02.109 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.109 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.110 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.110 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846220+0000", 2024-09-15T03:47:02.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1248999999999995e-05 2024-09-15T03:47:02.110 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.110 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.110 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846231+0000", 2024-09-15T03:47:02.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1379999999999999e-05 2024-09-15T03:47:02.110 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.110 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.110 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847412+0000", 2024-09-15T03:47:02.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001181407 2024-09-15T03:47:02.111 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.111 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847435+0000", 2024-09-15T03:47:02.111 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3084000000000001e-05 2024-09-15T03:47:02.111 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:02.111 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856526+0000", 2024-09-15T03:47:02.111 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0090901599999999999 2024-09-15T03:47:02.111 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.111 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:02.111 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.111 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.112 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.112 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s6 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:02.112 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.846101+0000", 2024-09-15T03:47:02.112 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.1689421449999999, 2024-09-15T03:47:02.112 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010577013999999999, 2024-09-15T03:47:02.112 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:02.112 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.112 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:02.112 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.112 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:02.112 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846101+0000", 2024-09-15T03:47:02.112 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.112 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:02.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846101+0000", 2024-09-15T03:47:02.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:02.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846102+0000", 2024-09-15T03:47:02.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5100000000000001e-07 2024-09-15T03:47:02.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:02.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846110+0000", 2024-09-15T03:47:02.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7069999999999998e-06 2024-09-15T03:47:02.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:02.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846115+0000", 2024-09-15T03:47:02.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2799999999999997e-06 2024-09-15T03:47:02.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846117+0000", 2024-09-15T03:47:02.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5909999999999998e-06 2024-09-15T03:47:02.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846259+0000", 2024-09-15T03:47:02.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014169900000000001 2024-09-15T03:47:02.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846268+0000", 2024-09-15T03:47:02.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5159999999999994e-06 2024-09-15T03:47:02.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846335+0000", 2024-09-15T03:47:02.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7936999999999997e-05 2024-09-15T03:47:02.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.116 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.116 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847415+0000", 2024-09-15T03:47:02.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001079874 2024-09-15T03:47:02.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.116 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.116 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847453+0000", 2024-09-15T03:47:02.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7808999999999997e-05 2024-09-15T03:47:02.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.116 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.116 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847477+0000", 2024-09-15T03:47:02.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3609e-05 2024-09-15T03:47:02.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.117 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:02.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856678+0000", 2024-09-15T03:47:02.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.009201641 2024-09-15T03:47:02.117 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.117 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:02.117 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.117 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 51/31 ECSubWrite(tid=93, reqid=client.4521.0:352, at_version=51'24, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:02.117 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.847310+0000", 2024-09-15T03:47:02.117 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.1677338390000003, 2024-09-15T03:47:02.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012670010000000001, 2024-09-15T03:47:02.118 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:02.118 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.118 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:02.118 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.118 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:02.118 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847310+0000", 2024-09-15T03:47:02.118 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.118 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.118 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.118 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:02.118 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847310+0000", 2024-09-15T03:47:02.118 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.118 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.118 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.119 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:02.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847310+0000", 2024-09-15T03:47:02.119 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8899999999999999e-07 2024-09-15T03:47:02.119 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.119 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:02.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847750+0000", 2024-09-15T03:47:02.119 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000439238 2024-09-15T03:47:02.119 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.119 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:02.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847767+0000", 2024-09-15T03:47:02.119 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7362000000000001e-05 2024-09-15T03:47:02.119 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.120 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.120 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.120 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847774+0000", 2024-09-15T03:47:02.120 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0169999999999999e-06 2024-09-15T03:47:02.120 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.120 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.120 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.120 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847831+0000", 2024-09-15T03:47:02.120 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7207999999999998e-05 2024-09-15T03:47:02.121 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.121 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.121 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.121 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847841+0000", 2024-09-15T03:47:02.121 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7829999999999993e-06 2024-09-15T03:47:02.121 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.121 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.121 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.121 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.848145+0000", 2024-09-15T03:47:02.121 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030431300000000001 2024-09-15T03:47:02.121 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.121 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.121 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.122 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.848179+0000", 2024-09-15T03:47:02.122 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3423999999999997e-05 2024-09-15T03:47:02.122 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.122 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.122 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.122 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850141+0000", 2024-09-15T03:47:02.122 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019623380000000001 2024-09-15T03:47:02.122 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.122 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.122 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.122 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850164+0000", 2024-09-15T03:47:02.122 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2969e-05 2024-09-15T03:47:02.122 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.123 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.123 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:02.123 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.859980+0000", 2024-09-15T03:47:02.123 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098156690000000008 2024-09-15T03:47:02.123 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.123 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:02.123 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.123 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.123 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.123 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 51/31 ECSubWrite(tid=94, reqid=client.4521.0:353, at_version=51'25, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:02.123 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.850957+0000", 2024-09-15T03:47:02.123 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.1640860499999999, 2024-09-15T03:47:02.123 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092043439999999997, 2024-09-15T03:47:02.124 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:02.124 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.124 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:02.124 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.124 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:02.124 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850957+0000", 2024-09-15T03:47:02.124 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.124 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.124 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.124 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:02.124 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850957+0000", 2024-09-15T03:47:02.124 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.124 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.124 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.125 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:02.125 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850958+0000", 2024-09-15T03:47:02.125 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9500000000000002e-07 2024-09-15T03:47:02.125 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.125 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.125 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:02.125 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851297+0000", 2024-09-15T03:47:02.125 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033833300000000001 2024-09-15T03:47:02.125 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.126 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.126 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:02.126 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851311+0000", 2024-09-15T03:47:02.126 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4144999999999999e-05 2024-09-15T03:47:02.126 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.126 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.126 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.126 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851318+0000", 2024-09-15T03:47:02.126 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8700000000000003e-06 2024-09-15T03:47:02.127 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.127 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.127 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.127 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851390+0000", 2024-09-15T03:47:02.127 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.203e-05 2024-09-15T03:47:02.127 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.127 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.127 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.127 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851402+0000", 2024-09-15T03:47:02.127 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2146e-05 2024-09-15T03:47:02.127 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.127 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.127 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.127 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851556+0000", 2024-09-15T03:47:02.128 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015461199999999999 2024-09-15T03:47:02.128 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.128 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.128 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.128 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851570+0000", 2024-09-15T03:47:02.128 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4106e-05 2024-09-15T03:47:02.128 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.128 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.128 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.128 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853231+0000", 2024-09-15T03:47:02.128 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001660559 2024-09-15T03:47:02.128 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.128 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.128 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.129 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853246+0000", 2024-09-15T03:47:02.129 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5457000000000001e-05 2024-09-15T03:47:02.129 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.129 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.129 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:02.129 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.860162+0000", 2024-09-15T03:47:02.129 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069153909999999999 2024-09-15T03:47:02.129 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.129 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:02.129 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.129 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.129 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.129 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 51/31 ECSubWrite(tid=95, reqid=client.4521.0:354, at_version=51'26, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:02.130 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.851558+0000", 2024-09-15T03:47:02.130 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.1634859119999996, 2024-09-15T03:47:02.130 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0087097059999999994, 2024-09-15T03:47:02.130 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:02.130 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.130 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:02.130 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.130 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:02.130 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851558+0000", 2024-09-15T03:47:02.130 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.130 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.130 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.130 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:02.130 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851558+0000", 2024-09-15T03:47:02.130 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.131 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.131 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.131 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:02.131 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851559+0000", 2024-09-15T03:47:02.131 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7599999999999996e-07 2024-09-15T03:47:02.131 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.131 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.131 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:02.131 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851599+0000", 2024-09-15T03:47:02.131 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0472000000000001e-05 2024-09-15T03:47:02.131 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.131 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.131 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:02.132 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851614+0000", 2024-09-15T03:47:02.132 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4640999999999999e-05 2024-09-15T03:47:02.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.132 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.132 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851620+0000", 2024-09-15T03:47:02.132 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3430000000000001e-06 2024-09-15T03:47:02.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.132 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.132 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851680+0000", 2024-09-15T03:47:02.132 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9648000000000002e-05 2024-09-15T03:47:02.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.133 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851691+0000", 2024-09-15T03:47:02.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1012999999999999e-05 2024-09-15T03:47:02.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.133 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.133 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851803+0000", 2024-09-15T03:47:02.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112089 2024-09-15T03:47:02.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.133 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.133 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853234+0000", 2024-09-15T03:47:02.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0014310810000000001 2024-09-15T03:47:02.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853304+0000", 2024-09-15T03:47:02.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0656999999999998e-05 2024-09-15T03:47:02.134 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853317+0000", 2024-09-15T03:47:02.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2186e-05 2024-09-15T03:47:02.134 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:02.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.860267+0000", 2024-09-15T03:47:02.135 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069506999999999998 2024-09-15T03:47:02.135 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.135 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:02.135 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.135 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.135 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 51/31 ECSubWrite(tid=96, reqid=client.4521.0:355, at_version=51'27, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:02.135 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.852498+0000", 2024-09-15T03:47:02.135 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.1625459979999997, 2024-09-15T03:47:02.135 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0078680220000000006, 2024-09-15T03:47:02.135 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:02.135 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.135 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:02.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.136 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:02.136 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.852498+0000", 2024-09-15T03:47:02.136 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.136 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.136 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.136 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:02.136 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.852498+0000", 2024-09-15T03:47:02.136 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:02.136 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.136 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.136 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:02.136 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.852498+0000", 2024-09-15T03:47:02.136 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8800000000000002e-07 2024-09-15T03:47:02.136 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:02.137 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.852512+0000", 2024-09-15T03:47:02.137 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3961000000000001e-05 2024-09-15T03:47:02.137 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:02.137 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.852524+0000", 2024-09-15T03:47:02.137 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2014000000000001e-05 2024-09-15T03:47:02.137 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.137 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.852531+0000", 2024-09-15T03:47:02.138 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7479999999999996e-06 2024-09-15T03:47:02.138 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.138 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.138 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:02.138 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.852577+0000", 2024-09-15T03:47:02.138 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6514999999999998e-05 2024-09-15T03:47:02.138 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.138 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.138 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.138 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.852585+0000", 2024-09-15T03:47:02.138 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9200000000000004e-06 2024-09-15T03:47:02.138 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.138 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.138 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.139 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.852651+0000", 2024-09-15T03:47:02.139 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5809000000000006e-05 2024-09-15T03:47:02.139 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.139 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.139 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.139 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853235+0000", 2024-09-15T03:47:02.139 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00058343299999999996 2024-09-15T03:47:02.139 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.139 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.139 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.139 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853305+0000", 2024-09-15T03:47:02.139 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0504000000000001e-05 2024-09-15T03:47:02.139 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.139 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.140 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.140 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853332+0000", 2024-09-15T03:47:02.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7189999999999999e-05 2024-09-15T03:47:02.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:02.140 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:02.140 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:02.140 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.860366+0000", 2024-09-15T03:47:02.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0070333399999999999 2024-09-15T03:47:02.140 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.140 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:02.140 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.140 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:02.140 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:02.140 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:02.142 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:47:02.143 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:02.141+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:02.243 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:02.243 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:02.243 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-15T03:47:02.243 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:02.243 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:02.242+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:02.251 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:47:02.343 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:02.342+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:02.351 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:02.351 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:02.351 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-15T03:47:02.352 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-15T03:47:02.352 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:02.360 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:47:02.445 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:02.443+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:02.462 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:02.462 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-15T03:47:02.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-15T03:47:02.462 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-15T03:47:02.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.462 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs2 33/31 ECSubWrite(tid=22, reqid=client.4521.0:187, at_version=33'8, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:02.462 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:40.107032+0000", 2024-09-15T03:47:02.462 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.353099791999998, 2024-09-15T03:47:02.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0038892639999999999, 2024-09-15T03:47:02.462 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.462 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.463 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107032+0000", 2024-09-15T03:47:02.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107032+0000", 2024-09-15T03:47:02.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107117+0000", 2024-09-15T03:47:02.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4419999999999995e-05 2024-09-15T03:47:02.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107140+0000", 2024-09-15T03:47:02.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3696000000000001e-05 2024-09-15T03:47:02.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107158+0000", 2024-09-15T03:47:02.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7672e-05 2024-09-15T03:47:02.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.465 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.465 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107166+0000", 2024-09-15T03:47:02.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1529999999999997e-06 2024-09-15T03:47:02.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.465 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.465 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107230+0000", 2024-09-15T03:47:02.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3924999999999995e-05 2024-09-15T03:47:02.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.465 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.465 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107241+0000", 2024-09-15T03:47:02.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0712e-05 2024-09-15T03:47:02.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107460+0000", 2024-09-15T03:47:02.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021953599999999999 2024-09-15T03:47:02.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107738+0000", 2024-09-15T03:47:02.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00027775800000000002 2024-09-15T03:47:02.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.466 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107795+0000", 2024-09-15T03:47:02.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6867999999999998e-05 2024-09-15T03:47:02.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108035+0000", 2024-09-15T03:47:02.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024024199999999999 2024-09-15T03:47:02.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.467 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.467 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.110922+0000", 2024-09-15T03:47:02.467 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002886282 2024-09-15T03:47:02.467 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.467 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.468 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.468 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs2 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:02.468 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:40.108717+0000", 2024-09-15T03:47:02.468 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.351415234000001, 2024-09-15T03:47:02.468 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041586089999999997, 2024-09-15T03:47:02.468 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.468 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.468 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.468 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.468 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108717+0000", 2024-09-15T03:47:02.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108717+0000", 2024-09-15T03:47:02.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.469 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108718+0000", 2024-09-15T03:47:02.469 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.412e-06 2024-09-15T03:47:02.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.469 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108755+0000", 2024-09-15T03:47:02.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6798000000000002e-05 2024-09-15T03:47:02.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108771+0000", 2024-09-15T03:47:02.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6002e-05 2024-09-15T03:47:02.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.470 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.470 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108779+0000", 2024-09-15T03:47:02.470 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5429999999999996e-06 2024-09-15T03:47:02.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108830+0000", 2024-09-15T03:47:02.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1127e-05 2024-09-15T03:47:02.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108840+0000", 2024-09-15T03:47:02.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8460000000000003e-06 2024-09-15T03:47:02.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.471 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.471 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108951+0000", 2024-09-15T03:47:02.471 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011156 2024-09-15T03:47:02.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.109109+0000", 2024-09-15T03:47:02.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015739700000000001 2024-09-15T03:47:02.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.472 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.472 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.109163+0000", 2024-09-15T03:47:02.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4026000000000001e-05 2024-09-15T03:47:02.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.110172+0000", 2024-09-15T03:47:02.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001009362 2024-09-15T03:47:02.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.112876+0000", 2024-09-15T03:47:02.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0027035359999999999 2024-09-15T03:47:02.473 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.474 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.474 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.474 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs2 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:02.474 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:40.108796+0000", 2024-09-15T03:47:02.474 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.351336337999999, 2024-09-15T03:47:02.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0042266949999999999, 2024-09-15T03:47:02.474 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.474 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.474 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108796+0000", 2024-09-15T03:47:02.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108796+0000", 2024-09-15T03:47:02.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108796+0000", 2024-09-15T03:47:02.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8599999999999998e-07 2024-09-15T03:47:02.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108818+0000", 2024-09-15T03:47:02.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1206999999999999e-05 2024-09-15T03:47:02.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108828+0000", 2024-09-15T03:47:02.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0321e-05 2024-09-15T03:47:02.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108834+0000", 2024-09-15T03:47:02.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0229999999999998e-06 2024-09-15T03:47:02.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108999+0000", 2024-09-15T03:47:02.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016466399999999999 2024-09-15T03:47:02.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.109005+0000", 2024-09-15T03:47:02.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.973e-06 2024-09-15T03:47:02.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.109086+0000", 2024-09-15T03:47:02.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0136999999999995e-05 2024-09-15T03:47:02.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.109109+0000", 2024-09-15T03:47:02.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3822999999999999e-05 2024-09-15T03:47:02.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.109163+0000", 2024-09-15T03:47:02.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4029000000000001e-05 2024-09-15T03:47:02.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.110210+0000", 2024-09-15T03:47:02.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010464859999999999 2024-09-15T03:47:02.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.479 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.479 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.113023+0000", 2024-09-15T03:47:02.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002812546 2024-09-15T03:47:02.479 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.479 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.479 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.479 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.es1 34/28 ECSubWrite(tid=33, reqid=client.4521.0:195, at_version=34'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-15T03:47:02.479 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:41.117958+0000", 2024-09-15T03:47:02.480 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.342174477, 2024-09-15T03:47:02.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0027557810000000001, 2024-09-15T03:47:02.480 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.480 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.480 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.480 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.117958+0000", 2024-09-15T03:47:02.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.480 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.117958+0000", 2024-09-15T03:47:02.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.481 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.117958+0000", 2024-09-15T03:47:02.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4299999999999998e-07 2024-09-15T03:47:02.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.481 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118032+0000", 2024-09-15T03:47:02.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3400999999999997e-05 2024-09-15T03:47:02.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118053+0000", 2024-09-15T03:47:02.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1123000000000001e-05 2024-09-15T03:47:02.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118062+0000", 2024-09-15T03:47:02.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1600000000000004e-06 2024-09-15T03:47:02.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118148+0000", 2024-09-15T03:47:02.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6266000000000004e-05 2024-09-15T03:47:02.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118159+0000", 2024-09-15T03:47:02.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1358999999999999e-05 2024-09-15T03:47:02.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118479+0000", 2024-09-15T03:47:02.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031943999999999999 2024-09-15T03:47:02.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118492+0000", 2024-09-15T03:47:02.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3532e-05 2024-09-15T03:47:02.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118607+0000", 2024-09-15T03:47:02.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114376 2024-09-15T03:47:02.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118654+0000", 2024-09-15T03:47:02.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7524000000000001e-05 2024-09-15T03:47:02.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.120713+0000", 2024-09-15T03:47:02.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020591569999999998 2024-09-15T03:47:02.485 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.485 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.485 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.485 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.5s7 38/31 ECSubWrite(tid=38, reqid=client.4521.0:223, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:47:02.485 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:45.146773+0000", 2024-09-15T03:47:02.485 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.313358715, 2024-09-15T03:47:02.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041833800000000004, 2024-09-15T03:47:02.485 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.485 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.485 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.146773+0000", 2024-09-15T03:47:02.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.146773+0000", 2024-09-15T03:47:02.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.146774+0000", 2024-09-15T03:47:02.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.37e-07 2024-09-15T03:47:02.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.146888+0000", 2024-09-15T03:47:02.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114219 2024-09-15T03:47:02.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.146901+0000", 2024-09-15T03:47:02.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2551999999999999e-05 2024-09-15T03:47:02.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.146912+0000", 2024-09-15T03:47:02.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0881e-05 2024-09-15T03:47:02.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.147069+0000", 2024-09-15T03:47:02.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015756900000000001 2024-09-15T03:47:02.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.147081+0000", 2024-09-15T03:47:02.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2153999999999999e-05 2024-09-15T03:47:02.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148051+0000", 2024-09-15T03:47:02.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00096960899999999997 2024-09-15T03:47:02.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148083+0000", 2024-09-15T03:47:02.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2141999999999998e-05 2024-09-15T03:47:02.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148249+0000", 2024-09-15T03:47:02.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000165833 2024-09-15T03:47:02.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148284+0000", 2024-09-15T03:47:02.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4953000000000002e-05 2024-09-15T03:47:02.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.150957+0000", 2024-09-15T03:47:02.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0026729309999999999 2024-09-15T03:47:02.490 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.490 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.490 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.490 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.es1 38/28 ECSubWrite(tid=45, reqid=client.4521.0:227, at_version=38'24, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:47:02.490 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:45.155077+0000", 2024-09-15T03:47:02.491 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.305055586999998, 2024-09-15T03:47:02.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0028777379999999999, 2024-09-15T03:47:02.491 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.491 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.491 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155077+0000", 2024-09-15T03:47:02.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155077+0000", 2024-09-15T03:47:02.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155077+0000", 2024-09-15T03:47:02.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7400000000000002e-07 2024-09-15T03:47:02.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155170+0000", 2024-09-15T03:47:02.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2720999999999995e-05 2024-09-15T03:47:02.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.493 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.493 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155187+0000", 2024-09-15T03:47:02.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6818000000000001e-05 2024-09-15T03:47:02.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.493 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.493 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155195+0000", 2024-09-15T03:47:02.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5499999999999997e-06 2024-09-15T03:47:02.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.493 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.493 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155415+0000", 2024-09-15T03:47:02.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022020199999999999 2024-09-15T03:47:02.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155450+0000", 2024-09-15T03:47:02.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4999999999999997e-05 2024-09-15T03:47:02.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.156476+0000", 2024-09-15T03:47:02.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010262380000000001 2024-09-15T03:47:02.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.156489+0000", 2024-09-15T03:47:02.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2937e-05 2024-09-15T03:47:02.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.156655+0000", 2024-09-15T03:47:02.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016607899999999999 2024-09-15T03:47:02.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.156681+0000", 2024-09-15T03:47:02.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5707e-05 2024-09-15T03:47:02.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.157954+0000", 2024-09-15T03:47:02.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012736119999999999 2024-09-15T03:47:02.496 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.496 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.496 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.496 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.4s4 38/31 ECSubWrite(tid=38, reqid=client.4521.0:237, at_version=38'6, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:02.496 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:45.158151+0000", 2024-09-15T03:47:02.496 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.301980771, 2024-09-15T03:47:02.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0035353099999999998, 2024-09-15T03:47:02.496 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.496 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.497 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158151+0000", 2024-09-15T03:47:02.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158151+0000", 2024-09-15T03:47:02.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158152+0000", 2024-09-15T03:47:02.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0500000000000003e-07 2024-09-15T03:47:02.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158174+0000", 2024-09-15T03:47:02.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2272999999999998e-05 2024-09-15T03:47:02.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158187+0000", 2024-09-15T03:47:02.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2291e-05 2024-09-15T03:47:02.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158193+0000", 2024-09-15T03:47:02.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.776e-06 2024-09-15T03:47:02.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158246+0000", 2024-09-15T03:47:02.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2858000000000001e-05 2024-09-15T03:47:02.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158254+0000", 2024-09-15T03:47:02.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1920000000000005e-06 2024-09-15T03:47:02.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158341+0000", 2024-09-15T03:47:02.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6136000000000003e-05 2024-09-15T03:47:02.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158368+0000", 2024-09-15T03:47:02.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6976000000000001e-05 2024-09-15T03:47:02.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158404+0000", 2024-09-15T03:47:02.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5988000000000002e-05 2024-09-15T03:47:02.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158415+0000", 2024-09-15T03:47:02.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1697e-05 2024-09-15T03:47:02.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.161687+0000", 2024-09-15T03:47:02.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032714179999999999 2024-09-15T03:47:02.501 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.501 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.501 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.502 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.4s4 38/31 ECSubWrite(tid=39, reqid=client.4521.0:238, at_version=38'7, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:02.502 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:45.159152+0000", 2024-09-15T03:47:02.502 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.300980065000001, 2024-09-15T03:47:02.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0027662569999999998, 2024-09-15T03:47:02.502 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.502 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.502 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.159152+0000", 2024-09-15T03:47:02.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.159152+0000", 2024-09-15T03:47:02.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.159153+0000", 2024-09-15T03:47:02.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1500000000000004e-07 2024-09-15T03:47:02.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.159172+0000", 2024-09-15T03:47:02.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9143e-05 2024-09-15T03:47:02.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.159190+0000", 2024-09-15T03:47:02.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8380999999999999e-05 2024-09-15T03:47:02.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.159197+0000", 2024-09-15T03:47:02.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9759999999999998e-06 2024-09-15T03:47:02.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.159293+0000", 2024-09-15T03:47:02.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5909999999999995e-05 2024-09-15T03:47:02.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.159302+0000", 2024-09-15T03:47:02.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6449999999999994e-06 2024-09-15T03:47:02.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.159389+0000", 2024-09-15T03:47:02.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7303000000000001e-05 2024-09-15T03:47:02.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.159414+0000", 2024-09-15T03:47:02.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4735e-05 2024-09-15T03:47:02.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.159455+0000", 2024-09-15T03:47:02.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1326999999999999e-05 2024-09-15T03:47:02.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.159469+0000", 2024-09-15T03:47:02.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4358999999999999e-05 2024-09-15T03:47:02.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.161918+0000", 2024-09-15T03:47:02.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024488629999999999 2024-09-15T03:47:02.507 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.507 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.507 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.508 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.4s4 39/31 ECSubWrite(tid=41, reqid=osd.6.0:41, at_version=39'9, trim_to=0'0, roll_forward_to=38'7))", 2024-09-15T03:47:02.508 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:46.146606+0000", 2024-09-15T03:47:02.508 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.313526206999999, 2024-09-15T03:47:02.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0048345970000000004, 2024-09-15T03:47:02.508 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.508 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.508 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.146606+0000", 2024-09-15T03:47:02.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.146606+0000", 2024-09-15T03:47:02.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.146606+0000", 2024-09-15T03:47:02.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3200000000000005e-07 2024-09-15T03:47:02.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.146636+0000", 2024-09-15T03:47:02.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9434e-05 2024-09-15T03:47:02.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.146666+0000", 2024-09-15T03:47:02.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9733000000000001e-05 2024-09-15T03:47:02.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.146674+0000", 2024-09-15T03:47:02.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2919999999999996e-06 2024-09-15T03:47:02.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150070+0000", 2024-09-15T03:47:02.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0033961590000000002 2024-09-15T03:47:02.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150080+0000", 2024-09-15T03:47:02.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0181e-05 2024-09-15T03:47:02.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150334+0000", 2024-09-15T03:47:02.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00025387100000000002 2024-09-15T03:47:02.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150370+0000", 2024-09-15T03:47:02.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6269999999999997e-05 2024-09-15T03:47:02.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150430+0000", 2024-09-15T03:47:02.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9404000000000002e-05 2024-09-15T03:47:02.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150480+0000", 2024-09-15T03:47:02.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0541000000000003e-05 2024-09-15T03:47:02.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.151441+0000", 2024-09-15T03:47:02.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00096018000000000004 2024-09-15T03:47:02.513 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.513 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.513 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.513 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.0s6 39/31 ECSubWrite(tid=43, reqid=client.4521.0:240, at_version=39'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:02.513 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:46.148496+0000", 2024-09-15T03:47:02.513 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.311635766999999, 2024-09-15T03:47:02.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0031001290000000001, 2024-09-15T03:47:02.514 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.514 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.514 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.148496+0000", 2024-09-15T03:47:02.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.148496+0000", 2024-09-15T03:47:02.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.148497+0000", 2024-09-15T03:47:02.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8299999999999998e-07 2024-09-15T03:47:02.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.148524+0000", 2024-09-15T03:47:02.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.705e-05 2024-09-15T03:47:02.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.148541+0000", 2024-09-15T03:47:02.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7238999999999999e-05 2024-09-15T03:47:02.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.148565+0000", 2024-09-15T03:47:02.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4045000000000001e-05 2024-09-15T03:47:02.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.148623+0000", 2024-09-15T03:47:02.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7635e-05 2024-09-15T03:47:02.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.148633+0000", 2024-09-15T03:47:02.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.012e-05 2024-09-15T03:47:02.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.148900+0000", 2024-09-15T03:47:02.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026675199999999998 2024-09-15T03:47:02.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149204+0000", 2024-09-15T03:47:02.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00030492699999999998 2024-09-15T03:47:02.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149261+0000", 2024-09-15T03:47:02.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6801e-05 2024-09-15T03:47:02.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149290+0000", 2024-09-15T03:47:02.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8688999999999999e-05 2024-09-15T03:47:02.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.151597+0000", 2024-09-15T03:47:02.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002306588 2024-09-15T03:47:02.519 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.519 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.519 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.519 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.0s6 39/31 ECSubWrite(tid=45, reqid=client.4521.0:241, at_version=39'17, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:02.519 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:46.153279+0000", 2024-09-15T03:47:02.519 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.306853500999999, 2024-09-15T03:47:02.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032250170000000002, 2024-09-15T03:47:02.519 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.519 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.519 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153279+0000", 2024-09-15T03:47:02.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153279+0000", 2024-09-15T03:47:02.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153279+0000", 2024-09-15T03:47:02.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8099999999999998e-07 2024-09-15T03:47:02.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153299+0000", 2024-09-15T03:47:02.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9947999999999999e-05 2024-09-15T03:47:02.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153311+0000", 2024-09-15T03:47:02.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.182e-05 2024-09-15T03:47:02.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153474+0000", 2024-09-15T03:47:02.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016328800000000001 2024-09-15T03:47:02.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153521+0000", 2024-09-15T03:47:02.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7033999999999997e-05 2024-09-15T03:47:02.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153530+0000", 2024-09-15T03:47:02.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2199999999999998e-06 2024-09-15T03:47:02.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153623+0000", 2024-09-15T03:47:02.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3095000000000002e-05 2024-09-15T03:47:02.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153644+0000", 2024-09-15T03:47:02.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0560999999999998e-05 2024-09-15T03:47:02.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153700+0000", 2024-09-15T03:47:02.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6397000000000002e-05 2024-09-15T03:47:02.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153723+0000", 2024-09-15T03:47:02.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2855000000000001e-05 2024-09-15T03:47:02.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.156504+0000", 2024-09-15T03:47:02.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0027804180000000002 2024-09-15T03:47:02.525 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.525 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.525 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.525 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.0s6 39/31 ECSubWrite(tid=46, reqid=client.4521.0:242, at_version=39'18, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:02.525 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:46.153652+0000", 2024-09-15T03:47:02.525 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.306479843999998, 2024-09-15T03:47:02.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032006410000000002, 2024-09-15T03:47:02.525 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.526 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.526 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153652+0000", 2024-09-15T03:47:02.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153652+0000", 2024-09-15T03:47:02.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153653+0000", 2024-09-15T03:47:02.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1900000000000003e-07 2024-09-15T03:47:02.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153679+0000", 2024-09-15T03:47:02.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6573000000000001e-05 2024-09-15T03:47:02.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153696+0000", 2024-09-15T03:47:02.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6308999999999999e-05 2024-09-15T03:47:02.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153703+0000", 2024-09-15T03:47:02.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8170000000000001e-06 2024-09-15T03:47:02.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153922+0000", 2024-09-15T03:47:02.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000219237 2024-09-15T03:47:02.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153931+0000", 2024-09-15T03:47:02.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9649999999999997e-06 2024-09-15T03:47:02.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.154023+0000", 2024-09-15T03:47:02.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1972999999999995e-05 2024-09-15T03:47:02.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.154051+0000", 2024-09-15T03:47:02.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7936000000000002e-05 2024-09-15T03:47:02.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.154097+0000", 2024-09-15T03:47:02.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6674000000000001e-05 2024-09-15T03:47:02.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.154144+0000", 2024-09-15T03:47:02.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6399000000000001e-05 2024-09-15T03:47:02.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.156853+0000", 2024-09-15T03:47:02.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002709239 2024-09-15T03:47:02.531 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.531 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.531 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.531 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.3s7 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:47:02.531 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:46.170728+0000", 2024-09-15T03:47:02.531 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.289403879999998, 2024-09-15T03:47:02.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0037502820000000002, 2024-09-15T03:47:02.532 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.532 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.532 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.170728+0000", 2024-09-15T03:47:02.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.170728+0000", 2024-09-15T03:47:02.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.170730+0000", 2024-09-15T03:47:02.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.483e-06 2024-09-15T03:47:02.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171114+0000", 2024-09-15T03:47:02.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00038467500000000001 2024-09-15T03:47:02.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171135+0000", 2024-09-15T03:47:02.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0962999999999999e-05 2024-09-15T03:47:02.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171145+0000", 2024-09-15T03:47:02.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2450000000000004e-06 2024-09-15T03:47:02.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171287+0000", 2024-09-15T03:47:02.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014246899999999999 2024-09-15T03:47:02.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171299+0000", 2024-09-15T03:47:02.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1574e-05 2024-09-15T03:47:02.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171685+0000", 2024-09-15T03:47:02.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00038670300000000001 2024-09-15T03:47:02.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171716+0000", 2024-09-15T03:47:02.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0919e-05 2024-09-15T03:47:02.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171821+0000", 2024-09-15T03:47:02.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000104609 2024-09-15T03:47:02.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171911+0000", 2024-09-15T03:47:02.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0348999999999998e-05 2024-09-15T03:47:02.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.174479+0000", 2024-09-15T03:47:02.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002567293 2024-09-15T03:47:02.536 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.537 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.537 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.537 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.3s7 47/31 ECSubWrite(tid=60, reqid=osd.6.0:60, at_version=47'30, trim_to=0'0, roll_forward_to=41'26))", 2024-09-15T03:47:02.537 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:54.192946+0000", 2024-09-15T03:47:02.537 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 8.2671860539999997, 2024-09-15T03:47:02.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003164895, 2024-09-15T03:47:02.537 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.537 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.537 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:54.192946+0000", 2024-09-15T03:47:02.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:54.192946+0000", 2024-09-15T03:47:02.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:54.192947+0000", 2024-09-15T03:47:02.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.15e-07 2024-09-15T03:47:02.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:54.192969+0000", 2024-09-15T03:47:02.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2362999999999999e-05 2024-09-15T03:47:02.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:54.192982+0000", 2024-09-15T03:47:02.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3124e-05 2024-09-15T03:47:02.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:54.192989+0000", 2024-09-15T03:47:02.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.579e-06 2024-09-15T03:47:02.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:54.193034+0000", 2024-09-15T03:47:02.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5658000000000002e-05 2024-09-15T03:47:02.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:54.193043+0000", 2024-09-15T03:47:02.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.704e-06 2024-09-15T03:47:02.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:54.193235+0000", 2024-09-15T03:47:02.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019195700000000001 2024-09-15T03:47:02.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:54.193263+0000", 2024-09-15T03:47:02.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7627000000000002e-05 2024-09-15T03:47:02.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:54.193297+0000", 2024-09-15T03:47:02.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4124000000000001e-05 2024-09-15T03:47:02.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:54.193310+0000", 2024-09-15T03:47:02.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3594999999999999e-05 2024-09-15T03:47:02.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:54.196111+0000", 2024-09-15T03:47:02.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0028007489999999999 2024-09-15T03:47:02.542 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.542 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.542 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.542 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s7 49/31 ECSubWrite(tid=80, reqid=client.4521.0:320, at_version=49'15, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:47:02.542 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.840854+0000", 2024-09-15T03:47:02.543 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.6192786410000002, 2024-09-15T03:47:02.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0033670699999999998, 2024-09-15T03:47:02.543 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.543 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.543 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.840854+0000", 2024-09-15T03:47:02.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.840854+0000", 2024-09-15T03:47:02.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.840854+0000", 2024-09-15T03:47:02.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8799999999999998e-07 2024-09-15T03:47:02.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841170+0000", 2024-09-15T03:47:02.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031579299999999998 2024-09-15T03:47:02.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841228+0000", 2024-09-15T03:47:02.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8634999999999997e-05 2024-09-15T03:47:02.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841238+0000", 2024-09-15T03:47:02.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8160000000000005e-06 2024-09-15T03:47:02.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841335+0000", 2024-09-15T03:47:02.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6521999999999995e-05 2024-09-15T03:47:02.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841347+0000", 2024-09-15T03:47:02.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2709e-05 2024-09-15T03:47:02.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841717+0000", 2024-09-15T03:47:02.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00036960399999999999 2024-09-15T03:47:02.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841779+0000", 2024-09-15T03:47:02.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1717000000000006e-05 2024-09-15T03:47:02.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.842283+0000", 2024-09-15T03:47:02.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00050401999999999999 2024-09-15T03:47:02.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.842477+0000", 2024-09-15T03:47:02.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000194224 2024-09-15T03:47:02.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844221+0000", 2024-09-15T03:47:02.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0017437419999999999 2024-09-15T03:47:02.548 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.548 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.548 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.548 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s2 49/31 ECSubWrite(tid=30, reqid=client.4521.0:325, at_version=49'12, trim_to=0'0, roll_forward_to=40'10))", 2024-09-15T03:47:02.548 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.844826+0000", 2024-09-15T03:47:02.548 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.6153061199999996, 2024-09-15T03:47:02.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003471156, 2024-09-15T03:47:02.548 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.548 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.548 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844826+0000", 2024-09-15T03:47:02.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844826+0000", 2024-09-15T03:47:02.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844826+0000", 2024-09-15T03:47:02.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.03e-07 2024-09-15T03:47:02.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844859+0000", 2024-09-15T03:47:02.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2305000000000003e-05 2024-09-15T03:47:02.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844882+0000", 2024-09-15T03:47:02.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3071e-05 2024-09-15T03:47:02.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844890+0000", 2024-09-15T03:47:02.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9689999999999994e-06 2024-09-15T03:47:02.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844949+0000", 2024-09-15T03:47:02.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9630999999999999e-05 2024-09-15T03:47:02.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844960+0000", 2024-09-15T03:47:02.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0808999999999999e-05 2024-09-15T03:47:02.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845279+0000", 2024-09-15T03:47:02.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031877099999999997 2024-09-15T03:47:02.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845293+0000", 2024-09-15T03:47:02.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4219999999999999e-05 2024-09-15T03:47:02.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845423+0000", 2024-09-15T03:47:02.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000130309 2024-09-15T03:47:02.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845444+0000", 2024-09-15T03:47:02.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0883e-05 2024-09-15T03:47:02.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.848297+0000", 2024-09-15T03:47:02.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0028528849999999999 2024-09-15T03:47:02.553 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.553 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.553 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.554 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.cs4 49/28 ECSubWrite(tid=15, reqid=client.4521.0:337, at_version=49'10, trim_to=0'0, roll_forward_to=43'8))", 2024-09-15T03:47:02.554 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.858111+0000", 2024-09-15T03:47:02.554 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.6020214169999996, 2024-09-15T03:47:02.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0031710559999999998, 2024-09-15T03:47:02.554 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.554 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.554 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858111+0000", 2024-09-15T03:47:02.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858111+0000", 2024-09-15T03:47:02.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858111+0000", 2024-09-15T03:47:02.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6600000000000002e-07 2024-09-15T03:47:02.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858138+0000", 2024-09-15T03:47:02.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6715000000000001e-05 2024-09-15T03:47:02.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.556 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858149+0000", 2024-09-15T03:47:02.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1567e-05 2024-09-15T03:47:02.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.556 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858156+0000", 2024-09-15T03:47:02.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8029999999999999e-06 2024-09-15T03:47:02.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.556 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858338+0000", 2024-09-15T03:47:02.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000181614 2024-09-15T03:47:02.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858347+0000", 2024-09-15T03:47:02.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4409999999999999e-06 2024-09-15T03:47:02.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858625+0000", 2024-09-15T03:47:02.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000277719 2024-09-15T03:47:02.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858652+0000", 2024-09-15T03:47:02.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7078999999999999e-05 2024-09-15T03:47:02.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858691+0000", 2024-09-15T03:47:02.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8899999999999997e-05 2024-09-15T03:47:02.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858746+0000", 2024-09-15T03:47:02.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5074999999999997e-05 2024-09-15T03:47:02.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861282+0000", 2024-09-15T03:47:02.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0025356770000000001 2024-09-15T03:47:02.560 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.560 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.560 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.560 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s7 49/31 ECSubWrite(tid=87, reqid=client.4521.0:335, at_version=49'21, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:02.560 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.858802+0000", 2024-09-15T03:47:02.561 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.6013303969999999, 2024-09-15T03:47:02.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0031070249999999998, 2024-09-15T03:47:02.561 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.561 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.561 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858802+0000", 2024-09-15T03:47:02.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858802+0000", 2024-09-15T03:47:02.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858803+0000", 2024-09-15T03:47:02.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.54e-07 2024-09-15T03:47:02.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858840+0000", 2024-09-15T03:47:02.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7577999999999998e-05 2024-09-15T03:47:02.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858859+0000", 2024-09-15T03:47:02.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8858000000000001e-05 2024-09-15T03:47:02.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858872+0000", 2024-09-15T03:47:02.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3198e-05 2024-09-15T03:47:02.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858916+0000", 2024-09-15T03:47:02.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3344999999999999e-05 2024-09-15T03:47:02.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858924+0000", 2024-09-15T03:47:02.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5790000000000004e-06 2024-09-15T03:47:02.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859014+0000", 2024-09-15T03:47:02.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9487999999999994e-05 2024-09-15T03:47:02.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859067+0000", 2024-09-15T03:47:02.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3331000000000002e-05 2024-09-15T03:47:02.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859167+0000", 2024-09-15T03:47:02.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000100348 2024-09-15T03:47:02.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859208+0000", 2024-09-15T03:47:02.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0345000000000001e-05 2024-09-15T03:47:02.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861909+0000", 2024-09-15T03:47:02.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002701101 2024-09-15T03:47:02.565 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.566 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.566 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.566 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s7 49/31 ECSubWrite(tid=88, reqid=client.4521.0:336, at_version=49'22, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:02.566 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.859255+0000", 2024-09-15T03:47:02.566 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.6008771609999997, 2024-09-15T03:47:02.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00308925, 2024-09-15T03:47:02.566 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.566 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.566 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859255+0000", 2024-09-15T03:47:02.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859255+0000", 2024-09-15T03:47:02.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859256+0000", 2024-09-15T03:47:02.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6400000000000001e-07 2024-09-15T03:47:02.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859276+0000", 2024-09-15T03:47:02.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0080000000000001e-05 2024-09-15T03:47:02.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859288+0000", 2024-09-15T03:47:02.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1846e-05 2024-09-15T03:47:02.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859299+0000", 2024-09-15T03:47:02.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0900000000000001e-05 2024-09-15T03:47:02.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.569 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859355+0000", 2024-09-15T03:47:02.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6721999999999997e-05 2024-09-15T03:47:02.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.569 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859365+0000", 2024-09-15T03:47:02.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0169999999999999e-05 2024-09-15T03:47:02.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.569 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859463+0000", 2024-09-15T03:47:02.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7226999999999996e-05 2024-09-15T03:47:02.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859486+0000", 2024-09-15T03:47:02.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3433000000000001e-05 2024-09-15T03:47:02.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859519+0000", 2024-09-15T03:47:02.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2441999999999999e-05 2024-09-15T03:47:02.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859533+0000", 2024-09-15T03:47:02.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4766000000000001e-05 2024-09-15T03:47:02.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.862344+0000", 2024-09-15T03:47:02.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0028108999999999999 2024-09-15T03:47:02.571 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.571 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.571 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.571 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.cs4 49/28 ECSubWrite(tid=16, reqid=client.4521.0:337, at_version=0'0, trim_to=0'0, roll_forward_to=49'10))", 2024-09-15T03:47:02.571 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.859334+0000", 2024-09-15T03:47:02.572 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 6.6007976819999996, 2024-09-15T03:47:02.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029848610000000001, 2024-09-15T03:47:02.572 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.572 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.572 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859334+0000", 2024-09-15T03:47:02.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859334+0000", 2024-09-15T03:47:02.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859335+0000", 2024-09-15T03:47:02.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.44e-07 2024-09-15T03:47:02.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859351+0000", 2024-09-15T03:47:02.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5409000000000001e-05 2024-09-15T03:47:02.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859363+0000", 2024-09-15T03:47:02.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2319e-05 2024-09-15T03:47:02.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859369+0000", 2024-09-15T03:47:02.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3679999999999998e-06 2024-09-15T03:47:02.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859425+0000", 2024-09-15T03:47:02.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6233000000000002e-05 2024-09-15T03:47:02.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.575 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859436+0000", 2024-09-15T03:47:02.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0696999999999999e-05 2024-09-15T03:47:02.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.575 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.575 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859549+0000", 2024-09-15T03:47:02.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011264000000000001 2024-09-15T03:47:02.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.575 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.575 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859560+0000", 2024-09-15T03:47:02.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0981e-05 2024-09-15T03:47:02.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859594+0000", 2024-09-15T03:47:02.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.43e-05 2024-09-15T03:47:02.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859611+0000", 2024-09-15T03:47:02.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7178000000000001e-05 2024-09-15T03:47:02.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.862319+0000", 2024-09-15T03:47:02.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0027080920000000001 2024-09-15T03:47:02.576 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.577 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.577 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.577 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.577 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.577 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:02.578 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:47:02.579 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:02.578+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:02.679 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:02.679 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-15T03:47:02.679 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:02.680 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:02.678+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:02.687 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:47:02.780 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:02.780+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:02.788 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:02.789 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:02.789 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-15T03:47:02.789 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-15T03:47:02.789 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:02.797 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:47:02.863 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-15T03:47:02.864 INFO:tasks.rados.rados.0.smithi001.stdout:148: done (1 left) 2024-09-15T03:47:02.864 INFO:tasks.rados.rados.0.smithi001.stdout:149: done (0 left) 2024-09-15T03:47:02.864 INFO:tasks.rados.rados.0.smithi001.stdout:150: read oid 4 snap -1 2024-09-15T03:47:02.864 INFO:tasks.rados.rados.0.smithi001.stdout:150: expect (ObjNum 64 snap 6 seq_num 64) 2024-09-15T03:47:02.864 INFO:tasks.rados.rados.0.smithi001.stdout:151: read oid 34 snap -1 2024-09-15T03:47:02.864 INFO:tasks.rados.rados.0.smithi001.stdout:151: expect (ObjNum 70 snap 8 seq_num 70) 2024-09-15T03:47:02.864 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 5 current snap is 18 2024-09-15T03:47:02.864 INFO:tasks.rados.rados.0.smithi001.stdout:152: seq_num 82 ranges {704512=622592,1327104=475136,1802240=294912} 2024-09-15T03:47:02.871 INFO:tasks.rados.rados.0.smithi001.stdout:152: writing smithi00132172-5 from 704512 to 1327104 tid 1 2024-09-15T03:47:02.874 INFO:tasks.rados.rados.0.smithi001.stdout:152: writing smithi00132172-5 from 1327104 to 1802240 tid 2 2024-09-15T03:47:02.876 INFO:tasks.rados.rados.0.smithi001.stdout:152: writing smithi00132172-5 from 1802240 to 2097152 tid 3 2024-09-15T03:47:02.876 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 25 current snap is 18 2024-09-15T03:47:02.876 INFO:tasks.rados.rados.0.smithi001.stdout:153: seq_num 83 ranges {1277952=638976,1916928=622592,2539520=147456} 2024-09-15T03:47:02.881 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:02.880+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:02.887 INFO:tasks.rados.rados.0.smithi001.stdout:153: writing smithi00132172-25 from 1277952 to 1916928 tid 1 2024-09-15T03:47:02.891 INFO:tasks.rados.rados.0.smithi001.stdout:153: writing smithi00132172-25 from 1916928 to 2539520 tid 2 2024-09-15T03:47:02.892 INFO:tasks.rados.rados.0.smithi001.stdout:153: writing smithi00132172-25 from 2539520 to 2686976 tid 3 2024-09-15T03:47:02.892 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 21 current snap is 18 2024-09-15T03:47:02.892 INFO:tasks.rados.rados.0.smithi001.stdout:154: seq_num 84 ranges {344064=655360,999424=638976,1638400=131072} 2024-09-15T03:47:02.898 INFO:tasks.rados.rados.0.smithi001.stdout:154: writing smithi00132172-21 from 344064 to 999424 tid 1 2024-09-15T03:47:02.902 INFO:tasks.rados.rados.0.smithi001.stdout:154: writing smithi00132172-21 from 999424 to 1638400 tid 2 2024-09-15T03:47:02.903 INFO:tasks.rados.rados.0.smithi001.stdout:154: writing smithi00132172-21 from 1638400 to 1769472 tid 3 2024-09-15T03:47:02.903 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 27 current snap is 18 2024-09-15T03:47:02.903 INFO:tasks.rados.rados.0.smithi001.stdout:155: seq_num 85 ranges {3768320=671744,4440064=770048} 2024-09-15T03:47:02.917 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:02.918 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-15T03:47:02.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-15T03:47:02.918 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-15T03:47:02.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.918 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=80, reqid=client.4521.0:320, at_version=49'15, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:47:02.918 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.840885+0000", 2024-09-15T03:47:02.918 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.071521272, 2024-09-15T03:47:02.918 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0033862459999998, 2024-09-15T03:47:02.918 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.918 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.918 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.918 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.919 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.840885+0000", 2024-09-15T03:47:02.919 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.919 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.840885+0000", 2024-09-15T03:47:02.919 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.919 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.919 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.840885+0000", 2024-09-15T03:47:02.919 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1100000000000002e-07 2024-09-15T03:47:02.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841181+0000", 2024-09-15T03:47:02.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029618199999999998 2024-09-15T03:47:02.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841195+0000", 2024-09-15T03:47:02.920 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3424000000000001e-05 2024-09-15T03:47:02.920 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.920 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.920 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841204+0000", 2024-09-15T03:47:02.921 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0170000000000002e-06 2024-09-15T03:47:02.921 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.921 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.921 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841274+0000", 2024-09-15T03:47:02.921 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0561000000000004e-05 2024-09-15T03:47:02.921 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.921 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.921 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841289+0000", 2024-09-15T03:47:02.922 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4127e-05 2024-09-15T03:47:02.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.922 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841717+0000", 2024-09-15T03:47:02.922 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00042811800000000001 2024-09-15T03:47:02.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.922 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.922 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841745+0000", 2024-09-15T03:47:02.922 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8297e-05 2024-09-15T03:47:02.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.923 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.923 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841852+0000", 2024-09-15T03:47:02.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010660199999999999 2024-09-15T03:47:02.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.923 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.923 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.842419+0000", 2024-09-15T03:47:02.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00056784499999999998 2024-09-15T03:47:02.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.923 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.923 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.923 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.844271+0000", 2024-09-15T03:47:02.923 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0018517619999998 2024-09-15T03:47:02.924 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.924 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.924 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.924 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=30, reqid=client.4521.0:325, at_version=49'12, trim_to=0'0, roll_forward_to=40'10))", 2024-09-15T03:47:02.924 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.844912+0000", 2024-09-15T03:47:02.924 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.0674946900000002, 2024-09-15T03:47:02.924 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0007286940000002, 2024-09-15T03:47:02.924 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.924 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.924 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.925 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.925 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844912+0000", 2024-09-15T03:47:02.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.925 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.925 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844912+0000", 2024-09-15T03:47:02.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.925 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.925 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.925 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844912+0000", 2024-09-15T03:47:02.925 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.84e-07 2024-09-15T03:47:02.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844937+0000", 2024-09-15T03:47:02.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4729e-05 2024-09-15T03:47:02.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.926 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844957+0000", 2024-09-15T03:47:02.926 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0285999999999999e-05 2024-09-15T03:47:02.926 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.926 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844965+0000", 2024-09-15T03:47:02.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5709999999999998e-06 2024-09-15T03:47:02.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.927 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845086+0000", 2024-09-15T03:47:02.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000120856 2024-09-15T03:47:02.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.927 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.927 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845121+0000", 2024-09-15T03:47:02.927 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4292999999999998e-05 2024-09-15T03:47:02.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.928 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.928 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845492+0000", 2024-09-15T03:47:02.928 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00037167400000000001 2024-09-15T03:47:02.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.928 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.928 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845529+0000", 2024-09-15T03:47:02.928 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7092000000000003e-05 2024-09-15T03:47:02.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.929 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.929 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845591+0000", 2024-09-15T03:47:02.929 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1824999999999998e-05 2024-09-15T03:47:02.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.929 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.930 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845612+0000", 2024-09-15T03:47:02.930 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.084e-05 2024-09-15T03:47:02.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.930 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.930 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845640+0000", 2024-09-15T03:47:02.930 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0000281439999998 2024-09-15T03:47:02.930 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.930 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.930 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.930 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=81, reqid=client.4521.0:321, at_version=49'16, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:47:02.931 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.845113+0000", 2024-09-15T03:47:02.931 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.0672932829999997, 2024-09-15T03:47:02.931 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.999459995, 2024-09-15T03:47:02.931 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.931 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.931 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.931 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.931 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845113+0000", 2024-09-15T03:47:02.931 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.931 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845113+0000", 2024-09-15T03:47:02.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845114+0000", 2024-09-15T03:47:02.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0210000000000001e-06 2024-09-15T03:47:02.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.932 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.932 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845422+0000", 2024-09-15T03:47:02.932 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00030762999999999998 2024-09-15T03:47:02.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.933 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845436+0000", 2024-09-15T03:47:02.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3998999999999999e-05 2024-09-15T03:47:02.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.933 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845445+0000", 2024-09-15T03:47:02.933 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5799999999999998e-06 2024-09-15T03:47:02.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.933 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.933 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845538+0000", 2024-09-15T03:47:02.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2739999999999996e-05 2024-09-15T03:47:02.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.934 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845548+0000", 2024-09-15T03:47:02.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0548e-05 2024-09-15T03:47:02.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.934 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.934 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845803+0000", 2024-09-15T03:47:02.934 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00025495599999999998 2024-09-15T03:47:02.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845998+0000", 2024-09-15T03:47:02.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019432799999999999 2024-09-15T03:47:02.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846122+0000", 2024-09-15T03:47:02.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012434600000000001 2024-09-15T03:47:02.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.935 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.935 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846178+0000", 2024-09-15T03:47:02.935 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5943999999999998e-05 2024-09-15T03:47:02.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.936 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.936 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.844573+0000", 2024-09-15T03:47:02.936 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9983949029999999 2024-09-15T03:47:02.936 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.936 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.936 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.936 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=82, reqid=client.4521.0:322, at_version=49'17, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:47:02.936 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.846151+0000", 2024-09-15T03:47:02.937 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.0662552610000002, 2024-09-15T03:47:02.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9989014119999999, 2024-09-15T03:47:02.937 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.937 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.937 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846151+0000", 2024-09-15T03:47:02.937 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.937 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.937 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846151+0000", 2024-09-15T03:47:02.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.938 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846152+0000", 2024-09-15T03:47:02.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0800000000000004e-07 2024-09-15T03:47:02.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.938 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846238+0000", 2024-09-15T03:47:02.938 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5866999999999996e-05 2024-09-15T03:47:02.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.938 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846255+0000", 2024-09-15T03:47:02.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6954999999999999e-05 2024-09-15T03:47:02.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846262+0000", 2024-09-15T03:47:02.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7680000000000008e-06 2024-09-15T03:47:02.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.939 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.939 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846325+0000", 2024-09-15T03:47:02.939 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2191000000000001e-05 2024-09-15T03:47:02.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846335+0000", 2024-09-15T03:47:02.940 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0767e-05 2024-09-15T03:47:02.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.940 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846446+0000", 2024-09-15T03:47:02.940 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110522 2024-09-15T03:47:02.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.940 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.941 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846458+0000", 2024-09-15T03:47:02.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2023e-05 2024-09-15T03:47:02.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.941 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846508+0000", 2024-09-15T03:47:02.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0479000000000002e-05 2024-09-15T03:47:02.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.941 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.941 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846526+0000", 2024-09-15T03:47:02.941 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7836e-05 2024-09-15T03:47:02.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.942 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.942 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845052+0000", 2024-09-15T03:47:02.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9985261960000003 2024-09-15T03:47:02.942 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.942 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.942 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.942 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=31, reqid=client.4521.0:325, at_version=0'0, trim_to=0'0, roll_forward_to=49'12))", 2024-09-15T03:47:02.942 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.847407+0000", 2024-09-15T03:47:02.942 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.0649990579999997, 2024-09-15T03:47:02.942 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9983494909999999, 2024-09-15T03:47:02.943 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.943 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.943 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.943 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.943 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847407+0000", 2024-09-15T03:47:02.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.943 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.943 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.943 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847407+0000", 2024-09-15T03:47:02.943 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.944 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.944 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847408+0000", 2024-09-15T03:47:02.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0070000000000001e-06 2024-09-15T03:47:02.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.944 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.944 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847421+0000", 2024-09-15T03:47:02.944 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3203999999999999e-05 2024-09-15T03:47:02.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.944 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847443+0000", 2024-09-15T03:47:02.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1539000000000002e-05 2024-09-15T03:47:02.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847452+0000", 2024-09-15T03:47:02.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6570000000000003e-06 2024-09-15T03:47:02.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.945 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.945 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847534+0000", 2024-09-15T03:47:02.945 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2309000000000001e-05 2024-09-15T03:47:02.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847544+0000", 2024-09-15T03:47:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0482000000000001e-05 2024-09-15T03:47:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847645+0000", 2024-09-15T03:47:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010021700000000001 2024-09-15T03:47:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.946 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847665+0000", 2024-09-15T03:47:02.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0271999999999998e-05 2024-09-15T03:47:02.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.947 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.947 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847708+0000", 2024-09-15T03:47:02.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2828999999999999e-05 2024-09-15T03:47:02.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.947 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.947 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847724+0000", 2024-09-15T03:47:02.947 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6334e-05 2024-09-15T03:47:02.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.947 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.948 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845757+0000", 2024-09-15T03:47:02.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.998032641 2024-09-15T03:47:02.948 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.948 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.948 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.948 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=83, reqid=client.4521.0:323, at_version=49'18, trim_to=0'0, roll_forward_to=49'15))", 2024-09-15T03:47:02.948 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.847691+0000", 2024-09-15T03:47:02.948 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.0647152110000002, 2024-09-15T03:47:02.948 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9981305969999998, 2024-09-15T03:47:02.948 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.949 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.949 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847691+0000", 2024-09-15T03:47:02.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.949 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.949 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847691+0000", 2024-09-15T03:47:02.949 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.950 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847692+0000", 2024-09-15T03:47:02.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9899999999999999e-07 2024-09-15T03:47:02.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.950 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847715+0000", 2024-09-15T03:47:02.950 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2818000000000001e-05 2024-09-15T03:47:02.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.950 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.950 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847731+0000", 2024-09-15T03:47:02.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6401000000000001e-05 2024-09-15T03:47:02.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847738+0000", 2024-09-15T03:47:02.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3699999999999997e-06 2024-09-15T03:47:02.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.951 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847791+0000", 2024-09-15T03:47:02.951 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2584999999999999e-05 2024-09-15T03:47:02.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.951 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.952 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847801+0000", 2024-09-15T03:47:02.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0295999999999999e-05 2024-09-15T03:47:02.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.952 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.952 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847931+0000", 2024-09-15T03:47:02.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0001295 2024-09-15T03:47:02.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.952 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.952 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847946+0000", 2024-09-15T03:47:02.952 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4732000000000001e-05 2024-09-15T03:47:02.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847989+0000", 2024-09-15T03:47:02.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3625000000000003e-05 2024-09-15T03:47:02.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.953 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.848004+0000", 2024-09-15T03:47:02.953 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4385e-05 2024-09-15T03:47:02.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.953 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.954 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845822+0000", 2024-09-15T03:47:02.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9978179860000003 2024-09-15T03:47:02.954 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.954 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.954 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.954 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=84, reqid=client.4521.0:323, at_version=0'0, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:02.954 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.849295+0000", 2024-09-15T03:47:02.954 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.0631111310000003, 2024-09-15T03:47:02.954 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9965517450000001, 2024-09-15T03:47:02.954 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.954 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.954 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849295+0000", 2024-09-15T03:47:02.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849295+0000", 2024-09-15T03:47:02.955 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.955 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.955 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849296+0000", 2024-09-15T03:47:02.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6300000000000004e-07 2024-09-15T03:47:02.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.956 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.956 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849308+0000", 2024-09-15T03:47:02.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2261e-05 2024-09-15T03:47:02.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.956 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.956 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849325+0000", 2024-09-15T03:47:02.956 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6588000000000001e-05 2024-09-15T03:47:02.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849332+0000", 2024-09-15T03:47:02.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.464e-06 2024-09-15T03:47:02.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849379+0000", 2024-09-15T03:47:02.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6498000000000001e-05 2024-09-15T03:47:02.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.957 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.957 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849391+0000", 2024-09-15T03:47:02.957 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1926000000000001e-05 2024-09-15T03:47:02.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849497+0000", 2024-09-15T03:47:02.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010634200000000001 2024-09-15T03:47:02.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849516+0000", 2024-09-15T03:47:02.958 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8995999999999999e-05 2024-09-15T03:47:02.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.958 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.958 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849565+0000", 2024-09-15T03:47:02.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8735e-05 2024-09-15T03:47:02.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.959 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.959 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849577+0000", 2024-09-15T03:47:02.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2585999999999999e-05 2024-09-15T03:47:02.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.959 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.959 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845847+0000", 2024-09-15T03:47:02.959 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9962694860000001 2024-09-15T03:47:02.959 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.959 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.960 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.960 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpRead(3.7s5 49/31 ECSubRead(tid=85, to_read={3:e40d1694:::smithi00132172-50:head=0,4096,0}, subchunks={3:e40d1694:::smithi00132172-50:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:47:02.960 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.849814+0000", 2024-09-15T03:47:02.960 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.0625918170000004, 2024-09-15T03:47:02.960 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9961114690000001, 2024-09-15T03:47:02.960 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.960 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.960 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.960 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.960 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849814+0000", 2024-09-15T03:47:02.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.961 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849814+0000", 2024-09-15T03:47:02.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.961 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849815+0000", 2024-09-15T03:47:02.961 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6499999999999999e-07 2024-09-15T03:47:02.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.961 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849826+0000", 2024-09-15T03:47:02.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0635000000000001e-05 2024-09-15T03:47:02.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.962 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.850154+0000", 2024-09-15T03:47:02.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00032866900000000002 2024-09-15T03:47:02.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.962 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.962 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.850169+0000", 2024-09-15T03:47:02.962 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5051e-05 2024-09-15T03:47:02.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.963 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.963 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.850226+0000", 2024-09-15T03:47:02.963 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6926999999999999e-05 2024-09-15T03:47:02.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.963 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.963 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845926+0000", 2024-09-15T03:47:02.963 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9956995219999998 2024-09-15T03:47:02.963 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.963 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.963 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.964 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=32, reqid=client.4521.0:329, at_version=49'13, trim_to=0'0, roll_forward_to=49'12))", 2024-09-15T03:47:02.964 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.853281+0000", 2024-09-15T03:47:02.964 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.0591253309999997, 2024-09-15T03:47:02.964 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9927023569999998, 2024-09-15T03:47:02.964 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.964 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.964 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.964 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.964 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853281+0000", 2024-09-15T03:47:02.964 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853281+0000", 2024-09-15T03:47:02.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853282+0000", 2024-09-15T03:47:02.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.347e-06 2024-09-15T03:47:02.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.965 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.965 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853401+0000", 2024-09-15T03:47:02.965 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118711 2024-09-15T03:47:02.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853441+0000", 2024-09-15T03:47:02.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0059999999999999e-05 2024-09-15T03:47:02.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853451+0000", 2024-09-15T03:47:02.966 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5850000000000006e-06 2024-09-15T03:47:02.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.966 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.966 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853617+0000", 2024-09-15T03:47:02.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016655899999999999 2024-09-15T03:47:02.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.967 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.967 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853632+0000", 2024-09-15T03:47:02.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4605e-05 2024-09-15T03:47:02.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.967 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.967 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853842+0000", 2024-09-15T03:47:02.967 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000210523 2024-09-15T03:47:02.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.967 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.968 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853871+0000", 2024-09-15T03:47:02.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8441000000000001e-05 2024-09-15T03:47:02.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.968 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.968 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854188+0000", 2024-09-15T03:47:02.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031700800000000003 2024-09-15T03:47:02.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.968 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.968 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854209+0000", 2024-09-15T03:47:02.968 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0795999999999999e-05 2024-09-15T03:47:02.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.969 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.969 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845983+0000", 2024-09-15T03:47:02.969 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9917747220000002 2024-09-15T03:47:02.969 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.969 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.969 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.969 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=33, reqid=client.4521.0:329, at_version=0'0, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:47:02.969 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.854592+0000", 2024-09-15T03:47:02.969 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.0578143200000003, 2024-09-15T03:47:02.969 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9914290499999998, 2024-09-15T03:47:02.970 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.970 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.970 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.970 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.970 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854592+0000", 2024-09-15T03:47:02.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.970 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.970 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854592+0000", 2024-09-15T03:47:02.970 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854593+0000", 2024-09-15T03:47:02.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1200000000000002e-07 2024-09-15T03:47:02.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854605+0000", 2024-09-15T03:47:02.971 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1946999999999999e-05 2024-09-15T03:47:02.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.971 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.971 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854624+0000", 2024-09-15T03:47:02.972 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9502000000000001e-05 2024-09-15T03:47:02.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.972 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.972 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854631+0000", 2024-09-15T03:47:02.972 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2490000000000004e-06 2024-09-15T03:47:02.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.972 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.972 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854683+0000", 2024-09-15T03:47:02.972 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1643e-05 2024-09-15T03:47:02.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.972 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.973 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854693+0000", 2024-09-15T03:47:02.973 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8199999999999992e-06 2024-09-15T03:47:02.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.973 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.973 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.973 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854768+0000", 2024-09-15T03:47:02.973 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5104999999999993e-05 2024-09-15T03:47:02.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.973 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.973 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.973 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854830+0000", 2024-09-15T03:47:02.973 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2191000000000001e-05 2024-09-15T03:47:02.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.974 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.974 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854934+0000", 2024-09-15T03:47:02.974 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010424099999999999 2024-09-15T03:47:02.974 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.974 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.974 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854952+0000", 2024-09-15T03:47:02.974 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7521e-05 2024-09-15T03:47:02.974 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.974 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.974 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.846021+0000", 2024-09-15T03:47:02.974 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9910690190000002 2024-09-15T03:47:02.975 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.975 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.975 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.975 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.975 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.975 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=34, reqid=client.4521.0:330, at_version=49'14, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:47:02.975 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.857132+0000", 2024-09-15T03:47:02.975 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.0552739390000001, 2024-09-15T03:47:02.975 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9890264310000001, 2024-09-15T03:47:02.975 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.975 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.975 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.975 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.975 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.976 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857132+0000", 2024-09-15T03:47:02.976 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.976 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.976 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857132+0000", 2024-09-15T03:47:02.976 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.976 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.976 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857133+0000", 2024-09-15T03:47:02.976 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.61e-07 2024-09-15T03:47:02.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.977 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.977 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857223+0000", 2024-09-15T03:47:02.977 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0476999999999994e-05 2024-09-15T03:47:02.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.977 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.977 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857245+0000", 2024-09-15T03:47:02.977 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1415000000000001e-05 2024-09-15T03:47:02.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.977 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.977 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857254+0000", 2024-09-15T03:47:02.978 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.329e-06 2024-09-15T03:47:02.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.978 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.978 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857312+0000", 2024-09-15T03:47:02.978 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7865000000000002e-05 2024-09-15T03:47:02.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.978 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.978 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857323+0000", 2024-09-15T03:47:02.978 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0655999999999999e-05 2024-09-15T03:47:02.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.978 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857817+0000", 2024-09-15T03:47:02.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00049391300000000001 2024-09-15T03:47:02.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857946+0000", 2024-09-15T03:47:02.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012926699999999999 2024-09-15T03:47:02.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.979 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.979 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.979 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858076+0000", 2024-09-15T03:47:02.979 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012957600000000001 2024-09-15T03:47:02.979 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.980 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.980 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858094+0000", 2024-09-15T03:47:02.980 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8715e-05 2024-09-15T03:47:02.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.980 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.980 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.846159+0000", 2024-09-15T03:47:02.980 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9880644570000001 2024-09-15T03:47:02.980 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.980 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.980 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.981 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=86, reqid=client.4521.0:334, at_version=49'20, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:02.981 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.857463+0000", 2024-09-15T03:47:02.981 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.0549436209999996, 2024-09-15T03:47:02.981 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0025860940000002, 2024-09-15T03:47:02.981 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.981 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.981 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.981 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.981 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857463+0000", 2024-09-15T03:47:02.981 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857463+0000", 2024-09-15T03:47:02.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.982 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857464+0000", 2024-09-15T03:47:02.982 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0210000000000001e-06 2024-09-15T03:47:02.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.982 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.982 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857489+0000", 2024-09-15T03:47:02.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5500999999999999e-05 2024-09-15T03:47:02.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.983 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857508+0000", 2024-09-15T03:47:02.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9025000000000002e-05 2024-09-15T03:47:02.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.983 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.983 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857515+0000", 2024-09-15T03:47:02.983 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9759999999999998e-06 2024-09-15T03:47:02.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.984 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.984 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845961+0000", 2024-09-15T03:47:02.984 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9884457740000001 2024-09-15T03:47:02.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.984 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.984 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845966+0000", 2024-09-15T03:47:02.984 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3750000000000002e-06 2024-09-15T03:47:02.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.984 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.984 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.846127+0000", 2024-09-15T03:47:02.984 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000160462 2024-09-15T03:47:02.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.985 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.985 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.846136+0000", 2024-09-15T03:47:02.985 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6260000000000007e-06 2024-09-15T03:47:02.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.985 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.985 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.846191+0000", 2024-09-15T03:47:02.985 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4274999999999998e-05 2024-09-15T03:47:02.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.985 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.985 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.853397+0000", 2024-09-15T03:47:02.986 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0072068519999999997 2024-09-15T03:47:02.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.986 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.986 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.860049+0000", 2024-09-15T03:47:02.986 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0066512070000000001 2024-09-15T03:47:02.986 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.986 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.986 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.986 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.cs3 49/28 ECSubWrite(tid=15, reqid=client.4521.0:337, at_version=49'10, trim_to=0'0, roll_forward_to=43'8))", 2024-09-15T03:47:02.986 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.857712+0000", 2024-09-15T03:47:02.987 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.0546943019999997, 2024-09-15T03:47:02.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9893153209999999, 2024-09-15T03:47:02.987 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.987 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.987 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.987 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.987 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857712+0000", 2024-09-15T03:47:02.987 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.987 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.987 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857712+0000", 2024-09-15T03:47:02.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.988 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857712+0000", 2024-09-15T03:47:02.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8000000000000002e-07 2024-09-15T03:47:02.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.988 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.988 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857755+0000", 2024-09-15T03:47:02.988 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2960999999999997e-05 2024-09-15T03:47:02.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.989 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.989 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857773+0000", 2024-09-15T03:47:02.989 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7635999999999999e-05 2024-09-15T03:47:02.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.989 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.989 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857780+0000", 2024-09-15T03:47:02.989 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.024e-06 2024-09-15T03:47:02.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.989 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.989 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857883+0000", 2024-09-15T03:47:02.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103224 2024-09-15T03:47:02.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857893+0000", 2024-09-15T03:47:02.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0265000000000001e-05 2024-09-15T03:47:02.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858180+0000", 2024-09-15T03:47:02.990 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000286602 2024-09-15T03:47:02.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.990 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.990 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858193+0000", 2024-09-15T03:47:02.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3342e-05 2024-09-15T03:47:02.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858233+0000", 2024-09-15T03:47:02.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9811999999999998e-05 2024-09-15T03:47:02.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.991 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.991 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858253+0000", 2024-09-15T03:47:02.991 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0259e-05 2024-09-15T03:47:02.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.992 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.992 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.847027+0000", 2024-09-15T03:47:02.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.988773916 2024-09-15T03:47:02.992 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.992 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.992 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.992 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=87, reqid=client.4521.0:335, at_version=49'21, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:02.992 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.858481+0000", 2024-09-15T03:47:02.992 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.0539251529999998, 2024-09-15T03:47:02.992 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.004544015, 2024-09-15T03:47:02.992 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.993 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.993 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.993 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.993 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858481+0000", 2024-09-15T03:47:02.993 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.993 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.993 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858481+0000", 2024-09-15T03:47:02.993 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.994 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:02.994 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858482+0000", 2024-09-15T03:47:02.994 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.09e-07 2024-09-15T03:47:02.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.994 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:02.994 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858505+0000", 2024-09-15T03:47:02.994 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3133e-05 2024-09-15T03:47:02.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.994 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:02.994 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858524+0000", 2024-09-15T03:47:02.994 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8706e-05 2024-09-15T03:47:02.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.995 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:02.995 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858532+0000", 2024-09-15T03:47:02.995 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5539999999999998e-06 2024-09-15T03:47:02.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.995 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:02.995 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851213+0000", 2024-09-15T03:47:02.995 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.992680526 2024-09-15T03:47:02.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.996 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:02.996 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851221+0000", 2024-09-15T03:47:02.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5380000000000002e-06 2024-09-15T03:47:02.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.996 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:02.996 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851309+0000", 2024-09-15T03:47:02.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8069e-05 2024-09-15T03:47:02.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.996 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:02.996 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851495+0000", 2024-09-15T03:47:02.996 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018516999999999999 2024-09-15T03:47:02.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:02.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851557+0000", 2024-09-15T03:47:02.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2031000000000003e-05 2024-09-15T03:47:02.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:02.997 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.853665+0000", 2024-09-15T03:47:02.997 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021079499999999999 2024-09-15T03:47:02.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.997 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:02.998 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.863025+0000", 2024-09-15T03:47:02.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0093605289999999994 2024-09-15T03:47:02.998 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.998 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:02.998 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:02.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.998 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.cs3 49/28 ECSubWrite(tid=16, reqid=client.4521.0:337, at_version=0'0, trim_to=0'0, roll_forward_to=49'10))", 2024-09-15T03:47:02.998 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.859219+0000", 2024-09-15T03:47:02.998 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.0531871820000003, 2024-09-15T03:47:02.998 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9879305710000001, 2024-09-15T03:47:02.998 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:02.998 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:02.999 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:02.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.999 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:02.999 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859219+0000", 2024-09-15T03:47:02.999 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.999 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:02.999 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859219+0000", 2024-09-15T03:47:02.999 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:02.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:02.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:02.999 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.000 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859220+0000", 2024-09-15T03:47:03.000 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3300000000000001e-07 2024-09-15T03:47:03.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.000 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.000 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859233+0000", 2024-09-15T03:47:03.000 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2781e-05 2024-09-15T03:47:03.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.000 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.000 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859251+0000", 2024-09-15T03:47:03.000 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8770000000000002e-05 2024-09-15T03:47:03.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859259+0000", 2024-09-15T03:47:03.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7309999999999993e-06 2024-09-15T03:47:03.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859342+0000", 2024-09-15T03:47:03.001 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2909000000000002e-05 2024-09-15T03:47:03.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.001 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:03.001 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859357+0000", 2024-09-15T03:47:03.002 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5245000000000001e-05 2024-09-15T03:47:03.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.002 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:03.002 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859453+0000", 2024-09-15T03:47:03.002 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5878999999999995e-05 2024-09-15T03:47:03.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.002 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:03.002 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859475+0000", 2024-09-15T03:47:03.002 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1591999999999999e-05 2024-09-15T03:47:03.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.003 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:03.003 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859518+0000", 2024-09-15T03:47:03.003 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3021e-05 2024-09-15T03:47:03.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.003 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:03.003 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859536+0000", 2024-09-15T03:47:03.003 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8420999999999999e-05 2024-09-15T03:47:03.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.003 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.003 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.847150+0000", 2024-09-15T03:47:03.003 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9876134890000001 2024-09-15T03:47:03.003 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.004 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.004 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.004 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=88, reqid=client.4521.0:336, at_version=49'22, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:03.004 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.859709+0000", 2024-09-15T03:47:03.004 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.052697405, 2024-09-15T03:47:03.004 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0037862020000001, 2024-09-15T03:47:03.004 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.004 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:03.004 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.004 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859709+0000", 2024-09-15T03:47:03.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859709+0000", 2024-09-15T03:47:03.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.005 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.005 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859710+0000", 2024-09-15T03:47:03.005 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.052e-06 2024-09-15T03:47:03.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.006 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859729+0000", 2024-09-15T03:47:03.006 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9409999999999999e-05 2024-09-15T03:47:03.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.006 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859759+0000", 2024-09-15T03:47:03.006 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9998e-05 2024-09-15T03:47:03.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.006 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.006 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859767+0000", 2024-09-15T03:47:03.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.14e-06 2024-09-15T03:47:03.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.007 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.007 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851350+0000", 2024-09-15T03:47:03.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.99158261 2024-09-15T03:47:03.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.007 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:03.007 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851358+0000", 2024-09-15T03:47:03.007 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1070000000000005e-06 2024-09-15T03:47:03.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.008 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:03.008 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851443+0000", 2024-09-15T03:47:03.008 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4760999999999997e-05 2024-09-15T03:47:03.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.008 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:03.008 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851495+0000", 2024-09-15T03:47:03.008 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2070999999999997e-05 2024-09-15T03:47:03.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.008 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:03.008 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851557+0000", 2024-09-15T03:47:03.008 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1829999999999996e-05 2024-09-15T03:47:03.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.009 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:03.009 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.853682+0000", 2024-09-15T03:47:03.009 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021249110000000002 2024-09-15T03:47:03.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.009 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.009 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.863495+0000", 2024-09-15T03:47:03.009 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0098133119999999994 2024-09-15T03:47:03.009 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.009 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.009 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.010 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=35, reqid=client.4521.0:331, at_version=49'15, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:47:03.010 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.860778+0000", 2024-09-15T03:47:03.010 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.0516285099999996, 2024-09-15T03:47:03.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9863441239999999, 2024-09-15T03:47:03.010 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.010 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:03.010 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.010 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.010 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860778+0000", 2024-09-15T03:47:03.010 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.011 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.011 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860778+0000", 2024-09-15T03:47:03.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.011 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.011 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860779+0000", 2024-09-15T03:47:03.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0109999999999999e-06 2024-09-15T03:47:03.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.011 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.011 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860922+0000", 2024-09-15T03:47:03.011 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014357399999999999 2024-09-15T03:47:03.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.012 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.012 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860944+0000", 2024-09-15T03:47:03.012 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1424e-05 2024-09-15T03:47:03.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.012 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.012 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860951+0000", 2024-09-15T03:47:03.012 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5610000000000001e-06 2024-09-15T03:47:03.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.012 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.013 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861022+0000", 2024-09-15T03:47:03.013 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1126999999999998e-05 2024-09-15T03:47:03.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.013 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:03.013 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861034+0000", 2024-09-15T03:47:03.013 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1810999999999999e-05 2024-09-15T03:47:03.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.013 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:03.013 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861164+0000", 2024-09-15T03:47:03.013 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000129727 2024-09-15T03:47:03.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.014 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:03.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861196+0000", 2024-09-15T03:47:03.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2240000000000003e-05 2024-09-15T03:47:03.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.014 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:03.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861259+0000", 2024-09-15T03:47:03.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2454999999999997e-05 2024-09-15T03:47:03.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.014 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:03.014 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861274+0000", 2024-09-15T03:47:03.014 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5588000000000001e-05 2024-09-15T03:47:03.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.015 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.015 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.847122+0000", 2024-09-15T03:47:03.015 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9858476060000001 2024-09-15T03:47:03.015 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.015 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.015 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.015 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=36, reqid=client.4521.0:332, at_version=49'16, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:47:03.015 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.861537+0000", 2024-09-15T03:47:03.015 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.0508689469999997, 2024-09-15T03:47:03.016 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.985662638, 2024-09-15T03:47:03.016 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.016 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:03.016 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.016 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.016 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861537+0000", 2024-09-15T03:47:03.016 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.016 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.016 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861537+0000", 2024-09-15T03:47:03.016 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.017 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.017 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861538+0000", 2024-09-15T03:47:03.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7000000000000003e-07 2024-09-15T03:47:03.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.017 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.017 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861571+0000", 2024-09-15T03:47:03.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2493000000000002e-05 2024-09-15T03:47:03.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.017 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.017 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861588+0000", 2024-09-15T03:47:03.017 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7076e-05 2024-09-15T03:47:03.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.018 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.018 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861595+0000", 2024-09-15T03:47:03.018 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3440000000000003e-06 2024-09-15T03:47:03.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.018 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.018 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861665+0000", 2024-09-15T03:47:03.018 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9814999999999995e-05 2024-09-15T03:47:03.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.018 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:03.019 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861674+0000", 2024-09-15T03:47:03.019 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9740000000000008e-06 2024-09-15T03:47:03.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.019 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:03.019 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861767+0000", 2024-09-15T03:47:03.019 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3572999999999993e-05 2024-09-15T03:47:03.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.019 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:03.019 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861778+0000", 2024-09-15T03:47:03.019 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1083e-05 2024-09-15T03:47:03.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.020 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:03.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861888+0000", 2024-09-15T03:47:03.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010937700000000001 2024-09-15T03:47:03.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.020 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:03.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861906+0000", 2024-09-15T03:47:03.020 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.863e-05 2024-09-15T03:47:03.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.020 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.020 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.847200+0000", 2024-09-15T03:47:03.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.985293403 2024-09-15T03:47:03.021 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.021 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.021 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.021 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=37, reqid=client.4521.0:332, at_version=0'0, trim_to=0'0, roll_forward_to=49'16))", 2024-09-15T03:47:03.021 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.864457+0000", 2024-09-15T03:47:03.021 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.0479492580000001, 2024-09-15T03:47:03.021 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9827806690000003, 2024-09-15T03:47:03.021 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.021 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:03.021 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864457+0000", 2024-09-15T03:47:03.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864457+0000", 2024-09-15T03:47:03.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.022 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.022 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864458+0000", 2024-09-15T03:47:03.022 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.029e-06 2024-09-15T03:47:03.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.023 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.023 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864472+0000", 2024-09-15T03:47:03.023 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4287000000000001e-05 2024-09-15T03:47:03.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.023 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.023 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864504+0000", 2024-09-15T03:47:03.023 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1435e-05 2024-09-15T03:47:03.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.024 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.024 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864511+0000", 2024-09-15T03:47:03.024 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4900000000000003e-06 2024-09-15T03:47:03.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.024 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.024 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864583+0000", 2024-09-15T03:47:03.024 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1828000000000004e-05 2024-09-15T03:47:03.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.024 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:03.024 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864593+0000", 2024-09-15T03:47:03.024 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0427e-05 2024-09-15T03:47:03.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.025 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:03.025 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864701+0000", 2024-09-15T03:47:03.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000107969 2024-09-15T03:47:03.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.025 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:03.025 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864721+0000", 2024-09-15T03:47:03.025 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9422000000000001e-05 2024-09-15T03:47:03.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.026 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:03.026 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864798+0000", 2024-09-15T03:47:03.026 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6731999999999996e-05 2024-09-15T03:47:03.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.026 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:03.026 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864815+0000", 2024-09-15T03:47:03.026 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7286999999999999e-05 2024-09-15T03:47:03.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.026 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.026 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.026 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.847238+0000", 2024-09-15T03:47:03.026 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9824227629999998 2024-09-15T03:47:03.027 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.027 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.027 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.027 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.bs4 49/31 ECSubWrite(tid=75, reqid=client.4521.0:338, at_version=49'58, trim_to=0'0, roll_forward_to=47'56))", 2024-09-15T03:47:03.027 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.870036+0000", 2024-09-15T03:47:03.027 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.0423706340000001, 2024-09-15T03:47:03.027 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.978548666, 2024-09-15T03:47:03.027 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.027 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:03.027 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.028 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.028 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870036+0000", 2024-09-15T03:47:03.028 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.028 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.028 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870036+0000", 2024-09-15T03:47:03.028 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.028 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.029 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870036+0000", 2024-09-15T03:47:03.029 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.61e-07 2024-09-15T03:47:03.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.029 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.029 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870051+0000", 2024-09-15T03:47:03.029 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5074e-05 2024-09-15T03:47:03.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.029 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.030 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870061+0000", 2024-09-15T03:47:03.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.066e-05 2024-09-15T03:47:03.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.030 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.030 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870067+0000", 2024-09-15T03:47:03.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1580000000000004e-06 2024-09-15T03:47:03.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.030 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.030 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870121+0000", 2024-09-15T03:47:03.030 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4605999999999999e-05 2024-09-15T03:47:03.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:03.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870134+0000", 2024-09-15T03:47:03.031 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2748999999999999e-05 2024-09-15T03:47:03.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:03.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870450+0000", 2024-09-15T03:47:03.031 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031624399999999998 2024-09-15T03:47:03.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.031 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:03.031 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870459+0000", 2024-09-15T03:47:03.031 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0499999999999997e-06 2024-09-15T03:47:03.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.032 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:03.032 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870496+0000", 2024-09-15T03:47:03.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6548999999999999e-05 2024-09-15T03:47:03.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.032 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:03.032 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870509+0000", 2024-09-15T03:47:03.032 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3216e-05 2024-09-15T03:47:03.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.033 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.033 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.848584+0000", 2024-09-15T03:47:03.033 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9780751990000001 2024-09-15T03:47:03.033 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.033 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.033 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.033 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.033 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.033 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:03.035 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:47:03.036 INFO:tasks.rados.rados.0.smithi001.stdout:155: writing smithi00132172-27 from 3768320 to 4440064 tid 1 2024-09-15T03:47:03.036 INFO:tasks.rados.rados.0.smithi001.stdout:155: writing smithi00132172-27 from 4440064 to 5210112 tid 2 2024-09-15T03:47:03.036 INFO:tasks.rados.rados.0.smithi001.stdout:156: snap_create 2024-09-15T03:47:03.036 INFO:tasks.rados.rados.0.smithi001.stdout:151: expect (ObjNum 70 snap 8 seq_num 70) 2024-09-15T03:47:03.036 INFO:tasks.rados.rados.0.smithi001.stdout:152: finishing write tid 1 to smithi00132172-5 2024-09-15T03:47:03.036 INFO:tasks.rados.rados.0.smithi001.stdout:150: expect (ObjNum 64 snap 6 seq_num 64) 2024-09-15T03:47:03.036 INFO:tasks.rados.rados.0.smithi001.stdout:152: finishing write tid 2 to smithi00132172-5 2024-09-15T03:47:03.036 INFO:tasks.rados.rados.0.smithi001.stdout:152: finishing write tid 3 to smithi00132172-5 2024-09-15T03:47:03.036 INFO:tasks.rados.rados.0.smithi001.stdout:152: finishing write tid 4 to smithi00132172-5 2024-09-15T03:47:03.037 INFO:tasks.rados.rados.0.smithi001.stdout:152: finishing write tid 5 to smithi00132172-5 2024-09-15T03:47:03.037 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 5 v 19 (ObjNum 82 snap 18 seq_num 82) dirty exists 2024-09-15T03:47:03.037 INFO:tasks.rados.rados.0.smithi001.stdout:152: left oid 5 (ObjNum 82 snap 18 seq_num 82) 2024-09-15T03:47:03.037 INFO:tasks.rados.rados.0.smithi001.stdout:153: finishing write tid 1 to smithi00132172-25 2024-09-15T03:47:03.037 INFO:tasks.rados.rados.0.smithi001.stdout:153: finishing write tid 2 to smithi00132172-25 2024-09-15T03:47:03.037 INFO:tasks.rados.rados.0.smithi001.stdout:153: finishing write tid 3 to smithi00132172-25 2024-09-15T03:47:03.037 INFO:tasks.rados.rados.0.smithi001.stdout:153: finishing write tid 4 to smithi00132172-25 2024-09-15T03:47:03.037 INFO:tasks.rados.rados.0.smithi001.stdout:153: finishing write tid 5 to smithi00132172-25 2024-09-15T03:47:03.037 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 25 v 44 (ObjNum 83 snap 18 seq_num 83) dirty exists 2024-09-15T03:47:03.037 INFO:tasks.rados.rados.0.smithi001.stdout:153: left oid 25 (ObjNum 83 snap 18 seq_num 83) 2024-09-15T03:47:03.037 INFO:tasks.rados.rados.0.smithi001.stdout:154: finishing write tid 1 to smithi00132172-21 2024-09-15T03:47:03.037 INFO:tasks.rados.rados.0.smithi001.stdout:154: finishing write tid 2 to smithi00132172-21 2024-09-15T03:47:03.037 INFO:tasks.rados.rados.0.smithi001.stdout:154: finishing write tid 3 to smithi00132172-21 2024-09-15T03:47:03.038 INFO:tasks.rados.rados.0.smithi001.stdout:154: finishing write tid 4 to smithi00132172-21 2024-09-15T03:47:03.038 INFO:tasks.rados.rados.0.smithi001.stdout:154: finishing write tid 5 to smithi00132172-21 2024-09-15T03:47:03.038 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 21 v 10 (ObjNum 84 snap 18 seq_num 84) dirty exists 2024-09-15T03:47:03.038 INFO:tasks.rados.rados.0.smithi001.stdout:154: left oid 21 (ObjNum 84 snap 18 seq_num 84) 2024-09-15T03:47:03.038 INFO:tasks.rados.rados.0.smithi001.stdout:155: finishing write tid 1 to smithi00132172-27 2024-09-15T03:47:03.038 INFO:tasks.rados.rados.0.smithi001.stdout:155: finishing write tid 2 to smithi00132172-27 2024-09-15T03:47:03.038 INFO:tasks.rados.rados.0.smithi001.stdout:155: finishing write tid 3 to smithi00132172-27 2024-09-15T03:47:03.038 INFO:tasks.rados.rados.0.smithi001.stdout:155: finishing write tid 4 to smithi00132172-27 2024-09-15T03:47:03.038 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 27 v 27 (ObjNum 85 snap 18 seq_num 85) dirty exists 2024-09-15T03:47:03.038 INFO:tasks.rados.rados.0.smithi001.stdout:155: left oid 27 (ObjNum 85 snap 18 seq_num 85) 2024-09-15T03:47:03.039 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:03.033+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:03.135 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:03.135+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:03.177 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:03.177 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:03.177 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-15T03:47:03.177 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:03.185 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:47:03.270 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:03.269+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:03.326 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:03.326 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:03.326 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-15T03:47:03.326 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-15T03:47:03.326 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:03.334 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:47:03.337 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:03.336+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:03.438 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:03.437+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:03.479 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:03.480 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-15T03:47:03.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-15T03:47:03.480 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-15T03:47:03.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.480 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:195 3.es0 3:7f0816e1:::smithi00132172-31:head [copy-from ver 4] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:47:03.480 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:41.112078+0000", 2024-09-15T03:47:03.480 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.365780445999999, 2024-09-15T03:47:03.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012535605, 2024-09-15T03:47:03.480 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.480 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:03.481 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:03.481 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:03.481 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:03.481 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 195 2024-09-15T03:47:03.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.481 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.481 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112078+0000", 2024-09-15T03:47:03.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112078+0000", 2024-09-15T03:47:03.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112080+0000", 2024-09-15T03:47:03.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4399999999999999e-06 2024-09-15T03:47:03.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112090+0000", 2024-09-15T03:47:03.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7310000000000005e-06 2024-09-15T03:47:03.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112107+0000", 2024-09-15T03:47:03.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7557000000000001e-05 2024-09-15T03:47:03.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112113+0000", 2024-09-15T03:47:03.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6910000000000002e-06 2024-09-15T03:47:03.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112176+0000", 2024-09-15T03:47:03.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3014000000000003e-05 2024-09-15T03:47:03.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:03.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112308+0000", 2024-09-15T03:47:03.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000131402 2024-09-15T03:47:03.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:03.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118088+0000", 2024-09-15T03:47:03.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0057801570000000002 2024-09-15T03:47:03.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:03.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118560+0000", 2024-09-15T03:47:03.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00047206199999999998 2024-09-15T03:47:03.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:03.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118573+0000", 2024-09-15T03:47:03.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3521e-05 2024-09-15T03:47:03.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:03.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118667+0000", 2024-09-15T03:47:03.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.323e-05 2024-09-15T03:47:03.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:03.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.119046+0000", 2024-09-15T03:47:03.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00037924699999999998 2024-09-15T03:47:03.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:03.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.124533+0000", 2024-09-15T03:47:03.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0054869849999999998 2024-09-15T03:47:03.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.124613+0000", 2024-09-15T03:47:03.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0568000000000005e-05 2024-09-15T03:47:03.487 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.487 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.487 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.487 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:196 3.es0 3:7f0816e1:::smithi00132172-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:47:03.488 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:41.112125+0000", 2024-09-15T03:47:03.488 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.365732829999999, 2024-09-15T03:47:03.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0068455030000000002, 2024-09-15T03:47:03.488 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.488 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:03.488 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:03.488 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:03.488 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:03.488 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 196 2024-09-15T03:47:03.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.488 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112125+0000", 2024-09-15T03:47:03.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112125+0000", 2024-09-15T03:47:03.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112126+0000", 2024-09-15T03:47:03.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4099999999999998e-07 2024-09-15T03:47:03.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112130+0000", 2024-09-15T03:47:03.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6909999999999999e-06 2024-09-15T03:47:03.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112138+0000", 2024-09-15T03:47:03.490 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6969999999999999e-06 2024-09-15T03:47:03.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.490 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.490 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112143+0000", 2024-09-15T03:47:03.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1620000000000001e-06 2024-09-15T03:47:03.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112506+0000", 2024-09-15T03:47:03.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00036344600000000002 2024-09-15T03:47:03.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.491 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:03.491 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112569+0000", 2024-09-15T03:47:03.491 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3367000000000005e-05 2024-09-15T03:47:03.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118741+0000", 2024-09-15T03:47:03.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0061715700000000004 2024-09-15T03:47:03.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:03.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118852+0000", 2024-09-15T03:47:03.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011062500000000001 2024-09-15T03:47:03.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.492 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.492 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118971+0000", 2024-09-15T03:47:03.492 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011920400000000001 2024-09-15T03:47:03.493 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.493 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.493 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.493 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(osd.7.17:1 3.es0 3:7e6c26d0:::smithi00132172-14:head [assert-version v22,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:03.493 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:45.131370+0000", 2024-09-15T03:47:03.493 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 18.346487689, 2024-09-15T03:47:03.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0052243280000000003, 2024-09-15T03:47:03.493 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.493 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:03.493 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.131370+0000", 2024-09-15T03:47:03.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.131370+0000", 2024-09-15T03:47:03.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.494 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.494 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.131372+0000", 2024-09-15T03:47:03.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9420000000000002e-06 2024-09-15T03:47:03.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.131379+0000", 2024-09-15T03:47:03.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8959999999999997e-06 2024-09-15T03:47:03.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.131398+0000", 2024-09-15T03:47:03.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8366999999999999e-05 2024-09-15T03:47:03.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.131406+0000", 2024-09-15T03:47:03.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0800000000000006e-06 2024-09-15T03:47:03.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.131465+0000", 2024-09-15T03:47:03.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8924e-05 2024-09-15T03:47:03.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:03.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.131582+0000", 2024-09-15T03:47:03.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011693600000000001 2024-09-15T03:47:03.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.136595+0000", 2024-09-15T03:47:03.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.005013183 2024-09-15T03:47:03.497 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.497 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.497 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.497 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.bs5 38/31 ECSubWrite(tid=36, reqid=client.4521.0:220, at_version=38'23, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:47:03.497 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:45.147807+0000", 2024-09-15T03:47:03.497 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 18.330051109999999, 2024-09-15T03:47:03.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0046676189999999996, 2024-09-15T03:47:03.498 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.498 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:03.498 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.147807+0000", 2024-09-15T03:47:03.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.147807+0000", 2024-09-15T03:47:03.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.147809+0000", 2024-09-15T03:47:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7600000000000001e-06 2024-09-15T03:47:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148346+0000", 2024-09-15T03:47:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00053685099999999997 2024-09-15T03:47:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148372+0000", 2024-09-15T03:47:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6094000000000001e-05 2024-09-15T03:47:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148382+0000", 2024-09-15T03:47:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0008e-05 2024-09-15T03:47:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148438+0000", 2024-09-15T03:47:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6242000000000001e-05 2024-09-15T03:47:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148450+0000", 2024-09-15T03:47:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2189e-05 2024-09-15T03:47:03.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148853+0000", 2024-09-15T03:47:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00040301399999999999 2024-09-15T03:47:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148880+0000", 2024-09-15T03:47:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6375000000000001e-05 2024-09-15T03:47:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:03.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.149135+0000", 2024-09-15T03:47:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00025514599999999999 2024-09-15T03:47:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.149161+0000", 2024-09-15T03:47:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6427999999999998e-05 2024-09-15T03:47:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.152475+0000", 2024-09-15T03:47:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0033135119999999998 2024-09-15T03:47:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.502 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:227 3.es0 3:77a7da34:::smithi00132172-4:head [append 0~704512] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:45.152975+0000", 2024-09-15T03:47:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 18.324882883000001, 2024-09-15T03:47:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0058200999999999999, 2024-09-15T03:47:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:03.503 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 227 2024-09-15T03:47:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.152975+0000", 2024-09-15T03:47:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.152975+0000", 2024-09-15T03:47:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.152977+0000", 2024-09-15T03:47:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7799999999999999e-06 2024-09-15T03:47:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.153247+0000", 2024-09-15T03:47:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00027026799999999999 2024-09-15T03:47:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.153262+0000", 2024-09-15T03:47:03.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4966999999999999e-05 2024-09-15T03:47:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.153268+0000", 2024-09-15T03:47:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4970000000000001e-06 2024-09-15T03:47:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.153325+0000", 2024-09-15T03:47:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7383999999999998e-05 2024-09-15T03:47:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:03.506 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.153443+0000", 2024-09-15T03:47:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000117327 2024-09-15T03:47:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155145+0000", 2024-09-15T03:47:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0017023609999999999 2024-09-15T03:47:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155506+0000", 2024-09-15T03:47:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00036064300000000001 2024-09-15T03:47:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155529+0000", 2024-09-15T03:47:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3229000000000001e-05 2024-09-15T03:47:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155709+0000", 2024-09-15T03:47:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018021000000000001 2024-09-15T03:47:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.157102+0000", 2024-09-15T03:47:03.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001392534 2024-09-15T03:47:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158717+0000", 2024-09-15T03:47:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001615989 2024-09-15T03:47:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158795+0000", 2024-09-15T03:47:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7911000000000007e-05 2024-09-15T03:47:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.509 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:228 3.es0 3:77a7da34:::smithi00132172-4:head [append 0~376832] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:45.155394+0000", 2024-09-15T03:47:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 18.322464239999999, 2024-09-15T03:47:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0046305629999999999, 2024-09-15T03:47:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 228 2024-09-15T03:47:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.510 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155394+0000", 2024-09-15T03:47:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155394+0000", 2024-09-15T03:47:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155396+0000", 2024-09-15T03:47:03.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3070000000000001e-06 2024-09-15T03:47:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155562+0000", 2024-09-15T03:47:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016546599999999999 2024-09-15T03:47:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155578+0000", 2024-09-15T03:47:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6392000000000002e-05 2024-09-15T03:47:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155585+0000", 2024-09-15T03:47:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.267e-06 2024-09-15T03:47:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155783+0000", 2024-09-15T03:47:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000197784 2024-09-15T03:47:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155887+0000", 2024-09-15T03:47:03.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000104066 2024-09-15T03:47:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.156917+0000", 2024-09-15T03:47:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001029709 2024-09-15T03:47:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.157046+0000", 2024-09-15T03:47:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012913399999999999 2024-09-15T03:47:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:03.514 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.157343+0000", 2024-09-15T03:47:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000296483 2024-09-15T03:47:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.157619+0000", 2024-09-15T03:47:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000276562 2024-09-15T03:47:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158027+0000", 2024-09-15T03:47:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00040807799999999998 2024-09-15T03:47:03.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.159959+0000", 2024-09-15T03:47:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001931938 2024-09-15T03:47:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.160025+0000", 2024-09-15T03:47:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5376999999999994e-05 2024-09-15T03:47:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:230 3.es0 3:77a7da34:::smithi00132172-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:45.155698+0000", 2024-09-15T03:47:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 18.322160027999999, 2024-09-15T03:47:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0062491680000000003, 2024-09-15T03:47:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 230 2024-09-15T03:47:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155698+0000", 2024-09-15T03:47:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155698+0000", 2024-09-15T03:47:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155699+0000", 2024-09-15T03:47:03.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2900000000000003e-07 2024-09-15T03:47:03.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155702+0000", 2024-09-15T03:47:03.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0809999999999998e-06 2024-09-15T03:47:03.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155708+0000", 2024-09-15T03:47:03.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0440000000000001e-06 2024-09-15T03:47:03.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.155712+0000", 2024-09-15T03:47:03.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.827e-06 2024-09-15T03:47:03.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158184+0000", 2024-09-15T03:47:03.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024722260000000001 2024-09-15T03:47:03.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:03.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158303+0000", 2024-09-15T03:47:03.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118845 2024-09-15T03:47:03.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.160521+0000", 2024-09-15T03:47:03.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022177239999999999 2024-09-15T03:47:03.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:03.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.160616+0000", 2024-09-15T03:47:03.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5948999999999999e-05 2024-09-15T03:47:03.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.161947+0000", 2024-09-15T03:47:03.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001330843 2024-09-15T03:47:03.521 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.522 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.522 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.522 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.0s1 39/31 ECSubWrite(tid=43, reqid=client.4521.0:240, at_version=39'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:03.522 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:46.149555+0000", 2024-09-15T03:47:03.522 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.328303346999999, 2024-09-15T03:47:03.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0056873840000000002, 2024-09-15T03:47:03.522 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.522 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:03.522 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149555+0000", 2024-09-15T03:47:03.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149555+0000", 2024-09-15T03:47:03.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149555+0000", 2024-09-15T03:47:03.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2300000000000002e-07 2024-09-15T03:47:03.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149613+0000", 2024-09-15T03:47:03.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7519000000000003e-05 2024-09-15T03:47:03.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149701+0000", 2024-09-15T03:47:03.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8067999999999998e-05 2024-09-15T03:47:03.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149778+0000", 2024-09-15T03:47:03.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7546000000000006e-05 2024-09-15T03:47:03.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149835+0000", 2024-09-15T03:47:03.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6489000000000001e-05 2024-09-15T03:47:03.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:03.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.149845+0000", 2024-09-15T03:47:03.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0583e-05 2024-09-15T03:47:03.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:03.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150114+0000", 2024-09-15T03:47:03.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026860399999999998 2024-09-15T03:47:03.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:03.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150145+0000", 2024-09-15T03:47:03.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0617999999999997e-05 2024-09-15T03:47:03.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:03.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150194+0000", 2024-09-15T03:47:03.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9573999999999997e-05 2024-09-15T03:47:03.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:03.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.151015+0000", 2024-09-15T03:47:03.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00082130099999999998 2024-09-15T03:47:03.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.155242+0000", 2024-09-15T03:47:03.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0042267590000000001 2024-09-15T03:47:03.527 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.527 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.528 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.528 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.0s1 39/31 ECSubWrite(tid=45, reqid=client.4521.0:241, at_version=39'17, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:03.528 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:46.153423+0000", 2024-09-15T03:47:03.528 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.324435249, 2024-09-15T03:47:03.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0046692590000000003, 2024-09-15T03:47:03.528 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.528 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:03.528 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153423+0000", 2024-09-15T03:47:03.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153423+0000", 2024-09-15T03:47:03.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153423+0000", 2024-09-15T03:47:03.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6100000000000001e-07 2024-09-15T03:47:03.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153445+0000", 2024-09-15T03:47:03.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1447999999999999e-05 2024-09-15T03:47:03.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153456+0000", 2024-09-15T03:47:03.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1199000000000001e-05 2024-09-15T03:47:03.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153463+0000", 2024-09-15T03:47:03.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6089999999999997e-06 2024-09-15T03:47:03.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153597+0000", 2024-09-15T03:47:03.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013469799999999999 2024-09-15T03:47:03.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:03.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153609+0000", 2024-09-15T03:47:03.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1293e-05 2024-09-15T03:47:03.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:03.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153733+0000", 2024-09-15T03:47:03.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000124127 2024-09-15T03:47:03.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:03.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153776+0000", 2024-09-15T03:47:03.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2731999999999997e-05 2024-09-15T03:47:03.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:03.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153821+0000", 2024-09-15T03:47:03.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5250000000000002e-05 2024-09-15T03:47:03.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:03.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.155216+0000", 2024-09-15T03:47:03.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001395164 2024-09-15T03:47:03.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.158092+0000", 2024-09-15T03:47:03.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0028761780000000001 2024-09-15T03:47:03.534 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.534 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.534 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.534 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.0s1 39/31 ECSubWrite(tid=46, reqid=client.4521.0:242, at_version=39'18, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:03.534 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:46.153470+0000", 2024-09-15T03:47:03.534 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.324388588000001, 2024-09-15T03:47:03.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0047847699999999998, 2024-09-15T03:47:03.534 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.535 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:03.535 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153470+0000", 2024-09-15T03:47:03.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153470+0000", 2024-09-15T03:47:03.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153470+0000", 2024-09-15T03:47:03.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.34e-07 2024-09-15T03:47:03.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153491+0000", 2024-09-15T03:47:03.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0565999999999999e-05 2024-09-15T03:47:03.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153499+0000", 2024-09-15T03:47:03.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9850000000000006e-06 2024-09-15T03:47:03.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153504+0000", 2024-09-15T03:47:03.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2900000000000002e-06 2024-09-15T03:47:03.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153776+0000", 2024-09-15T03:47:03.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000271959 2024-09-15T03:47:03.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:03.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.153784+0000", 2024-09-15T03:47:03.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3410000000000003e-06 2024-09-15T03:47:03.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:03.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.155184+0000", 2024-09-15T03:47:03.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0014001910000000001 2024-09-15T03:47:03.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:03.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.155579+0000", 2024-09-15T03:47:03.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00039442399999999999 2024-09-15T03:47:03.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:03.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.155844+0000", 2024-09-15T03:47:03.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026551999999999999 2024-09-15T03:47:03.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:03.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.155874+0000", 2024-09-15T03:47:03.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9388999999999999e-05 2024-09-15T03:47:03.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.158254+0000", 2024-09-15T03:47:03.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002380671 2024-09-15T03:47:03.540 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.540 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.540 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.540 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.3s5 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:47:03.540 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:46.170728+0000", 2024-09-15T03:47:03.540 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.307130313999998, 2024-09-15T03:47:03.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.004627585, 2024-09-15T03:47:03.540 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.540 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:03.540 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.170728+0000", 2024-09-15T03:47:03.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.170728+0000", 2024-09-15T03:47:03.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.170729+0000", 2024-09-15T03:47:03.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2999999999999999e-07 2024-09-15T03:47:03.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171187+0000", 2024-09-15T03:47:03.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00045862099999999999 2024-09-15T03:47:03.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171205+0000", 2024-09-15T03:47:03.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8073000000000001e-05 2024-09-15T03:47:03.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171212+0000", 2024-09-15T03:47:03.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9190000000000002e-06 2024-09-15T03:47:03.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171305+0000", 2024-09-15T03:47:03.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2961000000000007e-05 2024-09-15T03:47:03.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:03.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171315+0000", 2024-09-15T03:47:03.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6900000000000004e-06 2024-09-15T03:47:03.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:03.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171560+0000", 2024-09-15T03:47:03.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024454499999999998 2024-09-15T03:47:03.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:03.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171582+0000", 2024-09-15T03:47:03.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2246999999999999e-05 2024-09-15T03:47:03.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:03.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171710+0000", 2024-09-15T03:47:03.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012822300000000001 2024-09-15T03:47:03.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:03.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171733+0000", 2024-09-15T03:47:03.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.264e-05 2024-09-15T03:47:03.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.175355+0000", 2024-09-15T03:47:03.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0036227360000000001 2024-09-15T03:47:03.545 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.545 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.546 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.546 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s4 49/31 ECSubWrite(tid=80, reqid=client.4521.0:320, at_version=49'15, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:47:03.546 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.841069+0000", 2024-09-15T03:47:03.546 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.6367888839999996, 2024-09-15T03:47:03.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0054374860000000001, 2024-09-15T03:47:03.546 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.546 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:03.546 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841069+0000", 2024-09-15T03:47:03.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841069+0000", 2024-09-15T03:47:03.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841070+0000", 2024-09-15T03:47:03.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.27e-07 2024-09-15T03:47:03.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841549+0000", 2024-09-15T03:47:03.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000479166 2024-09-15T03:47:03.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841571+0000", 2024-09-15T03:47:03.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2061000000000001e-05 2024-09-15T03:47:03.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841579+0000", 2024-09-15T03:47:03.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9999999999999996e-06 2024-09-15T03:47:03.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841688+0000", 2024-09-15T03:47:03.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000109467 2024-09-15T03:47:03.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:03.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841700+0000", 2024-09-15T03:47:03.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1663e-05 2024-09-15T03:47:03.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:03.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.843886+0000", 2024-09-15T03:47:03.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021862169999999998 2024-09-15T03:47:03.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:03.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844020+0000", 2024-09-15T03:47:03.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013349499999999999 2024-09-15T03:47:03.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:03.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844124+0000", 2024-09-15T03:47:03.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010440100000000001 2024-09-15T03:47:03.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:03.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844156+0000", 2024-09-15T03:47:03.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1653999999999999e-05 2024-09-15T03:47:03.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846507+0000", 2024-09-15T03:47:03.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002351035 2024-09-15T03:47:03.551 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.551 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.551 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.551 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s1 49/31 ECSubWrite(tid=30, reqid=client.4521.0:325, at_version=49'12, trim_to=0'0, roll_forward_to=40'10))", 2024-09-15T03:47:03.552 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.844978+0000", 2024-09-15T03:47:03.552 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.6328802319999998, 2024-09-15T03:47:03.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0049714099999999999, 2024-09-15T03:47:03.552 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.552 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:03.552 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844978+0000", 2024-09-15T03:47:03.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844978+0000", 2024-09-15T03:47:03.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844978+0000", 2024-09-15T03:47:03.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5199999999999998e-07 2024-09-15T03:47:03.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845006+0000", 2024-09-15T03:47:03.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7492e-05 2024-09-15T03:47:03.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845024+0000", 2024-09-15T03:47:03.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8017e-05 2024-09-15T03:47:03.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845031+0000", 2024-09-15T03:47:03.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5830000000000001e-06 2024-09-15T03:47:03.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845277+0000", 2024-09-15T03:47:03.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024547499999999999 2024-09-15T03:47:03.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:03.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845286+0000", 2024-09-15T03:47:03.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5710000000000004e-06 2024-09-15T03:47:03.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:03.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846025+0000", 2024-09-15T03:47:03.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00073880099999999998 2024-09-15T03:47:03.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:03.556 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846059+0000", 2024-09-15T03:47:03.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4032999999999998e-05 2024-09-15T03:47:03.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:03.556 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846135+0000", 2024-09-15T03:47:03.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6045999999999997e-05 2024-09-15T03:47:03.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:03.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846367+0000", 2024-09-15T03:47:03.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023208899999999999 2024-09-15T03:47:03.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849949+0000", 2024-09-15T03:47:03.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0035819509999999999 2024-09-15T03:47:03.557 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.557 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.557 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.cs7 49/28 ECSubWrite(tid=15, reqid=client.4521.0:337, at_version=49'10, trim_to=0'0, roll_forward_to=43'8))", 2024-09-15T03:47:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.858004+0000", 2024-09-15T03:47:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.6198543579999996, 2024-09-15T03:47:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0048927559999999998, 2024-09-15T03:47:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858004+0000", 2024-09-15T03:47:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858004+0000", 2024-09-15T03:47:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858004+0000", 2024-09-15T03:47:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2300000000000001e-07 2024-09-15T03:47:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858050+0000", 2024-09-15T03:47:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5488999999999998e-05 2024-09-15T03:47:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858069+0000", 2024-09-15T03:47:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9457999999999999e-05 2024-09-15T03:47:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858116+0000", 2024-09-15T03:47:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6613000000000002e-05 2024-09-15T03:47:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858284+0000", 2024-09-15T03:47:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016815499999999999 2024-09-15T03:47:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858293+0000", 2024-09-15T03:47:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3279999999999995e-06 2024-09-15T03:47:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858569+0000", 2024-09-15T03:47:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00027534100000000002 2024-09-15T03:47:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858602+0000", 2024-09-15T03:47:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3577000000000001e-05 2024-09-15T03:47:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858650+0000", 2024-09-15T03:47:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7808e-05 2024-09-15T03:47:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858670+0000", 2024-09-15T03:47:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0224000000000002e-05 2024-09-15T03:47:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.862897+0000", 2024-09-15T03:47:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00422624 2024-09-15T03:47:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:384 3.es0 3.2c5be5ee (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-15T03:47:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:02.863548+0000", 2024-09-15T03:47:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.61431051599999997, 2024-09-15T03:47:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0076790490000000003, 2024-09-15T03:47:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 384 2024-09-15T03:47:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863548+0000", 2024-09-15T03:47:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863548+0000", 2024-09-15T03:47:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863551+0000", 2024-09-15T03:47:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9210000000000001e-06 2024-09-15T03:47:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863603+0000", 2024-09-15T03:47:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2713999999999997e-05 2024-09-15T03:47:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863621+0000", 2024-09-15T03:47:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7968000000000001e-05 2024-09-15T03:47:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863631+0000", 2024-09-15T03:47:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6479999999999998e-06 2024-09-15T03:47:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863698+0000", 2024-09-15T03:47:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7288999999999999e-05 2024-09-15T03:47:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-15T03:47:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863707+0000", 2024-09-15T03:47:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4919999999999993e-06 2024-09-15T03:47:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.871217+0000", 2024-09-15T03:47:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0075099880000000004 2024-09-15T03:47:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.871227+0000", 2024-09-15T03:47:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0029e-05 2024-09-15T03:47:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:385 3.es0 3.2c5be5ee (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-15T03:47:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:02.863745+0000", 2024-09-15T03:47:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.61411364499999999, 2024-09-15T03:47:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0075116790000000003, 2024-09-15T03:47:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 385 2024-09-15T03:47:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863745+0000", 2024-09-15T03:47:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863745+0000", 2024-09-15T03:47:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863745+0000", 2024-09-15T03:47:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.13e-07 2024-09-15T03:47:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863749+0000", 2024-09-15T03:47:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.33e-06 2024-09-15T03:47:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863758+0000", 2024-09-15T03:47:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6160000000000002e-06 2024-09-15T03:47:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863763+0000", 2024-09-15T03:47:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7979999999999996e-06 2024-09-15T03:47:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863797+0000", 2024-09-15T03:47:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4304999999999998e-05 2024-09-15T03:47:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863803+0000", 2024-09-15T03:47:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4110000000000004e-06 2024-09-15T03:47:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.871248+0000", 2024-09-15T03:47:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0074454830000000001 2024-09-15T03:47:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.871256+0000", 2024-09-15T03:47:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.123e-06 2024-09-15T03:47:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:386 3.es0 3.2c5be5ee (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-15T03:47:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:02.863811+0000", 2024-09-15T03:47:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.61404758800000003, 2024-09-15T03:47:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0074705300000000004, 2024-09-15T03:47:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 386 2024-09-15T03:47:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863811+0000", 2024-09-15T03:47:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863811+0000", 2024-09-15T03:47:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863811+0000", 2024-09-15T03:47:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9900000000000001e-07 2024-09-15T03:47:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863814+0000", 2024-09-15T03:47:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2799999999999999e-06 2024-09-15T03:47:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863821+0000", 2024-09-15T03:47:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9240000000000001e-06 2024-09-15T03:47:03.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863825+0000", 2024-09-15T03:47:03.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1200000000000004e-06 2024-09-15T03:47:03.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.581 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863889+0000", 2024-09-15T03:47:03.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3616999999999998e-05 2024-09-15T03:47:03.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.863896+0000", 2024-09-15T03:47:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7649999999999996e-06 2024-09-15T03:47:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.871274+0000", 2024-09-15T03:47:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0073780579999999998 2024-09-15T03:47:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.871281+0000", 2024-09-15T03:47:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.267e-06 2024-09-15T03:47:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:384 3.es0 3:77a7da34:::smithi00132172-4:head [read 0~1146880,checksum,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e56)", 2024-09-15T03:47:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:02.864202+0000", 2024-09-15T03:47:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.61365666699999999, 2024-09-15T03:47:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012715746999999999, 2024-09-15T03:47:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 384 2024-09-15T03:47:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864202+0000", 2024-09-15T03:47:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864202+0000", 2024-09-15T03:47:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864202+0000", 2024-09-15T03:47:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9400000000000004e-07 2024-09-15T03:47:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864206+0000", 2024-09-15T03:47:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9770000000000002e-06 2024-09-15T03:47:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864215+0000", 2024-09-15T03:47:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.337e-06 2024-09-15T03:47:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864221+0000", 2024-09-15T03:47:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4570000000000002e-06 2024-09-15T03:47:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864255+0000", 2024-09-15T03:47:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3725e-05 2024-09-15T03:47:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864260+0000", 2024-09-15T03:47:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.395e-06 2024-09-15T03:47:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.871305+0000", 2024-09-15T03:47:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0070446800000000002 2024-09-15T03:47:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.871413+0000", 2024-09-15T03:47:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000107731 2024-09-15T03:47:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.876917+0000", 2024-09-15T03:47:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0055046510000000002 2024-09-15T03:47:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:385 3.es0 3:77a7da34:::smithi00132172-4:head [read 0~1146880,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e56)", 2024-09-15T03:47:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:02.864230+0000", 2024-09-15T03:47:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.61362773599999998, 2024-09-15T03:47:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.014027420000000001, 2024-09-15T03:47:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 385 2024-09-15T03:47:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864230+0000", 2024-09-15T03:47:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864230+0000", 2024-09-15T03:47:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864231+0000", 2024-09-15T03:47:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7700000000000005e-07 2024-09-15T03:47:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864234+0000", 2024-09-15T03:47:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0369999999999999e-06 2024-09-15T03:47:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864242+0000", 2024-09-15T03:47:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4200000000000007e-06 2024-09-15T03:47:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864247+0000", 2024-09-15T03:47:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4789999999999998e-06 2024-09-15T03:47:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864290+0000", 2024-09-15T03:47:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3594000000000003e-05 2024-09-15T03:47:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864297+0000", 2024-09-15T03:47:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3579999999999999e-06 2024-09-15T03:47:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.872828+0000", 2024-09-15T03:47:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0085315449999999998 2024-09-15T03:47:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.872956+0000", 2024-09-15T03:47:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000127818 2024-09-15T03:47:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.878258+0000", 2024-09-15T03:47:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0053016920000000002 2024-09-15T03:47:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:386 3.es0 3:77a7da34:::smithi00132172-4:head [read 0~1146880,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e56)", 2024-09-15T03:47:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:02.864259+0000", 2024-09-15T03:47:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.61359894199999998, 2024-09-15T03:47:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.014933481, 2024-09-15T03:47:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 386 2024-09-15T03:47:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864259+0000", 2024-09-15T03:47:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864259+0000", 2024-09-15T03:47:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864260+0000", 2024-09-15T03:47:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3799999999999998e-07 2024-09-15T03:47:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864262+0000", 2024-09-15T03:47:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7209999999999999e-06 2024-09-15T03:47:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864270+0000", 2024-09-15T03:47:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6240000000000002e-06 2024-09-15T03:47:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864274+0000", 2024-09-15T03:47:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1699999999999999e-06 2024-09-15T03:47:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864316+0000", 2024-09-15T03:47:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1489999999999997e-05 2024-09-15T03:47:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864321+0000", 2024-09-15T03:47:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1100000000000002e-06 2024-09-15T03:47:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.873888+0000", 2024-09-15T03:47:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0095676790000000008 2024-09-15T03:47:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.874019+0000", 2024-09-15T03:47:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013085199999999999 2024-09-15T03:47:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.879193+0000", 2024-09-15T03:47:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0051733969999999997 2024-09-15T03:47:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:03.599 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:03.599 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:03.599 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:47:03.601 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:47:03.601 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:03.599+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:03.701 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:03.699+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:03.744 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:03.744 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:03.744 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-15T03:47:03.744 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:03.753 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:47:03.759 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:03.759 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:03.775 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:03.775 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:03.789 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:03.790 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:03.802 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:03.801+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:03.803 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:03.804 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:03.819 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:03.819 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:03.834 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:03.849 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:03.850 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:03.855 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:03.855 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:03.864 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:47:03.865 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:03.866 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:03.868 INFO:tasks.rados.rados.0.smithi001.stdout:150: done (6 left) 2024-09-15T03:47:03.869 INFO:tasks.rados.rados.0.smithi001.stdout:151: done (5 left) 2024-09-15T03:47:03.869 INFO:tasks.rados.rados.0.smithi001.stdout:152: done (4 left) 2024-09-15T03:47:03.869 INFO:tasks.rados.rados.0.smithi001.stdout:153: done (3 left) 2024-09-15T03:47:03.869 INFO:tasks.rados.rados.0.smithi001.stdout:154: done (2 left) 2024-09-15T03:47:03.869 INFO:tasks.rados.rados.0.smithi001.stdout:155: done (1 left) 2024-09-15T03:47:03.869 INFO:tasks.rados.rados.0.smithi001.stdout:156: done (0 left) 2024-09-15T03:47:03.869 INFO:tasks.rados.rados.0.smithi001.stdout:157: copy_from oid 25 from oid 31 current snap is 19 2024-09-15T03:47:03.869 INFO:tasks.rados.rados.0.smithi001.stdout:158: read oid 49 snap 15 2024-09-15T03:47:03.869 INFO:tasks.rados.rados.0.smithi001.stdout:158: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-15T03:47:03.869 INFO:tasks.rados.rados.0.smithi001.stdout:159: snap_remove snap 9 2024-09-15T03:47:03.884 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:03.884 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:03.901 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:03.901 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:03.902 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:03.901+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:03.916 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:47:03.916 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:47:03.932 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:03.932 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:03.970 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:03.970 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:03.978 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:47:04.023 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:04.021+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:04.097 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:04.097 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:04.104 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:04.104+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:04.107 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:47:04.205 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:04.205+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:04.249 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:04.249 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:04.257 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:47:04.306 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:04.304+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:04.357 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:04.358 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:04.366 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:47:04.407 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:04.407+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:04.507 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:04.507 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:04.508 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:04.508+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:04.515 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:47:04.609 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:04.609+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:04.616 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:04.616 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:04.624 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:47:04.710 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:04.708+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:04.725 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:04.725 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:04.733 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:47:04.818 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:04.816+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:04.831 INFO:tasks.rados.rados.0.smithi001.stdout:159: done (2 left) 2024-09-15T03:47:04.831 INFO:tasks.rados.rados.0.smithi001.stdout:160: read oid 4 snap -1 2024-09-15T03:47:04.832 INFO:tasks.rados.rados.0.smithi001.stdout:160: expect (ObjNum 64 snap 6 seq_num 64) 2024-09-15T03:47:04.832 INFO:tasks.rados.rados.0.smithi001.stdout:161: rmattr oid 32 current snap is 19 2024-09-15T03:47:04.832 INFO:tasks.rados.rados.0.smithi001.stdout:161: done (3 left) 2024-09-15T03:47:04.832 INFO:tasks.rados.rados.0.smithi001.stdout:162: snap_remove snap 17 2024-09-15T03:47:04.884 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T03:47:04.885 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:04.885 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:04.894 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:47:04.911 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:04.911+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:05.013 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:05.011+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:05.034 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:05.034 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:05.042 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:47:05.114 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:05.113+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:05.184 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:05.184 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:05.192 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:47:05.215 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:05.214+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:05.293 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:05.293 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:05.301 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:47:05.315 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:05.314+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:05.402 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:05.402 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:05.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:47:05.416 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:05.416+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:05.510 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:05.510 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:05.517 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:05.515+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:05.518 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:47:05.618 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:05.617+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:05.628 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:05.628 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:05.637 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:47:05.718 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:05.718+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:05.734 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:05.734 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:05.743 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:47:05.819 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:05.819+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:05.833 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-15T03:47:05.838 INFO:tasks.rados.rados.0.smithi001.stdout:162: done (3 left) 2024-09-15T03:47:05.838 INFO:tasks.rados.rados.0.smithi001.stdout:163: snap_remove snap 12 2024-09-15T03:47:05.845 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:05.845 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:05.853 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:47:05.921 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:05.919+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:05.943 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:47:05.991 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:05.991 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:05.999 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:47:06.021 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:06.020+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:06.069 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:06.083 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:06.083 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:06.096 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:06.096 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:06.098 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:06.098 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:06.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:47:06.112 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:06.112 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:06.127 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:06.127 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:06.142 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:06.142 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:06.148 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:06.148+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:06.157 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:06.172 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:06.172 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:06.187 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:06.187 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:06.202 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:06.202 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:06.203 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:06.204 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:06.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:06.204 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:47:06.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.204 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es7 34/28 ECSubWrite(tid=33, reqid=client.4521.0:195, at_version=34'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-15T03:47:06.204 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:41.118211+0000", 2024-09-15T03:47:06.204 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 25.083607228000002, 2024-09-15T03:47:06.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0070162009999999997, 2024-09-15T03:47:06.204 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:06.204 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:06.204 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:06.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:06.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118211+0000", 2024-09-15T03:47:06.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:06.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118211+0000", 2024-09-15T03:47:06.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:06.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118212+0000", 2024-09-15T03:47:06.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5599999999999999e-07 2024-09-15T03:47:06.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:06.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118422+0000", 2024-09-15T03:47:06.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021081 2024-09-15T03:47:06.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:06.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118433+0000", 2024-09-15T03:47:06.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0529e-05 2024-09-15T03:47:06.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:06.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118438+0000", 2024-09-15T03:47:06.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8210000000000001e-06 2024-09-15T03:47:06.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.207 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:06.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118479+0000", 2024-09-15T03:47:06.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1493999999999999e-05 2024-09-15T03:47:06.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.207 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:06.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118490+0000", 2024-09-15T03:47:06.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0550000000000001e-05 2024-09-15T03:47:06.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.207 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:06.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118700+0000", 2024-09-15T03:47:06.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021063500000000001 2024-09-15T03:47:06.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.208 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:06.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118711+0000", 2024-09-15T03:47:06.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0849000000000001e-05 2024-09-15T03:47:06.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.208 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:06.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118776+0000", 2024-09-15T03:47:06.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4702999999999994e-05 2024-09-15T03:47:06.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:06.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.118797+0000", 2024-09-15T03:47:06.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1265999999999999e-05 2024-09-15T03:47:06.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:06.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:41.125228+0000", 2024-09-15T03:47:06.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064303880000000004 2024-09-15T03:47:06.209 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.209 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:06.209 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.210 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s3 39/31 ECSubWrite(tid=44, reqid=osd.1.0:44, at_version=39'12, trim_to=0'0, roll_forward_to=38'10))", 2024-09-15T03:47:06.210 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.150872+0000", 2024-09-15T03:47:06.210 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 20.050946791000001, 2024-09-15T03:47:06.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075168539999999999, 2024-09-15T03:47:06.210 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:06.210 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:06.210 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:06.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:06.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150872+0000", 2024-09-15T03:47:06.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:06.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150872+0000", 2024-09-15T03:47:06.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:06.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150872+0000", 2024-09-15T03:47:06.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3300000000000002e-07 2024-09-15T03:47:06.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:06.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150909+0000", 2024-09-15T03:47:06.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6294000000000002e-05 2024-09-15T03:47:06.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.212 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:06.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150919+0000", 2024-09-15T03:47:06.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9690000000000006e-06 2024-09-15T03:47:06.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.212 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.212 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:06.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150924+0000", 2024-09-15T03:47:06.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7710000000000003e-06 2024-09-15T03:47:06.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.212 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.212 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:06.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150958+0000", 2024-09-15T03:47:06.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3531999999999997e-05 2024-09-15T03:47:06.213 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.213 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:06.213 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150970+0000", 2024-09-15T03:47:06.213 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1831000000000001e-05 2024-09-15T03:47:06.213 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.213 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:06.213 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152463+0000", 2024-09-15T03:47:06.213 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0014934239999999999 2024-09-15T03:47:06.213 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.213 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:06.213 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152471+0000", 2024-09-15T03:47:06.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2830000000000004e-06 2024-09-15T03:47:06.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:06.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152507+0000", 2024-09-15T03:47:06.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6739999999999997e-05 2024-09-15T03:47:06.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:06.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152517+0000", 2024-09-15T03:47:06.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8570000000000007e-06 2024-09-15T03:47:06.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:06.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.158389+0000", 2024-09-15T03:47:06.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058715199999999999 2024-09-15T03:47:06.215 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.215 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:06.215 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.215 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.215 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.0s2 39/31 ECSubWrite(tid=45, reqid=client.4521.0:241, at_version=39'17, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:06.215 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.150931+0000", 2024-09-15T03:47:06.215 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 20.050887672000002, 2024-09-15T03:47:06.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0072693200000000001, 2024-09-15T03:47:06.215 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:06.215 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:06.215 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:06.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:06.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150931+0000", 2024-09-15T03:47:06.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:06.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150931+0000", 2024-09-15T03:47:06.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:06.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150931+0000", 2024-09-15T03:47:06.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3699999999999999e-07 2024-09-15T03:47:06.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:06.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150942+0000", 2024-09-15T03:47:06.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1053000000000001e-05 2024-09-15T03:47:06.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:06.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150946+0000", 2024-09-15T03:47:06.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0890000000000002e-06 2024-09-15T03:47:06.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.218 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:06.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150949+0000", 2024-09-15T03:47:06.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5270000000000001e-06 2024-09-15T03:47:06.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.218 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.218 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:06.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150983+0000", 2024-09-15T03:47:06.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4258000000000003e-05 2024-09-15T03:47:06.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.218 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.218 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:06.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150994+0000", 2024-09-15T03:47:06.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0893999999999999e-05 2024-09-15T03:47:06.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.219 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.219 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:06.219 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.151057+0000", 2024-09-15T03:47:06.219 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3478000000000004e-05 2024-09-15T03:47:06.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.219 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.219 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:06.219 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.151063+0000", 2024-09-15T03:47:06.219 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7880000000000003e-06 2024-09-15T03:47:06.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.219 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.219 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:06.219 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152305+0000", 2024-09-15T03:47:06.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012418850000000001 2024-09-15T03:47:06.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:06.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152335+0000", 2024-09-15T03:47:06.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0049e-05 2024-09-15T03:47:06.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:06.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.158200+0000", 2024-09-15T03:47:06.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058650619999999999 2024-09-15T03:47:06.220 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.220 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:06.221 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.221 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.221 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.0s2 39/31 ECSubWrite(tid=46, reqid=client.4521.0:242, at_version=39'18, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:06.221 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.150954+0000", 2024-09-15T03:47:06.221 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 20.050864170000001, 2024-09-15T03:47:06.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.007342459, 2024-09-15T03:47:06.221 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:06.221 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:06.221 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:06.221 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.221 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:06.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150954+0000", 2024-09-15T03:47:06.222 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.222 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:06.222 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150954+0000", 2024-09-15T03:47:06.222 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.222 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:06.222 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150955+0000", 2024-09-15T03:47:06.222 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2700000000000001e-07 2024-09-15T03:47:06.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:06.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150964+0000", 2024-09-15T03:47:06.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2820000000000002e-06 2024-09-15T03:47:06.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:06.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150970+0000", 2024-09-15T03:47:06.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0700000000000003e-06 2024-09-15T03:47:06.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:06.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150973+0000", 2024-09-15T03:47:06.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.576e-06 2024-09-15T03:47:06.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:06.224 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.151082+0000", 2024-09-15T03:47:06.224 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000109261 2024-09-15T03:47:06.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:06.224 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.151088+0000", 2024-09-15T03:47:06.224 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2480000000000001e-06 2024-09-15T03:47:06.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:06.224 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.151145+0000", 2024-09-15T03:47:06.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6886999999999999e-05 2024-09-15T03:47:06.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:06.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152307+0000", 2024-09-15T03:47:06.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0011623180000000001 2024-09-15T03:47:06.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:06.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152336+0000", 2024-09-15T03:47:06.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8677999999999999e-05 2024-09-15T03:47:06.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.226 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:06.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152365+0000", 2024-09-15T03:47:06.226 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9170999999999999e-05 2024-09-15T03:47:06.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.226 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:06.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.158297+0000", 2024-09-15T03:47:06.226 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0059317409999999999 2024-09-15T03:47:06.226 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.226 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:06.226 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.227 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 39/28 ECSubWrite(tid=42, reqid=client.4521.0:255, at_version=39'19, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:47:06.227 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.170724+0000", 2024-09-15T03:47:06.227 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 20.031094583000002, 2024-09-15T03:47:06.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0086604270000000001, 2024-09-15T03:47:06.227 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:06.227 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:06.227 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:06.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:06.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170724+0000", 2024-09-15T03:47:06.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:06.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170724+0000", 2024-09-15T03:47:06.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:06.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170725+0000", 2024-09-15T03:47:06.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8999999999999995e-07 2024-09-15T03:47:06.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:06.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170742+0000", 2024-09-15T03:47:06.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7181e-05 2024-09-15T03:47:06.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:06.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170757+0000", 2024-09-15T03:47:06.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4693e-05 2024-09-15T03:47:06.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:06.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170763+0000", 2024-09-15T03:47:06.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4509999999999997e-06 2024-09-15T03:47:06.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:06.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170807+0000", 2024-09-15T03:47:06.230 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3574e-05 2024-09-15T03:47:06.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.230 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:06.230 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170815+0000", 2024-09-15T03:47:06.230 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3459999999999994e-06 2024-09-15T03:47:06.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.230 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:06.230 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170900+0000", 2024-09-15T03:47:06.230 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5436e-05 2024-09-15T03:47:06.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.231 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:06.231 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170906+0000", 2024-09-15T03:47:06.231 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9190000000000004e-06 2024-09-15T03:47:06.231 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.231 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.231 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:06.231 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173681+0000", 2024-09-15T03:47:06.231 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00277437 2024-09-15T03:47:06.231 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.231 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.231 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:06.231 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173695+0000", 2024-09-15T03:47:06.231 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4446e-05 2024-09-15T03:47:06.232 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.232 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.232 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:06.232 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179384+0000", 2024-09-15T03:47:06.232 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0056892210000000004 2024-09-15T03:47:06.232 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.232 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:06.232 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.232 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.232 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.232 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:47:06.232 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:06.233 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 20.030716748, 2024-09-15T03:47:06.233 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012183955999999999, 2024-09-15T03:47:06.233 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:06.233 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:06.233 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:06.233 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.233 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:06.233 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:06.233 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.233 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.233 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.233 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:06.233 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:06.233 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.234 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.234 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:06.234 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:06.234 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.17e-07 2024-09-15T03:47:06.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.234 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.234 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:06.234 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171454+0000", 2024-09-15T03:47:06.234 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00035226700000000002 2024-09-15T03:47:06.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.234 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.234 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:06.235 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171465+0000", 2024-09-15T03:47:06.235 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0943e-05 2024-09-15T03:47:06.235 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.235 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.235 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:06.235 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171471+0000", 2024-09-15T03:47:06.235 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7400000000000001e-06 2024-09-15T03:47:06.235 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.235 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.235 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:06.235 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171508+0000", 2024-09-15T03:47:06.235 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7147999999999997e-05 2024-09-15T03:47:06.235 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.235 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.236 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:06.236 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171517+0000", 2024-09-15T03:47:06.236 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2560000000000008e-06 2024-09-15T03:47:06.236 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.236 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.236 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:06.236 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171716+0000", 2024-09-15T03:47:06.236 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019899699999999999 2024-09-15T03:47:06.236 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.236 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.236 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:06.236 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173686+0000", 2024-09-15T03:47:06.236 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019696829999999999 2024-09-15T03:47:06.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.237 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.237 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:06.237 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173777+0000", 2024-09-15T03:47:06.237 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0637999999999995e-05 2024-09-15T03:47:06.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.237 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.237 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:06.237 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173814+0000", 2024-09-15T03:47:06.237 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7499000000000001e-05 2024-09-15T03:47:06.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.237 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.237 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:06.238 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.183286+0000", 2024-09-15T03:47:06.238 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094716680000000008 2024-09-15T03:47:06.238 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.238 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:06.238 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.238 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.238 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 39/28 ECSubWrite(tid=43, reqid=client.4521.0:256, at_version=39'20, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:47:06.238 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.171211+0000", 2024-09-15T03:47:06.238 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 20.030607655000001, 2024-09-15T03:47:06.239 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082752069999999997, 2024-09-15T03:47:06.239 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:06.239 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:06.239 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:06.239 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.240 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:06.240 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171211+0000", 2024-09-15T03:47:06.240 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.240 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.240 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.240 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:06.240 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171211+0000", 2024-09-15T03:47:06.240 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.240 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.240 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.240 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:06.240 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171211+0000", 2024-09-15T03:47:06.240 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.39e-07 2024-09-15T03:47:06.241 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.241 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.241 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:06.241 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171228+0000", 2024-09-15T03:47:06.241 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6254999999999999e-05 2024-09-15T03:47:06.241 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.241 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.241 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:06.241 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171241+0000", 2024-09-15T03:47:06.241 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3482999999999999e-05 2024-09-15T03:47:06.241 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.241 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.241 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:06.242 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171247+0000", 2024-09-15T03:47:06.242 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.088e-06 2024-09-15T03:47:06.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.242 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.242 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:06.242 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171292+0000", 2024-09-15T03:47:06.242 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5062000000000003e-05 2024-09-15T03:47:06.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.242 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.242 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:06.242 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171302+0000", 2024-09-15T03:47:06.242 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.302e-06 2024-09-15T03:47:06.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.242 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.243 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:06.243 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171382+0000", 2024-09-15T03:47:06.243 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0717999999999995e-05 2024-09-15T03:47:06.243 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.243 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.243 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:06.243 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173686+0000", 2024-09-15T03:47:06.243 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023032320000000001 2024-09-15T03:47:06.243 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.243 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.243 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:06.243 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173776+0000", 2024-09-15T03:47:06.243 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0302000000000004e-05 2024-09-15T03:47:06.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.244 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.244 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:06.244 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173805+0000", 2024-09-15T03:47:06.244 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8979000000000002e-05 2024-09-15T03:47:06.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.244 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.244 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:06.244 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179486+0000", 2024-09-15T03:47:06.244 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0056813469999999998 2024-09-15T03:47:06.244 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.244 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:06.244 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.245 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.245 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 39/31 ECSubWrite(tid=44, reqid=client.4521.0:243, at_version=0'0, trim_to=0'0, roll_forward_to=39'17))", 2024-09-15T03:47:06.245 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.176404+0000", 2024-09-15T03:47:06.245 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 20.025414858000001, 2024-09-15T03:47:06.245 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069470829999999997, 2024-09-15T03:47:06.245 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:06.245 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:06.245 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:06.245 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.245 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:06.245 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176404+0000", 2024-09-15T03:47:06.245 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.245 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.246 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:06.246 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176404+0000", 2024-09-15T03:47:06.246 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.246 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.246 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:06.246 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176404+0000", 2024-09-15T03:47:06.246 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.54e-07 2024-09-15T03:47:06.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.246 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.246 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:06.246 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176417+0000", 2024-09-15T03:47:06.246 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2785999999999999e-05 2024-09-15T03:47:06.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.247 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.247 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:06.247 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176430+0000", 2024-09-15T03:47:06.247 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2622999999999999e-05 2024-09-15T03:47:06.247 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.247 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.247 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:06.247 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176436+0000", 2024-09-15T03:47:06.247 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3570000000000003e-06 2024-09-15T03:47:06.247 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.247 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.247 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:06.247 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176479+0000", 2024-09-15T03:47:06.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2939999999999999e-05 2024-09-15T03:47:06.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.248 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.248 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:06.248 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176487+0000", 2024-09-15T03:47:06.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2979999999999992e-06 2024-09-15T03:47:06.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.248 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.248 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:06.248 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176580+0000", 2024-09-15T03:47:06.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2349999999999995e-05 2024-09-15T03:47:06.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.248 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.249 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:06.249 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176588+0000", 2024-09-15T03:47:06.249 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2360000000000004e-06 2024-09-15T03:47:06.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.249 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.249 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:06.249 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176618+0000", 2024-09-15T03:47:06.249 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9459999999999999e-05 2024-09-15T03:47:06.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.249 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.249 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:06.249 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.176629+0000", 2024-09-15T03:47:06.249 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1382999999999999e-05 2024-09-15T03:47:06.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.250 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.250 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:06.250 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.183351+0000", 2024-09-15T03:47:06.250 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0067218959999999998 2024-09-15T03:47:06.250 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.250 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:06.250 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.250 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.250 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.250 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:337 3.cs0 3:390b2f04:::smithi00132172-33:head [delete] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e49)", 2024-09-15T03:47:06.250 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.856819+0000", 2024-09-15T03:47:06.250 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.344999514, 2024-09-15T03:47:06.250 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0077070849999999998, 2024-09-15T03:47:06.250 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:06.251 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:06.251 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:06.251 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:06.251 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:06.251 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 337 2024-09-15T03:47:06.251 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.251 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:06.251 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.251 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:06.251 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856819+0000", 2024-09-15T03:47:06.251 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.251 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.251 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.252 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:06.252 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856819+0000", 2024-09-15T03:47:06.252 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.252 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.252 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.252 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:06.252 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856821+0000", 2024-09-15T03:47:06.252 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6700000000000001e-06 2024-09-15T03:47:06.252 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.252 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.252 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:06.252 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856827+0000", 2024-09-15T03:47:06.252 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8080000000000001e-06 2024-09-15T03:47:06.252 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.253 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.253 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:06.253 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856843+0000", 2024-09-15T03:47:06.253 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6430999999999999e-05 2024-09-15T03:47:06.253 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.253 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.253 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:06.253 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:06.253 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7920000000000004e-06 2024-09-15T03:47:06.253 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.253 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.253 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:06.253 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856889+0000", 2024-09-15T03:47:06.253 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8825000000000002e-05 2024-09-15T03:47:06.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.254 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.254 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:06.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856960+0000", 2024-09-15T03:47:06.254 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1470999999999994e-05 2024-09-15T03:47:06.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.254 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.254 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:06.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857667+0000", 2024-09-15T03:47:06.254 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00070671199999999997 2024-09-15T03:47:06.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.254 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.254 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:06.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857896+0000", 2024-09-15T03:47:06.255 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000229621 2024-09-15T03:47:06.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.255 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:06.255 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857912+0000", 2024-09-15T03:47:06.255 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5400000000000002e-05 2024-09-15T03:47:06.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.255 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:06.255 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857949+0000", 2024-09-15T03:47:06.255 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7239e-05 2024-09-15T03:47:06.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.256 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:06.256 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858108+0000", 2024-09-15T03:47:06.256 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015922399999999999 2024-09-15T03:47:06.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.256 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.256 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:06.256 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859023+0000", 2024-09-15T03:47:06.256 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00091487899999999995 2024-09-15T03:47:06.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.256 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.256 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:06.256 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.864526+0000", 2024-09-15T03:47:06.257 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0055030130000000002 2024-09-15T03:47:06.257 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.257 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:06.257 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.257 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.257 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.257 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 49/31 ECSubWrite(tid=87, reqid=client.4521.0:335, at_version=49'21, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:06.257 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.858786+0000", 2024-09-15T03:47:06.257 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.343032879000001, 2024-09-15T03:47:06.257 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0085806300000000005, 2024-09-15T03:47:06.257 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:06.257 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:06.257 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:06.257 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.258 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:06.258 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858786+0000", 2024-09-15T03:47:06.258 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.258 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.258 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.258 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:06.258 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858786+0000", 2024-09-15T03:47:06.258 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.258 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.258 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.258 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:06.258 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858786+0000", 2024-09-15T03:47:06.258 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3099999999999997e-07 2024-09-15T03:47:06.258 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:06.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858809+0000", 2024-09-15T03:47:06.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2331999999999999e-05 2024-09-15T03:47:06.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:06.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858819+0000", 2024-09-15T03:47:06.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0468e-05 2024-09-15T03:47:06.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:06.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858828+0000", 2024-09-15T03:47:06.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0499999999999997e-06 2024-09-15T03:47:06.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:06.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858864+0000", 2024-09-15T03:47:06.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6072000000000003e-05 2024-09-15T03:47:06.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:06.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858870+0000", 2024-09-15T03:47:06.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.896e-06 2024-09-15T03:47:06.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.261 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:06.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858938+0000", 2024-09-15T03:47:06.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7657000000000006e-05 2024-09-15T03:47:06.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.261 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.261 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:06.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858946+0000", 2024-09-15T03:47:06.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0609999999999996e-06 2024-09-15T03:47:06.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.261 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.261 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:06.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861287+0000", 2024-09-15T03:47:06.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023413779999999999 2024-09-15T03:47:06.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.262 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.262 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:06.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861301+0000", 2024-09-15T03:47:06.262 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3626e-05 2024-09-15T03:47:06.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.262 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.262 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:06.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.867366+0000", 2024-09-15T03:47:06.262 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0060654589999999996 2024-09-15T03:47:06.262 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.262 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:06.262 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.263 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 49/31 ECSubWrite(tid=88, reqid=client.4521.0:336, at_version=49'22, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:06.263 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.859239+0000", 2024-09-15T03:47:06.263 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.342579906999999, 2024-09-15T03:47:06.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082420800000000006, 2024-09-15T03:47:06.263 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:06.263 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:06.263 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:06.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.263 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:06.263 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859239+0000", 2024-09-15T03:47:06.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.264 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.264 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:06.264 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859239+0000", 2024-09-15T03:47:06.264 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.264 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.264 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:06.264 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859241+0000", 2024-09-15T03:47:06.264 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4880000000000001e-06 2024-09-15T03:47:06.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.264 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.264 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:06.264 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859256+0000", 2024-09-15T03:47:06.264 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5119999999999999e-05 2024-09-15T03:47:06.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:06.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859265+0000", 2024-09-15T03:47:06.265 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1500000000000005e-06 2024-09-15T03:47:06.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:06.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859274+0000", 2024-09-15T03:47:06.265 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2060000000000007e-06 2024-09-15T03:47:06.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:06.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859314+0000", 2024-09-15T03:47:06.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0250000000000003e-05 2024-09-15T03:47:06.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.266 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.266 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:06.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859324+0000", 2024-09-15T03:47:06.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0513000000000001e-05 2024-09-15T03:47:06.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.266 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.266 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:06.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859418+0000", 2024-09-15T03:47:06.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3397e-05 2024-09-15T03:47:06.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.267 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:06.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861290+0000", 2024-09-15T03:47:06.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0018723920000000001 2024-09-15T03:47:06.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.267 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:06.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861353+0000", 2024-09-15T03:47:06.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2706000000000006e-05 2024-09-15T03:47:06.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.267 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:06.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861365+0000", 2024-09-15T03:47:06.268 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2316e-05 2024-09-15T03:47:06.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:06.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.867481+0000", 2024-09-15T03:47:06.268 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0061155419999999999 2024-09-15T03:47:06.268 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.268 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:06.268 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.268 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.9s4 49/31 ECSubWrite(tid=35, reqid=client.4521.0:331, at_version=49'15, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:47:06.268 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.860386+0000", 2024-09-15T03:47:06.269 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.341432691, 2024-09-15T03:47:06.269 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0070832780000000001, 2024-09-15T03:47:06.269 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:06.269 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:06.269 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:06.269 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.269 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:06.269 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860386+0000", 2024-09-15T03:47:06.269 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.269 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.269 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.269 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:06.269 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860386+0000", 2024-09-15T03:47:06.269 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.270 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.270 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.270 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:06.270 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860386+0000", 2024-09-15T03:47:06.270 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1199999999999999e-07 2024-09-15T03:47:06.270 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.270 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.270 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:06.270 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860431+0000", 2024-09-15T03:47:06.270 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5210999999999998e-05 2024-09-15T03:47:06.270 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.270 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.270 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:06.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860438+0000", 2024-09-15T03:47:06.271 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3659999999999997e-06 2024-09-15T03:47:06.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.271 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:06.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860441+0000", 2024-09-15T03:47:06.271 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4709999999999999e-06 2024-09-15T03:47:06.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.271 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:06.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860476+0000", 2024-09-15T03:47:06.271 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4699000000000001e-05 2024-09-15T03:47:06.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:06.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860485+0000", 2024-09-15T03:47:06.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4190000000000008e-06 2024-09-15T03:47:06.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:06.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.860578+0000", 2024-09-15T03:47:06.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2390000000000001e-05 2024-09-15T03:47:06.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:06.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861291+0000", 2024-09-15T03:47:06.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00071288899999999997 2024-09-15T03:47:06.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:06.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861354+0000", 2024-09-15T03:47:06.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3004999999999997e-05 2024-09-15T03:47:06.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:06.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861400+0000", 2024-09-15T03:47:06.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6711000000000001e-05 2024-09-15T03:47:06.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:06.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.867469+0000", 2024-09-15T03:47:06.274 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006068805 2024-09-15T03:47:06.274 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.274 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:06.274 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.274 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.274 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 51/31 ECSubWrite(tid=67, reqid=client.4521.0:350, at_version=51'33, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:06.274 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.843852+0000", 2024-09-15T03:47:06.274 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 8.3579667870000005, 2024-09-15T03:47:06.274 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010895471, 2024-09-15T03:47:06.274 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:06.274 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:06.274 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:06.274 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.275 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:06.275 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843852+0000", 2024-09-15T03:47:06.275 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.275 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.275 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:06.275 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843852+0000", 2024-09-15T03:47:06.275 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.275 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.275 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:06.275 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843853+0000", 2024-09-15T03:47:06.276 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.359e-06 2024-09-15T03:47:06.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.276 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.276 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:06.276 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843882+0000", 2024-09-15T03:47:06.276 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8711999999999999e-05 2024-09-15T03:47:06.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.276 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.276 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:06.276 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843901+0000", 2024-09-15T03:47:06.276 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9477e-05 2024-09-15T03:47:06.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.276 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.277 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:06.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843911+0000", 2024-09-15T03:47:06.277 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2590000000000006e-06 2024-09-15T03:47:06.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.277 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.277 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:06.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843973+0000", 2024-09-15T03:47:06.277 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2310999999999993e-05 2024-09-15T03:47:06.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.277 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.277 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:06.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843984+0000", 2024-09-15T03:47:06.277 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.132e-05 2024-09-15T03:47:06.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.278 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.278 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:06.278 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844108+0000", 2024-09-15T03:47:06.278 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012384399999999999 2024-09-15T03:47:06.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.278 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.278 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:06.278 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844139+0000", 2024-09-15T03:47:06.278 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1148000000000001e-05 2024-09-15T03:47:06.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.278 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.278 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:06.278 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844185+0000", 2024-09-15T03:47:06.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5388e-05 2024-09-15T03:47:06.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.279 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:06.279 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844202+0000", 2024-09-15T03:47:06.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7708999999999999e-05 2024-09-15T03:47:06.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.279 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:06.279 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.854747+0000", 2024-09-15T03:47:06.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010544944000000001 2024-09-15T03:47:06.279 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.279 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:06.280 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.280 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.280 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s7 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:06.280 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844746+0000", 2024-09-15T03:47:06.280 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 8.3570721900000002, 2024-09-15T03:47:06.280 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095535919999999996, 2024-09-15T03:47:06.280 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:06.280 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:06.280 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:06.280 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.280 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:06.280 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844746+0000", 2024-09-15T03:47:06.281 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.281 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.281 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.281 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:06.281 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844746+0000", 2024-09-15T03:47:06.281 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.281 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.281 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.281 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:06.281 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844747+0000", 2024-09-15T03:47:06.281 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6899999999999996e-07 2024-09-15T03:47:06.281 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.281 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.282 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:06.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844769+0000", 2024-09-15T03:47:06.282 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1787999999999999e-05 2024-09-15T03:47:06.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.282 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.282 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:06.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844785+0000", 2024-09-15T03:47:06.282 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6212999999999999e-05 2024-09-15T03:47:06.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.282 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.282 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:06.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844808+0000", 2024-09-15T03:47:06.282 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2167e-05 2024-09-15T03:47:06.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.283 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.283 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:06.283 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844875+0000", 2024-09-15T03:47:06.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7683999999999998e-05 2024-09-15T03:47:06.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.283 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.283 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:06.283 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844893+0000", 2024-09-15T03:47:06.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7331000000000001e-05 2024-09-15T03:47:06.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.283 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.283 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:06.283 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844978+0000", 2024-09-15T03:47:06.284 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5711999999999995e-05 2024-09-15T03:47:06.284 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.284 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.284 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:06.284 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844989+0000", 2024-09-15T03:47:06.284 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1026000000000001e-05 2024-09-15T03:47:06.284 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.284 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.284 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:06.284 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847475+0000", 2024-09-15T03:47:06.284 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024854830000000001 2024-09-15T03:47:06.284 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.284 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.285 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:06.285 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847492+0000", 2024-09-15T03:47:06.285 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7189000000000001e-05 2024-09-15T03:47:06.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.285 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.285 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:06.285 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.854300+0000", 2024-09-15T03:47:06.285 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0068080299999999996 2024-09-15T03:47:06.285 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.285 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:06.285 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.285 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.286 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s7 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:06.286 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844831+0000", 2024-09-15T03:47:06.286 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 8.3569871459999998, 2024-09-15T03:47:06.286 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095609189999999993, 2024-09-15T03:47:06.286 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:06.286 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:06.286 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:06.286 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.286 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:06.286 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844831+0000", 2024-09-15T03:47:06.286 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.286 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.286 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.287 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:06.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844831+0000", 2024-09-15T03:47:06.287 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.287 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.287 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:06.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844832+0000", 2024-09-15T03:47:06.287 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1399999999999997e-07 2024-09-15T03:47:06.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.287 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.287 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:06.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844855+0000", 2024-09-15T03:47:06.287 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2904e-05 2024-09-15T03:47:06.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.288 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.288 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:06.288 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844866+0000", 2024-09-15T03:47:06.288 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0691e-05 2024-09-15T03:47:06.288 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.288 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.288 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:06.288 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844872+0000", 2024-09-15T03:47:06.288 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.409e-06 2024-09-15T03:47:06.288 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.288 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.288 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:06.288 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845030+0000", 2024-09-15T03:47:06.289 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015832800000000001 2024-09-15T03:47:06.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.289 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.289 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:06.289 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845037+0000", 2024-09-15T03:47:06.289 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1099999999999997e-06 2024-09-15T03:47:06.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.289 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.289 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:06.289 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845118+0000", 2024-09-15T03:47:06.289 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0840000000000005e-05 2024-09-15T03:47:06.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.290 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.290 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:06.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847478+0000", 2024-09-15T03:47:06.290 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023595869999999998 2024-09-15T03:47:06.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.290 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.290 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:06.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847529+0000", 2024-09-15T03:47:06.290 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1585000000000002e-05 2024-09-15T03:47:06.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.290 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.290 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:06.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847544+0000", 2024-09-15T03:47:06.291 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4365000000000001e-05 2024-09-15T03:47:06.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.291 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.291 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:06.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.854392+0000", 2024-09-15T03:47:06.291 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0068485860000000003 2024-09-15T03:47:06.291 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.291 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:06.291 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.291 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.291 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 51/31 ECSubWrite(tid=68, reqid=client.4521.0:351, at_version=51'34, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:06.291 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:06.291 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 8.3566029660000005, 2024-09-15T03:47:06.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098406970000000007, 2024-09-15T03:47:06.292 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:06.292 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:06.292 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:06.292 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.292 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:06.292 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:06.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.292 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.292 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:06.292 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:06.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.293 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.293 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.293 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:06.293 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:06.293 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1299999999999999e-07 2024-09-15T03:47:06.293 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.293 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.293 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:06.293 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845232+0000", 2024-09-15T03:47:06.293 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5366000000000002e-05 2024-09-15T03:47:06.293 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.293 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.293 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:06.293 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845242+0000", 2024-09-15T03:47:06.294 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0403e-05 2024-09-15T03:47:06.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.294 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.294 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:06.294 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845247+0000", 2024-09-15T03:47:06.294 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4870000000000004e-06 2024-09-15T03:47:06.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.294 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.294 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:06.294 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845292+0000", 2024-09-15T03:47:06.294 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5762e-05 2024-09-15T03:47:06.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.294 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.295 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:06.295 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845302+0000", 2024-09-15T03:47:06.295 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5969999999999998e-06 2024-09-15T03:47:06.295 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.295 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.295 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:06.295 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845378+0000", 2024-09-15T03:47:06.295 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5628000000000002e-05 2024-09-15T03:47:06.295 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.295 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.295 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:06.295 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847479+0000", 2024-09-15T03:47:06.295 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021009380000000001 2024-09-15T03:47:06.295 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.296 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.296 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:06.296 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847530+0000", 2024-09-15T03:47:06.296 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1548000000000002e-05 2024-09-15T03:47:06.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.296 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.296 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:06.296 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847544+0000", 2024-09-15T03:47:06.296 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4355000000000001e-05 2024-09-15T03:47:06.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.296 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.296 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:06.296 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.855056+0000", 2024-09-15T03:47:06.297 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075119000000000002 2024-09-15T03:47:06.297 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.297 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:06.297 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.297 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.297 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 51/31 ECSubWrite(tid=93, reqid=client.4521.0:352, at_version=51'24, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:06.297 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:06.297 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 8.3545390319999999, 2024-09-15T03:47:06.297 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0096136880000000004, 2024-09-15T03:47:06.297 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:06.297 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:06.297 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:06.298 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.298 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:06.298 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:06.298 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.298 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.298 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:06.298 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:06.298 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.298 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.298 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:06.298 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:06.299 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9100000000000003e-07 2024-09-15T03:47:06.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.299 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.299 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:06.299 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847669+0000", 2024-09-15T03:47:06.299 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00038855999999999998 2024-09-15T03:47:06.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.299 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.299 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:06.299 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847682+0000", 2024-09-15T03:47:06.299 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3112000000000001e-05 2024-09-15T03:47:06.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.299 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.300 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:06.300 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847687+0000", 2024-09-15T03:47:06.300 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0880000000000002e-06 2024-09-15T03:47:06.300 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.300 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.300 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:06.300 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847745+0000", 2024-09-15T03:47:06.300 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8400999999999999e-05 2024-09-15T03:47:06.300 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.300 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.300 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:06.301 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847756+0000", 2024-09-15T03:47:06.301 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0325e-05 2024-09-15T03:47:06.301 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.301 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.301 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:06.301 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.848092+0000", 2024-09-15T03:47:06.301 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033636099999999997 2024-09-15T03:47:06.301 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.301 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.301 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:06.301 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.848101+0000", 2024-09-15T03:47:06.301 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6240000000000008e-06 2024-09-15T03:47:06.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.302 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.302 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:06.302 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850132+0000", 2024-09-15T03:47:06.302 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020309239999999999 2024-09-15T03:47:06.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.302 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.302 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:06.302 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850174+0000", 2024-09-15T03:47:06.302 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2630999999999999e-05 2024-09-15T03:47:06.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.302 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.302 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:06.302 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856893+0000", 2024-09-15T03:47:06.303 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0067188710000000004 2024-09-15T03:47:06.303 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.303 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:06.303 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.303 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.303 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:389 3.cs0 3:3164c049:::smithi00132172-34:head [read 0~1179648,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-15T03:47:06.303 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:02.863691+0000", 2024-09-15T03:47:06.303 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 3.3381276839999998, 2024-09-15T03:47:06.303 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075454679999999996, 2024-09-15T03:47:06.303 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:06.303 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:06.303 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:06.304 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:06.304 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:06.304 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 389 2024-09-15T03:47:06.304 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.304 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:06.304 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.304 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:06.304 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:02.863691+0000", 2024-09-15T03:47:06.304 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.304 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.304 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.304 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:06.304 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:02.863691+0000", 2024-09-15T03:47:06.305 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.305 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.305 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:06.305 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:02.863691+0000", 2024-09-15T03:47:06.305 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4900000000000001e-07 2024-09-15T03:47:06.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.305 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.305 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:06.305 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:02.863694+0000", 2024-09-15T03:47:06.305 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2630000000000002e-06 2024-09-15T03:47:06.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.305 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.305 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:06.306 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:02.863699+0000", 2024-09-15T03:47:06.306 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5480000000000001e-06 2024-09-15T03:47:06.306 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.306 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.306 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:06.306 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:02.863705+0000", 2024-09-15T03:47:06.306 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3400000000000003e-06 2024-09-15T03:47:06.306 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.306 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.306 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:06.306 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:02.863809+0000", 2024-09-15T03:47:06.306 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010391 2024-09-15T03:47:06.306 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.307 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.307 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:06.307 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:02.863816+0000", 2024-09-15T03:47:06.307 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5209999999999999e-06 2024-09-15T03:47:06.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.307 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.307 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:06.307 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:02.865813+0000", 2024-09-15T03:47:06.307 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019971780000000001 2024-09-15T03:47:06.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.307 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.307 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:06.307 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:02.865874+0000", 2024-09-15T03:47:06.308 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1016e-05 2024-09-15T03:47:06.308 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.308 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.308 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:06.308 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:02.871236+0000", 2024-09-15T03:47:06.308 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0053623430000000003 2024-09-15T03:47:06.308 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.308 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:06.308 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.308 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.308 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.308 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 57/31 ECSubWrite(tid=90, reqid=client.4521.0:410, at_version=57'46, trim_to=0'0, roll_forward_to=56'44))", 2024-09-15T03:47:06.308 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:03.883223+0000", 2024-09-15T03:47:06.309 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 2.3185958680000001, 2024-09-15T03:47:06.309 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092188510000000001, 2024-09-15T03:47:06.309 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:06.309 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:06.309 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:06.309 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.309 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:06.309 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.883223+0000", 2024-09-15T03:47:06.309 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.309 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.309 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:06.309 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.883223+0000", 2024-09-15T03:47:06.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.310 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.310 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:06.310 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.883223+0000", 2024-09-15T03:47:06.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1600000000000001e-07 2024-09-15T03:47:06.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.310 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.310 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:06.310 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884196+0000", 2024-09-15T03:47:06.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00097307999999999997 2024-09-15T03:47:06.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.310 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:06.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884210+0000", 2024-09-15T03:47:06.311 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4459e-05 2024-09-15T03:47:06.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.311 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:06.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884217+0000", 2024-09-15T03:47:06.311 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2870000000000001e-06 2024-09-15T03:47:06.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.311 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:06.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884281+0000", 2024-09-15T03:47:06.311 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4079999999999996e-05 2024-09-15T03:47:06.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.312 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.312 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:06.312 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884293+0000", 2024-09-15T03:47:06.312 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2081e-05 2024-09-15T03:47:06.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.312 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.312 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:06.312 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884681+0000", 2024-09-15T03:47:06.312 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00038795600000000001 2024-09-15T03:47:06.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.312 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.312 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:06.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884695+0000", 2024-09-15T03:47:06.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4497e-05 2024-09-15T03:47:06.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.313 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.313 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:06.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884962+0000", 2024-09-15T03:47:06.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00026705199999999999 2024-09-15T03:47:06.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.313 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.313 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:06.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884982+0000", 2024-09-15T03:47:06.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9211999999999998e-05 2024-09-15T03:47:06.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.314 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.314 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:06.314 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.892442+0000", 2024-09-15T03:47:06.314 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0074600309999999998 2024-09-15T03:47:06.314 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.314 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:06.314 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.314 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.314 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 57/31 ECSubWrite(tid=91, reqid=client.4521.0:410, at_version=0'0, trim_to=0'0, roll_forward_to=57'46))", 2024-09-15T03:47:06.314 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:03.885488+0000", 2024-09-15T03:47:06.314 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 2.3163306280000002, 2024-09-15T03:47:06.314 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0072455319999999998, 2024-09-15T03:47:06.315 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:06.315 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:06.315 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:06.315 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.315 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:06.315 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.885488+0000", 2024-09-15T03:47:06.315 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.315 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.315 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:06.315 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.885488+0000", 2024-09-15T03:47:06.315 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:06.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.315 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.316 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:06.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.885489+0000", 2024-09-15T03:47:06.316 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3099999999999997e-07 2024-09-15T03:47:06.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.316 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.316 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:06.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.885502+0000", 2024-09-15T03:47:06.316 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3356999999999999e-05 2024-09-15T03:47:06.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.316 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.316 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:06.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.885512+0000", 2024-09-15T03:47:06.316 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6479999999999998e-06 2024-09-15T03:47:06.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.317 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.317 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:06.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.885517+0000", 2024-09-15T03:47:06.317 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9740000000000001e-06 2024-09-15T03:47:06.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.317 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.317 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:06.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.885556+0000", 2024-09-15T03:47:06.317 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9174000000000001e-05 2024-09-15T03:47:06.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.317 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.317 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:06.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.885564+0000", 2024-09-15T03:47:06.318 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2630000000000003e-06 2024-09-15T03:47:06.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.318 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.318 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:06.318 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.885647+0000", 2024-09-15T03:47:06.318 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2862999999999996e-05 2024-09-15T03:47:06.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.318 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.318 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:06.318 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.885656+0000", 2024-09-15T03:47:06.318 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.935e-06 2024-09-15T03:47:06.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.318 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.318 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:06.319 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.885689+0000", 2024-09-15T03:47:06.319 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2796999999999998e-05 2024-09-15T03:47:06.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.319 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.319 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:06.319 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.885703+0000", 2024-09-15T03:47:06.319 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4022000000000001e-05 2024-09-15T03:47:06.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:06.319 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:06.319 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:06.319 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.892733+0000", 2024-09-15T03:47:06.319 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.007030868 2024-09-15T03:47:06.319 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.320 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:06.320 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.320 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:06.320 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:06.320 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:06.321 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:47:06.321 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:47:06.322 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:06.322 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:47:06.323 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:06.324 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:06.320+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:06.422 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:06.421+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:06.461 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:06.461 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:06.461 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:06.461 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:06.469 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:47:06.523 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:06.522+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:06.565 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:06.565 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:06.565 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:06.566 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:06.566 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:06.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:47:06.624 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:06.623+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:06.671 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:06.671 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:06.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:06.671 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [] 2024-09-15T03:47:06.671 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:06.679 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:47:06.764 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:06.763+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:06.817 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:06.817 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:06.817 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:06.817 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:06.826 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:47:06.827 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:06.825+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:06.835 INFO:tasks.rados.rados.0.smithi001.stdout:163: done (3 left) 2024-09-15T03:47:06.835 INFO:tasks.rados.rados.0.smithi001.stdout:164: delete oid 37 current snap is 19 2024-09-15T03:47:06.835 INFO:tasks.rados.rados.0.smithi001.stdout:158: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-15T03:47:06.865 INFO:tasks.rados.rados.0.smithi001.stdout:157: finishing copy_from racing read to smithi00132172-25 2024-09-15T03:47:06.865 INFO:tasks.rados.rados.0.smithi001.stdout:157: finishing copy_from to smithi00132172-25 2024-09-15T03:47:06.865 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 25 v 38 (ObjNum 66 snap 6 seq_num 66) dirty exists 2024-09-15T03:47:06.865 INFO:tasks.rados.rados.0.smithi001.stdout:160: expect (ObjNum 64 snap 6 seq_num 64) 2024-09-15T03:47:06.874 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-15T03:47:06.874 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T03:47:06.874 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] 2024-09-15T03:47:06.875 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 6 2024-09-15T03:47:06.897 INFO:tasks.rados.rados.0.smithi001.stdout:157: done (3 left) 2024-09-15T03:47:06.897 INFO:tasks.rados.rados.0.smithi001.stdout:158: done (2 left) 2024-09-15T03:47:06.897 INFO:tasks.rados.rados.0.smithi001.stdout:160: done (1 left) 2024-09-15T03:47:06.897 INFO:tasks.rados.rados.0.smithi001.stdout:164: done (0 left) 2024-09-15T03:47:06.897 INFO:tasks.rados.rados.0.smithi001.stdout:165: setattr oid 50 current snap is 19 2024-09-15T03:47:06.898 INFO:tasks.rados.rados.0.smithi001.stdout:166: rollback oid 13 current snap is 19 2024-09-15T03:47:06.898 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 13 to 15 2024-09-15T03:47:06.899 INFO:tasks.rados.rados.0.smithi001.stdout:167: copy_from oid 30 from oid 3 current snap is 19 2024-09-15T03:47:06.899 INFO:tasks.rados.rados.0.smithi001.stdout:168: read oid 26 snap 0 2024-09-15T03:47:06.899 INFO:tasks.rados.rados.0.smithi001.stdout:168: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-15T03:47:06.899 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 38 current snap is 19 2024-09-15T03:47:06.899 INFO:tasks.rados.rados.0.smithi001.stdout:169: seq_num 87 ranges {1785856=704512,2490368=688128,3178496=81920} 2024-09-15T03:47:06.912 INFO:tasks.rados.rados.0.smithi001.stdout:169: writing smithi00132172-38 from 1785856 to 2490368 tid 1 2024-09-15T03:47:06.916 INFO:tasks.rados.rados.0.smithi001.stdout:169: writing smithi00132172-38 from 2490368 to 3178496 tid 2 2024-09-15T03:47:06.917 INFO:tasks.rados.rados.0.smithi001.stdout:169: writing smithi00132172-38 from 3178496 to 3260416 tid 3 2024-09-15T03:47:06.917 INFO:tasks.rados.rados.0.smithi001.stdout:167: finishing copy_from to smithi00132172-30 2024-09-15T03:47:06.918 INFO:tasks.rados.rados.0.smithi001.stdout:167: got expected ENOENT (src dne) 2024-09-15T03:47:06.918 INFO:tasks.rados.rados.0.smithi001.stdout:167: finishing copy_from racing read to smithi00132172-30 2024-09-15T03:47:06.918 INFO:tasks.rados.rados.0.smithi001.stdout:166: finishing rollback tid 0 to smithi00132172-13 2024-09-15T03:47:06.918 INFO:tasks.rados.rados.0.smithi001.stdout:166: finishing rollback tid 1 to smithi00132172-13 2024-09-15T03:47:06.918 INFO:tasks.rados.rados.0.smithi001.stdout:166: finishing rollback tid 2 to smithi00132172-13 2024-09-15T03:47:06.918 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 13 v 42 (ObjNum 68 snap 6 seq_num 68) dirty exists 2024-09-15T03:47:06.918 INFO:tasks.rados.rados.0.smithi001.stdout:168: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-15T03:47:06.926 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:06.925+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:06.929 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 50 v 41 (ObjNum 72 snap 12 seq_num 72) dirty exists 2024-09-15T03:47:06.929 INFO:tasks.rados.rados.0.smithi001.stdout:169: finishing write tid 1 to smithi00132172-38 2024-09-15T03:47:06.929 INFO:tasks.rados.rados.0.smithi001.stdout:169: finishing write tid 2 to smithi00132172-38 2024-09-15T03:47:06.929 INFO:tasks.rados.rados.0.smithi001.stdout:169: finishing write tid 3 to smithi00132172-38 2024-09-15T03:47:06.930 INFO:tasks.rados.rados.0.smithi001.stdout:169: finishing write tid 4 to smithi00132172-38 2024-09-15T03:47:06.930 INFO:tasks.rados.rados.0.smithi001.stdout:169: finishing write tid 5 to smithi00132172-38 2024-09-15T03:47:06.930 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 38 v 47 (ObjNum 87 snap 19 seq_num 87) dirty exists 2024-09-15T03:47:06.930 INFO:tasks.rados.rados.0.smithi001.stdout:169: left oid 38 (ObjNum 87 snap 19 seq_num 87) 2024-09-15T03:47:06.930 INFO:tasks.rados.rados.0.smithi001.stdout:165: done (4 left) 2024-09-15T03:47:06.930 INFO:tasks.rados.rados.0.smithi001.stdout:166: done (3 left) 2024-09-15T03:47:06.930 INFO:tasks.rados.rados.0.smithi001.stdout:167: done (2 left) 2024-09-15T03:47:06.930 INFO:tasks.rados.rados.0.smithi001.stdout:168: done (1 left) 2024-09-15T03:47:06.930 INFO:tasks.rados.rados.0.smithi001.stdout:169: done (0 left) 2024-09-15T03:47:06.930 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 20 current snap is 19 2024-09-15T03:47:06.930 INFO:tasks.rados.rados.0.smithi001.stdout:170: seq_num 88 ranges {327680=720896,1048576=442368,1490944=327680} 2024-09-15T03:47:06.935 INFO:tasks.rados.rados.0.smithi001.stdout:170: writing smithi00132172-20 from 327680 to 1048576 tid 1 2024-09-15T03:47:06.937 INFO:tasks.rados.rados.0.smithi001.stdout:170: writing smithi00132172-20 from 1048576 to 1490944 tid 2 2024-09-15T03:47:06.940 INFO:tasks.rados.rados.0.smithi001.stdout:170: writing smithi00132172-20 from 1490944 to 1818624 tid 3 2024-09-15T03:47:06.940 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 9 current snap is 19 2024-09-15T03:47:06.940 INFO:tasks.rados.rados.0.smithi001.stdout:171: seq_num 89 ranges {147456=737280,884736=573440,1458176=196608} 2024-09-15T03:47:06.946 INFO:tasks.rados.rados.0.smithi001.stdout:171: writing smithi00132172-9 from 147456 to 884736 tid 1 2024-09-15T03:47:06.949 INFO:tasks.rados.rados.0.smithi001.stdout:171: writing smithi00132172-9 from 884736 to 1458176 tid 2 2024-09-15T03:47:06.951 INFO:tasks.rados.rados.0.smithi001.stdout:171: writing smithi00132172-9 from 1458176 to 1654784 tid 3 2024-09-15T03:47:06.951 INFO:tasks.rados.rados.0.smithi001.stdout:172: copy_from oid 19 from oid 25 current snap is 19 2024-09-15T03:47:06.951 INFO:tasks.rados.rados.0.smithi001.stdout:170: finishing write tid 1 to smithi00132172-20 2024-09-15T03:47:06.951 INFO:tasks.rados.rados.0.smithi001.stdout:170: finishing write tid 2 to smithi00132172-20 2024-09-15T03:47:06.952 INFO:tasks.rados.rados.0.smithi001.stdout:170: finishing write tid 3 to smithi00132172-20 2024-09-15T03:47:06.952 INFO:tasks.rados.rados.0.smithi001.stdout:170: finishing write tid 4 to smithi00132172-20 2024-09-15T03:47:06.952 INFO:tasks.rados.rados.0.smithi001.stdout:170: finishing write tid 5 to smithi00132172-20 2024-09-15T03:47:06.952 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 20 v 15 (ObjNum 88 snap 19 seq_num 88) dirty exists 2024-09-15T03:47:06.952 INFO:tasks.rados.rados.0.smithi001.stdout:170: left oid 20 (ObjNum 88 snap 19 seq_num 88) 2024-09-15T03:47:06.952 INFO:tasks.rados.rados.0.smithi001.stdout:171: finishing write tid 1 to smithi00132172-9 2024-09-15T03:47:06.952 INFO:tasks.rados.rados.0.smithi001.stdout:170: done (2 left) 2024-09-15T03:47:06.952 INFO:tasks.rados.rados.0.smithi001.stdout:173: snap_remove snap 18 2024-09-15T03:47:06.973 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:06.973 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:06.973 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:06.973 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:06.973 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:06.982 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:47:07.027 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:07.027+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:07.080 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:07.080 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:07.080 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:07.080 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:47:07.080 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.080 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:185 3.as0 3:5515acd6:::smithi00132172-13:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:47:07.080 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:47:07.080 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 26.972871706999999, 2024-09-15T03:47:07.080 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01268643, 2024-09-15T03:47:07.080 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:07.081 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:07.081 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:07.081 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:07.081 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:07.081 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 185 2024-09-15T03:47:07.081 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.081 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:07.081 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.081 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:07.081 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:47:07.081 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.082 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.082 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.082 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:07.082 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:47:07.082 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.082 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.082 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.082 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:07.082 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104710+0000", 2024-09-15T03:47:07.082 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0200000000000003e-07 2024-09-15T03:47:07.082 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.082 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.082 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:07.082 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104713+0000", 2024-09-15T03:47:07.083 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2339999999999999e-06 2024-09-15T03:47:07.083 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.083 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.083 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:07.083 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104718+0000", 2024-09-15T03:47:07.083 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3449999999999996e-06 2024-09-15T03:47:07.083 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.083 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.083 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.083 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104722+0000", 2024-09-15T03:47:07.083 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8399999999999997e-06 2024-09-15T03:47:07.083 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.084 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.084 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104849+0000", 2024-09-15T03:47:07.084 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012709599999999999 2024-09-15T03:47:07.084 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.084 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:07.084 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104856+0000", 2024-09-15T03:47:07.084 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.652e-06 2024-09-15T03:47:07.084 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.084 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.084 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106851+0000", 2024-09-15T03:47:07.085 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019954840000000001 2024-09-15T03:47:07.085 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.085 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.085 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:07.085 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106935+0000", 2024-09-15T03:47:07.085 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4095e-05 2024-09-15T03:47:07.085 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.085 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.085 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.085 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107453+0000", 2024-09-15T03:47:07.085 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00051729499999999997 2024-09-15T03:47:07.085 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.085 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.086 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.086 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107554+0000", 2024-09-15T03:47:07.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000101251 2024-09-15T03:47:07.086 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.086 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.086 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.086 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108104+0000", 2024-09-15T03:47:07.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00054995100000000002 2024-09-15T03:47:07.086 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.086 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.086 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.086 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108503+0000", 2024-09-15T03:47:07.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039953500000000002 2024-09-15T03:47:07.087 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.087 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.087 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.087 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109405+0000", 2024-09-15T03:47:07.087 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00090170099999999998 2024-09-15T03:47:07.087 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.087 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.087 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:07.087 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114365+0000", 2024-09-15T03:47:07.087 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049602120000000003 2024-09-15T03:47:07.087 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.087 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.087 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:07.087 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117396+0000", 2024-09-15T03:47:07.088 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030303370000000001 2024-09-15T03:47:07.088 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.088 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:07.088 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.088 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.088 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.088 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:186 3.as0 3:5515acd6:::smithi00132172-13:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:47:07.088 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:47:07.088 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 26.972831917000001, 2024-09-15T03:47:07.088 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012817485999999999, 2024-09-15T03:47:07.088 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:07.088 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:07.089 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:07.089 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:07.089 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:07.089 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 186 2024-09-15T03:47:07.089 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.089 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:07.089 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.089 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:07.089 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:47:07.089 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.089 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.089 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.089 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:07.090 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:47:07.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.090 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.090 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:07.090 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104750+0000", 2024-09-15T03:47:07.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1500000000000004e-07 2024-09-15T03:47:07.090 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.090 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:07.090 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104754+0000", 2024-09-15T03:47:07.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.946e-06 2024-09-15T03:47:07.090 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.091 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.091 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:07.091 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104769+0000", 2024-09-15T03:47:07.091 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.507e-05 2024-09-15T03:47:07.091 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.091 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.091 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.091 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104776+0000", 2024-09-15T03:47:07.091 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7849999999999992e-06 2024-09-15T03:47:07.091 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.091 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.091 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.091 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104888+0000", 2024-09-15T03:47:07.091 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011164500000000001 2024-09-15T03:47:07.092 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.092 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:07.092 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104894+0000", 2024-09-15T03:47:07.092 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.942e-06 2024-09-15T03:47:07.092 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.092 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.092 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108172+0000", 2024-09-15T03:47:07.092 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032781939999999999 2024-09-15T03:47:07.092 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:07.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108257+0000", 2024-09-15T03:47:07.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4488999999999997e-05 2024-09-15T03:47:07.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108732+0000", 2024-09-15T03:47:07.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047547399999999998 2024-09-15T03:47:07.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108823+0000", 2024-09-15T03:47:07.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1106000000000005e-05 2024-09-15T03:47:07.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.094 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108961+0000", 2024-09-15T03:47:07.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013735099999999999 2024-09-15T03:47:07.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.094 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109065+0000", 2024-09-15T03:47:07.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104715 2024-09-15T03:47:07.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.094 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109406+0000", 2024-09-15T03:47:07.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034062300000000002 2024-09-15T03:47:07.095 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.095 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.095 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:07.095 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114529+0000", 2024-09-15T03:47:07.095 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.005123219 2024-09-15T03:47:07.095 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.095 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.095 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:07.095 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117566+0000", 2024-09-15T03:47:07.095 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030372120000000001 2024-09-15T03:47:07.095 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.095 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:07.095 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.095 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.096 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:188 3.fs0 3:fe9c5513:::smithi00132172-47:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:47:07.096 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:47:07.096 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 26.972749220000001, 2024-09-15T03:47:07.096 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012637061, 2024-09-15T03:47:07.096 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:07.096 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:07.096 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:07.096 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:07.096 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:07.096 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 188 2024-09-15T03:47:07.096 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.097 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:07.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.097 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:07.097 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:47:07.097 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.097 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:07.097 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:47:07.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.098 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.098 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.098 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:07.098 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:47:07.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2799999999999996e-07 2024-09-15T03:47:07.098 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.098 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.098 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:07.098 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104835+0000", 2024-09-15T03:47:07.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7190000000000001e-06 2024-09-15T03:47:07.098 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.098 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:07.099 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104842+0000", 2024-09-15T03:47:07.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4580000000000004e-06 2024-09-15T03:47:07.099 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.099 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104846+0000", 2024-09-15T03:47:07.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1250000000000003e-06 2024-09-15T03:47:07.099 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.099 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104884+0000", 2024-09-15T03:47:07.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7075e-05 2024-09-15T03:47:07.099 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.100 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:07.100 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104888+0000", 2024-09-15T03:47:07.100 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8520000000000003e-06 2024-09-15T03:47:07.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.100 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.100 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106860+0000", 2024-09-15T03:47:07.100 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019719440000000002 2024-09-15T03:47:07.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.100 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:07.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106938+0000", 2024-09-15T03:47:07.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7421000000000003e-05 2024-09-15T03:47:07.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.101 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107368+0000", 2024-09-15T03:47:07.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043042999999999998 2024-09-15T03:47:07.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.101 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107460+0000", 2024-09-15T03:47:07.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1392999999999997e-05 2024-09-15T03:47:07.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107473+0000", 2024-09-15T03:47:07.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3702e-05 2024-09-15T03:47:07.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107504+0000", 2024-09-15T03:47:07.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1000000000000001e-05 2024-09-15T03:47:07.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.103 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.103 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.103 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108376+0000", 2024-09-15T03:47:07.103 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00087162000000000003 2024-09-15T03:47:07.103 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.103 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.103 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:07.103 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.111657+0000", 2024-09-15T03:47:07.103 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032815589999999999 2024-09-15T03:47:07.103 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.103 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.103 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:07.103 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117469+0000", 2024-09-15T03:47:07.104 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058112349999999997 2024-09-15T03:47:07.104 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.104 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:07.104 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.104 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.104 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:189 3.fs0 3:fe9c5513:::smithi00132172-47:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:47:07.104 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:47:07.104 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 26.972730034000001, 2024-09-15T03:47:07.104 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012731009999999999, 2024-09-15T03:47:07.104 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:07.104 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:07.105 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:07.105 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:07.105 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:07.105 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 189 2024-09-15T03:47:07.105 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.105 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:07.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:07.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:47:07.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.105 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:07.106 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:47:07.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.106 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.106 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.106 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:07.106 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:47:07.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7899999999999999e-07 2024-09-15T03:47:07.106 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.106 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.106 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:07.106 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104853+0000", 2024-09-15T03:47:07.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.933e-06 2024-09-15T03:47:07.106 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:07.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104859+0000", 2024-09-15T03:47:07.107 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.592e-06 2024-09-15T03:47:07.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104863+0000", 2024-09-15T03:47:07.107 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4000000000000002e-06 2024-09-15T03:47:07.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.108 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104910+0000", 2024-09-15T03:47:07.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6678000000000003e-05 2024-09-15T03:47:07.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.108 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:07.108 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104916+0000", 2024-09-15T03:47:07.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1079999999999998e-06 2024-09-15T03:47:07.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.108 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.108 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107550+0000", 2024-09-15T03:47:07.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026344110000000001 2024-09-15T03:47:07.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.109 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.109 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:07.109 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107624+0000", 2024-09-15T03:47:07.109 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4134000000000005e-05 2024-09-15T03:47:07.109 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.109 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.109 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.109 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107988+0000", 2024-09-15T03:47:07.109 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036353600000000002 2024-09-15T03:47:07.109 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.109 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.109 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.110 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108066+0000", 2024-09-15T03:47:07.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8047999999999995e-05 2024-09-15T03:47:07.110 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.110 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.110 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108105+0000", 2024-09-15T03:47:07.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8868000000000001e-05 2024-09-15T03:47:07.110 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.110 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.110 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108504+0000", 2024-09-15T03:47:07.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039916600000000001 2024-09-15T03:47:07.110 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.111 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108955+0000", 2024-09-15T03:47:07.111 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045126200000000002 2024-09-15T03:47:07.111 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:07.111 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114394+0000", 2024-09-15T03:47:07.111 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0054382750000000002 2024-09-15T03:47:07.111 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:07.111 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117582+0000", 2024-09-15T03:47:07.112 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031882199999999999 2024-09-15T03:47:07.112 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.112 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:07.112 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.112 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.112 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.112 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 38/31 ECSubWrite(tid=36, reqid=client.4521.0:220, at_version=38'23, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:47:07.112 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.148035+0000", 2024-09-15T03:47:07.112 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.929545928, 2024-09-15T03:47:07.112 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010074534, 2024-09-15T03:47:07.112 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:07.112 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.112 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:07.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:07.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148035+0000", 2024-09-15T03:47:07.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:07.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148035+0000", 2024-09-15T03:47:07.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:07.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148036+0000", 2024-09-15T03:47:07.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3939999999999999e-06 2024-09-15T03:47:07.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:07.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148681+0000", 2024-09-15T03:47:07.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00064493899999999997 2024-09-15T03:47:07.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:07.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148695+0000", 2024-09-15T03:47:07.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4078e-05 2024-09-15T03:47:07.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148701+0000", 2024-09-15T03:47:07.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7230000000000001e-06 2024-09-15T03:47:07.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148742+0000", 2024-09-15T03:47:07.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0781000000000001e-05 2024-09-15T03:47:07.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148749+0000", 2024-09-15T03:47:07.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6109999999999996e-06 2024-09-15T03:47:07.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.116 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.116 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149077+0000", 2024-09-15T03:47:07.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000327597 2024-09-15T03:47:07.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.116 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.116 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149087+0000", 2024-09-15T03:47:07.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4739999999999994e-06 2024-09-15T03:47:07.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.117 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149230+0000", 2024-09-15T03:47:07.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014398300000000001 2024-09-15T03:47:07.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.117 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149247+0000", 2024-09-15T03:47:07.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6201e-05 2024-09-15T03:47:07.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.117 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:07.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158109+0000", 2024-09-15T03:47:07.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0088627529999999993 2024-09-15T03:47:07.118 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.118 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:07.118 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.118 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.118 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.118 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:231 3.fs0 3:f40d4749:::smithi00132172-35:head [copy-from ver 6] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:07.118 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.155669+0000", 2024-09-15T03:47:07.118 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.921912228, 2024-09-15T03:47:07.118 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010370569, 2024-09-15T03:47:07.118 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:07.118 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:07.118 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:07.118 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:07.119 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:07.119 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 231 2024-09-15T03:47:07.119 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.119 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:07.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.119 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:07.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155669+0000", 2024-09-15T03:47:07.119 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.119 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.119 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:07.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155669+0000", 2024-09-15T03:47:07.119 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.120 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.120 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.120 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:07.120 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155671+0000", 2024-09-15T03:47:07.120 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0829999999999998e-06 2024-09-15T03:47:07.120 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.120 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.120 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:07.120 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155676+0000", 2024-09-15T03:47:07.120 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6210000000000001e-06 2024-09-15T03:47:07.120 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.120 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.121 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:07.121 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155690+0000", 2024-09-15T03:47:07.121 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4166e-05 2024-09-15T03:47:07.121 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.121 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.121 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.121 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155698+0000", 2024-09-15T03:47:07.121 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5560000000000002e-06 2024-09-15T03:47:07.121 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.121 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.121 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.121 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155748+0000", 2024-09-15T03:47:07.121 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9851000000000001e-05 2024-09-15T03:47:07.121 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.122 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.122 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:07.122 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155872+0000", 2024-09-15T03:47:07.122 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000124091 2024-09-15T03:47:07.122 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.122 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.122 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.122 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163470+0000", 2024-09-15T03:47:07.122 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075976730000000001 2024-09-15T03:47:07.122 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.122 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.122 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.122 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163801+0000", 2024-09-15T03:47:07.123 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033157999999999999 2024-09-15T03:47:07.123 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.123 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.123 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.123 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163814+0000", 2024-09-15T03:47:07.123 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2659e-05 2024-09-15T03:47:07.123 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.123 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.123 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.123 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163935+0000", 2024-09-15T03:47:07.123 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000121155 2024-09-15T03:47:07.123 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.123 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.124 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.124 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.164131+0000", 2024-09-15T03:47:07.124 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019600499999999999 2024-09-15T03:47:07.124 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.124 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.124 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:07.124 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.165998+0000", 2024-09-15T03:47:07.124 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001866882 2024-09-15T03:47:07.124 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.124 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.124 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:07.124 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.166039+0000", 2024-09-15T03:47:07.124 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1247e-05 2024-09-15T03:47:07.125 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.125 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:07.125 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.125 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.125 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.125 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:232 3.fs0 3:f40d4749:::smithi00132172-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:07.125 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:47:07.125 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.921874938999999, 2024-09-15T03:47:07.125 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084205730000000006, 2024-09-15T03:47:07.125 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:07.125 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:07.125 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:07.125 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:07.126 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:07.126 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 232 2024-09-15T03:47:07.126 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.126 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:07.126 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.126 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:07.126 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:47:07.126 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.126 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.126 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.126 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:07.126 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:47:07.126 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.127 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.127 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.127 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:07.127 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:47:07.127 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7399999999999999e-07 2024-09-15T03:47:07.127 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.127 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.127 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:07.127 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155708+0000", 2024-09-15T03:47:07.127 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6500000000000001e-06 2024-09-15T03:47:07.127 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.127 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.127 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:07.128 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155713+0000", 2024-09-15T03:47:07.128 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2519999999999999e-06 2024-09-15T03:47:07.128 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.128 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.128 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.128 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155716+0000", 2024-09-15T03:47:07.128 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4430000000000002e-06 2024-09-15T03:47:07.128 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.128 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.128 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.128 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156060+0000", 2024-09-15T03:47:07.128 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034440000000000002 2024-09-15T03:47:07.129 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.129 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.129 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:07.129 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156112+0000", 2024-09-15T03:47:07.129 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1928000000000002e-05 2024-09-15T03:47:07.129 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.129 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.129 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.129 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163929+0000", 2024-09-15T03:47:07.129 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0078168999999999999 2024-09-15T03:47:07.129 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.129 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.129 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:07.130 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.164027+0000", 2024-09-15T03:47:07.130 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8109999999999994e-05 2024-09-15T03:47:07.130 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.130 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.130 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:07.130 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.164126+0000", 2024-09-15T03:47:07.130 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9516000000000001e-05 2024-09-15T03:47:07.130 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.130 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:07.130 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.130 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.130 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.130 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 39/31 ECSubWrite(tid=43, reqid=client.4521.0:252, at_version=39'27, trim_to=0'0, roll_forward_to=39'25))", 2024-09-15T03:47:07.131 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:47:07.131 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 20.899235246, 2024-09-15T03:47:07.131 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082900309999999998, 2024-09-15T03:47:07.131 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:07.131 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.131 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:07.131 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.131 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:07.131 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:47:07.131 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.131 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.131 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.131 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:07.131 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:47:07.132 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.132 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:07.132 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:47:07.132 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6199999999999998e-07 2024-09-15T03:47:07.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.132 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:07.132 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178642+0000", 2024-09-15T03:47:07.132 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029559399999999999 2024-09-15T03:47:07.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.133 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.133 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:07.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178653+0000", 2024-09-15T03:47:07.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0865e-05 2024-09-15T03:47:07.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.133 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.133 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178658+0000", 2024-09-15T03:47:07.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3720000000000004e-06 2024-09-15T03:47:07.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.133 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.133 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178695+0000", 2024-09-15T03:47:07.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6708999999999997e-05 2024-09-15T03:47:07.134 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178701+0000", 2024-09-15T03:47:07.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7989999999999996e-06 2024-09-15T03:47:07.134 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178926+0000", 2024-09-15T03:47:07.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022405299999999999 2024-09-15T03:47:07.134 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.135 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178935+0000", 2024-09-15T03:47:07.135 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9649999999999997e-06 2024-09-15T03:47:07.135 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.135 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.135 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179230+0000", 2024-09-15T03:47:07.135 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029551000000000002 2024-09-15T03:47:07.135 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.135 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.135 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179247+0000", 2024-09-15T03:47:07.135 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6685999999999999e-05 2024-09-15T03:47:07.135 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.136 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:07.136 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.186636+0000", 2024-09-15T03:47:07.136 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0073889159999999997 2024-09-15T03:47:07.136 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.136 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:07.136 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.136 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.136 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.136 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 49/31 ECSubWrite(tid=87, reqid=client.4521.0:335, at_version=49'21, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:07.136 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.858534+0000", 2024-09-15T03:47:07.136 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.21904711, 2024-09-15T03:47:07.136 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094628430000000003, 2024-09-15T03:47:07.136 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:07.136 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858534+0000", 2024-09-15T03:47:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858534+0000", 2024-09-15T03:47:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858534+0000", 2024-09-15T03:47:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2300000000000001e-07 2024-09-15T03:47:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858553+0000", 2024-09-15T03:47:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8221000000000001e-05 2024-09-15T03:47:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858564+0000", 2024-09-15T03:47:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1471000000000001e-05 2024-09-15T03:47:07.138 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858570+0000", 2024-09-15T03:47:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7640000000000002e-06 2024-09-15T03:47:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858621+0000", 2024-09-15T03:47:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0943999999999999e-05 2024-09-15T03:47:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858626+0000", 2024-09-15T03:47:07.139 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4949999999999999e-06 2024-09-15T03:47:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858715+0000", 2024-09-15T03:47:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9061000000000007e-05 2024-09-15T03:47:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858722+0000", 2024-09-15T03:47:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9689999999999997e-06 2024-09-15T03:47:07.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858748+0000", 2024-09-15T03:47:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6179000000000001e-05 2024-09-15T03:47:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858761+0000", 2024-09-15T03:47:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2159e-05 2024-09-15T03:47:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.141 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.867997+0000", 2024-09-15T03:47:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092359569999999995 2024-09-15T03:47:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 49/31 ECSubWrite(tid=88, reqid=client.4521.0:336, at_version=49'22, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:47:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.218319468000001, 2024-09-15T03:47:07.142 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0088255839999999992, 2024-09-15T03:47:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:47:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:47:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859262+0000", 2024-09-15T03:47:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2399999999999999e-07 2024-09-15T03:47:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859275+0000", 2024-09-15T03:47:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3395e-05 2024-09-15T03:47:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.144 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859285+0000", 2024-09-15T03:47:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8959999999999998e-06 2024-09-15T03:47:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859289+0000", 2024-09-15T03:47:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2549999999999999e-06 2024-09-15T03:47:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.145 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859327+0000", 2024-09-15T03:47:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7753000000000003e-05 2024-09-15T03:47:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859335+0000", 2024-09-15T03:47:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3000000000000002e-06 2024-09-15T03:47:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859391+0000", 2024-09-15T03:47:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6085000000000003e-05 2024-09-15T03:47:07.146 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859415+0000", 2024-09-15T03:47:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3295999999999999e-05 2024-09-15T03:47:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859432+0000", 2024-09-15T03:47:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7087000000000001e-05 2024-09-15T03:47:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.147 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859442+0000", 2024-09-15T03:47:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0339e-05 2024-09-15T03:47:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.868087+0000", 2024-09-15T03:47:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0086447539999999993 2024-09-15T03:47:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s6 51/31 ECSubWrite(tid=90, reqid=client.4521.0:346, at_version=51'12, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:07.148 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.842187+0000", 2024-09-15T03:47:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.2353934090000003, 2024-09-15T03:47:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082230800000000007, 2024-09-15T03:47:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842187+0000", 2024-09-15T03:47:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.149 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842187+0000", 2024-09-15T03:47:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842188+0000", 2024-09-15T03:47:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2899999999999999e-07 2024-09-15T03:47:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842218+0000", 2024-09-15T03:47:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0216999999999999e-05 2024-09-15T03:47:07.150 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842233+0000", 2024-09-15T03:47:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5444999999999999e-05 2024-09-15T03:47:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842242+0000", 2024-09-15T03:47:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6090000000000001e-06 2024-09-15T03:47:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842310+0000", 2024-09-15T03:47:07.151 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7986999999999998e-05 2024-09-15T03:47:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842322+0000", 2024-09-15T03:47:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2160999999999999e-05 2024-09-15T03:47:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842617+0000", 2024-09-15T03:47:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029521700000000002 2024-09-15T03:47:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.152 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842640+0000", 2024-09-15T03:47:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2433999999999999e-05 2024-09-15T03:47:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842682+0000", 2024-09-15T03:47:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1826000000000002e-05 2024-09-15T03:47:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842842+0000", 2024-09-15T03:47:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016048500000000001 2024-09-15T03:47:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.153 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850411+0000", 2024-09-15T03:47:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075683699999999996 2024-09-15T03:47:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s6 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.846035+0000", 2024-09-15T03:47:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.2315462410000002, 2024-09-15T03:47:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010491004, 2024-09-15T03:47:07.154 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846035+0000", 2024-09-15T03:47:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846035+0000", 2024-09-15T03:47:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.155 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:07.156 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846035+0000", 2024-09-15T03:47:07.156 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8599999999999997e-07 2024-09-15T03:47:07.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.156 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:07.156 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846068+0000", 2024-09-15T03:47:07.156 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2138000000000003e-05 2024-09-15T03:47:07.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.156 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:07.156 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846081+0000", 2024-09-15T03:47:07.156 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3127e-05 2024-09-15T03:47:07.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846092+0000", 2024-09-15T03:47:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1634000000000001e-05 2024-09-15T03:47:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846136+0000", 2024-09-15T03:47:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3999999999999999e-05 2024-09-15T03:47:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846148+0000", 2024-09-15T03:47:07.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1939e-05 2024-09-15T03:47:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846220+0000", 2024-09-15T03:47:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1248999999999995e-05 2024-09-15T03:47:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846231+0000", 2024-09-15T03:47:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1379999999999999e-05 2024-09-15T03:47:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.158 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847412+0000", 2024-09-15T03:47:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001181407 2024-09-15T03:47:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847435+0000", 2024-09-15T03:47:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3084000000000001e-05 2024-09-15T03:47:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856526+0000", 2024-09-15T03:47:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0090901599999999999 2024-09-15T03:47:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.159 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s6 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.846101+0000", 2024-09-15T03:47:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.2314794330000005, 2024-09-15T03:47:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010577013999999999, 2024-09-15T03:47:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:07.160 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846101+0000", 2024-09-15T03:47:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846101+0000", 2024-09-15T03:47:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846102+0000", 2024-09-15T03:47:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5100000000000001e-07 2024-09-15T03:47:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.161 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846110+0000", 2024-09-15T03:47:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7069999999999998e-06 2024-09-15T03:47:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846115+0000", 2024-09-15T03:47:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2799999999999997e-06 2024-09-15T03:47:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846117+0000", 2024-09-15T03:47:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5909999999999998e-06 2024-09-15T03:47:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.162 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846259+0000", 2024-09-15T03:47:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014169900000000001 2024-09-15T03:47:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846268+0000", 2024-09-15T03:47:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5159999999999994e-06 2024-09-15T03:47:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846335+0000", 2024-09-15T03:47:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7936999999999997e-05 2024-09-15T03:47:07.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847415+0000", 2024-09-15T03:47:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001079874 2024-09-15T03:47:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847453+0000", 2024-09-15T03:47:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7808999999999997e-05 2024-09-15T03:47:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.164 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847477+0000", 2024-09-15T03:47:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3609e-05 2024-09-15T03:47:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856678+0000", 2024-09-15T03:47:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.009201641 2024-09-15T03:47:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 51/31 ECSubWrite(tid=93, reqid=client.4521.0:352, at_version=51'24, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.847310+0000", 2024-09-15T03:47:07.165 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.230271127, 2024-09-15T03:47:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012670010000000001, 2024-09-15T03:47:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847310+0000", 2024-09-15T03:47:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847310+0000", 2024-09-15T03:47:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847310+0000", 2024-09-15T03:47:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8899999999999999e-07 2024-09-15T03:47:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847750+0000", 2024-09-15T03:47:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000439238 2024-09-15T03:47:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:07.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847767+0000", 2024-09-15T03:47:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7362000000000001e-05 2024-09-15T03:47:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847774+0000", 2024-09-15T03:47:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0169999999999999e-06 2024-09-15T03:47:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847831+0000", 2024-09-15T03:47:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7207999999999998e-05 2024-09-15T03:47:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847841+0000", 2024-09-15T03:47:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7829999999999993e-06 2024-09-15T03:47:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.848145+0000", 2024-09-15T03:47:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030431300000000001 2024-09-15T03:47:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.848179+0000", 2024-09-15T03:47:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3423999999999997e-05 2024-09-15T03:47:07.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850141+0000", 2024-09-15T03:47:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019623380000000001 2024-09-15T03:47:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850164+0000", 2024-09-15T03:47:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2969e-05 2024-09-15T03:47:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:07.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.859980+0000", 2024-09-15T03:47:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098156690000000008 2024-09-15T03:47:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 51/31 ECSubWrite(tid=94, reqid=client.4521.0:353, at_version=51'25, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.850957+0000", 2024-09-15T03:47:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.2266233379999996, 2024-09-15T03:47:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092043439999999997, 2024-09-15T03:47:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:07.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850957+0000", 2024-09-15T03:47:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850957+0000", 2024-09-15T03:47:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850958+0000", 2024-09-15T03:47:07.172 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9500000000000002e-07 2024-09-15T03:47:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851297+0000", 2024-09-15T03:47:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033833300000000001 2024-09-15T03:47:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851311+0000", 2024-09-15T03:47:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4144999999999999e-05 2024-09-15T03:47:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.173 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851318+0000", 2024-09-15T03:47:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8700000000000003e-06 2024-09-15T03:47:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851390+0000", 2024-09-15T03:47:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.203e-05 2024-09-15T03:47:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851402+0000", 2024-09-15T03:47:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2146e-05 2024-09-15T03:47:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.174 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851556+0000", 2024-09-15T03:47:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015461199999999999 2024-09-15T03:47:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851570+0000", 2024-09-15T03:47:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4106e-05 2024-09-15T03:47:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853231+0000", 2024-09-15T03:47:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001660559 2024-09-15T03:47:07.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853246+0000", 2024-09-15T03:47:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5457000000000001e-05 2024-09-15T03:47:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.860162+0000", 2024-09-15T03:47:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069153909999999999 2024-09-15T03:47:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 51/31 ECSubWrite(tid=95, reqid=client.4521.0:354, at_version=51'26, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.851558+0000", 2024-09-15T03:47:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.2260232000000002, 2024-09-15T03:47:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0087097059999999994, 2024-09-15T03:47:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851558+0000", 2024-09-15T03:47:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851558+0000", 2024-09-15T03:47:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851559+0000", 2024-09-15T03:47:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7599999999999996e-07 2024-09-15T03:47:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851599+0000", 2024-09-15T03:47:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0472000000000001e-05 2024-09-15T03:47:07.178 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.179 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.179 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:07.179 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851614+0000", 2024-09-15T03:47:07.179 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4640999999999999e-05 2024-09-15T03:47:07.179 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.179 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.179 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.179 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851620+0000", 2024-09-15T03:47:07.179 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3430000000000001e-06 2024-09-15T03:47:07.179 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.179 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.179 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.179 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851680+0000", 2024-09-15T03:47:07.179 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9648000000000002e-05 2024-09-15T03:47:07.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.180 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.180 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851691+0000", 2024-09-15T03:47:07.180 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1012999999999999e-05 2024-09-15T03:47:07.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.180 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.180 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851803+0000", 2024-09-15T03:47:07.180 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112089 2024-09-15T03:47:07.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.180 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.180 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.181 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853234+0000", 2024-09-15T03:47:07.181 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0014310810000000001 2024-09-15T03:47:07.181 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.181 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.181 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853304+0000", 2024-09-15T03:47:07.181 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0656999999999998e-05 2024-09-15T03:47:07.181 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.181 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.181 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853317+0000", 2024-09-15T03:47:07.181 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2186e-05 2024-09-15T03:47:07.181 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.182 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:07.182 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.860267+0000", 2024-09-15T03:47:07.182 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069506999999999998 2024-09-15T03:47:07.182 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.182 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:07.182 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.182 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.182 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.182 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:424 3.as0 3:5515acd6:::smithi00132172-13:head [rollback 17] snapc 1a=[1a,18,17,16,15,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:47:07.182 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.898858+0000", 2024-09-15T03:47:07.182 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.17872299899999999, 2024-09-15T03:47:07.182 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013029552999999999, 2024-09-15T03:47:07.182 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:07.183 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:07.183 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:07.183 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:07.183 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:07.183 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 424 2024-09-15T03:47:07.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.183 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:07.183 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.183 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:07.183 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898858+0000", 2024-09-15T03:47:07.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.183 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.183 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:07.184 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898858+0000", 2024-09-15T03:47:07.184 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.184 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:07.184 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898858+0000", 2024-09-15T03:47:07.184 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2300000000000002e-07 2024-09-15T03:47:07.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.184 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:07.184 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898860+0000", 2024-09-15T03:47:07.184 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.623e-06 2024-09-15T03:47:07.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.185 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.185 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:07.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898863+0000", 2024-09-15T03:47:07.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5870000000000001e-06 2024-09-15T03:47:07.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.185 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.185 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898865+0000", 2024-09-15T03:47:07.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.075e-06 2024-09-15T03:47:07.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.185 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.185 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.899786+0000", 2024-09-15T03:47:07.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00092033499999999999 2024-09-15T03:47:07.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.186 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.186 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:07.186 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.899886+0000", 2024-09-15T03:47:07.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000100516 2024-09-15T03:47:07.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.186 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.186 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.186 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900376+0000", 2024-09-15T03:47:07.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00048932700000000001 2024-09-15T03:47:07.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.186 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.186 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900486+0000", 2024-09-15T03:47:07.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011024 2024-09-15T03:47:07.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900492+0000", 2024-09-15T03:47:07.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.908e-06 2024-09-15T03:47:07.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900724+0000", 2024-09-15T03:47:07.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000232382 2024-09-15T03:47:07.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901279+0000", 2024-09-15T03:47:07.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00055503599999999996 2024-09-15T03:47:07.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:07.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.902348+0000", 2024-09-15T03:47:07.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010692130000000001 2024-09-15T03:47:07.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:07.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.911887+0000", 2024-09-15T03:47:07.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.009538988 2024-09-15T03:47:07.189 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.189 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:07.189 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.189 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.189 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.189 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:425 3.as0 3:5515acd6:::smithi00132172-13:head [append 0~0] snapc 1a=[1a,18,17,16,15,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:47:07.189 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.898884+0000", 2024-09-15T03:47:07.189 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.17869660900000001, 2024-09-15T03:47:07.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013107466999999999, 2024-09-15T03:47:07.189 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:07.189 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:07.189 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:07.190 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:07.190 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:07.190 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 425 2024-09-15T03:47:07.190 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.190 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:07.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.190 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:07.190 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898884+0000", 2024-09-15T03:47:07.190 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.190 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.190 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:07.190 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898884+0000", 2024-09-15T03:47:07.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:07.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898884+0000", 2024-09-15T03:47:07.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4499999999999998e-07 2024-09-15T03:47:07.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:07.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898886+0000", 2024-09-15T03:47:07.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0140000000000001e-06 2024-09-15T03:47:07.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:07.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898890+0000", 2024-09-15T03:47:07.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9179999999999996e-06 2024-09-15T03:47:07.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898893+0000", 2024-09-15T03:47:07.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2900000000000001e-06 2024-09-15T03:47:07.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900560+0000", 2024-09-15T03:47:07.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00166749 2024-09-15T03:47:07.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:07.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900717+0000", 2024-09-15T03:47:07.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000157174 2024-09-15T03:47:07.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901127+0000", 2024-09-15T03:47:07.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00040965100000000003 2024-09-15T03:47:07.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901239+0000", 2024-09-15T03:47:07.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112324 2024-09-15T03:47:07.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.194 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901246+0000", 2024-09-15T03:47:07.194 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8730000000000001e-06 2024-09-15T03:47:07.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.194 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903551+0000", 2024-09-15T03:47:07.194 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023044459999999999 2024-09-15T03:47:07.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.194 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903565+0000", 2024-09-15T03:47:07.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4066999999999999e-05 2024-09-15T03:47:07.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.195 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.195 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:07.195 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903748+0000", 2024-09-15T03:47:07.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018303899999999999 2024-09-15T03:47:07.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.195 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.195 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:07.195 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.911992+0000", 2024-09-15T03:47:07.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082439360000000003 2024-09-15T03:47:07.195 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.195 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:07.196 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.196 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 60/31 ECSubWrite(tid=129, reqid=client.4521.0:422, at_version=60'41, trim_to=0'0, roll_forward_to=60'39))", 2024-09-15T03:47:07.196 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.901153+0000", 2024-09-15T03:47:07.196 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.17642738299999999, 2024-09-15T03:47:07.196 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017516662999999998, 2024-09-15T03:47:07.196 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:07.196 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.196 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:07.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.196 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:07.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901153+0000", 2024-09-15T03:47:07.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:07.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901153+0000", 2024-09-15T03:47:07.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:07.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901154+0000", 2024-09-15T03:47:07.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3000000000000001e-07 2024-09-15T03:47:07.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:07.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901593+0000", 2024-09-15T03:47:07.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043868200000000003 2024-09-15T03:47:07.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:07.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901605+0000", 2024-09-15T03:47:07.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1987000000000001e-05 2024-09-15T03:47:07.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.199 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901611+0000", 2024-09-15T03:47:07.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1720000000000004e-06 2024-09-15T03:47:07.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.199 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901648+0000", 2024-09-15T03:47:07.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7734999999999997e-05 2024-09-15T03:47:07.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.199 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901657+0000", 2024-09-15T03:47:07.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.242e-06 2024-09-15T03:47:07.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.200 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903437+0000", 2024-09-15T03:47:07.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001780404 2024-09-15T03:47:07.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.200 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903553+0000", 2024-09-15T03:47:07.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000115927 2024-09-15T03:47:07.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.201 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.201 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906054+0000", 2024-09-15T03:47:07.201 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002500604 2024-09-15T03:47:07.201 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.201 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.201 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.201 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906067+0000", 2024-09-15T03:47:07.201 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3699e-05 2024-09-15T03:47:07.201 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.201 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.201 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:07.201 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.918670+0000", 2024-09-15T03:47:07.201 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012602781 2024-09-15T03:47:07.201 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.202 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:07.202 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.202 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 60/31 ECSubWrite(tid=132, reqid=client.4521.0:422, at_version=0'0, trim_to=0'0, roll_forward_to=60'41))", 2024-09-15T03:47:07.202 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.906920+0000", 2024-09-15T03:47:07.202 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.17066055799999999, 2024-09-15T03:47:07.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011799888, 2024-09-15T03:47:07.202 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:07.202 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.202 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:07.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:07.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906920+0000", 2024-09-15T03:47:07.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:07.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906920+0000", 2024-09-15T03:47:07.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:07.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:07.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906921+0000", 2024-09-15T03:47:07.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4600000000000005e-07 2024-09-15T03:47:07.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:07.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906929+0000", 2024-09-15T03:47:07.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7840000000000004e-06 2024-09-15T03:47:07.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:07.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906938+0000", 2024-09-15T03:47:07.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4299999999999995e-06 2024-09-15T03:47:07.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906943+0000", 2024-09-15T03:47:07.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1209999999999998e-06 2024-09-15T03:47:07.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:07.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906988+0000", 2024-09-15T03:47:07.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5086999999999997e-05 2024-09-15T03:47:07.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906998+0000", 2024-09-15T03:47:07.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5610000000000004e-06 2024-09-15T03:47:07.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.907078+0000", 2024-09-15T03:47:07.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9989999999999998e-05 2024-09-15T03:47:07.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.907087+0000", 2024-09-15T03:47:07.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4110000000000002e-06 2024-09-15T03:47:07.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.908905+0000", 2024-09-15T03:47:07.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0018175229999999999 2024-09-15T03:47:07.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.207 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.908914+0000", 2024-09-15T03:47:07.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5829999999999996e-06 2024-09-15T03:47:07.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:07.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:07.207 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:07.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.918720+0000", 2024-09-15T03:47:07.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098058520000000003 2024-09-15T03:47:07.207 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.207 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:07.207 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.207 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:07.207 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:07.207 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:07.208 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-09-15T03:47:07.174+0000 7ff94fdf1640 -1 mon.a@0(leader).osd e60 definitely_dead 0 2024-09-15T03:47:07.209 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:07.208+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:07.210 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:47:07.309 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:07.308+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:07.311 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:07.311 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:07.311 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-15T03:47:07.311 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:07.320 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:47:07.410 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:07.410+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:07.421 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:07.422 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:07.422 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-15T03:47:07.422 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-15T03:47:07.422 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:07.431 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:47:07.511 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:07.511+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:07.533 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:07.534 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-15T03:47:07.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-15T03:47:07.534 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-15T03:47:07.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.534 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs2 33/31 ECSubWrite(tid=22, reqid=client.4521.0:187, at_version=33'8, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:07.534 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:40.107032+0000", 2024-09-15T03:47:07.534 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.424946882, 2024-09-15T03:47:07.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0038892639999999999, 2024-09-15T03:47:07.534 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:07.534 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.534 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:07.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:07.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107032+0000", 2024-09-15T03:47:07.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:07.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107032+0000", 2024-09-15T03:47:07.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:07.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107117+0000", 2024-09-15T03:47:07.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4419999999999995e-05 2024-09-15T03:47:07.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:07.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107140+0000", 2024-09-15T03:47:07.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3696000000000001e-05 2024-09-15T03:47:07.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:07.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107158+0000", 2024-09-15T03:47:07.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7672e-05 2024-09-15T03:47:07.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107166+0000", 2024-09-15T03:47:07.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1529999999999997e-06 2024-09-15T03:47:07.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:07.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107230+0000", 2024-09-15T03:47:07.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3924999999999995e-05 2024-09-15T03:47:07.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107241+0000", 2024-09-15T03:47:07.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0712e-05 2024-09-15T03:47:07.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107460+0000", 2024-09-15T03:47:07.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00021953599999999999 2024-09-15T03:47:07.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107738+0000", 2024-09-15T03:47:07.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00027775800000000002 2024-09-15T03:47:07.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.107795+0000", 2024-09-15T03:47:07.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6867999999999998e-05 2024-09-15T03:47:07.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108035+0000", 2024-09-15T03:47:07.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024024199999999999 2024-09-15T03:47:07.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:07.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.110922+0000", 2024-09-15T03:47:07.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002886282 2024-09-15T03:47:07.539 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.540 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:07.540 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.540 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs2 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:07.540 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:40.108717+0000", 2024-09-15T03:47:07.540 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.423262324, 2024-09-15T03:47:07.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041586089999999997, 2024-09-15T03:47:07.540 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:07.540 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.540 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:07.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:07.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108717+0000", 2024-09-15T03:47:07.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:07.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108717+0000", 2024-09-15T03:47:07.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:07.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108718+0000", 2024-09-15T03:47:07.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.412e-06 2024-09-15T03:47:07.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:07.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108755+0000", 2024-09-15T03:47:07.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6798000000000002e-05 2024-09-15T03:47:07.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:07.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108771+0000", 2024-09-15T03:47:07.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6002e-05 2024-09-15T03:47:07.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108779+0000", 2024-09-15T03:47:07.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5429999999999996e-06 2024-09-15T03:47:07.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:07.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108830+0000", 2024-09-15T03:47:07.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1127e-05 2024-09-15T03:47:07.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108840+0000", 2024-09-15T03:47:07.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8460000000000003e-06 2024-09-15T03:47:07.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108951+0000", 2024-09-15T03:47:07.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011156 2024-09-15T03:47:07.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.109109+0000", 2024-09-15T03:47:07.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015739700000000001 2024-09-15T03:47:07.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.109163+0000", 2024-09-15T03:47:07.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4026000000000001e-05 2024-09-15T03:47:07.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.110172+0000", 2024-09-15T03:47:07.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001009362 2024-09-15T03:47:07.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:07.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.112876+0000", 2024-09-15T03:47:07.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0027035359999999999 2024-09-15T03:47:07.546 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.546 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:07.546 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.546 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs2 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:07.546 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:40.108796+0000", 2024-09-15T03:47:07.547 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.423183428000002, 2024-09-15T03:47:07.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0042266949999999999, 2024-09-15T03:47:07.547 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:07.547 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.547 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:07.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:07.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108796+0000", 2024-09-15T03:47:07.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:07.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108796+0000", 2024-09-15T03:47:07.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:07.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108796+0000", 2024-09-15T03:47:07.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8599999999999998e-07 2024-09-15T03:47:07.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:07.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108818+0000", 2024-09-15T03:47:07.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1206999999999999e-05 2024-09-15T03:47:07.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:07.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108828+0000", 2024-09-15T03:47:07.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0321e-05 2024-09-15T03:47:07.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108834+0000", 2024-09-15T03:47:07.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0229999999999998e-06 2024-09-15T03:47:07.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:07.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.108999+0000", 2024-09-15T03:47:07.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016466399999999999 2024-09-15T03:47:07.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.109005+0000", 2024-09-15T03:47:07.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.973e-06 2024-09-15T03:47:07.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.109086+0000", 2024-09-15T03:47:07.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0136999999999995e-05 2024-09-15T03:47:07.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.109109+0000", 2024-09-15T03:47:07.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3822999999999999e-05 2024-09-15T03:47:07.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.109163+0000", 2024-09-15T03:47:07.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4029000000000001e-05 2024-09-15T03:47:07.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.110210+0000", 2024-09-15T03:47:07.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010464859999999999 2024-09-15T03:47:07.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:07.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:40.113023+0000", 2024-09-15T03:47:07.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002812546 2024-09-15T03:47:07.552 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.552 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:07.552 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.552 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.5s7 38/31 ECSubWrite(tid=38, reqid=client.4521.0:223, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:47:07.552 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:45.146773+0000", 2024-09-15T03:47:07.552 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.385205804999998, 2024-09-15T03:47:07.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041833800000000004, 2024-09-15T03:47:07.553 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:07.553 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.553 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:07.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:07.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.146773+0000", 2024-09-15T03:47:07.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:07.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.146773+0000", 2024-09-15T03:47:07.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:07.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.146774+0000", 2024-09-15T03:47:07.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.37e-07 2024-09-15T03:47:07.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:07.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.146888+0000", 2024-09-15T03:47:07.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000114219 2024-09-15T03:47:07.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:07.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.146901+0000", 2024-09-15T03:47:07.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2551999999999999e-05 2024-09-15T03:47:07.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.146912+0000", 2024-09-15T03:47:07.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0881e-05 2024-09-15T03:47:07.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:07.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.147069+0000", 2024-09-15T03:47:07.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015756900000000001 2024-09-15T03:47:07.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.556 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.147081+0000", 2024-09-15T03:47:07.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2153999999999999e-05 2024-09-15T03:47:07.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.556 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148051+0000", 2024-09-15T03:47:07.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00096960899999999997 2024-09-15T03:47:07.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148083+0000", 2024-09-15T03:47:07.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2141999999999998e-05 2024-09-15T03:47:07.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148249+0000", 2024-09-15T03:47:07.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000165833 2024-09-15T03:47:07.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.148284+0000", 2024-09-15T03:47:07.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4953000000000002e-05 2024-09-15T03:47:07.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:07.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.150957+0000", 2024-09-15T03:47:07.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0026729309999999999 2024-09-15T03:47:07.559 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.559 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:07.559 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.559 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.4s4 38/31 ECSubWrite(tid=38, reqid=client.4521.0:237, at_version=38'6, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:07.560 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:45.158151+0000", 2024-09-15T03:47:07.560 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.373827860999999, 2024-09-15T03:47:07.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0035353099999999998, 2024-09-15T03:47:07.560 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:07.560 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.560 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:07.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:07.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158151+0000", 2024-09-15T03:47:07.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:07.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158151+0000", 2024-09-15T03:47:07.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:07.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158152+0000", 2024-09-15T03:47:07.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0500000000000003e-07 2024-09-15T03:47:07.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:07.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158174+0000", 2024-09-15T03:47:07.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2272999999999998e-05 2024-09-15T03:47:07.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:07.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158187+0000", 2024-09-15T03:47:07.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2291e-05 2024-09-15T03:47:07.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158193+0000", 2024-09-15T03:47:07.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.776e-06 2024-09-15T03:47:07.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:07.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158246+0000", 2024-09-15T03:47:07.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2858000000000001e-05 2024-09-15T03:47:07.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158254+0000", 2024-09-15T03:47:07.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1920000000000005e-06 2024-09-15T03:47:07.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158341+0000", 2024-09-15T03:47:07.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6136000000000003e-05 2024-09-15T03:47:07.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158368+0000", 2024-09-15T03:47:07.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6976000000000001e-05 2024-09-15T03:47:07.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158404+0000", 2024-09-15T03:47:07.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5988000000000002e-05 2024-09-15T03:47:07.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.158415+0000", 2024-09-15T03:47:07.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1697e-05 2024-09-15T03:47:07.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:07.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:45.161687+0000", 2024-09-15T03:47:07.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032714179999999999 2024-09-15T03:47:07.565 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.565 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:07.565 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.566 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.4s4 39/31 ECSubWrite(tid=41, reqid=osd.6.0:41, at_version=39'9, trim_to=0'0, roll_forward_to=38'7))", 2024-09-15T03:47:07.566 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:46.146606+0000", 2024-09-15T03:47:07.566 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.385373297000001, 2024-09-15T03:47:07.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0048345970000000004, 2024-09-15T03:47:07.566 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:07.566 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.566 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:07.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:07.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.146606+0000", 2024-09-15T03:47:07.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:07.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.146606+0000", 2024-09-15T03:47:07.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:07.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.146606+0000", 2024-09-15T03:47:07.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3200000000000005e-07 2024-09-15T03:47:07.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:07.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.146636+0000", 2024-09-15T03:47:07.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9434e-05 2024-09-15T03:47:07.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.569 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:07.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.146666+0000", 2024-09-15T03:47:07.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9733000000000001e-05 2024-09-15T03:47:07.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.569 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.146674+0000", 2024-09-15T03:47:07.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2919999999999996e-06 2024-09-15T03:47:07.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.569 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:07.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150070+0000", 2024-09-15T03:47:07.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0033961590000000002 2024-09-15T03:47:07.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150080+0000", 2024-09-15T03:47:07.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0181e-05 2024-09-15T03:47:07.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150334+0000", 2024-09-15T03:47:07.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00025387100000000002 2024-09-15T03:47:07.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150370+0000", 2024-09-15T03:47:07.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6269999999999997e-05 2024-09-15T03:47:07.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150430+0000", 2024-09-15T03:47:07.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9404000000000002e-05 2024-09-15T03:47:07.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150480+0000", 2024-09-15T03:47:07.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0541000000000003e-05 2024-09-15T03:47:07.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:07.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.151441+0000", 2024-09-15T03:47:07.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00096018000000000004 2024-09-15T03:47:07.572 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.572 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:07.572 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.572 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.3s7 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:47:07.572 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:46.170728+0000", 2024-09-15T03:47:07.573 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.36125097, 2024-09-15T03:47:07.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0037502820000000002, 2024-09-15T03:47:07.573 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:07.573 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.573 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:07.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:07.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.170728+0000", 2024-09-15T03:47:07.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:07.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.170728+0000", 2024-09-15T03:47:07.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:07.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.170730+0000", 2024-09-15T03:47:07.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.483e-06 2024-09-15T03:47:07.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:07.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171114+0000", 2024-09-15T03:47:07.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00038467500000000001 2024-09-15T03:47:07.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.575 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:07.575 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171135+0000", 2024-09-15T03:47:07.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0962999999999999e-05 2024-09-15T03:47:07.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.575 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.575 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171145+0000", 2024-09-15T03:47:07.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2450000000000004e-06 2024-09-15T03:47:07.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.575 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:07.575 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171287+0000", 2024-09-15T03:47:07.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014246899999999999 2024-09-15T03:47:07.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171299+0000", 2024-09-15T03:47:07.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1574e-05 2024-09-15T03:47:07.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171685+0000", 2024-09-15T03:47:07.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00038670300000000001 2024-09-15T03:47:07.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171716+0000", 2024-09-15T03:47:07.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0919e-05 2024-09-15T03:47:07.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171821+0000", 2024-09-15T03:47:07.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000104609 2024-09-15T03:47:07.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.171911+0000", 2024-09-15T03:47:07.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0348999999999998e-05 2024-09-15T03:47:07.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:07.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.174479+0000", 2024-09-15T03:47:07.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002567293 2024-09-15T03:47:07.578 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.578 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:07.578 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.579 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.3s7 56/31 ECSubWrite(tid=83, reqid=client.4521.0:395, at_version=56'41, trim_to=0'0, roll_forward_to=54'39))", 2024-09-15T03:47:07.579 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:02.889686+0000", 2024-09-15T03:47:07.579 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 4.6422935870000002, 2024-09-15T03:47:07.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041337730000000003, 2024-09-15T03:47:07.579 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:07.579 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.579 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:07.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:07.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.889686+0000", 2024-09-15T03:47:07.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:07.580 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.889686+0000", 2024-09-15T03:47:07.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.580 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:07.580 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.889686+0000", 2024-09-15T03:47:07.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4700000000000001e-07 2024-09-15T03:47:07.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.580 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:07.580 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.890009+0000", 2024-09-15T03:47:07.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00032358300000000001 2024-09-15T03:47:07.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.581 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:07.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.890022+0000", 2024-09-15T03:47:07.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2839e-05 2024-09-15T03:47:07.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.581 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.890031+0000", 2024-09-15T03:47:07.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7839999999999992e-06 2024-09-15T03:47:07.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.581 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:07.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.890078+0000", 2024-09-15T03:47:07.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7243000000000002e-05 2024-09-15T03:47:07.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.890086+0000", 2024-09-15T03:47:07.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3189999999999997e-06 2024-09-15T03:47:07.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.890350+0000", 2024-09-15T03:47:07.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026462700000000003 2024-09-15T03:47:07.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.890360+0000", 2024-09-15T03:47:07.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0061000000000001e-05 2024-09-15T03:47:07.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.890521+0000", 2024-09-15T03:47:07.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000160851 2024-09-15T03:47:07.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.890539+0000", 2024-09-15T03:47:07.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7737e-05 2024-09-15T03:47:07.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:07.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.893819+0000", 2024-09-15T03:47:07.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032805820000000002 2024-09-15T03:47:07.584 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.584 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:07.584 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.584 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs2 60/31 ECSubWrite(tid=70, reqid=client.4521.0:421, at_version=60'22, trim_to=0'0, roll_forward_to=47'20))", 2024-09-15T03:47:07.584 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.837529+0000", 2024-09-15T03:47:07.584 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.69445042400000001, 2024-09-15T03:47:07.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0088963150000000001, 2024-09-15T03:47:07.584 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:07.585 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.585 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:07.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:07.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837529+0000", 2024-09-15T03:47:07.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:07.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837529+0000", 2024-09-15T03:47:07.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:07.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837529+0000", 2024-09-15T03:47:07.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8199999999999999e-07 2024-09-15T03:47:07.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:07.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837562+0000", 2024-09-15T03:47:07.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3080000000000002e-05 2024-09-15T03:47:07.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.587 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:07.587 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837572+0000", 2024-09-15T03:47:07.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7219999999999994e-06 2024-09-15T03:47:07.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.587 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.587 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837580+0000", 2024-09-15T03:47:07.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4470000000000006e-06 2024-09-15T03:47:07.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.587 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:07.587 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837727+0000", 2024-09-15T03:47:07.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014676199999999999 2024-09-15T03:47:07.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837738+0000", 2024-09-15T03:47:07.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0946e-05 2024-09-15T03:47:07.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.838091+0000", 2024-09-15T03:47:07.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00035338600000000002 2024-09-15T03:47:07.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.589 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.839733+0000", 2024-09-15T03:47:07.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016420409999999999 2024-09-15T03:47:07.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.589 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.839815+0000", 2024-09-15T03:47:07.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1433000000000005e-05 2024-09-15T03:47:07.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.589 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.839841+0000", 2024-09-15T03:47:07.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6529e-05 2024-09-15T03:47:07.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:07.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.846425+0000", 2024-09-15T03:47:07.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0065837869999999998 2024-09-15T03:47:07.590 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.590 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:07.590 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.590 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.ds4 60/31 ECSubWrite(tid=122, reqid=osd.8.0:122, at_version=60'16, trim_to=0'0, roll_forward_to=59'14))", 2024-09-15T03:47:07.590 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.840427+0000", 2024-09-15T03:47:07.590 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.69155192399999998, 2024-09-15T03:47:07.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0064843990000000001, 2024-09-15T03:47:07.590 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:07.590 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.591 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:07.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:07.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840427+0000", 2024-09-15T03:47:07.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:07.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840427+0000", 2024-09-15T03:47:07.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:07.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840428+0000", 2024-09-15T03:47:07.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1099999999999996e-07 2024-09-15T03:47:07.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:07.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840460+0000", 2024-09-15T03:47:07.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1869999999999998e-05 2024-09-15T03:47:07.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:07.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840487+0000", 2024-09-15T03:47:07.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7778999999999999e-05 2024-09-15T03:47:07.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840496+0000", 2024-09-15T03:47:07.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9660000000000002e-06 2024-09-15T03:47:07.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:07.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840565+0000", 2024-09-15T03:47:07.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8464e-05 2024-09-15T03:47:07.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840577+0000", 2024-09-15T03:47:07.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1789e-05 2024-09-15T03:47:07.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.594 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840853+0000", 2024-09-15T03:47:07.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00027612699999999998 2024-09-15T03:47:07.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.594 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840900+0000", 2024-09-15T03:47:07.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6801000000000001e-05 2024-09-15T03:47:07.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840948+0000", 2024-09-15T03:47:07.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8406999999999999e-05 2024-09-15T03:47:07.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840972+0000", 2024-09-15T03:47:07.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4377999999999999e-05 2024-09-15T03:47:07.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:07.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.846912+0000", 2024-09-15T03:47:07.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0059393069999999996 2024-09-15T03:47:07.595 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.596 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:07.596 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.596 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.5s7 60/31 ECSubWrite(tid=78, reqid=osd.1.0:78, at_version=60'33, trim_to=0'0, roll_forward_to=59'31))", 2024-09-15T03:47:07.596 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.842455+0000", 2024-09-15T03:47:07.596 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.68952425500000003, 2024-09-15T03:47:07.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0049164439999999998, 2024-09-15T03:47:07.596 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:07.596 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.596 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:07.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:07.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.842455+0000", 2024-09-15T03:47:07.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:07.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.842455+0000", 2024-09-15T03:47:07.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:07.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.842455+0000", 2024-09-15T03:47:07.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9800000000000004e-07 2024-09-15T03:47:07.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:07.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.842486+0000", 2024-09-15T03:47:07.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0309000000000001e-05 2024-09-15T03:47:07.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:07.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.842498+0000", 2024-09-15T03:47:07.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2221e-05 2024-09-15T03:47:07.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.842506+0000", 2024-09-15T03:47:07.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9310000000000008e-06 2024-09-15T03:47:07.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.599 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:07.599 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.842578+0000", 2024-09-15T03:47:07.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2080000000000001e-05 2024-09-15T03:47:07.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.599 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.599 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.842590+0000", 2024-09-15T03:47:07.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1646e-05 2024-09-15T03:47:07.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.599 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.600 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.842906+0000", 2024-09-15T03:47:07.600 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031659899999999998 2024-09-15T03:47:07.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.600 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.600 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.842943+0000", 2024-09-15T03:47:07.600 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7061000000000003e-05 2024-09-15T03:47:07.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.600 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.600 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.842994+0000", 2024-09-15T03:47:07.600 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0213999999999997e-05 2024-09-15T03:47:07.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.601 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843010+0000", 2024-09-15T03:47:07.601 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6716999999999999e-05 2024-09-15T03:47:07.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:07.601 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.847371+0000", 2024-09-15T03:47:07.601 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0043611680000000003 2024-09-15T03:47:07.601 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.601 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:07.601 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.602 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs2 60/31 ECSubWrite(tid=71, reqid=client.4521.0:421, at_version=0'0, trim_to=0'0, roll_forward_to=60'22))", 2024-09-15T03:47:07.602 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.843064+0000", 2024-09-15T03:47:07.602 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.68891481600000004, 2024-09-15T03:47:07.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0066467000000000002, 2024-09-15T03:47:07.602 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:07.602 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.602 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:07.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.602 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:07.602 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843064+0000", 2024-09-15T03:47:07.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.602 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:07.603 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843064+0000", 2024-09-15T03:47:07.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.603 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:07.603 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843065+0000", 2024-09-15T03:47:07.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5600000000000004e-07 2024-09-15T03:47:07.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.603 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:07.603 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843077+0000", 2024-09-15T03:47:07.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1336e-05 2024-09-15T03:47:07.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.604 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:07.604 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843089+0000", 2024-09-15T03:47:07.604 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2869e-05 2024-09-15T03:47:07.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.604 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.604 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843097+0000", 2024-09-15T03:47:07.604 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.503e-06 2024-09-15T03:47:07.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.604 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:07.604 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843157+0000", 2024-09-15T03:47:07.604 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9580999999999998e-05 2024-09-15T03:47:07.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.605 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.605 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.605 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843167+0000", 2024-09-15T03:47:07.605 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0406e-05 2024-09-15T03:47:07.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.605 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.605 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.605 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843279+0000", 2024-09-15T03:47:07.605 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112427 2024-09-15T03:47:07.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.605 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.605 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.605 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843339+0000", 2024-09-15T03:47:07.606 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0031000000000002e-05 2024-09-15T03:47:07.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.606 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843380+0000", 2024-09-15T03:47:07.606 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0683000000000003e-05 2024-09-15T03:47:07.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.606 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843393+0000", 2024-09-15T03:47:07.606 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3334e-05 2024-09-15T03:47:07.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:07.607 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.849711+0000", 2024-09-15T03:47:07.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0063176739999999997 2024-09-15T03:47:07.607 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.607 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:07.607 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.607 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.3s7 60/31 ECSubWrite(tid=99, reqid=osd.6.0:99, at_version=60'54, trim_to=0'0, roll_forward_to=59'52))", 2024-09-15T03:47:07.607 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.843197+0000", 2024-09-15T03:47:07.607 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.68878252699999998, 2024-09-15T03:47:07.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0048142999999999997, 2024-09-15T03:47:07.607 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:07.607 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.607 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:07.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.608 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:07.608 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843197+0000", 2024-09-15T03:47:07.608 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.608 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:07.608 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843197+0000", 2024-09-15T03:47:07.608 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.608 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:07.608 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843197+0000", 2024-09-15T03:47:07.608 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6199999999999998e-07 2024-09-15T03:47:07.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.609 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:07.609 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843226+0000", 2024-09-15T03:47:07.609 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8636000000000002e-05 2024-09-15T03:47:07.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.609 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:07.609 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843242+0000", 2024-09-15T03:47:07.609 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6194000000000001e-05 2024-09-15T03:47:07.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.609 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.610 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843250+0000", 2024-09-15T03:47:07.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1729999999999995e-06 2024-09-15T03:47:07.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.610 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:07.610 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843322+0000", 2024-09-15T03:47:07.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1835999999999995e-05 2024-09-15T03:47:07.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.610 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.610 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843332+0000", 2024-09-15T03:47:07.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0353000000000001e-05 2024-09-15T03:47:07.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.611 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.611 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843604+0000", 2024-09-15T03:47:07.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000271284 2024-09-15T03:47:07.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.611 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.611 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843660+0000", 2024-09-15T03:47:07.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6147999999999998e-05 2024-09-15T03:47:07.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.611 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.611 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843705+0000", 2024-09-15T03:47:07.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5627999999999997e-05 2024-09-15T03:47:07.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.612 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.612 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843725+0000", 2024-09-15T03:47:07.612 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0001999999999999e-05 2024-09-15T03:47:07.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.612 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:07.612 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.848011+0000", 2024-09-15T03:47:07.612 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0042855840000000003 2024-09-15T03:47:07.612 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.612 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:07.613 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.613 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.4s4 60/31 ECSubWrite(tid=100, reqid=osd.6.0:100, at_version=60'24, trim_to=0'0, roll_forward_to=58'22))", 2024-09-15T03:47:07.613 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.843647+0000", 2024-09-15T03:47:07.613 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.68833241300000003, 2024-09-15T03:47:07.613 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0047118129999999996, 2024-09-15T03:47:07.613 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:07.613 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.613 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:07.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.613 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:07.613 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843647+0000", 2024-09-15T03:47:07.613 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:07.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843647+0000", 2024-09-15T03:47:07.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:07.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843647+0000", 2024-09-15T03:47:07.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0699999999999997e-07 2024-09-15T03:47:07.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:07.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843678+0000", 2024-09-15T03:47:07.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0842e-05 2024-09-15T03:47:07.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:07.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843691+0000", 2024-09-15T03:47:07.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2651e-05 2024-09-15T03:47:07.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843702+0000", 2024-09-15T03:47:07.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1021e-05 2024-09-15T03:47:07.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:07.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843766+0000", 2024-09-15T03:47:07.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3683999999999995e-05 2024-09-15T03:47:07.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843776+0000", 2024-09-15T03:47:07.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0158e-05 2024-09-15T03:47:07.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844112+0000", 2024-09-15T03:47:07.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000336387 2024-09-15T03:47:07.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.617 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844297+0000", 2024-09-15T03:47:07.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018529899999999999 2024-09-15T03:47:07.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.617 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844370+0000", 2024-09-15T03:47:07.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2857999999999999e-05 2024-09-15T03:47:07.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844457+0000", 2024-09-15T03:47:07.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6947000000000006e-05 2024-09-15T03:47:07.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:07.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.848359+0000", 2024-09-15T03:47:07.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0039014589999999999 2024-09-15T03:47:07.618 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.618 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:07.618 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.619 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s7 60/31 ECSubWrite(tid=125, reqid=osd.4.0:125, at_version=60'39, trim_to=0'0, roll_forward_to=58'37))", 2024-09-15T03:47:07.619 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.844020+0000", 2024-09-15T03:47:07.619 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.68795956199999997, 2024-09-15T03:47:07.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.004682238, 2024-09-15T03:47:07.619 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:07.619 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.619 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:07.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:07.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844020+0000", 2024-09-15T03:47:07.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:07.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844020+0000", 2024-09-15T03:47:07.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:07.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844020+0000", 2024-09-15T03:47:07.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9700000000000003e-07 2024-09-15T03:47:07.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:07.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844045+0000", 2024-09-15T03:47:07.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5296999999999999e-05 2024-09-15T03:47:07.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.621 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:07.621 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844059+0000", 2024-09-15T03:47:07.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4133e-05 2024-09-15T03:47:07.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.621 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.621 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844068+0000", 2024-09-15T03:47:07.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3000000000000002e-06 2024-09-15T03:47:07.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.621 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:07.621 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844222+0000", 2024-09-15T03:47:07.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000154282 2024-09-15T03:47:07.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844233+0000", 2024-09-15T03:47:07.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1243999999999999e-05 2024-09-15T03:47:07.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844493+0000", 2024-09-15T03:47:07.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00025969200000000001 2024-09-15T03:47:07.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844512+0000", 2024-09-15T03:47:07.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9304e-05 2024-09-15T03:47:07.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844555+0000", 2024-09-15T03:47:07.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3105999999999997e-05 2024-09-15T03:47:07.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844591+0000", 2024-09-15T03:47:07.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5896999999999999e-05 2024-09-15T03:47:07.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:07.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.848702+0000", 2024-09-15T03:47:07.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041106859999999997 2024-09-15T03:47:07.624 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.624 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:07.624 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.624 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.3s7 60/31 ECSubWrite(tid=101, reqid=osd.6.0:101, at_version=60'56, trim_to=0'0, roll_forward_to=59'52))", 2024-09-15T03:47:07.624 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.844152+0000", 2024-09-15T03:47:07.624 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.68782710499999999, 2024-09-15T03:47:07.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0052846519999999999, 2024-09-15T03:47:07.625 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:07.625 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.625 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:07.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.625 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:07.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844152+0000", 2024-09-15T03:47:07.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.625 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:07.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844152+0000", 2024-09-15T03:47:07.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:07.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844165+0000", 2024-09-15T03:47:07.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2847000000000001e-05 2024-09-15T03:47:07.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:07.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844199+0000", 2024-09-15T03:47:07.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4418000000000001e-05 2024-09-15T03:47:07.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:07.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844226+0000", 2024-09-15T03:47:07.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6665999999999998e-05 2024-09-15T03:47:07.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844234+0000", 2024-09-15T03:47:07.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5909999999999998e-06 2024-09-15T03:47:07.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:07.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844284+0000", 2024-09-15T03:47:07.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0183999999999999e-05 2024-09-15T03:47:07.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844293+0000", 2024-09-15T03:47:07.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.448e-06 2024-09-15T03:47:07.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844548+0000", 2024-09-15T03:47:07.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00025473999999999999 2024-09-15T03:47:07.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844558+0000", 2024-09-15T03:47:07.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0373999999999999e-05 2024-09-15T03:47:07.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844651+0000", 2024-09-15T03:47:07.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2225000000000006e-05 2024-09-15T03:47:07.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844683+0000", 2024-09-15T03:47:07.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2032000000000001e-05 2024-09-15T03:47:07.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:07.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.849437+0000", 2024-09-15T03:47:07.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0047541270000000004 2024-09-15T03:47:07.630 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.630 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:07.630 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.630 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.8s6 60/31 ECSubWrite(tid=47, reqid=osd.3.0:47, at_version=60'10, trim_to=0'0, roll_forward_to=58'8))", 2024-09-15T03:47:07.630 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.844673+0000", 2024-09-15T03:47:07.630 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.68730598499999995, 2024-09-15T03:47:07.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0078207390000000002, 2024-09-15T03:47:07.631 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:07.631 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.631 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:07.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:07.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844673+0000", 2024-09-15T03:47:07.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:07.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844673+0000", 2024-09-15T03:47:07.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:07.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844674+0000", 2024-09-15T03:47:07.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4299999999999999e-07 2024-09-15T03:47:07.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:07.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844699+0000", 2024-09-15T03:47:07.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5003000000000001e-05 2024-09-15T03:47:07.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:07.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844708+0000", 2024-09-15T03:47:07.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8109999999999997e-06 2024-09-15T03:47:07.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844715+0000", 2024-09-15T03:47:07.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.127e-06 2024-09-15T03:47:07.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:07.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.848363+0000", 2024-09-15T03:47:07.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0036482390000000002 2024-09-15T03:47:07.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.848374+0000", 2024-09-15T03:47:07.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1137e-05 2024-09-15T03:47:07.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.848615+0000", 2024-09-15T03:47:07.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024119899999999999 2024-09-15T03:47:07.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.851231+0000", 2024-09-15T03:47:07.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0026161079999999998 2024-09-15T03:47:07.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.851513+0000", 2024-09-15T03:47:07.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000282102 2024-09-15T03:47:07.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.851556+0000", 2024-09-15T03:47:07.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2960999999999997e-05 2024-09-15T03:47:07.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:07.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.852494+0000", 2024-09-15T03:47:07.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000937709 2024-09-15T03:47:07.636 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.636 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:07.636 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.636 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.ds4 60/31 ECSubWrite(tid=123, reqid=osd.8.0:122, at_version=0'0, trim_to=0'0, roll_forward_to=60'16))", 2024-09-15T03:47:07.636 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.846028+0000", 2024-09-15T03:47:07.636 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.68595129799999999, 2024-09-15T03:47:07.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003527021, 2024-09-15T03:47:07.636 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:07.636 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.636 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:07.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:07.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.846028+0000", 2024-09-15T03:47:07.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:07.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.846028+0000", 2024-09-15T03:47:07.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:07.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.846029+0000", 2024-09-15T03:47:07.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.049e-06 2024-09-15T03:47:07.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:07.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.846042+0000", 2024-09-15T03:47:07.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2746e-05 2024-09-15T03:47:07.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:07.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.846061+0000", 2024-09-15T03:47:07.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9562999999999999e-05 2024-09-15T03:47:07.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.846069+0000", 2024-09-15T03:47:07.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5349999999999999e-06 2024-09-15T03:47:07.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:07.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.846244+0000", 2024-09-15T03:47:07.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017505900000000001 2024-09-15T03:47:07.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.846271+0000", 2024-09-15T03:47:07.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7328e-05 2024-09-15T03:47:07.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.846383+0000", 2024-09-15T03:47:07.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011179999999999999 2024-09-15T03:47:07.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.846407+0000", 2024-09-15T03:47:07.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3676000000000002e-05 2024-09-15T03:47:07.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.846525+0000", 2024-09-15T03:47:07.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118248 2024-09-15T03:47:07.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.846595+0000", 2024-09-15T03:47:07.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9673999999999997e-05 2024-09-15T03:47:07.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:07.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.849555+0000", 2024-09-15T03:47:07.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029603429999999998 2024-09-15T03:47:07.641 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.641 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:07.642 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.642 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.3s7 60/31 ECSubWrite(tid=102, reqid=osd.6.0:101, at_version=0'0, trim_to=0'0, roll_forward_to=60'56))", 2024-09-15T03:47:07.642 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.848675+0000", 2024-09-15T03:47:07.642 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.68330462599999997, 2024-09-15T03:47:07.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0036572169999999999, 2024-09-15T03:47:07.642 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:07.642 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.642 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:07.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:07.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.848675+0000", 2024-09-15T03:47:07.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:07.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.848675+0000", 2024-09-15T03:47:07.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:07.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.848675+0000", 2024-09-15T03:47:07.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5899999999999996e-07 2024-09-15T03:47:07.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:07.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.848688+0000", 2024-09-15T03:47:07.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2497000000000001e-05 2024-09-15T03:47:07.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:07.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.848700+0000", 2024-09-15T03:47:07.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2343e-05 2024-09-15T03:47:07.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.848707+0000", 2024-09-15T03:47:07.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0690000000000004e-06 2024-09-15T03:47:07.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:07.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.849998+0000", 2024-09-15T03:47:07.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001290694 2024-09-15T03:47:07.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.850202+0000", 2024-09-15T03:47:07.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000204584 2024-09-15T03:47:07.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.850339+0000", 2024-09-15T03:47:07.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013652299999999999 2024-09-15T03:47:07.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.851234+0000", 2024-09-15T03:47:07.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00089541200000000005 2024-09-15T03:47:07.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.851516+0000", 2024-09-15T03:47:07.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00028137000000000003 2024-09-15T03:47:07.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.851598+0000", 2024-09-15T03:47:07.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2684999999999999e-05 2024-09-15T03:47:07.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:07.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.852332+0000", 2024-09-15T03:47:07.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00073338099999999996 2024-09-15T03:47:07.648 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.649 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:07.649 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.649 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s7 60/31 ECSubWrite(tid=129, reqid=client.4521.0:422, at_version=60'41, trim_to=0'0, roll_forward_to=60'39))", 2024-09-15T03:47:07.649 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.900924+0000", 2024-09-15T03:47:07.649 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.63105517300000002, 2024-09-15T03:47:07.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041415929999999998, 2024-09-15T03:47:07.649 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:07.649 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:07.649 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:07.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:07.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.900924+0000", 2024-09-15T03:47:07.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:07.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.900924+0000", 2024-09-15T03:47:07.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:07.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:07.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.900926+0000", 2024-09-15T03:47:07.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5820000000000001e-06 2024-09-15T03:47:07.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:07.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.901122+0000", 2024-09-15T03:47:07.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019655700000000001 2024-09-15T03:47:07.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:07.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.901146+0000", 2024-09-15T03:47:07.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4139e-05 2024-09-15T03:47:07.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:07.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.901156+0000", 2024-09-15T03:47:07.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7480000000000005e-06 2024-09-15T03:47:07.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:07.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.901248+0000", 2024-09-15T03:47:07.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1599000000000002e-05 2024-09-15T03:47:07.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:07.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.901260+0000", 2024-09-15T03:47:07.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2045000000000001e-05 2024-09-15T03:47:07.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:07.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.901611+0000", 2024-09-15T03:47:07.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00035114 2024-09-15T03:47:07.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:07.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.901625+0000", 2024-09-15T03:47:07.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4224e-05 2024-09-15T03:47:07.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:07.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.901704+0000", 2024-09-15T03:47:07.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9105999999999998e-05 2024-09-15T03:47:07.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:07.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.901726+0000", 2024-09-15T03:47:07.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1883e-05 2024-09-15T03:47:07.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:07.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:07.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:07.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.905066+0000", 2024-09-15T03:47:07.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00333957 2024-09-15T03:47:07.654 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.654 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:07.654 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.655 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:07.655 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:07.655 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:07.656 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:47:07.657 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:07.656+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:07.757 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:07.756+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:07.797 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:07.797 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:07.797 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-15T03:47:07.798 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:07.806 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:47:07.856 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T03:47:07.858 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:07.857+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:07.884 INFO:teuthology.orchestra.run.smithi001.stderr:marked out osd.6. 2024-09-15T03:47:07.891 INFO:tasks.rados.rados.0.smithi001.stdout:173: done (2 left) 2024-09-15T03:47:07.891 INFO:tasks.rados.rados.0.smithi001.stdout:174: copy_from oid 49 from oid 45 current snap is 19 2024-09-15T03:47:07.891 INFO:tasks.rados.rados.0.smithi001.stdout:171: finishing write tid 2 to smithi00132172-9 2024-09-15T03:47:07.891 INFO:tasks.rados.rados.0.smithi001.stdout:171: finishing write tid 3 to smithi00132172-9 2024-09-15T03:47:07.891 INFO:tasks.rados.rados.0.smithi001.stdout:171: finishing write tid 4 to smithi00132172-9 2024-09-15T03:47:07.891 INFO:tasks.rados.rados.0.smithi001.stdout:171: finishing write tid 5 to smithi00132172-9 2024-09-15T03:47:07.891 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 9 v 61 (ObjNum 89 snap 19 seq_num 89) dirty exists 2024-09-15T03:47:07.891 INFO:tasks.rados.rados.0.smithi001.stdout:171: left oid 9 (ObjNum 89 snap 19 seq_num 89) 2024-09-15T03:47:07.892 INFO:tasks.rados.rados.0.smithi001.stdout:172: finishing copy_from racing read to smithi00132172-19 2024-09-15T03:47:07.892 INFO:tasks.rados.rados.0.smithi001.stdout:172: finishing copy_from to smithi00132172-19 2024-09-15T03:47:07.892 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 19 v 38 (ObjNum 66 snap 6 seq_num 66) dirty exists 2024-09-15T03:47:07.892 INFO:tasks.rados.rados.0.smithi001.stdout:171: done (2 left) 2024-09-15T03:47:07.892 INFO:tasks.rados.rados.0.smithi001.stdout:172: done (1 left) 2024-09-15T03:47:07.892 INFO:tasks.rados.rados.0.smithi001.stdout:175: snap_remove snap 13 2024-09-15T03:47:07.920 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:07.920 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:07.920 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-15T03:47:07.920 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-15T03:47:07.920 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:07.929 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:47:07.959 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:07.958+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:08.059 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:08.059+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:08.075 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:08.076 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-15T03:47:08.076 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-15T03:47:08.076 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-15T03:47:08.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.076 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=80, reqid=client.4521.0:320, at_version=49'15, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:47:08.076 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.840885+0000", 2024-09-15T03:47:08.076 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.232162873, 2024-09-15T03:47:08.076 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0033862459999998, 2024-09-15T03:47:08.076 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:08.076 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:08.076 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:08.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.077 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:08.077 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.840885+0000", 2024-09-15T03:47:08.077 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:08.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.840885+0000", 2024-09-15T03:47:08.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:08.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.840885+0000", 2024-09-15T03:47:08.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1100000000000002e-07 2024-09-15T03:47:08.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:08.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841181+0000", 2024-09-15T03:47:08.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029618199999999998 2024-09-15T03:47:08.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:08.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841195+0000", 2024-09-15T03:47:08.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3424000000000001e-05 2024-09-15T03:47:08.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:08.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841204+0000", 2024-09-15T03:47:08.083 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0170000000000002e-06 2024-09-15T03:47:08.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.083 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:08.083 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841274+0000", 2024-09-15T03:47:08.083 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0561000000000004e-05 2024-09-15T03:47:08.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.083 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:08.083 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841289+0000", 2024-09-15T03:47:08.083 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4127e-05 2024-09-15T03:47:08.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.084 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:08.084 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841717+0000", 2024-09-15T03:47:08.084 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00042811800000000001 2024-09-15T03:47:08.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.084 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:08.084 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841745+0000", 2024-09-15T03:47:08.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8297e-05 2024-09-15T03:47:08.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:08.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841852+0000", 2024-09-15T03:47:08.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010660199999999999 2024-09-15T03:47:08.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:08.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.842419+0000", 2024-09-15T03:47:08.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00056784499999999998 2024-09-15T03:47:08.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.086 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:08.086 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.844271+0000", 2024-09-15T03:47:08.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0018517619999998 2024-09-15T03:47:08.086 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.086 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:08.086 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.086 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=30, reqid=client.4521.0:325, at_version=49'12, trim_to=0'0, roll_forward_to=40'10))", 2024-09-15T03:47:08.086 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.844912+0000", 2024-09-15T03:47:08.086 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.228136291, 2024-09-15T03:47:08.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0007286940000002, 2024-09-15T03:47:08.087 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:08.087 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:08.087 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:08.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:08.087 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844912+0000", 2024-09-15T03:47:08.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:08.087 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844912+0000", 2024-09-15T03:47:08.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:08.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844912+0000", 2024-09-15T03:47:08.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.84e-07 2024-09-15T03:47:08.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:08.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844937+0000", 2024-09-15T03:47:08.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4729e-05 2024-09-15T03:47:08.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:08.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844957+0000", 2024-09-15T03:47:08.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0285999999999999e-05 2024-09-15T03:47:08.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:08.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844965+0000", 2024-09-15T03:47:08.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5709999999999998e-06 2024-09-15T03:47:08.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:08.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845086+0000", 2024-09-15T03:47:08.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000120856 2024-09-15T03:47:08.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:08.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845121+0000", 2024-09-15T03:47:08.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4292999999999998e-05 2024-09-15T03:47:08.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:08.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845492+0000", 2024-09-15T03:47:08.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00037167400000000001 2024-09-15T03:47:08.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:08.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845529+0000", 2024-09-15T03:47:08.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7092000000000003e-05 2024-09-15T03:47:08.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.091 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:08.091 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845591+0000", 2024-09-15T03:47:08.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1824999999999998e-05 2024-09-15T03:47:08.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.091 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:08.091 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845612+0000", 2024-09-15T03:47:08.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.084e-05 2024-09-15T03:47:08.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.092 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:08.092 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845640+0000", 2024-09-15T03:47:08.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0000281439999998 2024-09-15T03:47:08.092 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.092 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:08.092 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.092 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=81, reqid=client.4521.0:321, at_version=49'16, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:47:08.092 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.845113+0000", 2024-09-15T03:47:08.092 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.227934884, 2024-09-15T03:47:08.093 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.999459995, 2024-09-15T03:47:08.093 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:08.093 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:08.093 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:08.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.093 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:08.093 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845113+0000", 2024-09-15T03:47:08.093 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.093 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.093 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:08.093 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845113+0000", 2024-09-15T03:47:08.093 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:08.094 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845114+0000", 2024-09-15T03:47:08.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0210000000000001e-06 2024-09-15T03:47:08.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:08.094 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845422+0000", 2024-09-15T03:47:08.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00030762999999999998 2024-09-15T03:47:08.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:08.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845436+0000", 2024-09-15T03:47:08.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3998999999999999e-05 2024-09-15T03:47:08.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.095 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:08.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845445+0000", 2024-09-15T03:47:08.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5799999999999998e-06 2024-09-15T03:47:08.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.095 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:08.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845538+0000", 2024-09-15T03:47:08.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2739999999999996e-05 2024-09-15T03:47:08.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:08.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845548+0000", 2024-09-15T03:47:08.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0548e-05 2024-09-15T03:47:08.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:08.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845803+0000", 2024-09-15T03:47:08.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00025495599999999998 2024-09-15T03:47:08.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:08.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845998+0000", 2024-09-15T03:47:08.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019432799999999999 2024-09-15T03:47:08.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:08.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846122+0000", 2024-09-15T03:47:08.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012434600000000001 2024-09-15T03:47:08.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:08.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846178+0000", 2024-09-15T03:47:08.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5943999999999998e-05 2024-09-15T03:47:08.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:08.098 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.844573+0000", 2024-09-15T03:47:08.098 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9983949029999999 2024-09-15T03:47:08.098 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.098 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:08.098 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.098 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=82, reqid=client.4521.0:322, at_version=49'17, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:47:08.098 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.846151+0000", 2024-09-15T03:47:08.098 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.226896862, 2024-09-15T03:47:08.098 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9989014119999999, 2024-09-15T03:47:08.098 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:08.098 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:08.098 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:08.099 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.099 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:08.099 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846151+0000", 2024-09-15T03:47:08.099 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.099 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.099 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:08.099 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846151+0000", 2024-09-15T03:47:08.099 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.099 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.099 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:08.099 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846152+0000", 2024-09-15T03:47:08.099 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0800000000000004e-07 2024-09-15T03:47:08.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.100 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:08.100 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846238+0000", 2024-09-15T03:47:08.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5866999999999996e-05 2024-09-15T03:47:08.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.100 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:08.100 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846255+0000", 2024-09-15T03:47:08.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6954999999999999e-05 2024-09-15T03:47:08.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.101 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:08.101 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846262+0000", 2024-09-15T03:47:08.101 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7680000000000008e-06 2024-09-15T03:47:08.101 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.101 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:08.101 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846325+0000", 2024-09-15T03:47:08.101 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2191000000000001e-05 2024-09-15T03:47:08.101 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:08.102 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846335+0000", 2024-09-15T03:47:08.102 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0767e-05 2024-09-15T03:47:08.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:08.102 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846446+0000", 2024-09-15T03:47:08.102 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110522 2024-09-15T03:47:08.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:08.103 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846458+0000", 2024-09-15T03:47:08.103 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2023e-05 2024-09-15T03:47:08.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.103 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.103 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:08.103 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846508+0000", 2024-09-15T03:47:08.103 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0479000000000002e-05 2024-09-15T03:47:08.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.103 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.103 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:08.103 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846526+0000", 2024-09-15T03:47:08.103 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7836e-05 2024-09-15T03:47:08.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.103 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.104 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:08.104 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845052+0000", 2024-09-15T03:47:08.104 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9985261960000003 2024-09-15T03:47:08.104 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.104 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:08.104 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.104 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.104 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.104 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=31, reqid=client.4521.0:325, at_version=0'0, trim_to=0'0, roll_forward_to=49'12))", 2024-09-15T03:47:08.104 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.847407+0000", 2024-09-15T03:47:08.104 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.225640659, 2024-09-15T03:47:08.104 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9983494909999999, 2024-09-15T03:47:08.104 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:08.104 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:08.105 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:08.105 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.105 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:08.105 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847407+0000", 2024-09-15T03:47:08.105 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.105 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.105 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.105 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:08.105 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847407+0000", 2024-09-15T03:47:08.105 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.105 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.105 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.105 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:08.106 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847408+0000", 2024-09-15T03:47:08.106 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0070000000000001e-06 2024-09-15T03:47:08.106 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.106 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.106 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:08.106 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847421+0000", 2024-09-15T03:47:08.106 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3203999999999999e-05 2024-09-15T03:47:08.106 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.106 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.106 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:08.106 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847443+0000", 2024-09-15T03:47:08.106 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1539000000000002e-05 2024-09-15T03:47:08.106 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.106 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.106 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:08.107 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847452+0000", 2024-09-15T03:47:08.107 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6570000000000003e-06 2024-09-15T03:47:08.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.107 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:08.107 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847534+0000", 2024-09-15T03:47:08.107 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2309000000000001e-05 2024-09-15T03:47:08.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.107 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:08.107 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847544+0000", 2024-09-15T03:47:08.107 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0482000000000001e-05 2024-09-15T03:47:08.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.107 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:08.107 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847645+0000", 2024-09-15T03:47:08.108 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010021700000000001 2024-09-15T03:47:08.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.108 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:08.108 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847665+0000", 2024-09-15T03:47:08.108 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0271999999999998e-05 2024-09-15T03:47:08.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.108 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:08.108 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847708+0000", 2024-09-15T03:47:08.108 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2828999999999999e-05 2024-09-15T03:47:08.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.108 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:08.109 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847724+0000", 2024-09-15T03:47:08.109 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6334e-05 2024-09-15T03:47:08.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.109 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:08.109 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845757+0000", 2024-09-15T03:47:08.109 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.998032641 2024-09-15T03:47:08.109 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.109 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:08.109 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.109 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=83, reqid=client.4521.0:323, at_version=49'18, trim_to=0'0, roll_forward_to=49'15))", 2024-09-15T03:47:08.109 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.847691+0000", 2024-09-15T03:47:08.110 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.225356811999999, 2024-09-15T03:47:08.110 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9981305969999998, 2024-09-15T03:47:08.110 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:08.110 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:08.110 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:08.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.110 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:08.110 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847691+0000", 2024-09-15T03:47:08.110 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.110 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:08.110 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847691+0000", 2024-09-15T03:47:08.110 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.111 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:08.111 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847692+0000", 2024-09-15T03:47:08.111 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9899999999999999e-07 2024-09-15T03:47:08.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.111 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:08.111 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847715+0000", 2024-09-15T03:47:08.111 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2818000000000001e-05 2024-09-15T03:47:08.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.111 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:08.111 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847731+0000", 2024-09-15T03:47:08.111 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6401000000000001e-05 2024-09-15T03:47:08.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.112 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:08.112 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847738+0000", 2024-09-15T03:47:08.112 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3699999999999997e-06 2024-09-15T03:47:08.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.112 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:08.112 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847791+0000", 2024-09-15T03:47:08.112 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2584999999999999e-05 2024-09-15T03:47:08.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.113 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.113 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:08.113 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847801+0000", 2024-09-15T03:47:08.113 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0295999999999999e-05 2024-09-15T03:47:08.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.113 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.113 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:08.113 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847931+0000", 2024-09-15T03:47:08.113 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0001295 2024-09-15T03:47:08.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.113 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.113 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:08.114 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847946+0000", 2024-09-15T03:47:08.114 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4732000000000001e-05 2024-09-15T03:47:08.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.114 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:08.114 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847989+0000", 2024-09-15T03:47:08.114 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3625000000000003e-05 2024-09-15T03:47:08.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.114 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:08.114 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.848004+0000", 2024-09-15T03:47:08.114 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4385e-05 2024-09-15T03:47:08.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.115 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:08.115 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845822+0000", 2024-09-15T03:47:08.115 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9978179860000003 2024-09-15T03:47:08.115 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.115 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:08.115 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.115 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=84, reqid=client.4521.0:323, at_version=0'0, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:08.115 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.849295+0000", 2024-09-15T03:47:08.115 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.223752731999999, 2024-09-15T03:47:08.115 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9965517450000001, 2024-09-15T03:47:08.116 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:08.116 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:08.116 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:08.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.116 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:08.116 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849295+0000", 2024-09-15T03:47:08.116 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.116 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:08.116 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849295+0000", 2024-09-15T03:47:08.116 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.117 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:08.117 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849296+0000", 2024-09-15T03:47:08.117 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6300000000000004e-07 2024-09-15T03:47:08.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.117 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.117 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:08.117 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849308+0000", 2024-09-15T03:47:08.117 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2261e-05 2024-09-15T03:47:08.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.117 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.117 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:08.117 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849325+0000", 2024-09-15T03:47:08.117 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6588000000000001e-05 2024-09-15T03:47:08.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.118 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:08.118 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849332+0000", 2024-09-15T03:47:08.118 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.464e-06 2024-09-15T03:47:08.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.118 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:08.118 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849379+0000", 2024-09-15T03:47:08.118 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6498000000000001e-05 2024-09-15T03:47:08.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.119 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:08.119 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849391+0000", 2024-09-15T03:47:08.119 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1926000000000001e-05 2024-09-15T03:47:08.119 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.119 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:08.119 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849497+0000", 2024-09-15T03:47:08.119 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010634200000000001 2024-09-15T03:47:08.119 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.119 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:08.119 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849516+0000", 2024-09-15T03:47:08.119 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8995999999999999e-05 2024-09-15T03:47:08.119 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.120 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:08.120 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849565+0000", 2024-09-15T03:47:08.120 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8735e-05 2024-09-15T03:47:08.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.120 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.120 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:08.120 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849577+0000", 2024-09-15T03:47:08.120 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2585999999999999e-05 2024-09-15T03:47:08.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.120 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.120 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:08.120 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845847+0000", 2024-09-15T03:47:08.120 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9962694860000001 2024-09-15T03:47:08.120 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.120 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:08.121 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.121 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.121 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.121 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpRead(3.7s5 49/31 ECSubRead(tid=85, to_read={3:e40d1694:::smithi00132172-50:head=0,4096,0}, subchunks={3:e40d1694:::smithi00132172-50:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:47:08.121 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.849814+0000", 2024-09-15T03:47:08.121 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.223233418, 2024-09-15T03:47:08.121 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9961114690000001, 2024-09-15T03:47:08.121 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:08.121 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:08.121 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:08.121 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.121 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:08.121 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849814+0000", 2024-09-15T03:47:08.121 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.122 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.122 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:08.122 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849814+0000", 2024-09-15T03:47:08.122 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.122 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.122 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:08.122 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849815+0000", 2024-09-15T03:47:08.122 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6499999999999999e-07 2024-09-15T03:47:08.122 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.122 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:08.122 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849826+0000", 2024-09-15T03:47:08.122 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0635000000000001e-05 2024-09-15T03:47:08.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.123 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.123 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:08.123 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.850154+0000", 2024-09-15T03:47:08.123 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00032866900000000002 2024-09-15T03:47:08.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.123 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.123 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:08.123 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.850169+0000", 2024-09-15T03:47:08.123 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5051e-05 2024-09-15T03:47:08.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.123 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.124 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:08.124 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.850226+0000", 2024-09-15T03:47:08.124 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6926999999999999e-05 2024-09-15T03:47:08.124 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.124 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.124 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:08.124 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845926+0000", 2024-09-15T03:47:08.124 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9956995219999998 2024-09-15T03:47:08.124 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.124 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:08.124 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.124 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.124 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.124 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=32, reqid=client.4521.0:329, at_version=49'13, trim_to=0'0, roll_forward_to=49'12))", 2024-09-15T03:47:08.125 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.853281+0000", 2024-09-15T03:47:08.125 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.219766932000001, 2024-09-15T03:47:08.125 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9927023569999998, 2024-09-15T03:47:08.125 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:08.125 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:08.125 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:08.125 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.125 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:08.125 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853281+0000", 2024-09-15T03:47:08.125 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.125 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.125 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.125 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:08.125 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853281+0000", 2024-09-15T03:47:08.126 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.126 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.126 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:08.126 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853282+0000", 2024-09-15T03:47:08.126 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.347e-06 2024-09-15T03:47:08.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.126 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.126 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:08.126 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853401+0000", 2024-09-15T03:47:08.126 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118711 2024-09-15T03:47:08.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.126 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.127 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:08.127 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853441+0000", 2024-09-15T03:47:08.127 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0059999999999999e-05 2024-09-15T03:47:08.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.127 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:08.127 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853451+0000", 2024-09-15T03:47:08.127 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5850000000000006e-06 2024-09-15T03:47:08.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.127 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:08.127 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853617+0000", 2024-09-15T03:47:08.127 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016655899999999999 2024-09-15T03:47:08.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.128 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:08.128 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853632+0000", 2024-09-15T03:47:08.128 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4605e-05 2024-09-15T03:47:08.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.128 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:08.128 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853842+0000", 2024-09-15T03:47:08.128 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000210523 2024-09-15T03:47:08.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.128 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:08.129 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853871+0000", 2024-09-15T03:47:08.129 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8441000000000001e-05 2024-09-15T03:47:08.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.129 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:08.129 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854188+0000", 2024-09-15T03:47:08.129 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031700800000000003 2024-09-15T03:47:08.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.129 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:08.129 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854209+0000", 2024-09-15T03:47:08.129 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0795999999999999e-05 2024-09-15T03:47:08.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.130 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:08.130 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845983+0000", 2024-09-15T03:47:08.130 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9917747220000002 2024-09-15T03:47:08.130 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.130 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:08.130 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.130 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=33, reqid=client.4521.0:329, at_version=0'0, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:47:08.130 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.854592+0000", 2024-09-15T03:47:08.130 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.218455921, 2024-09-15T03:47:08.130 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9914290499999998, 2024-09-15T03:47:08.130 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:08.130 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:08.130 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:08.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.131 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:08.131 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854592+0000", 2024-09-15T03:47:08.131 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.131 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:08.131 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854592+0000", 2024-09-15T03:47:08.131 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.131 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:08.131 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854593+0000", 2024-09-15T03:47:08.132 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1200000000000002e-07 2024-09-15T03:47:08.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.132 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:08.132 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854605+0000", 2024-09-15T03:47:08.132 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1946999999999999e-05 2024-09-15T03:47:08.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.132 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:08.132 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854624+0000", 2024-09-15T03:47:08.132 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9502000000000001e-05 2024-09-15T03:47:08.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.132 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:08.133 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854631+0000", 2024-09-15T03:47:08.133 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2490000000000004e-06 2024-09-15T03:47:08.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.133 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:08.133 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854683+0000", 2024-09-15T03:47:08.133 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1643e-05 2024-09-15T03:47:08.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.133 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:08.133 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854693+0000", 2024-09-15T03:47:08.133 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8199999999999992e-06 2024-09-15T03:47:08.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.134 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:08.134 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854768+0000", 2024-09-15T03:47:08.134 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5104999999999993e-05 2024-09-15T03:47:08.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.134 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:08.134 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854830+0000", 2024-09-15T03:47:08.134 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2191000000000001e-05 2024-09-15T03:47:08.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.134 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:08.134 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854934+0000", 2024-09-15T03:47:08.134 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010424099999999999 2024-09-15T03:47:08.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.135 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:08.135 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854952+0000", 2024-09-15T03:47:08.135 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7521e-05 2024-09-15T03:47:08.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.135 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:08.135 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.846021+0000", 2024-09-15T03:47:08.135 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9910690190000002 2024-09-15T03:47:08.135 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.135 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:08.135 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.136 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=34, reqid=client.4521.0:330, at_version=49'14, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:47:08.136 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.857132+0000", 2024-09-15T03:47:08.136 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.215915539999999, 2024-09-15T03:47:08.136 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9890264310000001, 2024-09-15T03:47:08.136 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:08.136 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:08.136 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:08.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.136 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:08.136 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857132+0000", 2024-09-15T03:47:08.136 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.137 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:08.137 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857132+0000", 2024-09-15T03:47:08.137 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.137 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:08.137 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857133+0000", 2024-09-15T03:47:08.137 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.61e-07 2024-09-15T03:47:08.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.138 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:08.138 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857223+0000", 2024-09-15T03:47:08.138 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0476999999999994e-05 2024-09-15T03:47:08.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.138 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.138 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:08.138 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857245+0000", 2024-09-15T03:47:08.138 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1415000000000001e-05 2024-09-15T03:47:08.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.138 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.138 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:08.138 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857254+0000", 2024-09-15T03:47:08.139 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.329e-06 2024-09-15T03:47:08.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.139 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:08.139 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857312+0000", 2024-09-15T03:47:08.139 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7865000000000002e-05 2024-09-15T03:47:08.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.139 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:08.139 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857323+0000", 2024-09-15T03:47:08.139 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0655999999999999e-05 2024-09-15T03:47:08.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.139 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:08.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857817+0000", 2024-09-15T03:47:08.140 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00049391300000000001 2024-09-15T03:47:08.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.140 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:08.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857946+0000", 2024-09-15T03:47:08.140 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012926699999999999 2024-09-15T03:47:08.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.140 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:08.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858076+0000", 2024-09-15T03:47:08.140 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012957600000000001 2024-09-15T03:47:08.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.141 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:08.141 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858094+0000", 2024-09-15T03:47:08.141 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8715e-05 2024-09-15T03:47:08.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.141 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:08.141 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.846159+0000", 2024-09-15T03:47:08.141 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9880644570000001 2024-09-15T03:47:08.141 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.141 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:08.141 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.142 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=86, reqid=client.4521.0:334, at_version=49'20, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:08.142 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.857463+0000", 2024-09-15T03:47:08.142 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.215585222, 2024-09-15T03:47:08.142 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0025860940000002, 2024-09-15T03:47:08.142 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:08.142 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:08.142 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:08.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.142 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:08.142 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857463+0000", 2024-09-15T03:47:08.142 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.143 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:08.143 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857463+0000", 2024-09-15T03:47:08.143 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.143 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:08.143 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857464+0000", 2024-09-15T03:47:08.143 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0210000000000001e-06 2024-09-15T03:47:08.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.143 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:08.143 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857489+0000", 2024-09-15T03:47:08.144 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5500999999999999e-05 2024-09-15T03:47:08.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.144 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:08.144 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857508+0000", 2024-09-15T03:47:08.144 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9025000000000002e-05 2024-09-15T03:47:08.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.144 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:08.144 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857515+0000", 2024-09-15T03:47:08.144 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9759999999999998e-06 2024-09-15T03:47:08.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:08.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845961+0000", 2024-09-15T03:47:08.145 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9884457740000001 2024-09-15T03:47:08.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:08.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845966+0000", 2024-09-15T03:47:08.145 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3750000000000002e-06 2024-09-15T03:47:08.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:08.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.846127+0000", 2024-09-15T03:47:08.146 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000160462 2024-09-15T03:47:08.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.146 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:08.146 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.846136+0000", 2024-09-15T03:47:08.146 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6260000000000007e-06 2024-09-15T03:47:08.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.146 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:08.146 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.846191+0000", 2024-09-15T03:47:08.146 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4274999999999998e-05 2024-09-15T03:47:08.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.147 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:08.147 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.853397+0000", 2024-09-15T03:47:08.147 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0072068519999999997 2024-09-15T03:47:08.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.147 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:08.147 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.860049+0000", 2024-09-15T03:47:08.147 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0066512070000000001 2024-09-15T03:47:08.147 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.147 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:08.147 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.148 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.cs3 49/28 ECSubWrite(tid=15, reqid=client.4521.0:337, at_version=49'10, trim_to=0'0, roll_forward_to=43'8))", 2024-09-15T03:47:08.148 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.857712+0000", 2024-09-15T03:47:08.148 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.215335903, 2024-09-15T03:47:08.148 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9893153209999999, 2024-09-15T03:47:08.148 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:08.148 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:08.148 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:08.148 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.148 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:08.148 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857712+0000", 2024-09-15T03:47:08.148 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.148 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.148 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.149 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:08.149 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857712+0000", 2024-09-15T03:47:08.149 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.149 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:08.149 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857712+0000", 2024-09-15T03:47:08.149 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8000000000000002e-07 2024-09-15T03:47:08.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.149 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:08.149 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857755+0000", 2024-09-15T03:47:08.149 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2960999999999997e-05 2024-09-15T03:47:08.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.150 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:08.150 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857773+0000", 2024-09-15T03:47:08.150 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7635999999999999e-05 2024-09-15T03:47:08.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.150 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:08.150 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857780+0000", 2024-09-15T03:47:08.150 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.024e-06 2024-09-15T03:47:08.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.150 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:08.150 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857883+0000", 2024-09-15T03:47:08.150 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103224 2024-09-15T03:47:08.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.151 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:08.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857893+0000", 2024-09-15T03:47:08.151 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0265000000000001e-05 2024-09-15T03:47:08.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.151 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:08.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858180+0000", 2024-09-15T03:47:08.151 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000286602 2024-09-15T03:47:08.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.151 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:08.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858193+0000", 2024-09-15T03:47:08.151 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3342e-05 2024-09-15T03:47:08.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.152 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:08.152 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858233+0000", 2024-09-15T03:47:08.152 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9811999999999998e-05 2024-09-15T03:47:08.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.152 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:08.152 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858253+0000", 2024-09-15T03:47:08.152 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0259e-05 2024-09-15T03:47:08.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.152 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:08.153 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.847027+0000", 2024-09-15T03:47:08.153 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.988773916 2024-09-15T03:47:08.153 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.153 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:08.153 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.153 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=87, reqid=client.4521.0:335, at_version=49'21, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:08.153 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.858481+0000", 2024-09-15T03:47:08.153 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.214566754, 2024-09-15T03:47:08.153 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.004544015, 2024-09-15T03:47:08.153 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:08.153 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:08.154 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:08.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.154 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:08.154 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858481+0000", 2024-09-15T03:47:08.154 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.154 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.154 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:08.154 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858481+0000", 2024-09-15T03:47:08.154 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.154 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.154 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:08.154 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858482+0000", 2024-09-15T03:47:08.155 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.09e-07 2024-09-15T03:47:08.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.155 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:08.155 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858505+0000", 2024-09-15T03:47:08.155 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3133e-05 2024-09-15T03:47:08.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.155 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:08.155 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858524+0000", 2024-09-15T03:47:08.155 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8706e-05 2024-09-15T03:47:08.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.155 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:08.156 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858532+0000", 2024-09-15T03:47:08.156 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5539999999999998e-06 2024-09-15T03:47:08.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.156 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:08.156 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851213+0000", 2024-09-15T03:47:08.156 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.992680526 2024-09-15T03:47:08.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.156 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:08.156 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851221+0000", 2024-09-15T03:47:08.156 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5380000000000002e-06 2024-09-15T03:47:08.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.157 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:08.157 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851309+0000", 2024-09-15T03:47:08.157 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8069e-05 2024-09-15T03:47:08.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.157 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:08.157 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851495+0000", 2024-09-15T03:47:08.157 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018516999999999999 2024-09-15T03:47:08.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.157 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:08.157 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851557+0000", 2024-09-15T03:47:08.157 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2031000000000003e-05 2024-09-15T03:47:08.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.158 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:08.158 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.853665+0000", 2024-09-15T03:47:08.158 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021079499999999999 2024-09-15T03:47:08.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.158 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:08.158 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.863025+0000", 2024-09-15T03:47:08.158 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0093605289999999994 2024-09-15T03:47:08.158 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.158 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:08.158 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.159 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.cs3 49/28 ECSubWrite(tid=16, reqid=client.4521.0:337, at_version=0'0, trim_to=0'0, roll_forward_to=49'10))", 2024-09-15T03:47:08.159 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.859219+0000", 2024-09-15T03:47:08.159 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.213828783, 2024-09-15T03:47:08.159 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9879305710000001, 2024-09-15T03:47:08.159 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:08.159 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:08.159 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:08.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.159 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:08.159 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859219+0000", 2024-09-15T03:47:08.159 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.159 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:08.160 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859219+0000", 2024-09-15T03:47:08.160 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.160 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:08.160 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859220+0000", 2024-09-15T03:47:08.160 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3300000000000001e-07 2024-09-15T03:47:08.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.160 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:08.160 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859233+0000", 2024-09-15T03:47:08.160 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2781e-05 2024-09-15T03:47:08.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:08.161 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859251+0000", 2024-09-15T03:47:08.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8770000000000002e-05 2024-09-15T03:47:08.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:08.161 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859259+0000", 2024-09-15T03:47:08.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7309999999999993e-06 2024-09-15T03:47:08.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:08.161 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859342+0000", 2024-09-15T03:47:08.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2909000000000002e-05 2024-09-15T03:47:08.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.162 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:08.162 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859357+0000", 2024-09-15T03:47:08.162 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5245000000000001e-05 2024-09-15T03:47:08.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.162 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:08.162 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859453+0000", 2024-09-15T03:47:08.162 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5878999999999995e-05 2024-09-15T03:47:08.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.162 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:08.162 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859475+0000", 2024-09-15T03:47:08.162 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1591999999999999e-05 2024-09-15T03:47:08.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.163 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:08.163 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859518+0000", 2024-09-15T03:47:08.163 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3021e-05 2024-09-15T03:47:08.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.163 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:08.163 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859536+0000", 2024-09-15T03:47:08.163 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8420999999999999e-05 2024-09-15T03:47:08.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.163 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:08.163 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.847150+0000", 2024-09-15T03:47:08.164 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9876134890000001 2024-09-15T03:47:08.164 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.164 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:08.164 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.164 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=88, reqid=client.4521.0:336, at_version=49'22, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:08.164 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.859709+0000", 2024-09-15T03:47:08.164 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.213339006, 2024-09-15T03:47:08.164 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0037862020000001, 2024-09-15T03:47:08.164 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:08.164 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:08.165 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:08.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.165 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:08.165 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859709+0000", 2024-09-15T03:47:08.165 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.165 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:08.165 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859709+0000", 2024-09-15T03:47:08.165 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.165 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:08.165 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859710+0000", 2024-09-15T03:47:08.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.052e-06 2024-09-15T03:47:08.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.166 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:08.166 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859729+0000", 2024-09-15T03:47:08.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9409999999999999e-05 2024-09-15T03:47:08.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.166 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:08.166 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859759+0000", 2024-09-15T03:47:08.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9998e-05 2024-09-15T03:47:08.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.167 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:08.167 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859767+0000", 2024-09-15T03:47:08.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.14e-06 2024-09-15T03:47:08.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.167 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:08.167 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851350+0000", 2024-09-15T03:47:08.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.99158261 2024-09-15T03:47:08.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.167 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:08.167 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851358+0000", 2024-09-15T03:47:08.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1070000000000005e-06 2024-09-15T03:47:08.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.168 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:08.168 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851443+0000", 2024-09-15T03:47:08.168 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4760999999999997e-05 2024-09-15T03:47:08.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.168 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:08.168 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851495+0000", 2024-09-15T03:47:08.168 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2070999999999997e-05 2024-09-15T03:47:08.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.168 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:08.168 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851557+0000", 2024-09-15T03:47:08.169 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1829999999999996e-05 2024-09-15T03:47:08.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.169 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:08.169 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.853682+0000", 2024-09-15T03:47:08.169 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021249110000000002 2024-09-15T03:47:08.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.169 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:08.169 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.863495+0000", 2024-09-15T03:47:08.169 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0098133119999999994 2024-09-15T03:47:08.169 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.169 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:08.169 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.170 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=35, reqid=client.4521.0:331, at_version=49'15, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:47:08.170 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.860778+0000", 2024-09-15T03:47:08.170 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.212270111, 2024-09-15T03:47:08.170 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9863441239999999, 2024-09-15T03:47:08.170 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:08.170 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:08.170 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:08.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.170 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:08.170 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860778+0000", 2024-09-15T03:47:08.170 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.171 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:08.171 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860778+0000", 2024-09-15T03:47:08.171 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.171 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:08.171 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860779+0000", 2024-09-15T03:47:08.171 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0109999999999999e-06 2024-09-15T03:47:08.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.171 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:08.172 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860922+0000", 2024-09-15T03:47:08.172 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014357399999999999 2024-09-15T03:47:08.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.172 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:08.172 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860944+0000", 2024-09-15T03:47:08.172 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1424e-05 2024-09-15T03:47:08.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.172 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:08.172 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860951+0000", 2024-09-15T03:47:08.172 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5610000000000001e-06 2024-09-15T03:47:08.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.173 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:08.173 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861022+0000", 2024-09-15T03:47:08.173 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1126999999999998e-05 2024-09-15T03:47:08.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.173 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:08.173 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861034+0000", 2024-09-15T03:47:08.173 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1810999999999999e-05 2024-09-15T03:47:08.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.174 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:08.174 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861164+0000", 2024-09-15T03:47:08.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000129727 2024-09-15T03:47:08.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.175 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:08.175 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861196+0000", 2024-09-15T03:47:08.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2240000000000003e-05 2024-09-15T03:47:08.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.175 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:08.175 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861259+0000", 2024-09-15T03:47:08.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2454999999999997e-05 2024-09-15T03:47:08.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.176 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:08.176 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861274+0000", 2024-09-15T03:47:08.176 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5588000000000001e-05 2024-09-15T03:47:08.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.176 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:08.176 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.847122+0000", 2024-09-15T03:47:08.176 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9858476060000001 2024-09-15T03:47:08.176 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.176 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:08.176 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.177 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=36, reqid=client.4521.0:332, at_version=49'16, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:47:08.177 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.861537+0000", 2024-09-15T03:47:08.177 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.211510548, 2024-09-15T03:47:08.177 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.985662638, 2024-09-15T03:47:08.177 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:08.177 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:08.177 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:08.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.177 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:08.177 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861537+0000", 2024-09-15T03:47:08.177 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.178 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:08.178 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861537+0000", 2024-09-15T03:47:08.178 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.178 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:08.178 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861538+0000", 2024-09-15T03:47:08.178 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7000000000000003e-07 2024-09-15T03:47:08.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.178 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:08.179 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861571+0000", 2024-09-15T03:47:08.179 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2493000000000002e-05 2024-09-15T03:47:08.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.179 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:08.179 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861588+0000", 2024-09-15T03:47:08.179 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7076e-05 2024-09-15T03:47:08.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.179 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:08.179 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861595+0000", 2024-09-15T03:47:08.179 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3440000000000003e-06 2024-09-15T03:47:08.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.180 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:08.180 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861665+0000", 2024-09-15T03:47:08.180 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9814999999999995e-05 2024-09-15T03:47:08.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.180 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:08.180 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861674+0000", 2024-09-15T03:47:08.180 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9740000000000008e-06 2024-09-15T03:47:08.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.180 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:08.180 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861767+0000", 2024-09-15T03:47:08.180 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3572999999999993e-05 2024-09-15T03:47:08.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:08.181 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861778+0000", 2024-09-15T03:47:08.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1083e-05 2024-09-15T03:47:08.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:08.181 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861888+0000", 2024-09-15T03:47:08.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010937700000000001 2024-09-15T03:47:08.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:08.182 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861906+0000", 2024-09-15T03:47:08.182 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.863e-05 2024-09-15T03:47:08.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.182 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:08.182 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.847200+0000", 2024-09-15T03:47:08.182 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.985293403 2024-09-15T03:47:08.182 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.182 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:08.182 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.182 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=37, reqid=client.4521.0:332, at_version=0'0, trim_to=0'0, roll_forward_to=49'16))", 2024-09-15T03:47:08.182 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.864457+0000", 2024-09-15T03:47:08.183 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.208590858999999, 2024-09-15T03:47:08.183 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9827806690000003, 2024-09-15T03:47:08.183 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:08.183 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:08.183 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:08.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.183 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:08.183 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864457+0000", 2024-09-15T03:47:08.183 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.183 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:08.184 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864457+0000", 2024-09-15T03:47:08.184 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.184 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:08.184 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864458+0000", 2024-09-15T03:47:08.184 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.029e-06 2024-09-15T03:47:08.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.184 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:08.184 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864472+0000", 2024-09-15T03:47:08.184 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4287000000000001e-05 2024-09-15T03:47:08.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:08.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864504+0000", 2024-09-15T03:47:08.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1435e-05 2024-09-15T03:47:08.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:08.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864511+0000", 2024-09-15T03:47:08.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4900000000000003e-06 2024-09-15T03:47:08.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:08.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864583+0000", 2024-09-15T03:47:08.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1828000000000004e-05 2024-09-15T03:47:08.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:08.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864593+0000", 2024-09-15T03:47:08.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0427e-05 2024-09-15T03:47:08.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:08.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864701+0000", 2024-09-15T03:47:08.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000107969 2024-09-15T03:47:08.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:08.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864721+0000", 2024-09-15T03:47:08.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9422000000000001e-05 2024-09-15T03:47:08.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:08.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864798+0000", 2024-09-15T03:47:08.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6731999999999996e-05 2024-09-15T03:47:08.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:08.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864815+0000", 2024-09-15T03:47:08.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7286999999999999e-05 2024-09-15T03:47:08.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.188 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:08.188 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.847238+0000", 2024-09-15T03:47:08.188 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9824227629999998 2024-09-15T03:47:08.188 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.188 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:08.188 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.188 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.bs4 49/31 ECSubWrite(tid=75, reqid=client.4521.0:338, at_version=49'58, trim_to=0'0, roll_forward_to=47'56))", 2024-09-15T03:47:08.188 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.870036+0000", 2024-09-15T03:47:08.188 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 12.203012234999999, 2024-09-15T03:47:08.188 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.978548666, 2024-09-15T03:47:08.188 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:08.188 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:08.189 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:08.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.189 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:08.189 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870036+0000", 2024-09-15T03:47:08.189 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.189 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:08.189 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870036+0000", 2024-09-15T03:47:08.189 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:08.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.189 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:08.190 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870036+0000", 2024-09-15T03:47:08.190 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.61e-07 2024-09-15T03:47:08.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.190 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:08.190 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870051+0000", 2024-09-15T03:47:08.190 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5074e-05 2024-09-15T03:47:08.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.190 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:08.190 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870061+0000", 2024-09-15T03:47:08.190 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.066e-05 2024-09-15T03:47:08.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.191 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:08.191 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870067+0000", 2024-09-15T03:47:08.191 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1580000000000004e-06 2024-09-15T03:47:08.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.191 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:08.191 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870121+0000", 2024-09-15T03:47:08.191 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4605999999999999e-05 2024-09-15T03:47:08.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.191 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:08.191 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870134+0000", 2024-09-15T03:47:08.191 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2748999999999999e-05 2024-09-15T03:47:08.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870450+0000", 2024-09-15T03:47:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031624399999999998 2024-09-15T03:47:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870459+0000", 2024-09-15T03:47:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0499999999999997e-06 2024-09-15T03:47:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870496+0000", 2024-09-15T03:47:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6548999999999999e-05 2024-09-15T03:47:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870509+0000", 2024-09-15T03:47:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3216e-05 2024-09-15T03:47:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.848584+0000", 2024-09-15T03:47:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9780751990000001 2024-09-15T03:47:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:08.194 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:08.194 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:08.195 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:47:08.196 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:08.194+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:08.296 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:08.295+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:08.324 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:47:08.337 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:08.337 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:08.337 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-15T03:47:08.337 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:08.346 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:47:08.396 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:08.396+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:08.441 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:08.441 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:08.455 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:08.455 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:08.471 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:08.471 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:08.485 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:08.486 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:08.493 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:08.493 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:08.498 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:47:08.498 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:08.497+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:08.501 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:08.501 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:08.516 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:08.516 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:08.531 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:08.532 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:08.546 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:08.547 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:08.561 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:08.562 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:08.577 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:08.577 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:08.592 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:47:08.592 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:47:08.598 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:08.598+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:08.601 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:08.602 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:08.607 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:08.607 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:08.610 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:47:08.701 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:08.700+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:08.758 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:08.759 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:08.766 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:47:08.808 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:08.807+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:08.892 INFO:tasks.rados.rados.0.smithi001.stdout:175: done (1 left) 2024-09-15T03:47:08.892 INFO:tasks.rados.rados.0.smithi001.stdout:176: setattr oid 29 current snap is 19 2024-09-15T03:47:08.892 INFO:tasks.rados.rados.0.smithi001.stdout:177: snap_create 2024-09-15T03:47:08.893 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-15T03:47:08.901 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:08.900+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:08.918 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:08.918 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:08.927 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:47:09.011 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:09.010+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:09.068 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:09.068 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:09.077 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:47:09.121 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:09.120+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:09.191 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:09.191 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:09.199 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:47:09.203 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:09.202+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:09.304 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:09.303+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:09.343 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:09.343 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:09.351 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:47:09.405 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:09.404+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:09.451 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:09.452 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:09.460 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:47:09.505 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:09.504+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:09.560 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:09.560 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:09.569 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:47:09.607 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:09.606+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:09.708 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:09.707+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:09.709 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:09.709 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:09.718 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:47:09.809 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:09.809+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:09.818 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:09.818 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:09.827 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:47:09.833 INFO:tasks.rados.rados.0.smithi001.stdout:177: done (2 left) 2024-09-15T03:47:09.833 INFO:tasks.rados.rados.0.smithi001.stdout:178: read oid 48 snap -1 2024-09-15T03:47:09.834 INFO:tasks.rados.rados.0.smithi001.stdout:178: expect (ObjNum 47 snap 0 seq_num 47) 2024-09-15T03:47:09.834 INFO:tasks.rados.rados.0.smithi001.stdout:179: snap_create 2024-09-15T03:47:09.851 INFO:tasks.rados.rados.0.smithi001.stdout:178: expect (ObjNum 47 snap 0 seq_num 47) 2024-09-15T03:47:09.881 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 29 v 70 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-15T03:47:09.882 INFO:tasks.rados.rados.0.smithi001.stdout:174: finishing copy_from racing read to smithi00132172-49 2024-09-15T03:47:09.882 INFO:tasks.rados.rados.0.smithi001.stdout:174: finishing copy_from to smithi00132172-49 2024-09-15T03:47:09.882 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 49 v 20 (ObjNum 79 snap 16 seq_num 79) dirty exists 2024-09-15T03:47:09.910 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:09.909+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:09.959 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:09.960 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:09.968 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:47:10.011 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:10.011+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:10.068 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:10.069 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:10.077 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:47:10.112 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:10.112+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:10.178 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:10.179 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:10.186 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:47:10.231 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:10.230+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:10.287 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:10.287 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:10.295 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:47:10.314 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:10.314+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:10.415 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:10.415+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:10.437 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:10.437 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:10.445 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:47:10.516 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:10.516+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:10.545 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:10.545 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:10.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:47:10.617 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:10.617+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:10.619 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:47:10.663 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:10.663 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:10.672 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:47:10.718 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:10.717+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:10.748 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:10.748 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:10.762 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:10.762 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:10.768 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:10.769 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:10.777 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:47:10.778 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:10.778 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:10.791 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:10.791 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:10.806 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:10.806 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:10.818 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:10.818+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:10.820 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:10.820 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:10.833 INFO:tasks.rados.rados.0.smithi001.stdout:174: done (3 left) 2024-09-15T03:47:10.833 INFO:tasks.rados.rados.0.smithi001.stdout:176: done (2 left) 2024-09-15T03:47:10.833 INFO:tasks.rados.rados.0.smithi001.stdout:178: done (1 left) 2024-09-15T03:47:10.833 INFO:tasks.rados.rados.0.smithi001.stdout:179: done (0 left) 2024-09-15T03:47:10.833 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 18 current snap is 21 2024-09-15T03:47:10.833 INFO:tasks.rados.rados.0.smithi001.stdout:180: seq_num 90 ranges {294912=753664,1048576=720896,1769472=49152} 2024-09-15T03:47:10.840 INFO:tasks.rados.rados.0.smithi001.stdout:180: writing smithi00132172-18 from 294912 to 1048576 tid 1 2024-09-15T03:47:10.845 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:10.846 INFO:tasks.rados.rados.0.smithi001.stdout:180: writing smithi00132172-18 from 1048576 to 1769472 tid 2 2024-09-15T03:47:10.847 INFO:tasks.rados.rados.0.smithi001.stdout:180: writing smithi00132172-18 from 1769472 to 1818624 tid 3 2024-09-15T03:47:10.847 INFO:tasks.rados.rados.0.smithi001.stdout:181: rollback oid 47 current snap is 21 2024-09-15T03:47:10.847 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 47 to 7 2024-09-15T03:47:10.847 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 35 current snap is 21 2024-09-15T03:47:10.847 INFO:tasks.rados.rados.0.smithi001.stdout:182: seq_num 91 ranges {720896=770048,1490944=475136,1966080=294912} 2024-09-15T03:47:10.855 INFO:tasks.rados.rados.0.smithi001.stdout:182: writing smithi00132172-35 from 720896 to 1490944 tid 1 2024-09-15T03:47:10.858 INFO:tasks.rados.rados.0.smithi001.stdout:182: writing smithi00132172-35 from 1490944 to 1966080 tid 2 2024-09-15T03:47:10.860 INFO:tasks.rados.rados.0.smithi001.stdout:182: writing smithi00132172-35 from 1966080 to 2260992 tid 3 2024-09-15T03:47:10.860 INFO:tasks.rados.rados.0.smithi001.stdout:183: rollback oid 48 current snap is 21 2024-09-15T03:47:10.860 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 48 to 20 2024-09-15T03:47:10.860 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 40 current snap is 21 2024-09-15T03:47:10.861 INFO:tasks.rados.rados.0.smithi001.stdout:184: seq_num 92 ranges {1802240=786432,2588672=491520,3080192=278528} 2024-09-15T03:47:10.861 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:10.862 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:10.874 INFO:tasks.rados.rados.0.smithi001.stdout:184: writing smithi00132172-40 from 1802240 to 2588672 tid 1 2024-09-15T03:47:10.877 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:10.877 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:10.878 INFO:tasks.rados.rados.0.smithi001.stdout:184: writing smithi00132172-40 from 2588672 to 3080192 tid 2 2024-09-15T03:47:10.880 INFO:tasks.rados.rados.0.smithi001.stdout:184: writing smithi00132172-40 from 3080192 to 3358720 tid 3 2024-09-15T03:47:10.880 INFO:tasks.rados.rados.0.smithi001.stdout:185: read oid 20 snap 10 2024-09-15T03:47:10.880 INFO:tasks.rados.rados.0.smithi001.stdout:185: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-15T03:47:10.880 INFO:tasks.rados.rados.0.smithi001.stdout:186: read oid 49 snap -1 2024-09-15T03:47:10.880 INFO:tasks.rados.rados.0.smithi001.stdout:186: expect (ObjNum 79 snap 16 seq_num 79) 2024-09-15T03:47:10.881 INFO:tasks.rados.rados.0.smithi001.stdout:187: delete oid 24 current snap is 21 2024-09-15T03:47:10.881 INFO:tasks.rados.rados.0.smithi001.stdout:183: finishing rollback tid 0 to smithi00132172-48 2024-09-15T03:47:10.881 INFO:tasks.rados.rados.0.smithi001.stdout:183: finishing rollback tid 1 to smithi00132172-48 2024-09-15T03:47:10.881 INFO:tasks.rados.rados.0.smithi001.stdout:183: finishing rollback tid 2 to smithi00132172-48 2024-09-15T03:47:10.881 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 48 v 49 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-15T03:47:10.882 INFO:tasks.rados.rados.0.smithi001.stdout:183: done (7 left) 2024-09-15T03:47:10.882 INFO:tasks.rados.rados.0.smithi001.stdout:187: done (6 left) 2024-09-15T03:47:10.882 INFO:tasks.rados.rados.0.smithi001.stdout:188: delete oid 17 current snap is 21 2024-09-15T03:47:10.883 INFO:tasks.rados.rados.0.smithi001.stdout:185: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-15T03:47:10.887 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:10.887 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:10.887 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:10.887 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:47:10.887 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.888 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:47:10.888 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:10.888 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 52.982353654999997, 2024-09-15T03:47:10.888 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10988437500000001, 2024-09-15T03:47:10.888 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:10.888 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:10.888 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:10.888 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:47:10.889 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:47:10.889 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1 2024-09-15T03:47:10.889 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.889 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:10.889 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.889 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:10.889 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:10.889 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.890 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.890 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.890 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:10.890 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:10.890 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.890 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.890 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.890 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:10.890 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902355+0000", 2024-09-15T03:47:10.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.429e-06 2024-09-15T03:47:10.891 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:10.891 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902362+0000", 2024-09-15T03:47:10.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7290000000000003e-06 2024-09-15T03:47:10.891 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:10.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902373+0000", 2024-09-15T03:47:10.892 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0533999999999999e-05 2024-09-15T03:47:10.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:10.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902379+0000", 2024-09-15T03:47:10.892 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4350000000000002e-06 2024-09-15T03:47:10.893 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.893 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.893 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:10.893 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902428+0000", 2024-09-15T03:47:10.893 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8571e-05 2024-09-15T03:47:10.893 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.893 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.893 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:47:10.894 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902434+0000", 2024-09-15T03:47:10.894 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5610000000000004e-06 2024-09-15T03:47:10.894 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.894 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.894 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:10.894 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011307+0000", 2024-09-15T03:47:10.894 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.108872306 2024-09-15T03:47:10.894 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.895 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.895 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:10.895 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011448+0000", 2024-09-15T03:47:10.895 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014137199999999999 2024-09-15T03:47:10.895 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.895 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.895 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:47:10.895 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011569+0000", 2024-09-15T03:47:10.896 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120925 2024-09-15T03:47:10.896 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.896 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.896 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:10.896 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011695+0000", 2024-09-15T03:47:10.896 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012586800000000001 2024-09-15T03:47:10.896 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.896 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.897 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:10.897 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011703+0000", 2024-09-15T03:47:10.897 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4190000000000002e-06 2024-09-15T03:47:10.897 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.897 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.897 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:10.897 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011739+0000", 2024-09-15T03:47:10.897 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5474e-05 2024-09-15T03:47:10.898 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.898 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.898 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:47:10.898 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011940+0000", 2024-09-15T03:47:10.898 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020108699999999999 2024-09-15T03:47:10.898 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.898 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.898 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:47:10.899 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012179+0000", 2024-09-15T03:47:10.899 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023950299999999999 2024-09-15T03:47:10.899 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.899 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:10.899 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012215+0000", 2024-09-15T03:47:10.899 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5332e-05 2024-09-15T03:47:10.899 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.899 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:10.899 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012238+0000", 2024-09-15T03:47:10.899 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3830000000000001e-05 2024-09-15T03:47:10.899 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.900 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:10.900 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.900 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.900 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.900 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4263.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2024-09-15T03:47:10.900 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:10.900 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 46.840956968999997, 2024-09-15T03:47:10.900 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019495788999999999, 2024-09-15T03:47:10.900 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:10.900 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:10.900 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:10.900 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4263", 2024-09-15T03:47:10.900 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/646873808", 2024-09-15T03:47:10.901 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2 2024-09-15T03:47:10.901 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.901 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:10.901 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.901 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:10.901 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:10.901 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.901 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.901 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.901 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:10.901 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:10.901 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.902 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.902 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.902 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:10.902 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043752+0000", 2024-09-15T03:47:10.902 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.815e-06 2024-09-15T03:47:10.902 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.902 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.902 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:10.902 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043757+0000", 2024-09-15T03:47:10.902 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4240000000000003e-06 2024-09-15T03:47:10.902 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.902 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.902 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:10.903 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043766+0000", 2024-09-15T03:47:10.903 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8629999999999997e-06 2024-09-15T03:47:10.903 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.903 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.903 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:10.903 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043773+0000", 2024-09-15T03:47:10.903 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0099999999999998e-06 2024-09-15T03:47:10.903 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.903 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.903 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:10.903 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043830+0000", 2024-09-15T03:47:10.903 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7089999999999997e-05 2024-09-15T03:47:10.903 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.904 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.904 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:47:10.904 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043837+0000", 2024-09-15T03:47:10.904 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.08e-06 2024-09-15T03:47:10.904 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.904 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.904 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:10.904 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063103+0000", 2024-09-15T03:47:10.904 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019265619000000001 2024-09-15T03:47:10.904 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.904 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.905 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:10.905 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063246+0000", 2024-09-15T03:47:10.905 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000143888 2024-09-15T03:47:10.905 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.905 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:10.905 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.905 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.905 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.906 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.fs5 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:10.906 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:47:10.906 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 30.777109063000001, 2024-09-15T03:47:10.906 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098369180000000001, 2024-09-15T03:47:10.906 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:10.906 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:10.906 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:10.906 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.906 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:10.906 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:47:10.906 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.906 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.906 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.907 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:10.907 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:47:10.907 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.907 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.907 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.907 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:10.907 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:47:10.907 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4499999999999998e-07 2024-09-15T03:47:10.907 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.907 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.907 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:10.907 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107630+0000", 2024-09-15T03:47:10.907 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0877999999999998e-05 2024-09-15T03:47:10.907 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.907 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.908 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:10.908 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107642+0000", 2024-09-15T03:47:10.908 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2567e-05 2024-09-15T03:47:10.908 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.908 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.908 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:10.908 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107648+0000", 2024-09-15T03:47:10.908 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8690000000000001e-06 2024-09-15T03:47:10.908 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.908 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.908 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:10.908 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107894+0000", 2024-09-15T03:47:10.909 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024540400000000001 2024-09-15T03:47:10.909 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.909 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.909 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:10.909 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107900+0000", 2024-09-15T03:47:10.909 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5710000000000003e-06 2024-09-15T03:47:10.909 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.909 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.909 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:10.909 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107968+0000", 2024-09-15T03:47:10.909 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7957e-05 2024-09-15T03:47:10.909 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.909 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.910 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:10.910 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108274+0000", 2024-09-15T03:47:10.910 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030591700000000001 2024-09-15T03:47:10.910 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.910 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.910 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:10.910 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108504+0000", 2024-09-15T03:47:10.910 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022952199999999999 2024-09-15T03:47:10.910 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.910 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.910 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:10.910 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109438+0000", 2024-09-15T03:47:10.910 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00093439100000000002 2024-09-15T03:47:10.910 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.911 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.911 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:10.911 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117435+0000", 2024-09-15T03:47:10.911 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0079974969999999992 2024-09-15T03:47:10.911 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.911 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:10.911 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.911 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.911 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.911 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:10.911 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:47:10.911 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 30.777051811, 2024-09-15T03:47:10.911 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084894540000000004, 2024-09-15T03:47:10.912 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:10.912 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:10.912 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:10.912 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.912 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:10.912 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:47:10.912 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.912 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.912 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.912 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:10.912 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:47:10.913 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.913 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.913 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.913 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:10.913 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:47:10.913 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4700000000000002e-07 2024-09-15T03:47:10.913 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.913 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.913 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:10.913 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107670+0000", 2024-09-15T03:47:10.913 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3892e-05 2024-09-15T03:47:10.913 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.913 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.914 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:10.914 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107678+0000", 2024-09-15T03:47:10.914 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9589999999999995e-06 2024-09-15T03:47:10.914 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.914 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.914 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:10.914 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107683+0000", 2024-09-15T03:47:10.914 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7199999999999997e-06 2024-09-15T03:47:10.914 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.914 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.914 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:10.914 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107768+0000", 2024-09-15T03:47:10.914 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4968000000000004e-05 2024-09-15T03:47:10.915 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.915 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.915 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:10.915 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107776+0000", 2024-09-15T03:47:10.915 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9109999999999993e-06 2024-09-15T03:47:10.915 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.915 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.915 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:10.915 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107858+0000", 2024-09-15T03:47:10.915 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2349999999999996e-05 2024-09-15T03:47:10.915 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.915 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.915 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:10.916 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108273+0000", 2024-09-15T03:47:10.916 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00041525600000000002 2024-09-15T03:47:10.916 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.916 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.916 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:10.916 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108503+0000", 2024-09-15T03:47:10.916 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022990599999999999 2024-09-15T03:47:10.916 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.916 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.916 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:10.916 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108705+0000", 2024-09-15T03:47:10.916 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020173000000000001 2024-09-15T03:47:10.916 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.917 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.917 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:10.917 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.116145+0000", 2024-09-15T03:47:10.917 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0074404149999999997 2024-09-15T03:47:10.917 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.917 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:10.917 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.917 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.917 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.917 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.fs5 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:10.917 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.109087+0000", 2024-09-15T03:47:10.917 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 30.775620817, 2024-09-15T03:47:10.917 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084235809999999994, 2024-09-15T03:47:10.918 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:10.918 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:10.918 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:10.918 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.918 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:10.918 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109087+0000", 2024-09-15T03:47:10.918 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.918 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.918 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.918 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:10.918 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109087+0000", 2024-09-15T03:47:10.918 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.918 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.918 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.919 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:10.919 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109087+0000", 2024-09-15T03:47:10.919 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7599999999999997e-07 2024-09-15T03:47:10.919 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.919 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.919 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:10.919 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109107+0000", 2024-09-15T03:47:10.919 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9511999999999999e-05 2024-09-15T03:47:10.919 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.919 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.919 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:10.919 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109127+0000", 2024-09-15T03:47:10.920 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0268e-05 2024-09-15T03:47:10.920 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.920 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.920 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:10.920 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109136+0000", 2024-09-15T03:47:10.920 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1330000000000005e-06 2024-09-15T03:47:10.920 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.920 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.920 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:10.920 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109594+0000", 2024-09-15T03:47:10.920 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045802100000000002 2024-09-15T03:47:10.920 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.921 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.921 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:10.921 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109602+0000", 2024-09-15T03:47:10.921 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7020000000000001e-06 2024-09-15T03:47:10.921 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.921 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.921 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:10.921 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109678+0000", 2024-09-15T03:47:10.921 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5428999999999999e-05 2024-09-15T03:47:10.921 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.921 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.921 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:10.921 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109699+0000", 2024-09-15T03:47:10.922 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1759999999999998e-05 2024-09-15T03:47:10.922 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.922 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.922 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:10.922 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109728+0000", 2024-09-15T03:47:10.922 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8886000000000001e-05 2024-09-15T03:47:10.922 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.922 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.922 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:10.922 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109736+0000", 2024-09-15T03:47:10.922 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6310000000000003e-06 2024-09-15T03:47:10.922 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.922 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.922 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:10.923 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117510+0000", 2024-09-15T03:47:10.923 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.007774563 2024-09-15T03:47:10.923 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.923 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:10.923 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.923 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.924 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.924 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/31 ECSubWrite(tid=38, reqid=client.4521.0:223, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:47:10.924 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:47:10.924 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 25.737500632, 2024-09-15T03:47:10.924 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010504493, 2024-09-15T03:47:10.924 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:10.924 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:10.924 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:10.924 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.924 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:10.924 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:47:10.924 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.925 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.925 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.925 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:10.925 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:47:10.925 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.925 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.925 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.925 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:10.925 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:47:10.925 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4999999999999999e-07 2024-09-15T03:47:10.925 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.925 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.925 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:10.926 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147648+0000", 2024-09-15T03:47:10.926 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00044030399999999998 2024-09-15T03:47:10.926 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:10.926 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147660+0000", 2024-09-15T03:47:10.926 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2517000000000001e-05 2024-09-15T03:47:10.926 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:10.926 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147667+0000", 2024-09-15T03:47:10.926 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.737e-06 2024-09-15T03:47:10.926 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.927 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.927 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:10.927 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147713+0000", 2024-09-15T03:47:10.927 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6405e-05 2024-09-15T03:47:10.927 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.927 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.927 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:10.927 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147719+0000", 2024-09-15T03:47:10.927 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0580000000000003e-06 2024-09-15T03:47:10.927 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.927 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.927 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:10.927 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147905+0000", 2024-09-15T03:47:10.928 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018574600000000001 2024-09-15T03:47:10.928 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.928 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.928 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:10.928 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147917+0000", 2024-09-15T03:47:10.928 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1749999999999999e-05 2024-09-15T03:47:10.928 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.928 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.928 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:10.928 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148022+0000", 2024-09-15T03:47:10.928 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104863 2024-09-15T03:47:10.928 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.928 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.928 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:10.929 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148037+0000", 2024-09-15T03:47:10.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5302999999999999e-05 2024-09-15T03:47:10.929 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.929 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.929 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:10.929 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157711+0000", 2024-09-15T03:47:10.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0096745600000000005 2024-09-15T03:47:10.929 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.929 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:10.929 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.929 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.930 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.930 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 39/28 ECSubWrite(tid=42, reqid=client.4521.0:255, at_version=39'19, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:47:10.930 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.170708+0000", 2024-09-15T03:47:10.930 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 24.713999845, 2024-09-15T03:47:10.930 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0093655969999999998, 2024-09-15T03:47:10.930 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:10.930 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:10.930 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:10.930 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.930 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:10.930 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170708+0000", 2024-09-15T03:47:10.930 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.930 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.930 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.931 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:10.931 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170708+0000", 2024-09-15T03:47:10.931 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.931 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.931 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.931 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:10.931 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170708+0000", 2024-09-15T03:47:10.931 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8899999999999999e-07 2024-09-15T03:47:10.931 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.931 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.931 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:10.931 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170728+0000", 2024-09-15T03:47:10.931 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9184000000000001e-05 2024-09-15T03:47:10.931 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.932 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.932 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:10.932 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170741+0000", 2024-09-15T03:47:10.932 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3696999999999999e-05 2024-09-15T03:47:10.932 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.932 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.932 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:10.932 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170747+0000", 2024-09-15T03:47:10.932 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1569999999999997e-06 2024-09-15T03:47:10.932 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.932 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.932 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:10.932 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170803+0000", 2024-09-15T03:47:10.933 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5974000000000003e-05 2024-09-15T03:47:10.933 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.933 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.933 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:10.933 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170811+0000", 2024-09-15T03:47:10.933 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8259999999999993e-06 2024-09-15T03:47:10.933 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.933 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.933 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:10.933 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170902+0000", 2024-09-15T03:47:10.933 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1116e-05 2024-09-15T03:47:10.933 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.933 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.933 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:10.934 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170909+0000", 2024-09-15T03:47:10.934 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3489999999999997e-06 2024-09-15T03:47:10.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.934 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:10.934 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173637+0000", 2024-09-15T03:47:10.934 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027282270000000002 2024-09-15T03:47:10.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.934 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:10.934 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173649+0000", 2024-09-15T03:47:10.934 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2561000000000001e-05 2024-09-15T03:47:10.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.935 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:10.935 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180073+0000", 2024-09-15T03:47:10.935 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064239170000000003 2024-09-15T03:47:10.935 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.935 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:10.935 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.935 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.935 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.935 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:47:10.935 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:10.936 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 24.713883580000001, 2024-09-15T03:47:10.936 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011878227, 2024-09-15T03:47:10.936 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:10.936 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:10.936 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:10.936 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.936 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:10.936 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:10.936 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.936 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.936 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.936 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:10.937 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:10.937 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.937 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.937 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.937 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:10.937 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170825+0000", 2024-09-15T03:47:10.937 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9599999999999999e-07 2024-09-15T03:47:10.937 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.937 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.937 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:10.937 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171100+0000", 2024-09-15T03:47:10.937 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027548500000000001 2024-09-15T03:47:10.937 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.938 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:10.938 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171112+0000", 2024-09-15T03:47:10.938 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2152e-05 2024-09-15T03:47:10.938 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.938 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:10.938 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171119+0000", 2024-09-15T03:47:10.938 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5050000000000004e-06 2024-09-15T03:47:10.938 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.938 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:10.939 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171162+0000", 2024-09-15T03:47:10.939 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2908000000000003e-05 2024-09-15T03:47:10.939 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.939 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.939 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:10.939 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171169+0000", 2024-09-15T03:47:10.939 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9739999999999996e-06 2024-09-15T03:47:10.940 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.940 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.940 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:10.940 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171398+0000", 2024-09-15T03:47:10.940 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022954799999999999 2024-09-15T03:47:10.940 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.940 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.940 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:10.940 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173640+0000", 2024-09-15T03:47:10.940 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022417000000000001 2024-09-15T03:47:10.940 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.940 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.940 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:10.941 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173705+0000", 2024-09-15T03:47:10.941 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4907999999999996e-05 2024-09-15T03:47:10.941 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.941 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.941 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:10.941 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173742+0000", 2024-09-15T03:47:10.944 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6940999999999997e-05 2024-09-15T03:47:10.944 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.945 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.945 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:10.945 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.182702+0000", 2024-09-15T03:47:10.945 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0089604100000000002 2024-09-15T03:47:10.945 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.945 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:10.945 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.945 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.946 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.946 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 39/28 ECSubWrite(tid=43, reqid=client.4521.0:256, at_version=39'20, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:47:10.946 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.171196+0000", 2024-09-15T03:47:10.946 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 24.713511494999999, 2024-09-15T03:47:10.946 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0090205690000000009, 2024-09-15T03:47:10.946 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:10.946 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:10.946 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:10.947 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.947 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:10.947 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171196+0000", 2024-09-15T03:47:10.947 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.947 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.947 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.947 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:10.947 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171196+0000", 2024-09-15T03:47:10.947 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.947 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.947 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.947 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:10.947 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171197+0000", 2024-09-15T03:47:10.947 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3e-07 2024-09-15T03:47:10.948 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.948 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.948 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:10.948 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171218+0000", 2024-09-15T03:47:10.948 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0775e-05 2024-09-15T03:47:10.948 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.948 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.948 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:10.948 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171241+0000", 2024-09-15T03:47:10.948 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3538999999999999e-05 2024-09-15T03:47:10.948 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.948 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.948 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:10.949 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171248+0000", 2024-09-15T03:47:10.949 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6220000000000003e-06 2024-09-15T03:47:10.949 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.949 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.949 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:10.949 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171291+0000", 2024-09-15T03:47:10.949 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3115000000000003e-05 2024-09-15T03:47:10.949 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.949 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.949 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:10.950 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171301+0000", 2024-09-15T03:47:10.950 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.662e-06 2024-09-15T03:47:10.950 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.950 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.950 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:10.950 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171395+0000", 2024-09-15T03:47:10.950 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4368000000000002e-05 2024-09-15T03:47:10.950 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.950 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.950 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:10.950 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173639+0000", 2024-09-15T03:47:10.951 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022444040000000002 2024-09-15T03:47:10.951 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.951 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.951 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:10.951 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173704+0000", 2024-09-15T03:47:10.951 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5031999999999997e-05 2024-09-15T03:47:10.951 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.951 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.951 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:10.951 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173718+0000", 2024-09-15T03:47:10.951 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3159000000000001e-05 2024-09-15T03:47:10.951 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.952 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.952 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:10.952 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180217+0000", 2024-09-15T03:47:10.952 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064991629999999996 2024-09-15T03:47:10.952 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.952 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:10.952 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.952 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.952 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.952 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 39/31 ECSubWrite(tid=43, reqid=client.4521.0:252, at_version=39'27, trim_to=0'0, roll_forward_to=39'25))", 2024-09-15T03:47:10.953 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:10.953 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 24.706166262, 2024-09-15T03:47:10.953 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011286757, 2024-09-15T03:47:10.953 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:10.953 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:10.953 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:10.953 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.953 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:10.953 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:10.953 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.953 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.953 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.954 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:10.954 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:10.954 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.954 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.954 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.954 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:10.954 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178542+0000", 2024-09-15T03:47:10.954 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9400000000000004e-07 2024-09-15T03:47:10.954 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.954 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.954 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:10.954 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178754+0000", 2024-09-15T03:47:10.954 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021228800000000001 2024-09-15T03:47:10.954 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.955 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.955 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:10.955 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178765+0000", 2024-09-15T03:47:10.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0176e-05 2024-09-15T03:47:10.955 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.955 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.955 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:10.955 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178769+0000", 2024-09-15T03:47:10.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1359999999999999e-06 2024-09-15T03:47:10.955 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.955 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.955 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:10.955 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178810+0000", 2024-09-15T03:47:10.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0825999999999998e-05 2024-09-15T03:47:10.956 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.956 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.956 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:10.956 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178820+0000", 2024-09-15T03:47:10.956 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0237999999999999e-05 2024-09-15T03:47:10.956 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.956 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.957 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:10.957 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179033+0000", 2024-09-15T03:47:10.957 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021361500000000001 2024-09-15T03:47:10.957 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.957 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.958 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:10.958 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179040+0000", 2024-09-15T03:47:10.958 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0509999999999999e-06 2024-09-15T03:47:10.958 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.958 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.958 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:10.958 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179102+0000", 2024-09-15T03:47:10.958 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1977000000000006e-05 2024-09-15T03:47:10.958 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.958 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.958 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:10.958 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179114+0000", 2024-09-15T03:47:10.959 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1223000000000001e-05 2024-09-15T03:47:10.959 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.959 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.959 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:10.959 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.189828+0000", 2024-09-15T03:47:10.959 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010714433000000001 2024-09-15T03:47:10.959 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.959 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:10.959 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.959 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.959 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.959 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 39/31 ECSubWrite(tid=44, reqid=client.4521.0:252, at_version=0'0, trim_to=0'0, roll_forward_to=39'27))", 2024-09-15T03:47:10.960 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.179948+0000", 2024-09-15T03:47:10.960 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 24.704759162999999, 2024-09-15T03:47:10.960 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010068739, 2024-09-15T03:47:10.960 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:10.961 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:10.961 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:10.961 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:10.961 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179948+0000", 2024-09-15T03:47:10.961 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.961 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.961 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:10.961 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179948+0000", 2024-09-15T03:47:10.961 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.961 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.961 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:10.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179949+0000", 2024-09-15T03:47:10.962 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6199999999999998e-07 2024-09-15T03:47:10.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:10.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179960+0000", 2024-09-15T03:47:10.962 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1216000000000001e-05 2024-09-15T03:47:10.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:10.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179970+0000", 2024-09-15T03:47:10.962 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8980000000000008e-06 2024-09-15T03:47:10.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:10.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179975+0000", 2024-09-15T03:47:10.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5650000000000003e-06 2024-09-15T03:47:10.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:10.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180004+0000", 2024-09-15T03:47:10.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9224999999999999e-05 2024-09-15T03:47:10.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:10.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180009+0000", 2024-09-15T03:47:10.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9250000000000003e-06 2024-09-15T03:47:10.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.964 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:10.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180061+0000", 2024-09-15T03:47:10.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1758000000000001e-05 2024-09-15T03:47:10.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.964 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:10.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180067+0000", 2024-09-15T03:47:10.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3269999999999997e-06 2024-09-15T03:47:10.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.965 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:10.965 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180101+0000", 2024-09-15T03:47:10.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3701000000000001e-05 2024-09-15T03:47:10.965 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.965 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:10.965 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180112+0000", 2024-09-15T03:47:10.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1287000000000001e-05 2024-09-15T03:47:10.965 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.965 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:10.965 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.190017+0000", 2024-09-15T03:47:10.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0099052749999999998 2024-09-15T03:47:10.966 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.966 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:10.966 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.966 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.966 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/31 ECSubWrite(tid=90, reqid=client.4521.0:346, at_version=51'12, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:10.966 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.841846+0000", 2024-09-15T03:47:10.966 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 13.042861368000001, 2024-09-15T03:47:10.966 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0088888219999999993, 2024-09-15T03:47:10.966 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:10.966 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:10.967 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:10.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.967 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:10.967 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.841846+0000", 2024-09-15T03:47:10.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.967 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:10.967 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.841846+0000", 2024-09-15T03:47:10.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:10.968 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.841846+0000", 2024-09-15T03:47:10.968 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8300000000000001e-07 2024-09-15T03:47:10.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:10.968 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.841882+0000", 2024-09-15T03:47:10.968 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5574999999999997e-05 2024-09-15T03:47:10.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:10.968 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.841896+0000", 2024-09-15T03:47:10.968 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4100999999999999e-05 2024-09-15T03:47:10.969 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.969 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.969 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:10.969 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.841906+0000", 2024-09-15T03:47:10.969 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0356e-05 2024-09-15T03:47:10.969 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.969 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.969 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:10.969 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.841983+0000", 2024-09-15T03:47:10.969 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6477000000000007e-05 2024-09-15T03:47:10.969 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.969 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.969 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:10.970 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.841996+0000", 2024-09-15T03:47:10.970 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3360000000000001e-05 2024-09-15T03:47:10.970 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.970 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.970 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:10.970 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842321+0000", 2024-09-15T03:47:10.970 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00032513000000000003 2024-09-15T03:47:10.970 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.970 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.970 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:10.970 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842474+0000", 2024-09-15T03:47:10.971 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015222099999999999 2024-09-15T03:47:10.971 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.971 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.971 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:10.971 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842529+0000", 2024-09-15T03:47:10.971 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5262e-05 2024-09-15T03:47:10.971 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.971 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.971 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:10.971 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842553+0000", 2024-09-15T03:47:10.971 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3978999999999999e-05 2024-09-15T03:47:10.971 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:10.972 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850735+0000", 2024-09-15T03:47:10.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.008182178 2024-09-15T03:47:10.972 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.972 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:10.972 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.972 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 51/31 ECSubWrite(tid=66, reqid=client.4521.0:349, at_version=51'32, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:10.972 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.842006+0000", 2024-09-15T03:47:10.972 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 13.042701508, 2024-09-15T03:47:10.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095108069999999996, 2024-09-15T03:47:10.973 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:10.973 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:10.973 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:10.973 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.973 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:10.973 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842006+0000", 2024-09-15T03:47:10.973 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.973 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.973 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.973 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:10.973 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842006+0000", 2024-09-15T03:47:10.973 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.974 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.974 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.974 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:10.974 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842006+0000", 2024-09-15T03:47:10.974 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9000000000000001e-07 2024-09-15T03:47:10.974 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.974 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.974 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:10.974 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842021+0000", 2024-09-15T03:47:10.974 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5089000000000001e-05 2024-09-15T03:47:10.974 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.974 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.974 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:10.975 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842030+0000", 2024-09-15T03:47:10.975 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2709999999999992e-06 2024-09-15T03:47:10.975 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.975 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:10.975 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842034+0000", 2024-09-15T03:47:10.975 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5469999999999998e-06 2024-09-15T03:47:10.975 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.975 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:10.975 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842099+0000", 2024-09-15T03:47:10.975 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4911000000000002e-05 2024-09-15T03:47:10.975 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:10.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842111+0000", 2024-09-15T03:47:10.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1681e-05 2024-09-15T03:47:10.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:10.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842399+0000", 2024-09-15T03:47:10.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00028861199999999999 2024-09-15T03:47:10.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:10.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842475+0000", 2024-09-15T03:47:10.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5420999999999995e-05 2024-09-15T03:47:10.977 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.977 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.977 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:10.977 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842530+0000", 2024-09-15T03:47:10.977 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4863e-05 2024-09-15T03:47:10.977 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:10.978 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842603+0000", 2024-09-15T03:47:10.978 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3615000000000006e-05 2024-09-15T03:47:10.978 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:10.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851517+0000", 2024-09-15T03:47:10.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0089136070000000005 2024-09-15T03:47:10.979 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.979 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:10.979 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.979 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 51/31 ECSubWrite(tid=67, reqid=client.4521.0:350, at_version=51'33, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:10.979 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:10.979 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 13.040866221, 2024-09-15T03:47:10.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.015569239, 2024-09-15T03:47:10.979 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:10.979 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:10.979 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:10.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:10.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:10.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:10.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:10.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:10.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843843+0000", 2024-09-15T03:47:10.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.195e-06 2024-09-15T03:47:10.981 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.981 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.981 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:10.981 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843880+0000", 2024-09-15T03:47:10.981 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7527999999999997e-05 2024-09-15T03:47:10.981 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.981 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.981 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:10.981 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843898+0000", 2024-09-15T03:47:10.981 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7696000000000001e-05 2024-09-15T03:47:10.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.982 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:10.982 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843907+0000", 2024-09-15T03:47:10.982 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8820000000000007e-06 2024-09-15T03:47:10.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.982 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:10.982 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843978+0000", 2024-09-15T03:47:10.982 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1537000000000003e-05 2024-09-15T03:47:10.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.982 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:10.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843989+0000", 2024-09-15T03:47:10.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0720000000000001e-05 2024-09-15T03:47:10.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:10.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844102+0000", 2024-09-15T03:47:10.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000113406 2024-09-15T03:47:10.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:10.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844131+0000", 2024-09-15T03:47:10.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8762e-05 2024-09-15T03:47:10.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:10.984 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844180+0000", 2024-09-15T03:47:10.984 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9329999999999997e-05 2024-09-15T03:47:10.984 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.984 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:10.984 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844201+0000", 2024-09-15T03:47:10.984 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0772999999999999e-05 2024-09-15T03:47:10.984 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.984 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:10.984 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.859411+0000", 2024-09-15T03:47:10.984 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01520941 2024-09-15T03:47:10.984 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.984 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:10.985 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.985 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.985 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.985 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:10.985 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:10.985 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 13.039987310000001, 2024-09-15T03:47:10.985 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012156452, 2024-09-15T03:47:10.985 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:10.985 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:10.985 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:10.985 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.985 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:10.986 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:10.986 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.986 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.986 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.986 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:10.987 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:10.987 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.987 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.987 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.987 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:10.987 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844721+0000", 2024-09-15T03:47:10.988 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1100000000000005e-07 2024-09-15T03:47:10.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.988 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.988 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:10.988 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844743+0000", 2024-09-15T03:47:10.988 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1999000000000001e-05 2024-09-15T03:47:10.989 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.989 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.989 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:10.989 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844758+0000", 2024-09-15T03:47:10.989 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4718999999999999e-05 2024-09-15T03:47:10.989 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.989 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.989 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:10.989 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844772+0000", 2024-09-15T03:47:10.989 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4637000000000001e-05 2024-09-15T03:47:10.989 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.989 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.990 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:10.990 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844842+0000", 2024-09-15T03:47:10.990 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9596000000000002e-05 2024-09-15T03:47:10.990 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.990 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.991 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:10.991 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844853+0000", 2024-09-15T03:47:10.991 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0912e-05 2024-09-15T03:47:10.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.991 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.991 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:10.991 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844966+0000", 2024-09-15T03:47:10.991 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011276 2024-09-15T03:47:10.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.992 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.992 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:10.992 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844977+0000", 2024-09-15T03:47:10.992 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1676e-05 2024-09-15T03:47:10.992 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.992 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.992 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:10.992 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847414+0000", 2024-09-15T03:47:10.992 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024363760000000001 2024-09-15T03:47:10.992 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.992 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.992 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:10.993 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847438+0000", 2024-09-15T03:47:10.993 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3737e-05 2024-09-15T03:47:10.993 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.993 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.993 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:10.993 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856877+0000", 2024-09-15T03:47:10.993 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094392290000000004 2024-09-15T03:47:10.993 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.993 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:10.993 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:10.994 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.994 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.994 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:10.994 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:10.994 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 13.039917915, 2024-09-15T03:47:10.994 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012258211, 2024-09-15T03:47:10.994 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:10.994 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:10.994 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:10.994 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.994 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:10.994 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:10.994 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.995 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:10.995 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:10.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:10.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.995 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:10.995 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:10.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.58e-07 2024-09-15T03:47:10.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.995 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:10.996 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844818+0000", 2024-09-15T03:47:10.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.796e-05 2024-09-15T03:47:10.996 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.996 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:10.996 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844829+0000", 2024-09-15T03:47:10.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0895e-05 2024-09-15T03:47:10.996 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.996 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:10.996 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844836+0000", 2024-09-15T03:47:10.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.618e-06 2024-09-15T03:47:10.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:10.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845023+0000", 2024-09-15T03:47:10.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018743500000000001 2024-09-15T03:47:10.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.998 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:10.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845035+0000", 2024-09-15T03:47:10.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1958e-05 2024-09-15T03:47:10.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.999 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:10.999 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845147+0000", 2024-09-15T03:47:10.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112281 2024-09-15T03:47:10.999 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:10.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:10.999 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:10.999 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847417+0000", 2024-09-15T03:47:10.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022693969999999998 2024-09-15T03:47:11.000 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.000 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.000 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847462+0000", 2024-09-15T03:47:11.000 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5466999999999997e-05 2024-09-15T03:47:11.000 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.000 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.000 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847503+0000", 2024-09-15T03:47:11.000 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1022999999999997e-05 2024-09-15T03:47:11.000 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.857048+0000", 2024-09-15T03:47:11.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095447190000000001 2024-09-15T03:47:11.001 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.001 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.001 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.001 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 51/31 ECSubWrite(tid=68, reqid=client.4521.0:351, at_version=51'34, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:11.001 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:11.001 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 13.039499797, 2024-09-15T03:47:11.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.014544793, 2024-09-15T03:47:11.002 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.002 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:11.002 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:11.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:11.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.003 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.003 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845209+0000", 2024-09-15T03:47:11.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5300000000000003e-07 2024-09-15T03:47:11.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.003 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.003 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845227+0000", 2024-09-15T03:47:11.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8759000000000001e-05 2024-09-15T03:47:11.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.003 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.004 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845241+0000", 2024-09-15T03:47:11.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3740000000000001e-05 2024-09-15T03:47:11.004 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.004 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.004 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845247+0000", 2024-09-15T03:47:11.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2779999999999999e-06 2024-09-15T03:47:11.004 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845308+0000", 2024-09-15T03:47:11.005 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0702000000000003e-05 2024-09-15T03:47:11.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845317+0000", 2024-09-15T03:47:11.005 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3770000000000002e-06 2024-09-15T03:47:11.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845413+0000", 2024-09-15T03:47:11.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5356e-05 2024-09-15T03:47:11.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847421+0000", 2024-09-15T03:47:11.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020077200000000002 2024-09-15T03:47:11.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847466+0000", 2024-09-15T03:47:11.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5754999999999998e-05 2024-09-15T03:47:11.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847572+0000", 2024-09-15T03:47:11.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010594000000000001 2024-09-15T03:47:11.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.859753+0000", 2024-09-15T03:47:11.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012180412999999999 2024-09-15T03:47:11.007 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.008 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.008 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.008 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 60/28 ECSubWrite(tid=73, reqid=client.4521.0:424, at_version=60'41, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:11.008 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:11.008 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 3.9842965430000001, 2024-09-15T03:47:11.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01137208, 2024-09-15T03:47:11.008 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.008 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:11.008 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:11.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:11.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900412+0000", 2024-09-15T03:47:11.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7599999999999996e-07 2024-09-15T03:47:11.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:11.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7986999999999999e-05 2024-09-15T03:47:11.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900445+0000", 2024-09-15T03:47:11.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.468e-05 2024-09-15T03:47:11.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900451+0000", 2024-09-15T03:47:11.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5799999999999997e-06 2024-09-15T03:47:11.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900507+0000", 2024-09-15T03:47:11.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5855e-05 2024-09-15T03:47:11.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900530+0000", 2024-09-15T03:47:11.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2929e-05 2024-09-15T03:47:11.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.012 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.012 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.012 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900640+0000", 2024-09-15T03:47:11.012 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000110536 2024-09-15T03:47:11.012 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.012 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.012 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.012 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900648+0000", 2024-09-15T03:47:11.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9470000000000003e-06 2024-09-15T03:47:11.013 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.013 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.013 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900764+0000", 2024-09-15T03:47:11.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011518099999999999 2024-09-15T03:47:11.013 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.013 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.013 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900803+0000", 2024-09-15T03:47:11.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9941999999999998e-05 2024-09-15T03:47:11.013 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.014 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.014 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.911783+0000", 2024-09-15T03:47:11.014 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010979667 2024-09-15T03:47:11.014 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.014 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.014 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.014 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.014 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.014 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 60/28 ECSubWrite(tid=74, reqid=client.4521.0:425, at_version=60'42, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:11.014 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.901183+0000", 2024-09-15T03:47:11.014 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 3.9835245449999999, 2024-09-15T03:47:11.014 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010721191999999999, 2024-09-15T03:47:11.014 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.015 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:11.015 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.015 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.015 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.015 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901183+0000", 2024-09-15T03:47:11.015 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.015 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.015 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.015 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.015 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901183+0000", 2024-09-15T03:47:11.015 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.015 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.015 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.015 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.016 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901184+0000", 2024-09-15T03:47:11.016 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6599999999999996e-07 2024-09-15T03:47:11.016 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.016 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.016 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.016 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901202+0000", 2024-09-15T03:47:11.016 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8658999999999999e-05 2024-09-15T03:47:11.016 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.016 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.016 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.016 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901217+0000", 2024-09-15T03:47:11.016 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4371e-05 2024-09-15T03:47:11.016 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.017 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901223+0000", 2024-09-15T03:47:11.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2489999999999998e-06 2024-09-15T03:47:11.017 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.017 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901274+0000", 2024-09-15T03:47:11.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0602000000000001e-05 2024-09-15T03:47:11.017 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.017 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901284+0000", 2024-09-15T03:47:11.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.995e-06 2024-09-15T03:47:11.018 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901372+0000", 2024-09-15T03:47:11.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8399000000000005e-05 2024-09-15T03:47:11.018 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.019 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901380+0000", 2024-09-15T03:47:11.019 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8040000000000002e-06 2024-09-15T03:47:11.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.019 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903553+0000", 2024-09-15T03:47:11.019 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021731580000000001 2024-09-15T03:47:11.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.019 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903570+0000", 2024-09-15T03:47:11.019 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7218999999999999e-05 2024-09-15T03:47:11.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.020 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.020 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.911904+0000", 2024-09-15T03:47:11.020 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0083340700000000007 2024-09-15T03:47:11.020 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.020 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.020 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.020 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.020 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.020 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es5 60/28 ECSubWrite(tid=89, reqid=client.4521.0:446, at_version=60'57, trim_to=0'0, roll_forward_to=60'55))", 2024-09-15T03:47:11.020 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.962207+0000", 2024-09-15T03:47:11.020 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 3.9225006260000002, 2024-09-15T03:47:11.020 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010047031, 2024-09-15T03:47:11.020 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.021 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:11.021 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.021 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.962207+0000", 2024-09-15T03:47:11.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.021 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.021 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.962207+0000", 2024-09-15T03:47:11.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.021 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.022 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.022 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.962208+0000", 2024-09-15T03:47:11.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6199999999999997e-07 2024-09-15T03:47:11.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.022 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.022 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.963166+0000", 2024-09-15T03:47:11.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00095856700000000001 2024-09-15T03:47:11.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.023 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.023 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.963182+0000", 2024-09-15T03:47:11.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5865000000000001e-05 2024-09-15T03:47:11.023 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.023 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.023 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.023 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.963189+0000", 2024-09-15T03:47:11.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9190000000000002e-06 2024-09-15T03:47:11.023 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.023 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.023 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.023 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.963238+0000", 2024-09-15T03:47:11.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8724999999999999e-05 2024-09-15T03:47:11.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.963245+0000", 2024-09-15T03:47:11.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3189999999999997e-06 2024-09-15T03:47:11.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.963529+0000", 2024-09-15T03:47:11.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00028405200000000002 2024-09-15T03:47:11.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.963537+0000", 2024-09-15T03:47:11.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3250000000000008e-06 2024-09-15T03:47:11.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.963687+0000", 2024-09-15T03:47:11.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015002599999999999 2024-09-15T03:47:11.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.963707+0000", 2024-09-15T03:47:11.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9241000000000001e-05 2024-09-15T03:47:11.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.026 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.026 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.972254+0000", 2024-09-15T03:47:11.026 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0085473300000000006 2024-09-15T03:47:11.026 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.026 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.026 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.026 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.026 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.026 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:11.026 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T03:47:11.028 INFO:tasks.rados.rados.0.smithi001.stdout:186: expect (ObjNum 79 snap 16 seq_num 79) 2024-09-15T03:47:11.028 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:11.029 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:11.029 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:47:11.030 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:47:11.030 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:47:11.030 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:11.031 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:11.032 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:11.027+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:11.128 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:11.127+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:11.129 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:11.129 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:11.129 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:11.129 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:11.138 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:47:11.229 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:11.228+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:11.235 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:11.235 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:11.235 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:11.236 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:11.236 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:11.244 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:47:11.330 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:11.329+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:11.341 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:11.342 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:11.342 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:11.342 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:47:11.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.342 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s3 39/31 ECSubWrite(tid=44, reqid=osd.1.0:44, at_version=39'12, trim_to=0'0, roll_forward_to=38'10))", 2024-09-15T03:47:11.342 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.150872+0000", 2024-09-15T03:47:11.342 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 25.188914540999999, 2024-09-15T03:47:11.342 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075168539999999999, 2024-09-15T03:47:11.342 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.342 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:11.342 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150872+0000", 2024-09-15T03:47:11.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150872+0000", 2024-09-15T03:47:11.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150872+0000", 2024-09-15T03:47:11.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3300000000000002e-07 2024-09-15T03:47:11.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150909+0000", 2024-09-15T03:47:11.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6294000000000002e-05 2024-09-15T03:47:11.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150919+0000", 2024-09-15T03:47:11.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9690000000000006e-06 2024-09-15T03:47:11.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150924+0000", 2024-09-15T03:47:11.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7710000000000003e-06 2024-09-15T03:47:11.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150958+0000", 2024-09-15T03:47:11.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3531999999999997e-05 2024-09-15T03:47:11.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.150970+0000", 2024-09-15T03:47:11.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1831000000000001e-05 2024-09-15T03:47:11.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152463+0000", 2024-09-15T03:47:11.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0014934239999999999 2024-09-15T03:47:11.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152471+0000", 2024-09-15T03:47:11.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2830000000000004e-06 2024-09-15T03:47:11.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152507+0000", 2024-09-15T03:47:11.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6739999999999997e-05 2024-09-15T03:47:11.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.152517+0000", 2024-09-15T03:47:11.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8570000000000007e-06 2024-09-15T03:47:11.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.158389+0000", 2024-09-15T03:47:11.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058715199999999999 2024-09-15T03:47:11.348 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.348 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.348 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.348 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 39/28 ECSubWrite(tid=42, reqid=client.4521.0:255, at_version=39'19, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:47:11.348 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.170724+0000", 2024-09-15T03:47:11.348 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 25.169062332999999, 2024-09-15T03:47:11.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0086604270000000001, 2024-09-15T03:47:11.348 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.348 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:11.348 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170724+0000", 2024-09-15T03:47:11.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170724+0000", 2024-09-15T03:47:11.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170725+0000", 2024-09-15T03:47:11.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8999999999999995e-07 2024-09-15T03:47:11.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.350 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.350 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170742+0000", 2024-09-15T03:47:11.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7181e-05 2024-09-15T03:47:11.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.350 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.350 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170757+0000", 2024-09-15T03:47:11.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4693e-05 2024-09-15T03:47:11.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.350 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.350 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170763+0000", 2024-09-15T03:47:11.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4509999999999997e-06 2024-09-15T03:47:11.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170807+0000", 2024-09-15T03:47:11.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3574e-05 2024-09-15T03:47:11.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170815+0000", 2024-09-15T03:47:11.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3459999999999994e-06 2024-09-15T03:47:11.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170900+0000", 2024-09-15T03:47:11.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5436e-05 2024-09-15T03:47:11.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170906+0000", 2024-09-15T03:47:11.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9190000000000004e-06 2024-09-15T03:47:11.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173681+0000", 2024-09-15T03:47:11.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00277437 2024-09-15T03:47:11.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173695+0000", 2024-09-15T03:47:11.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4446e-05 2024-09-15T03:47:11.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179384+0000", 2024-09-15T03:47:11.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0056892210000000004 2024-09-15T03:47:11.353 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.353 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.354 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.354 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:47:11.354 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:11.354 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 25.168684498000001, 2024-09-15T03:47:11.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012183955999999999, 2024-09-15T03:47:11.354 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.354 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:11.354 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.354 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:11.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:11.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:11.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.17e-07 2024-09-15T03:47:11.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171454+0000", 2024-09-15T03:47:11.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00035226700000000002 2024-09-15T03:47:11.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171465+0000", 2024-09-15T03:47:11.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0943e-05 2024-09-15T03:47:11.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171471+0000", 2024-09-15T03:47:11.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7400000000000001e-06 2024-09-15T03:47:11.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171508+0000", 2024-09-15T03:47:11.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7147999999999997e-05 2024-09-15T03:47:11.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171517+0000", 2024-09-15T03:47:11.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2560000000000008e-06 2024-09-15T03:47:11.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171716+0000", 2024-09-15T03:47:11.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019899699999999999 2024-09-15T03:47:11.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173686+0000", 2024-09-15T03:47:11.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019696829999999999 2024-09-15T03:47:11.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173777+0000", 2024-09-15T03:47:11.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0637999999999995e-05 2024-09-15T03:47:11.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173814+0000", 2024-09-15T03:47:11.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7499000000000001e-05 2024-09-15T03:47:11.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.183286+0000", 2024-09-15T03:47:11.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094716680000000008 2024-09-15T03:47:11.359 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.359 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.359 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.360 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 39/28 ECSubWrite(tid=43, reqid=client.4521.0:256, at_version=39'20, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:47:11.360 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.171211+0000", 2024-09-15T03:47:11.360 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 25.168575404999999, 2024-09-15T03:47:11.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082752069999999997, 2024-09-15T03:47:11.360 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.360 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:11.360 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171211+0000", 2024-09-15T03:47:11.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171211+0000", 2024-09-15T03:47:11.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171211+0000", 2024-09-15T03:47:11.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.39e-07 2024-09-15T03:47:11.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171228+0000", 2024-09-15T03:47:11.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6254999999999999e-05 2024-09-15T03:47:11.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171241+0000", 2024-09-15T03:47:11.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3482999999999999e-05 2024-09-15T03:47:11.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171247+0000", 2024-09-15T03:47:11.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.088e-06 2024-09-15T03:47:11.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.363 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171292+0000", 2024-09-15T03:47:11.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5062000000000003e-05 2024-09-15T03:47:11.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.363 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171302+0000", 2024-09-15T03:47:11.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.302e-06 2024-09-15T03:47:11.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.364 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.364 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171382+0000", 2024-09-15T03:47:11.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0717999999999995e-05 2024-09-15T03:47:11.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.364 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.364 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173686+0000", 2024-09-15T03:47:11.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023032320000000001 2024-09-15T03:47:11.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.364 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.364 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173776+0000", 2024-09-15T03:47:11.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0302000000000004e-05 2024-09-15T03:47:11.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173805+0000", 2024-09-15T03:47:11.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8979000000000002e-05 2024-09-15T03:47:11.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179486+0000", 2024-09-15T03:47:11.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0056813469999999998 2024-09-15T03:47:11.365 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.365 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.366 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.366 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:337 3.cs0 3:390b2f04:::smithi00132172-33:head [delete] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e49)", 2024-09-15T03:47:11.366 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.856819+0000", 2024-09-15T03:47:11.366 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.482967263999999, 2024-09-15T03:47:11.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0077070849999999998, 2024-09-15T03:47:11.366 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.366 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:11.366 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:11.366 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:11.366 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:11.366 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 337 2024-09-15T03:47:11.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.367 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856819+0000", 2024-09-15T03:47:11.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856819+0000", 2024-09-15T03:47:11.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856821+0000", 2024-09-15T03:47:11.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6700000000000001e-06 2024-09-15T03:47:11.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856827+0000", 2024-09-15T03:47:11.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8080000000000001e-06 2024-09-15T03:47:11.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856843+0000", 2024-09-15T03:47:11.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6430999999999999e-05 2024-09-15T03:47:11.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:11.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7920000000000004e-06 2024-09-15T03:47:11.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856889+0000", 2024-09-15T03:47:11.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8825000000000002e-05 2024-09-15T03:47:11.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:11.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.856960+0000", 2024-09-15T03:47:11.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1470999999999994e-05 2024-09-15T03:47:11.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857667+0000", 2024-09-15T03:47:11.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00070671199999999997 2024-09-15T03:47:11.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857896+0000", 2024-09-15T03:47:11.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000229621 2024-09-15T03:47:11.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857912+0000", 2024-09-15T03:47:11.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5400000000000002e-05 2024-09-15T03:47:11.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.857949+0000", 2024-09-15T03:47:11.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7239e-05 2024-09-15T03:47:11.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858108+0000", 2024-09-15T03:47:11.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015922399999999999 2024-09-15T03:47:11.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.372 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:11.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859023+0000", 2024-09-15T03:47:11.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00091487899999999995 2024-09-15T03:47:11.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.372 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.864526+0000", 2024-09-15T03:47:11.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0055030130000000002 2024-09-15T03:47:11.372 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.372 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.372 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.373 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 49/31 ECSubWrite(tid=87, reqid=client.4521.0:335, at_version=49'21, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:11.373 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.858786+0000", 2024-09-15T03:47:11.373 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.481000629, 2024-09-15T03:47:11.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0085806300000000005, 2024-09-15T03:47:11.373 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.373 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:11.373 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858786+0000", 2024-09-15T03:47:11.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858786+0000", 2024-09-15T03:47:11.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858786+0000", 2024-09-15T03:47:11.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3099999999999997e-07 2024-09-15T03:47:11.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858809+0000", 2024-09-15T03:47:11.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2331999999999999e-05 2024-09-15T03:47:11.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858819+0000", 2024-09-15T03:47:11.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0468e-05 2024-09-15T03:47:11.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858828+0000", 2024-09-15T03:47:11.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0499999999999997e-06 2024-09-15T03:47:11.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858864+0000", 2024-09-15T03:47:11.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6072000000000003e-05 2024-09-15T03:47:11.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858870+0000", 2024-09-15T03:47:11.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.896e-06 2024-09-15T03:47:11.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858938+0000", 2024-09-15T03:47:11.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7657000000000006e-05 2024-09-15T03:47:11.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858946+0000", 2024-09-15T03:47:11.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0609999999999996e-06 2024-09-15T03:47:11.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861287+0000", 2024-09-15T03:47:11.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023413779999999999 2024-09-15T03:47:11.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861301+0000", 2024-09-15T03:47:11.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3626e-05 2024-09-15T03:47:11.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.378 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.867366+0000", 2024-09-15T03:47:11.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0060654589999999996 2024-09-15T03:47:11.378 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.378 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.378 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.378 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 49/31 ECSubWrite(tid=88, reqid=client.4521.0:336, at_version=49'22, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:11.378 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.859239+0000", 2024-09-15T03:47:11.378 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.480547657000001, 2024-09-15T03:47:11.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082420800000000006, 2024-09-15T03:47:11.379 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.379 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:11.379 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859239+0000", 2024-09-15T03:47:11.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859239+0000", 2024-09-15T03:47:11.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859241+0000", 2024-09-15T03:47:11.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4880000000000001e-06 2024-09-15T03:47:11.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859256+0000", 2024-09-15T03:47:11.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5119999999999999e-05 2024-09-15T03:47:11.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859265+0000", 2024-09-15T03:47:11.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1500000000000005e-06 2024-09-15T03:47:11.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859274+0000", 2024-09-15T03:47:11.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2060000000000007e-06 2024-09-15T03:47:11.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859314+0000", 2024-09-15T03:47:11.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0250000000000003e-05 2024-09-15T03:47:11.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859324+0000", 2024-09-15T03:47:11.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0513000000000001e-05 2024-09-15T03:47:11.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859418+0000", 2024-09-15T03:47:11.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3397e-05 2024-09-15T03:47:11.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861290+0000", 2024-09-15T03:47:11.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0018723920000000001 2024-09-15T03:47:11.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861353+0000", 2024-09-15T03:47:11.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2706000000000006e-05 2024-09-15T03:47:11.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861365+0000", 2024-09-15T03:47:11.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2316e-05 2024-09-15T03:47:11.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.867481+0000", 2024-09-15T03:47:11.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0061155419999999999 2024-09-15T03:47:11.384 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.384 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.384 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.384 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 51/31 ECSubWrite(tid=67, reqid=client.4521.0:350, at_version=51'33, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:11.384 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.843852+0000", 2024-09-15T03:47:11.384 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 13.495934537, 2024-09-15T03:47:11.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010895471, 2024-09-15T03:47:11.384 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.384 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:11.384 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843852+0000", 2024-09-15T03:47:11.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843852+0000", 2024-09-15T03:47:11.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843853+0000", 2024-09-15T03:47:11.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.359e-06 2024-09-15T03:47:11.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843882+0000", 2024-09-15T03:47:11.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8711999999999999e-05 2024-09-15T03:47:11.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843901+0000", 2024-09-15T03:47:11.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9477e-05 2024-09-15T03:47:11.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843911+0000", 2024-09-15T03:47:11.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2590000000000006e-06 2024-09-15T03:47:11.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843973+0000", 2024-09-15T03:47:11.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2310999999999993e-05 2024-09-15T03:47:11.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843984+0000", 2024-09-15T03:47:11.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.132e-05 2024-09-15T03:47:11.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844108+0000", 2024-09-15T03:47:11.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012384399999999999 2024-09-15T03:47:11.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844139+0000", 2024-09-15T03:47:11.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1148000000000001e-05 2024-09-15T03:47:11.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844185+0000", 2024-09-15T03:47:11.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5388e-05 2024-09-15T03:47:11.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844202+0000", 2024-09-15T03:47:11.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7708999999999999e-05 2024-09-15T03:47:11.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.854747+0000", 2024-09-15T03:47:11.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010544944000000001 2024-09-15T03:47:11.389 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.389 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.390 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.390 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s7 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:11.390 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844746+0000", 2024-09-15T03:47:11.390 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 13.49503994, 2024-09-15T03:47:11.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095535919999999996, 2024-09-15T03:47:11.390 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.390 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:11.390 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844746+0000", 2024-09-15T03:47:11.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844746+0000", 2024-09-15T03:47:11.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844747+0000", 2024-09-15T03:47:11.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6899999999999996e-07 2024-09-15T03:47:11.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844769+0000", 2024-09-15T03:47:11.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1787999999999999e-05 2024-09-15T03:47:11.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844785+0000", 2024-09-15T03:47:11.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6212999999999999e-05 2024-09-15T03:47:11.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844808+0000", 2024-09-15T03:47:11.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2167e-05 2024-09-15T03:47:11.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844875+0000", 2024-09-15T03:47:11.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7683999999999998e-05 2024-09-15T03:47:11.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844893+0000", 2024-09-15T03:47:11.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7331000000000001e-05 2024-09-15T03:47:11.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844978+0000", 2024-09-15T03:47:11.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5711999999999995e-05 2024-09-15T03:47:11.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844989+0000", 2024-09-15T03:47:11.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1026000000000001e-05 2024-09-15T03:47:11.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847475+0000", 2024-09-15T03:47:11.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024854830000000001 2024-09-15T03:47:11.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847492+0000", 2024-09-15T03:47:11.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7189000000000001e-05 2024-09-15T03:47:11.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.854300+0000", 2024-09-15T03:47:11.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0068080299999999996 2024-09-15T03:47:11.395 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.395 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.396 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.396 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s7 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:11.396 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844831+0000", 2024-09-15T03:47:11.396 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 13.494954895999999, 2024-09-15T03:47:11.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095609189999999993, 2024-09-15T03:47:11.396 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.396 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:11.396 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844831+0000", 2024-09-15T03:47:11.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844831+0000", 2024-09-15T03:47:11.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844832+0000", 2024-09-15T03:47:11.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1399999999999997e-07 2024-09-15T03:47:11.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844855+0000", 2024-09-15T03:47:11.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2904e-05 2024-09-15T03:47:11.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844866+0000", 2024-09-15T03:47:11.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0691e-05 2024-09-15T03:47:11.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844872+0000", 2024-09-15T03:47:11.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.409e-06 2024-09-15T03:47:11.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845030+0000", 2024-09-15T03:47:11.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015832800000000001 2024-09-15T03:47:11.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845037+0000", 2024-09-15T03:47:11.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1099999999999997e-06 2024-09-15T03:47:11.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845118+0000", 2024-09-15T03:47:11.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0840000000000005e-05 2024-09-15T03:47:11.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847478+0000", 2024-09-15T03:47:11.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023595869999999998 2024-09-15T03:47:11.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847529+0000", 2024-09-15T03:47:11.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1585000000000002e-05 2024-09-15T03:47:11.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847544+0000", 2024-09-15T03:47:11.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4365000000000001e-05 2024-09-15T03:47:11.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.854392+0000", 2024-09-15T03:47:11.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0068485860000000003 2024-09-15T03:47:11.401 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.401 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.401 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.402 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 51/31 ECSubWrite(tid=68, reqid=client.4521.0:351, at_version=51'34, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:11.402 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:11.402 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 13.494570716, 2024-09-15T03:47:11.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098406970000000007, 2024-09-15T03:47:11.402 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.402 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:11.402 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.402 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.402 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:11.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:11.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:11.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1299999999999999e-07 2024-09-15T03:47:11.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845232+0000", 2024-09-15T03:47:11.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5366000000000002e-05 2024-09-15T03:47:11.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845242+0000", 2024-09-15T03:47:11.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0403e-05 2024-09-15T03:47:11.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845247+0000", 2024-09-15T03:47:11.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4870000000000004e-06 2024-09-15T03:47:11.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845292+0000", 2024-09-15T03:47:11.405 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5762e-05 2024-09-15T03:47:11.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845302+0000", 2024-09-15T03:47:11.405 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5969999999999998e-06 2024-09-15T03:47:11.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845378+0000", 2024-09-15T03:47:11.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5628000000000002e-05 2024-09-15T03:47:11.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.406 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.406 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847479+0000", 2024-09-15T03:47:11.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021009380000000001 2024-09-15T03:47:11.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.406 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.406 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847530+0000", 2024-09-15T03:47:11.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1548000000000002e-05 2024-09-15T03:47:11.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.406 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.407 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.407 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847544+0000", 2024-09-15T03:47:11.407 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4355000000000001e-05 2024-09-15T03:47:11.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.407 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.407 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.855056+0000", 2024-09-15T03:47:11.407 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075119000000000002 2024-09-15T03:47:11.407 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.407 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.407 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.407 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 51/31 ECSubWrite(tid=93, reqid=client.4521.0:352, at_version=51'24, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:11.408 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:11.408 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 13.492506782, 2024-09-15T03:47:11.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0096136880000000004, 2024-09-15T03:47:11.408 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.408 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:11.408 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.408 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:11.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:11.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:11.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9100000000000003e-07 2024-09-15T03:47:11.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847669+0000", 2024-09-15T03:47:11.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00038855999999999998 2024-09-15T03:47:11.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847682+0000", 2024-09-15T03:47:11.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3112000000000001e-05 2024-09-15T03:47:11.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847687+0000", 2024-09-15T03:47:11.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0880000000000002e-06 2024-09-15T03:47:11.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847745+0000", 2024-09-15T03:47:11.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8400999999999999e-05 2024-09-15T03:47:11.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847756+0000", 2024-09-15T03:47:11.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0325e-05 2024-09-15T03:47:11.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.848092+0000", 2024-09-15T03:47:11.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033636099999999997 2024-09-15T03:47:11.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.848101+0000", 2024-09-15T03:47:11.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6240000000000008e-06 2024-09-15T03:47:11.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850132+0000", 2024-09-15T03:47:11.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020309239999999999 2024-09-15T03:47:11.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.413 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850174+0000", 2024-09-15T03:47:11.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2630999999999999e-05 2024-09-15T03:47:11.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.413 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.413 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856893+0000", 2024-09-15T03:47:11.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0067188710000000004 2024-09-15T03:47:11.413 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.413 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.413 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.413 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:389 3.cs0 3:3164c049:::smithi00132172-34:head [read 0~1179648,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-15T03:47:11.413 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:02.863691+0000", 2024-09-15T03:47:11.414 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 8.4760954339999994, 2024-09-15T03:47:11.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075454679999999996, 2024-09-15T03:47:11.414 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.414 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:11.414 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:11.414 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:11.414 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:11.414 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 389 2024-09-15T03:47:11.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.414 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:02.863691+0000", 2024-09-15T03:47:11.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:02.863691+0000", 2024-09-15T03:47:11.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:02.863691+0000", 2024-09-15T03:47:11.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4900000000000001e-07 2024-09-15T03:47:11.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:02.863694+0000", 2024-09-15T03:47:11.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2630000000000002e-06 2024-09-15T03:47:11.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:02.863699+0000", 2024-09-15T03:47:11.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5480000000000001e-06 2024-09-15T03:47:11.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:02.863705+0000", 2024-09-15T03:47:11.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3400000000000003e-06 2024-09-15T03:47:11.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:02.863809+0000", 2024-09-15T03:47:11.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010391 2024-09-15T03:47:11.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:11.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:02.863816+0000", 2024-09-15T03:47:11.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5209999999999999e-06 2024-09-15T03:47:11.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.418 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:02.865813+0000", 2024-09-15T03:47:11.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019971780000000001 2024-09-15T03:47:11.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.418 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:11.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:02.865874+0000", 2024-09-15T03:47:11.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1016e-05 2024-09-15T03:47:11.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.418 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:02.871236+0000", 2024-09-15T03:47:11.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0053623430000000003 2024-09-15T03:47:11.419 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.419 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.419 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.419 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 57/31 ECSubWrite(tid=90, reqid=client.4521.0:410, at_version=57'46, trim_to=0'0, roll_forward_to=56'44))", 2024-09-15T03:47:11.419 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:03.883223+0000", 2024-09-15T03:47:11.419 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 7.4565636179999997, 2024-09-15T03:47:11.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092188510000000001, 2024-09-15T03:47:11.419 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.419 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:11.419 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.883223+0000", 2024-09-15T03:47:11.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.883223+0000", 2024-09-15T03:47:11.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.883223+0000", 2024-09-15T03:47:11.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1600000000000001e-07 2024-09-15T03:47:11.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884196+0000", 2024-09-15T03:47:11.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00097307999999999997 2024-09-15T03:47:11.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884210+0000", 2024-09-15T03:47:11.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4459e-05 2024-09-15T03:47:11.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884217+0000", 2024-09-15T03:47:11.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2870000000000001e-06 2024-09-15T03:47:11.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884281+0000", 2024-09-15T03:47:11.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4079999999999996e-05 2024-09-15T03:47:11.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884293+0000", 2024-09-15T03:47:11.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2081e-05 2024-09-15T03:47:11.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884681+0000", 2024-09-15T03:47:11.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00038795600000000001 2024-09-15T03:47:11.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884695+0000", 2024-09-15T03:47:11.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4497e-05 2024-09-15T03:47:11.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884962+0000", 2024-09-15T03:47:11.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00026705199999999999 2024-09-15T03:47:11.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.424 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.424 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.424 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884982+0000", 2024-09-15T03:47:11.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9211999999999998e-05 2024-09-15T03:47:11.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.424 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.424 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.424 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.892442+0000", 2024-09-15T03:47:11.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0074600309999999998 2024-09-15T03:47:11.424 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.424 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.424 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.425 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.425 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 60/28 ECSubWrite(tid=73, reqid=client.4521.0:424, at_version=60'41, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:11.425 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.900505+0000", 2024-09-15T03:47:11.425 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.4392811190000003, 2024-09-15T03:47:11.425 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011912449, 2024-09-15T03:47:11.425 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.425 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:11.425 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.426 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.426 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.426 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900505+0000", 2024-09-15T03:47:11.426 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.426 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.426 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.426 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.426 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900505+0000", 2024-09-15T03:47:11.426 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.426 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.426 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.426 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900506+0000", 2024-09-15T03:47:11.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0800000000000003e-07 2024-09-15T03:47:11.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.427 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900527+0000", 2024-09-15T03:47:11.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0682999999999998e-05 2024-09-15T03:47:11.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.427 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900541+0000", 2024-09-15T03:47:11.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4217e-05 2024-09-15T03:47:11.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900547+0000", 2024-09-15T03:47:11.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3010000000000003e-06 2024-09-15T03:47:11.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900596+0000", 2024-09-15T03:47:11.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8612000000000001e-05 2024-09-15T03:47:11.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900607+0000", 2024-09-15T03:47:11.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.063e-05 2024-09-15T03:47:11.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900714+0000", 2024-09-15T03:47:11.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000107321 2024-09-15T03:47:11.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900721+0000", 2024-09-15T03:47:11.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1709999999999996e-06 2024-09-15T03:47:11.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900753+0000", 2024-09-15T03:47:11.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1565e-05 2024-09-15T03:47:11.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900766+0000", 2024-09-15T03:47:11.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3406e-05 2024-09-15T03:47:11.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.912418+0000", 2024-09-15T03:47:11.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011651635 2024-09-15T03:47:11.431 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.431 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.431 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.431 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 60/28 ECSubWrite(tid=74, reqid=client.4521.0:425, at_version=60'42, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:11.431 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.901156+0000", 2024-09-15T03:47:11.431 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.438630367, 2024-09-15T03:47:11.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011338225, 2024-09-15T03:47:11.431 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.431 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:11.431 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.432 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901156+0000", 2024-09-15T03:47:11.432 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.432 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.432 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901156+0000", 2024-09-15T03:47:11.432 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.432 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.432 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901157+0000", 2024-09-15T03:47:11.432 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9400000000000005e-07 2024-09-15T03:47:11.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901168+0000", 2024-09-15T03:47:11.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1924e-05 2024-09-15T03:47:11.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901178+0000", 2024-09-15T03:47:11.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7040000000000006e-06 2024-09-15T03:47:11.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901182+0000", 2024-09-15T03:47:11.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3080000000000001e-06 2024-09-15T03:47:11.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901226+0000", 2024-09-15T03:47:11.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3436999999999998e-05 2024-09-15T03:47:11.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901235+0000", 2024-09-15T03:47:11.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9400000000000008e-06 2024-09-15T03:47:11.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901329+0000", 2024-09-15T03:47:11.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3686999999999999e-05 2024-09-15T03:47:11.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901337+0000", 2024-09-15T03:47:11.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0129999999999993e-06 2024-09-15T03:47:11.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903268+0000", 2024-09-15T03:47:11.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001931545 2024-09-15T03:47:11.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903279+0000", 2024-09-15T03:47:11.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0626e-05 2024-09-15T03:47:11.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.912494+0000", 2024-09-15T03:47:11.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092154469999999999 2024-09-15T03:47:11.436 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.436 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.436 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.436 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 60/31 ECSubWrite(tid=129, reqid=client.4521.0:422, at_version=60'41, trim_to=0'0, roll_forward_to=60'39))", 2024-09-15T03:47:11.436 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.901373+0000", 2024-09-15T03:47:11.437 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.4384131900000003, 2024-09-15T03:47:11.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013085485000000001, 2024-09-15T03:47:11.437 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.437 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:11.437 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901373+0000", 2024-09-15T03:47:11.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901373+0000", 2024-09-15T03:47:11.438 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.438 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.438 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.438 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901374+0000", 2024-09-15T03:47:11.438 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5600000000000005e-07 2024-09-15T03:47:11.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.438 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.438 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.438 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901713+0000", 2024-09-15T03:47:11.438 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033933299999999998 2024-09-15T03:47:11.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.438 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.438 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901726+0000", 2024-09-15T03:47:11.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2843999999999999e-05 2024-09-15T03:47:11.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901730+0000", 2024-09-15T03:47:11.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4660000000000001e-06 2024-09-15T03:47:11.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901773+0000", 2024-09-15T03:47:11.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2542e-05 2024-09-15T03:47:11.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.440 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901783+0000", 2024-09-15T03:47:11.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.041e-05 2024-09-15T03:47:11.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.440 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906331+0000", 2024-09-15T03:47:11.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0045479049999999997 2024-09-15T03:47:11.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.440 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906340+0000", 2024-09-15T03:47:11.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8920000000000006e-06 2024-09-15T03:47:11.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906403+0000", 2024-09-15T03:47:11.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2793999999999995e-05 2024-09-15T03:47:11.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906413+0000", 2024-09-15T03:47:11.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0370999999999999e-05 2024-09-15T03:47:11.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.442 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.442 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.914459+0000", 2024-09-15T03:47:11.442 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0080452719999999991 2024-09-15T03:47:11.442 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.442 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.442 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.442 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.442 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 60/28 ECSubWrite(tid=75, reqid=client.4521.0:425, at_version=0'0, trim_to=0'0, roll_forward_to=60'42))", 2024-09-15T03:47:11.442 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.903981+0000", 2024-09-15T03:47:11.442 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.4358049770000001, 2024-09-15T03:47:11.442 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0086389139999999993, 2024-09-15T03:47:11.442 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.442 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:11.443 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903981+0000", 2024-09-15T03:47:11.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903981+0000", 2024-09-15T03:47:11.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903982+0000", 2024-09-15T03:47:11.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9699999999999996e-07 2024-09-15T03:47:11.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903990+0000", 2024-09-15T03:47:11.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6660000000000008e-06 2024-09-15T03:47:11.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.904001+0000", 2024-09-15T03:47:11.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1083e-05 2024-09-15T03:47:11.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.904005+0000", 2024-09-15T03:47:11.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1309999999999999e-06 2024-09-15T03:47:11.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.904044+0000", 2024-09-15T03:47:11.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8736999999999999e-05 2024-09-15T03:47:11.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.904055+0000", 2024-09-15T03:47:11.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1858999999999999e-05 2024-09-15T03:47:11.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.446 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.904181+0000", 2024-09-15T03:47:11.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000125284 2024-09-15T03:47:11.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.446 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.904187+0000", 2024-09-15T03:47:11.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3149999999999997e-06 2024-09-15T03:47:11.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.446 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906069+0000", 2024-09-15T03:47:11.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001882416 2024-09-15T03:47:11.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906084+0000", 2024-09-15T03:47:11.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4620999999999999e-05 2024-09-15T03:47:11.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.912620+0000", 2024-09-15T03:47:11.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0065362049999999998 2024-09-15T03:47:11.447 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.447 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.448 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.448 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 60/31 ECSubWrite(tid=132, reqid=client.4521.0:422, at_version=0'0, trim_to=0'0, roll_forward_to=60'41))", 2024-09-15T03:47:11.448 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.906928+0000", 2024-09-15T03:47:11.448 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.4328585619999998, 2024-09-15T03:47:11.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075791410000000002, 2024-09-15T03:47:11.448 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.448 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:11.448 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.448 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906928+0000", 2024-09-15T03:47:11.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906928+0000", 2024-09-15T03:47:11.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906929+0000", 2024-09-15T03:47:11.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1500000000000003e-07 2024-09-15T03:47:11.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906939+0000", 2024-09-15T03:47:11.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0835e-05 2024-09-15T03:47:11.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906955+0000", 2024-09-15T03:47:11.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5489e-05 2024-09-15T03:47:11.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906963+0000", 2024-09-15T03:47:11.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0960000000000001e-06 2024-09-15T03:47:11.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.907014+0000", 2024-09-15T03:47:11.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1112999999999997e-05 2024-09-15T03:47:11.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.907022+0000", 2024-09-15T03:47:11.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5220000000000001e-06 2024-09-15T03:47:11.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.907116+0000", 2024-09-15T03:47:11.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4845000000000004e-05 2024-09-15T03:47:11.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.907123+0000", 2024-09-15T03:47:11.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1330000000000004e-06 2024-09-15T03:47:11.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.907149+0000", 2024-09-15T03:47:11.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6336999999999999e-05 2024-09-15T03:47:11.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.907159+0000", 2024-09-15T03:47:11.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0202e-05 2024-09-15T03:47:11.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.914507+0000", 2024-09-15T03:47:11.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0073477539999999997 2024-09-15T03:47:11.453 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.453 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.453 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.453 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es7 60/28 ECSubWrite(tid=89, reqid=client.4521.0:446, at_version=60'57, trim_to=0'0, roll_forward_to=60'55))", 2024-09-15T03:47:11.453 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.961882+0000", 2024-09-15T03:47:11.453 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.3779046409999998, 2024-09-15T03:47:11.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0076377520000000003, 2024-09-15T03:47:11.454 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:11.454 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:11.454 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:11.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:11.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.961882+0000", 2024-09-15T03:47:11.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:11.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.961882+0000", 2024-09-15T03:47:11.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:11.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:11.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.961882+0000", 2024-09-15T03:47:11.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5000000000000002e-07 2024-09-15T03:47:11.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:11.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.962730+0000", 2024-09-15T03:47:11.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00084732700000000004 2024-09-15T03:47:11.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:11.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.962746+0000", 2024-09-15T03:47:11.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5981000000000002e-05 2024-09-15T03:47:11.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:11.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.962752+0000", 2024-09-15T03:47:11.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1550000000000004e-06 2024-09-15T03:47:11.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:11.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.962798+0000", 2024-09-15T03:47:11.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6581e-05 2024-09-15T03:47:11.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:11.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.962806+0000", 2024-09-15T03:47:11.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4590000000000001e-06 2024-09-15T03:47:11.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:11.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.963147+0000", 2024-09-15T03:47:11.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034129899999999999 2024-09-15T03:47:11.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:11.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.963156+0000", 2024-09-15T03:47:11.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2640000000000008e-06 2024-09-15T03:47:11.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:11.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.963314+0000", 2024-09-15T03:47:11.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000158345 2024-09-15T03:47:11.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:11.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.963329+0000", 2024-09-15T03:47:11.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5238999999999999e-05 2024-09-15T03:47:11.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:11.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:11.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:11.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.969519+0000", 2024-09-15T03:47:11.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0061903519999999997 2024-09-15T03:47:11.458 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.458 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.459 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.459 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:11.459 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:11.459 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:11.460 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:47:11.461 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:11.460+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:11.561 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:11.560+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:11.598 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:11.598 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:11.598 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:11.598 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:11.606 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:47:11.661 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:11.661+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:11.701 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:11.701 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:11.701 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:11.701 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:11.701 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:11.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:47:11.764 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:11.762+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:11.806 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:11.806 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:11.806 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:11.806 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [] 2024-09-15T03:47:11.806 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:11.814 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:47:11.835 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-15T03:47:11.861 INFO:tasks.rados.rados.0.smithi001.stdout:181: finishing rollback tid 0 to smithi00132172-47 2024-09-15T03:47:11.861 INFO:tasks.rados.rados.0.smithi001.stdout:184: finishing write tid 1 to smithi00132172-40 2024-09-15T03:47:11.862 INFO:tasks.rados.rados.0.smithi001.stdout:184: finishing write tid 2 to smithi00132172-40 2024-09-15T03:47:11.862 INFO:tasks.rados.rados.0.smithi001.stdout:181: finishing rollback tid 1 to smithi00132172-47 2024-09-15T03:47:11.862 INFO:tasks.rados.rados.0.smithi001.stdout:181: finishing rollback tid 2 to smithi00132172-47 2024-09-15T03:47:11.862 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 47 v 26 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-15T03:47:11.862 INFO:tasks.rados.rados.0.smithi001.stdout:184: finishing write tid 3 to smithi00132172-40 2024-09-15T03:47:11.863 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:11.862+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:11.865 INFO:tasks.rados.rados.0.smithi001.stdout:184: finishing write tid 4 to smithi00132172-40 2024-09-15T03:47:11.865 INFO:tasks.rados.rados.0.smithi001.stdout:182: finishing write tid 1 to smithi00132172-35 2024-09-15T03:47:11.871 INFO:tasks.rados.rados.0.smithi001.stdout:182: finishing write tid 2 to smithi00132172-35 2024-09-15T03:47:11.871 INFO:tasks.rados.rados.0.smithi001.stdout:182: finishing write tid 3 to smithi00132172-35 2024-09-15T03:47:11.871 INFO:tasks.rados.rados.0.smithi001.stdout:182: finishing write tid 4 to smithi00132172-35 2024-09-15T03:47:11.871 INFO:tasks.rados.rados.0.smithi001.stdout:181: done (6 left) 2024-09-15T03:47:11.871 INFO:tasks.rados.rados.0.smithi001.stdout:185: done (5 left) 2024-09-15T03:47:11.871 INFO:tasks.rados.rados.0.smithi001.stdout:186: done (4 left) 2024-09-15T03:47:11.871 INFO:tasks.rados.rados.0.smithi001.stdout:188: done (3 left) 2024-09-15T03:47:11.871 INFO:tasks.rados.rados.0.smithi001.stdout:189: rmattr oid 1 current snap is 21 2024-09-15T03:47:11.871 INFO:tasks.rados.rados.0.smithi001.stdout:189: done (3 left) 2024-09-15T03:47:11.871 INFO:tasks.rados.rados.0.smithi001.stdout:190: rollback oid 19 current snap is 21 2024-09-15T03:47:11.871 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 19 to 11 2024-09-15T03:47:11.871 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 31 current snap is 21 2024-09-15T03:47:11.871 INFO:tasks.rados.rados.0.smithi001.stdout:191: seq_num 95 ranges {1409024=442368,1851392=540672,2392064=622592} 2024-09-15T03:47:11.882 INFO:tasks.rados.rados.0.smithi001.stdout:191: writing smithi00132172-31 from 1409024 to 1851392 tid 1 2024-09-15T03:47:11.883 INFO:tasks.rados.rados.0.smithi001.stdout:191: writing smithi00132172-31 from 1851392 to 2392064 tid 2 2024-09-15T03:47:11.889 INFO:tasks.rados.rados.0.smithi001.stdout:191: writing smithi00132172-31 from 2392064 to 3014656 tid 3 2024-09-15T03:47:11.889 INFO:tasks.rados.rados.0.smithi001.stdout:190: finishing rollback tid 0 to smithi00132172-19 2024-09-15T03:47:11.889 INFO:tasks.rados.rados.0.smithi001.stdout:190: finishing rollback tid 1 to smithi00132172-19 2024-09-15T03:47:11.889 INFO:tasks.rados.rados.0.smithi001.stdout:184: finishing write tid 5 to smithi00132172-40 2024-09-15T03:47:11.889 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 40 v 75 (ObjNum 92 snap 21 seq_num 92) dirty exists 2024-09-15T03:47:11.889 INFO:tasks.rados.rados.0.smithi001.stdout:184: left oid 40 (ObjNum 92 snap 21 seq_num 92) 2024-09-15T03:47:11.889 INFO:tasks.rados.rados.0.smithi001.stdout:180: finishing write tid 1 to smithi00132172-18 2024-09-15T03:47:11.889 INFO:tasks.rados.rados.0.smithi001.stdout:180: finishing write tid 2 to smithi00132172-18 2024-09-15T03:47:11.889 INFO:tasks.rados.rados.0.smithi001.stdout:180: finishing write tid 3 to smithi00132172-18 2024-09-15T03:47:11.890 INFO:tasks.rados.rados.0.smithi001.stdout:184: done (4 left) 2024-09-15T03:47:11.890 INFO:tasks.rados.rados.0.smithi001.stdout:192: copy_from oid 3 from oid 14 current snap is 21 2024-09-15T03:47:11.890 INFO:tasks.rados.rados.0.smithi001.stdout:182: finishing write tid 5 to smithi00132172-35 2024-09-15T03:47:11.890 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 35 v 31 (ObjNum 91 snap 21 seq_num 91) dirty exists 2024-09-15T03:47:11.890 INFO:tasks.rados.rados.0.smithi001.stdout:182: left oid 35 (ObjNum 91 snap 21 seq_num 91) 2024-09-15T03:47:11.890 INFO:tasks.rados.rados.0.smithi001.stdout:190: finishing rollback tid 2 to smithi00132172-19 2024-09-15T03:47:11.890 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 19 v 69 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-15T03:47:11.890 INFO:tasks.rados.rados.0.smithi001.stdout:180: finishing write tid 4 to smithi00132172-18 2024-09-15T03:47:11.890 INFO:tasks.rados.rados.0.smithi001.stdout:180: finishing write tid 5 to smithi00132172-18 2024-09-15T03:47:11.890 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 18 v 25 (ObjNum 90 snap 21 seq_num 90) dirty exists 2024-09-15T03:47:11.890 INFO:tasks.rados.rados.0.smithi001.stdout:180: left oid 18 (ObjNum 90 snap 21 seq_num 90) 2024-09-15T03:47:11.890 INFO:tasks.rados.rados.0.smithi001.stdout:191: finishing write tid 1 to smithi00132172-31 2024-09-15T03:47:11.891 INFO:tasks.rados.rados.0.smithi001.stdout:180: done (4 left) 2024-09-15T03:47:11.891 INFO:tasks.rados.rados.0.smithi001.stdout:182: done (3 left) 2024-09-15T03:47:11.891 INFO:tasks.rados.rados.0.smithi001.stdout:190: done (2 left) 2024-09-15T03:47:11.891 INFO:tasks.rados.rados.0.smithi001.stdout:193: read oid 29 snap 11 2024-09-15T03:47:11.891 INFO:tasks.rados.rados.0.smithi001.stdout:193: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-15T03:47:11.891 INFO:tasks.rados.rados.0.smithi001.stdout:194: rollback oid 44 current snap is 21 2024-09-15T03:47:11.891 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 44 to 1 2024-09-15T03:47:11.891 INFO:tasks.rados.rados.0.smithi001.stdout:191: finishing write tid 2 to smithi00132172-31 2024-09-15T03:47:11.891 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 10 current snap is 21 2024-09-15T03:47:11.891 INFO:tasks.rados.rados.0.smithi001.stdout:195: seq_num 96 ranges {1523712=458752,1982464=688128,2670592=475136} 2024-09-15T03:47:11.900 INFO:tasks.rados.rados.0.smithi001.stdout:195: writing smithi00132172-10 from 1523712 to 1982464 tid 1 2024-09-15T03:47:11.904 INFO:tasks.rados.rados.0.smithi001.stdout:195: writing smithi00132172-10 from 1982464 to 2670592 tid 2 2024-09-15T03:47:11.908 INFO:tasks.rados.rados.0.smithi001.stdout:195: writing smithi00132172-10 from 2670592 to 3145728 tid 3 2024-09-15T03:47:11.908 INFO:tasks.rados.rados.0.smithi001.stdout:196: snap_create 2024-09-15T03:47:11.908 INFO:tasks.rados.rados.0.smithi001.stdout:193: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-15T03:47:11.918 INFO:tasks.rados.rados.0.smithi001.stdout:191: finishing write tid 3 to smithi00132172-31 2024-09-15T03:47:11.919 INFO:tasks.rados.rados.0.smithi001.stdout:191: finishing write tid 4 to smithi00132172-31 2024-09-15T03:47:11.919 INFO:tasks.rados.rados.0.smithi001.stdout:194: finishing rollback tid 0 to smithi00132172-44 2024-09-15T03:47:11.919 INFO:tasks.rados.rados.0.smithi001.stdout:191: finishing write tid 5 to smithi00132172-31 2024-09-15T03:47:11.919 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 31 v 74 (ObjNum 95 snap 21 seq_num 95) dirty exists 2024-09-15T03:47:11.919 INFO:tasks.rados.rados.0.smithi001.stdout:191: left oid 31 (ObjNum 95 snap 21 seq_num 95) 2024-09-15T03:47:11.920 INFO:tasks.rados.rados.0.smithi001.stdout:194: finishing rollback tid 1 to smithi00132172-44 2024-09-15T03:47:11.920 INFO:tasks.rados.rados.0.smithi001.stdout:194: finishing rollback tid 2 to smithi00132172-44 2024-09-15T03:47:11.920 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 44 v 80 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-15T03:47:11.920 INFO:tasks.rados.rados.0.smithi001.stdout:195: finishing write tid 1 to smithi00132172-10 2024-09-15T03:47:11.920 INFO:tasks.rados.rados.0.smithi001.stdout:192: finishing copy_from racing read to smithi00132172-3 2024-09-15T03:47:11.920 INFO:tasks.rados.rados.0.smithi001.stdout:192: finishing copy_from to smithi00132172-3 2024-09-15T03:47:11.920 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 3 v 51 (ObjNum 81 snap 16 seq_num 81) dirty exists 2024-09-15T03:47:11.920 INFO:tasks.rados.rados.0.smithi001.stdout:195: finishing write tid 2 to smithi00132172-10 2024-09-15T03:47:11.920 INFO:tasks.rados.rados.0.smithi001.stdout:195: finishing write tid 3 to smithi00132172-10 2024-09-15T03:47:11.920 INFO:tasks.rados.rados.0.smithi001.stdout:195: finishing write tid 4 to smithi00132172-10 2024-09-15T03:47:11.920 INFO:tasks.rados.rados.0.smithi001.stdout:195: finishing write tid 5 to smithi00132172-10 2024-09-15T03:47:11.920 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 10 v 74 (ObjNum 96 snap 21 seq_num 96) dirty exists 2024-09-15T03:47:11.921 INFO:tasks.rados.rados.0.smithi001.stdout:195: left oid 10 (ObjNum 96 snap 21 seq_num 96) 2024-09-15T03:47:11.938 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:11.939 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:11.939 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:11.939 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:11.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:47:11.988 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:11.987+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:12.065 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:12.064+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:12.083 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:12.083 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:12.084 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:12.084 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:12.084 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:12.092 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:47:12.166 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:12.166+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:12.190 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:12.190 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:12.190 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:12.190 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:47:12.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.190 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:185 3.as0 3:5515acd6:::smithi00132172-13:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:47:12.191 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:47:12.191 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 32.083586381000003, 2024-09-15T03:47:12.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01268643, 2024-09-15T03:47:12.191 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:12.191 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:12.191 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:12.191 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:12.191 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:12.191 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 185 2024-09-15T03:47:12.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.191 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:12.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:12.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:47:12.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:12.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:47:12.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:12.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104710+0000", 2024-09-15T03:47:12.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0200000000000003e-07 2024-09-15T03:47:12.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:12.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104713+0000", 2024-09-15T03:47:12.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2339999999999999e-06 2024-09-15T03:47:12.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:12.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104718+0000", 2024-09-15T03:47:12.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3449999999999996e-06 2024-09-15T03:47:12.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.194 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104722+0000", 2024-09-15T03:47:12.194 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8399999999999997e-06 2024-09-15T03:47:12.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.194 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104849+0000", 2024-09-15T03:47:12.194 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012709599999999999 2024-09-15T03:47:12.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.194 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:12.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104856+0000", 2024-09-15T03:47:12.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.652e-06 2024-09-15T03:47:12.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.195 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.195 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.195 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106851+0000", 2024-09-15T03:47:12.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019954840000000001 2024-09-15T03:47:12.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.195 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.195 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:12.196 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106935+0000", 2024-09-15T03:47:12.196 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4095e-05 2024-09-15T03:47:12.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.196 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.196 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107453+0000", 2024-09-15T03:47:12.196 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00051729499999999997 2024-09-15T03:47:12.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107554+0000", 2024-09-15T03:47:12.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000101251 2024-09-15T03:47:12.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108104+0000", 2024-09-15T03:47:12.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00054995100000000002 2024-09-15T03:47:12.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108503+0000", 2024-09-15T03:47:12.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039953500000000002 2024-09-15T03:47:12.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109405+0000", 2024-09-15T03:47:12.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00090170099999999998 2024-09-15T03:47:12.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:12.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114365+0000", 2024-09-15T03:47:12.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049602120000000003 2024-09-15T03:47:12.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:12.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117396+0000", 2024-09-15T03:47:12.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030303370000000001 2024-09-15T03:47:12.199 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.199 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:12.199 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.199 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:186 3.as0 3:5515acd6:::smithi00132172-13:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:47:12.199 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:47:12.199 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 32.083546591000001, 2024-09-15T03:47:12.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012817485999999999, 2024-09-15T03:47:12.199 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:12.199 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:12.200 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:12.200 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:12.200 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:12.200 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 186 2024-09-15T03:47:12.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.200 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:12.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.200 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:12.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:47:12.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.200 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:12.201 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:47:12.201 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.201 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.201 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.201 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:12.201 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104750+0000", 2024-09-15T03:47:12.201 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1500000000000004e-07 2024-09-15T03:47:12.201 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.201 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.201 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:12.201 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104754+0000", 2024-09-15T03:47:12.201 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.946e-06 2024-09-15T03:47:12.201 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.201 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:12.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104769+0000", 2024-09-15T03:47:12.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.507e-05 2024-09-15T03:47:12.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104776+0000", 2024-09-15T03:47:12.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7849999999999992e-06 2024-09-15T03:47:12.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104888+0000", 2024-09-15T03:47:12.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011164500000000001 2024-09-15T03:47:12.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:12.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104894+0000", 2024-09-15T03:47:12.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.942e-06 2024-09-15T03:47:12.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108172+0000", 2024-09-15T03:47:12.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032781939999999999 2024-09-15T03:47:12.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:12.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108257+0000", 2024-09-15T03:47:12.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4488999999999997e-05 2024-09-15T03:47:12.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108732+0000", 2024-09-15T03:47:12.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047547399999999998 2024-09-15T03:47:12.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108823+0000", 2024-09-15T03:47:12.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1106000000000005e-05 2024-09-15T03:47:12.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108961+0000", 2024-09-15T03:47:12.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013735099999999999 2024-09-15T03:47:12.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109065+0000", 2024-09-15T03:47:12.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104715 2024-09-15T03:47:12.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109406+0000", 2024-09-15T03:47:12.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034062300000000002 2024-09-15T03:47:12.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:12.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114529+0000", 2024-09-15T03:47:12.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.005123219 2024-09-15T03:47:12.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:12.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117566+0000", 2024-09-15T03:47:12.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030372120000000001 2024-09-15T03:47:12.206 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.206 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:12.207 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.207 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:188 3.fs0 3:fe9c5513:::smithi00132172-47:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:47:12.207 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:47:12.207 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 32.083463893999998, 2024-09-15T03:47:12.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012637061, 2024-09-15T03:47:12.207 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:12.207 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:12.207 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:12.207 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:12.207 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:12.207 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 188 2024-09-15T03:47:12.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.208 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:12.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.208 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:12.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:47:12.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.208 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:12.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:47:12.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:12.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:47:12.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2799999999999996e-07 2024-09-15T03:47:12.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:12.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104835+0000", 2024-09-15T03:47:12.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7190000000000001e-06 2024-09-15T03:47:12.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:12.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104842+0000", 2024-09-15T03:47:12.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4580000000000004e-06 2024-09-15T03:47:12.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104846+0000", 2024-09-15T03:47:12.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1250000000000003e-06 2024-09-15T03:47:12.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104884+0000", 2024-09-15T03:47:12.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7075e-05 2024-09-15T03:47:12.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:12.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104888+0000", 2024-09-15T03:47:12.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8520000000000003e-06 2024-09-15T03:47:12.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106860+0000", 2024-09-15T03:47:12.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019719440000000002 2024-09-15T03:47:12.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:12.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106938+0000", 2024-09-15T03:47:12.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7421000000000003e-05 2024-09-15T03:47:12.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.212 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.212 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107368+0000", 2024-09-15T03:47:12.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043042999999999998 2024-09-15T03:47:12.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.212 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.212 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107460+0000", 2024-09-15T03:47:12.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1392999999999997e-05 2024-09-15T03:47:12.213 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.213 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.213 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107473+0000", 2024-09-15T03:47:12.213 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3702e-05 2024-09-15T03:47:12.213 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.213 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.213 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107504+0000", 2024-09-15T03:47:12.213 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1000000000000001e-05 2024-09-15T03:47:12.213 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.213 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108376+0000", 2024-09-15T03:47:12.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00087162000000000003 2024-09-15T03:47:12.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:12.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.111657+0000", 2024-09-15T03:47:12.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032815589999999999 2024-09-15T03:47:12.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:12.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117469+0000", 2024-09-15T03:47:12.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058112349999999997 2024-09-15T03:47:12.215 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.215 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:12.215 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.215 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.215 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:189 3.fs0 3:fe9c5513:::smithi00132172-47:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:47:12.215 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:47:12.215 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 32.083444708000002, 2024-09-15T03:47:12.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012731009999999999, 2024-09-15T03:47:12.215 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:12.215 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:12.215 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:12.215 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:12.216 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:12.216 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 189 2024-09-15T03:47:12.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.216 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:12.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:12.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:47:12.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:12.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:47:12.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:12.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:47:12.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7899999999999999e-07 2024-09-15T03:47:12.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:12.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104853+0000", 2024-09-15T03:47:12.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.933e-06 2024-09-15T03:47:12.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:12.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104859+0000", 2024-09-15T03:47:12.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.592e-06 2024-09-15T03:47:12.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.218 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.218 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104863+0000", 2024-09-15T03:47:12.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4000000000000002e-06 2024-09-15T03:47:12.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.218 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.218 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104910+0000", 2024-09-15T03:47:12.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6678000000000003e-05 2024-09-15T03:47:12.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.218 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.219 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:12.219 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104916+0000", 2024-09-15T03:47:12.219 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1079999999999998e-06 2024-09-15T03:47:12.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.219 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.219 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.219 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107550+0000", 2024-09-15T03:47:12.219 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026344110000000001 2024-09-15T03:47:12.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.219 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.219 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:12.219 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107624+0000", 2024-09-15T03:47:12.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4134000000000005e-05 2024-09-15T03:47:12.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107988+0000", 2024-09-15T03:47:12.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036353600000000002 2024-09-15T03:47:12.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108066+0000", 2024-09-15T03:47:12.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8047999999999995e-05 2024-09-15T03:47:12.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.221 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108105+0000", 2024-09-15T03:47:12.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8868000000000001e-05 2024-09-15T03:47:12.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.221 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.221 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108504+0000", 2024-09-15T03:47:12.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039916600000000001 2024-09-15T03:47:12.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.221 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.221 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108955+0000", 2024-09-15T03:47:12.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045126200000000002 2024-09-15T03:47:12.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.222 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:12.222 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114394+0000", 2024-09-15T03:47:12.222 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0054382750000000002 2024-09-15T03:47:12.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.222 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:12.222 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117582+0000", 2024-09-15T03:47:12.222 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031882199999999999 2024-09-15T03:47:12.222 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.222 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:12.223 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.223 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 38/31 ECSubWrite(tid=36, reqid=client.4521.0:220, at_version=38'23, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:47:12.223 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.148035+0000", 2024-09-15T03:47:12.223 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 27.040260602, 2024-09-15T03:47:12.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010074534, 2024-09-15T03:47:12.223 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:12.223 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.223 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:12.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:12.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148035+0000", 2024-09-15T03:47:12.224 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:12.224 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148035+0000", 2024-09-15T03:47:12.224 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:12.224 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148036+0000", 2024-09-15T03:47:12.224 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3939999999999999e-06 2024-09-15T03:47:12.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:12.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148681+0000", 2024-09-15T03:47:12.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00064493899999999997 2024-09-15T03:47:12.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:12.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148695+0000", 2024-09-15T03:47:12.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4078e-05 2024-09-15T03:47:12.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148701+0000", 2024-09-15T03:47:12.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7230000000000001e-06 2024-09-15T03:47:12.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.226 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148742+0000", 2024-09-15T03:47:12.226 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0781000000000001e-05 2024-09-15T03:47:12.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.226 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148749+0000", 2024-09-15T03:47:12.226 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6109999999999996e-06 2024-09-15T03:47:12.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.226 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149077+0000", 2024-09-15T03:47:12.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000327597 2024-09-15T03:47:12.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149087+0000", 2024-09-15T03:47:12.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4739999999999994e-06 2024-09-15T03:47:12.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149230+0000", 2024-09-15T03:47:12.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014398300000000001 2024-09-15T03:47:12.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149247+0000", 2024-09-15T03:47:12.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6201e-05 2024-09-15T03:47:12.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:12.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158109+0000", 2024-09-15T03:47:12.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0088627529999999993 2024-09-15T03:47:12.228 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.228 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:12.228 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.228 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:231 3.fs0 3:f40d4749:::smithi00132172-35:head [copy-from ver 6] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:12.229 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.155669+0000", 2024-09-15T03:47:12.229 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 27.032626902000001, 2024-09-15T03:47:12.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010370569, 2024-09-15T03:47:12.229 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:12.229 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:12.229 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:12.229 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:12.229 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:12.229 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 231 2024-09-15T03:47:12.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.229 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:12.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:12.230 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155669+0000", 2024-09-15T03:47:12.230 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.230 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:12.230 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155669+0000", 2024-09-15T03:47:12.230 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.230 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:12.230 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155671+0000", 2024-09-15T03:47:12.230 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0829999999999998e-06 2024-09-15T03:47:12.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.231 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:12.231 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155676+0000", 2024-09-15T03:47:12.231 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6210000000000001e-06 2024-09-15T03:47:12.231 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.231 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.231 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:12.231 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155690+0000", 2024-09-15T03:47:12.231 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4166e-05 2024-09-15T03:47:12.231 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.231 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.231 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.231 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155698+0000", 2024-09-15T03:47:12.231 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5560000000000002e-06 2024-09-15T03:47:12.231 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.232 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.232 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.232 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155748+0000", 2024-09-15T03:47:12.232 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9851000000000001e-05 2024-09-15T03:47:12.232 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.232 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.232 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:12.232 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155872+0000", 2024-09-15T03:47:12.232 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000124091 2024-09-15T03:47:12.232 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.232 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.232 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.232 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163470+0000", 2024-09-15T03:47:12.233 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075976730000000001 2024-09-15T03:47:12.233 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.233 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.233 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.233 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163801+0000", 2024-09-15T03:47:12.233 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033157999999999999 2024-09-15T03:47:12.233 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.233 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.233 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.233 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163814+0000", 2024-09-15T03:47:12.233 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2659e-05 2024-09-15T03:47:12.233 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.233 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.233 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.234 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163935+0000", 2024-09-15T03:47:12.234 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000121155 2024-09-15T03:47:12.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.234 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.234 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.234 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.164131+0000", 2024-09-15T03:47:12.234 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019600499999999999 2024-09-15T03:47:12.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.234 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.234 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:12.234 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.165998+0000", 2024-09-15T03:47:12.234 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001866882 2024-09-15T03:47:12.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.234 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.235 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:12.235 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.166039+0000", 2024-09-15T03:47:12.235 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1247e-05 2024-09-15T03:47:12.235 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.235 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:12.235 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.235 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.235 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.235 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:232 3.fs0 3:f40d4749:::smithi00132172-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:12.235 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:47:12.235 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 27.032589612999999, 2024-09-15T03:47:12.235 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084205730000000006, 2024-09-15T03:47:12.236 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:12.236 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:12.236 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:12.236 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:12.236 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:12.236 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 232 2024-09-15T03:47:12.236 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.236 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:12.236 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.236 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:12.236 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:47:12.236 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.236 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.237 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.237 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:12.237 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:47:12.237 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.237 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.237 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:12.237 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:47:12.237 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7399999999999999e-07 2024-09-15T03:47:12.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.237 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.237 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:12.237 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155708+0000", 2024-09-15T03:47:12.237 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6500000000000001e-06 2024-09-15T03:47:12.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.238 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.238 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:12.238 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155713+0000", 2024-09-15T03:47:12.238 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2519999999999999e-06 2024-09-15T03:47:12.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.238 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.238 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.238 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155716+0000", 2024-09-15T03:47:12.238 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4430000000000002e-06 2024-09-15T03:47:12.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.238 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.238 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.239 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156060+0000", 2024-09-15T03:47:12.239 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034440000000000002 2024-09-15T03:47:12.239 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.239 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.239 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:12.239 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156112+0000", 2024-09-15T03:47:12.239 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1928000000000002e-05 2024-09-15T03:47:12.239 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.239 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.239 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.239 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163929+0000", 2024-09-15T03:47:12.239 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0078168999999999999 2024-09-15T03:47:12.239 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.239 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.240 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:12.240 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.164027+0000", 2024-09-15T03:47:12.240 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8109999999999994e-05 2024-09-15T03:47:12.240 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.240 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.240 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:12.240 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.164126+0000", 2024-09-15T03:47:12.240 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9516000000000001e-05 2024-09-15T03:47:12.240 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.240 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:12.240 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.240 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.240 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.241 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 39/31 ECSubWrite(tid=43, reqid=client.4521.0:252, at_version=39'27, trim_to=0'0, roll_forward_to=39'25))", 2024-09-15T03:47:12.241 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:47:12.241 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 26.00994992, 2024-09-15T03:47:12.241 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082900309999999998, 2024-09-15T03:47:12.241 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:12.241 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.241 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:12.241 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.241 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:12.241 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:47:12.241 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.241 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.241 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.242 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:12.242 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:47:12.242 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.242 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.242 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:12.242 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:47:12.242 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6199999999999998e-07 2024-09-15T03:47:12.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.242 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.242 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:12.242 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178642+0000", 2024-09-15T03:47:12.242 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029559399999999999 2024-09-15T03:47:12.243 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.243 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.243 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:12.243 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178653+0000", 2024-09-15T03:47:12.243 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0865e-05 2024-09-15T03:47:12.243 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.243 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.243 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.243 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178658+0000", 2024-09-15T03:47:12.243 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3720000000000004e-06 2024-09-15T03:47:12.243 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.243 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.243 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.244 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178695+0000", 2024-09-15T03:47:12.244 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6708999999999997e-05 2024-09-15T03:47:12.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.244 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.244 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.244 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178701+0000", 2024-09-15T03:47:12.244 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7989999999999996e-06 2024-09-15T03:47:12.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.244 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.244 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.244 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178926+0000", 2024-09-15T03:47:12.244 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022405299999999999 2024-09-15T03:47:12.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.244 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.245 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.245 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178935+0000", 2024-09-15T03:47:12.245 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9649999999999997e-06 2024-09-15T03:47:12.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.245 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.245 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.245 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179230+0000", 2024-09-15T03:47:12.245 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029551000000000002 2024-09-15T03:47:12.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.245 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.245 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.245 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179247+0000", 2024-09-15T03:47:12.245 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6685999999999999e-05 2024-09-15T03:47:12.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.246 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.246 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:12.246 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.186636+0000", 2024-09-15T03:47:12.246 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0073889159999999997 2024-09-15T03:47:12.246 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.246 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:12.246 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.246 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.246 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 49/31 ECSubWrite(tid=87, reqid=client.4521.0:335, at_version=49'21, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:12.246 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.858534+0000", 2024-09-15T03:47:12.246 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.329761783999999, 2024-09-15T03:47:12.247 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094628430000000003, 2024-09-15T03:47:12.247 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:12.247 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.247 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:12.247 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.247 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:12.247 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858534+0000", 2024-09-15T03:47:12.247 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.247 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.247 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.247 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:12.247 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858534+0000", 2024-09-15T03:47:12.247 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.247 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.248 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.248 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:12.248 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858534+0000", 2024-09-15T03:47:12.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2300000000000001e-07 2024-09-15T03:47:12.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.248 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.248 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:12.248 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858553+0000", 2024-09-15T03:47:12.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8221000000000001e-05 2024-09-15T03:47:12.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.248 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.248 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:12.248 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858564+0000", 2024-09-15T03:47:12.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1471000000000001e-05 2024-09-15T03:47:12.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.249 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.249 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.249 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858570+0000", 2024-09-15T03:47:12.249 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7640000000000002e-06 2024-09-15T03:47:12.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.249 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.249 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.249 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858621+0000", 2024-09-15T03:47:12.249 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0943999999999999e-05 2024-09-15T03:47:12.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.249 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.250 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.250 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858626+0000", 2024-09-15T03:47:12.250 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4949999999999999e-06 2024-09-15T03:47:12.250 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.250 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.250 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.250 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858715+0000", 2024-09-15T03:47:12.250 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9061000000000007e-05 2024-09-15T03:47:12.250 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.250 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.250 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.250 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858722+0000", 2024-09-15T03:47:12.250 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9689999999999997e-06 2024-09-15T03:47:12.251 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.251 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.251 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.251 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858748+0000", 2024-09-15T03:47:12.251 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6179000000000001e-05 2024-09-15T03:47:12.251 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.251 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.251 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.251 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858761+0000", 2024-09-15T03:47:12.251 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2159e-05 2024-09-15T03:47:12.251 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.251 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.251 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:12.251 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.867997+0000", 2024-09-15T03:47:12.252 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092359569999999995 2024-09-15T03:47:12.252 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.252 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:12.252 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.252 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.252 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.252 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 49/31 ECSubWrite(tid=88, reqid=client.4521.0:336, at_version=49'22, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:12.252 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:47:12.252 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.329034142000001, 2024-09-15T03:47:12.252 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0088255839999999992, 2024-09-15T03:47:12.252 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:12.252 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.253 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:12.253 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.253 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:12.253 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:47:12.253 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.253 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.253 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.253 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:12.253 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:47:12.253 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.253 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.253 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.253 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:12.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859262+0000", 2024-09-15T03:47:12.254 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2399999999999999e-07 2024-09-15T03:47:12.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.254 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.254 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:12.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859275+0000", 2024-09-15T03:47:12.254 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3395e-05 2024-09-15T03:47:12.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.254 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.254 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:12.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859285+0000", 2024-09-15T03:47:12.254 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8959999999999998e-06 2024-09-15T03:47:12.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.255 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.255 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859289+0000", 2024-09-15T03:47:12.255 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2549999999999999e-06 2024-09-15T03:47:12.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.255 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.255 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859327+0000", 2024-09-15T03:47:12.255 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7753000000000003e-05 2024-09-15T03:47:12.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.255 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.255 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859335+0000", 2024-09-15T03:47:12.256 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3000000000000002e-06 2024-09-15T03:47:12.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.256 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.256 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.256 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859391+0000", 2024-09-15T03:47:12.256 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6085000000000003e-05 2024-09-15T03:47:12.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.256 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.256 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.256 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859415+0000", 2024-09-15T03:47:12.256 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3295999999999999e-05 2024-09-15T03:47:12.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.256 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.256 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.257 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859432+0000", 2024-09-15T03:47:12.257 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7087000000000001e-05 2024-09-15T03:47:12.257 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.257 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.257 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.257 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859442+0000", 2024-09-15T03:47:12.257 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0339e-05 2024-09-15T03:47:12.257 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.257 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.257 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:12.257 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.868087+0000", 2024-09-15T03:47:12.257 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0086447539999999993 2024-09-15T03:47:12.257 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.258 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:12.258 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.258 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.258 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.258 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s6 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:12.258 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.846035+0000", 2024-09-15T03:47:12.258 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.342260915000001, 2024-09-15T03:47:12.258 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010491004, 2024-09-15T03:47:12.258 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:12.258 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.258 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:12.258 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.258 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:12.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846035+0000", 2024-09-15T03:47:12.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:12.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846035+0000", 2024-09-15T03:47:12.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:12.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846035+0000", 2024-09-15T03:47:12.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8599999999999997e-07 2024-09-15T03:47:12.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:12.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846068+0000", 2024-09-15T03:47:12.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2138000000000003e-05 2024-09-15T03:47:12.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:12.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846081+0000", 2024-09-15T03:47:12.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3127e-05 2024-09-15T03:47:12.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846092+0000", 2024-09-15T03:47:12.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1634000000000001e-05 2024-09-15T03:47:12.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.261 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.261 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846136+0000", 2024-09-15T03:47:12.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3999999999999999e-05 2024-09-15T03:47:12.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.261 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.261 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846148+0000", 2024-09-15T03:47:12.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1939e-05 2024-09-15T03:47:12.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.261 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.262 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846220+0000", 2024-09-15T03:47:12.262 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1248999999999995e-05 2024-09-15T03:47:12.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.262 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.262 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846231+0000", 2024-09-15T03:47:12.262 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1379999999999999e-05 2024-09-15T03:47:12.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.262 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.262 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847412+0000", 2024-09-15T03:47:12.262 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001181407 2024-09-15T03:47:12.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.263 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.263 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847435+0000", 2024-09-15T03:47:12.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3084000000000001e-05 2024-09-15T03:47:12.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.263 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:12.263 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856526+0000", 2024-09-15T03:47:12.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0090901599999999999 2024-09-15T03:47:12.263 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.263 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:12.263 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.264 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s6 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:12.264 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.846101+0000", 2024-09-15T03:47:12.264 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.342194106999999, 2024-09-15T03:47:12.264 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010577013999999999, 2024-09-15T03:47:12.264 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:12.264 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.264 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:12.264 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.264 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:12.264 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846101+0000", 2024-09-15T03:47:12.264 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:12.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846101+0000", 2024-09-15T03:47:12.265 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:12.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846102+0000", 2024-09-15T03:47:12.265 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5100000000000001e-07 2024-09-15T03:47:12.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:12.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846110+0000", 2024-09-15T03:47:12.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7069999999999998e-06 2024-09-15T03:47:12.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.266 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.266 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:12.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846115+0000", 2024-09-15T03:47:12.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2799999999999997e-06 2024-09-15T03:47:12.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.266 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.266 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846117+0000", 2024-09-15T03:47:12.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5909999999999998e-06 2024-09-15T03:47:12.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.266 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846259+0000", 2024-09-15T03:47:12.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014169900000000001 2024-09-15T03:47:12.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.267 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846268+0000", 2024-09-15T03:47:12.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5159999999999994e-06 2024-09-15T03:47:12.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.267 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846335+0000", 2024-09-15T03:47:12.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7936999999999997e-05 2024-09-15T03:47:12.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847415+0000", 2024-09-15T03:47:12.268 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001079874 2024-09-15T03:47:12.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847453+0000", 2024-09-15T03:47:12.268 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7808999999999997e-05 2024-09-15T03:47:12.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847477+0000", 2024-09-15T03:47:12.269 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3609e-05 2024-09-15T03:47:12.269 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.269 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.269 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:12.269 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856678+0000", 2024-09-15T03:47:12.269 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.009201641 2024-09-15T03:47:12.269 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.269 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:12.269 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.269 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.269 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.269 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 51/31 ECSubWrite(tid=93, reqid=client.4521.0:352, at_version=51'24, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:12.269 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.847310+0000", 2024-09-15T03:47:12.270 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.340985801, 2024-09-15T03:47:12.270 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012670010000000001, 2024-09-15T03:47:12.270 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:12.270 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.270 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:12.270 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.270 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:12.270 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847310+0000", 2024-09-15T03:47:12.270 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.270 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.270 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.270 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:12.270 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847310+0000", 2024-09-15T03:47:12.271 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.271 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:12.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847310+0000", 2024-09-15T03:47:12.271 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8899999999999999e-07 2024-09-15T03:47:12.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.271 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:12.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847750+0000", 2024-09-15T03:47:12.271 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000439238 2024-09-15T03:47:12.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:12.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847767+0000", 2024-09-15T03:47:12.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7362000000000001e-05 2024-09-15T03:47:12.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847774+0000", 2024-09-15T03:47:12.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0169999999999999e-06 2024-09-15T03:47:12.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847831+0000", 2024-09-15T03:47:12.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7207999999999998e-05 2024-09-15T03:47:12.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847841+0000", 2024-09-15T03:47:12.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7829999999999993e-06 2024-09-15T03:47:12.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.848145+0000", 2024-09-15T03:47:12.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030431300000000001 2024-09-15T03:47:12.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.848179+0000", 2024-09-15T03:47:12.274 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3423999999999997e-05 2024-09-15T03:47:12.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.274 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.274 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.274 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850141+0000", 2024-09-15T03:47:12.274 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019623380000000001 2024-09-15T03:47:12.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.274 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.274 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.274 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850164+0000", 2024-09-15T03:47:12.274 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2969e-05 2024-09-15T03:47:12.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.274 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.275 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:12.275 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.859980+0000", 2024-09-15T03:47:12.275 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098156690000000008 2024-09-15T03:47:12.275 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.275 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:12.275 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.275 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.275 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 51/31 ECSubWrite(tid=94, reqid=client.4521.0:353, at_version=51'25, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:12.275 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.850957+0000", 2024-09-15T03:47:12.275 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.337338012, 2024-09-15T03:47:12.275 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092043439999999997, 2024-09-15T03:47:12.275 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:12.276 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.276 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:12.276 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.276 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:12.276 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850957+0000", 2024-09-15T03:47:12.276 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.276 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.276 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:12.276 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850957+0000", 2024-09-15T03:47:12.276 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.276 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.277 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:12.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850958+0000", 2024-09-15T03:47:12.277 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9500000000000002e-07 2024-09-15T03:47:12.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.277 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.277 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:12.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851297+0000", 2024-09-15T03:47:12.277 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033833300000000001 2024-09-15T03:47:12.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.277 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.277 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:12.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851311+0000", 2024-09-15T03:47:12.277 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4144999999999999e-05 2024-09-15T03:47:12.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.278 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.278 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.278 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851318+0000", 2024-09-15T03:47:12.278 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8700000000000003e-06 2024-09-15T03:47:12.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.278 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.278 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.278 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851390+0000", 2024-09-15T03:47:12.278 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.203e-05 2024-09-15T03:47:12.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.278 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.278 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.279 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851402+0000", 2024-09-15T03:47:12.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2146e-05 2024-09-15T03:47:12.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.279 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.279 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851556+0000", 2024-09-15T03:47:12.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015461199999999999 2024-09-15T03:47:12.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.279 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.279 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851570+0000", 2024-09-15T03:47:12.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4106e-05 2024-09-15T03:47:12.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.280 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.280 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.280 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853231+0000", 2024-09-15T03:47:12.280 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001660559 2024-09-15T03:47:12.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.280 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.280 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.280 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853246+0000", 2024-09-15T03:47:12.280 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5457000000000001e-05 2024-09-15T03:47:12.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.280 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.280 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:12.280 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.860162+0000", 2024-09-15T03:47:12.281 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069153909999999999 2024-09-15T03:47:12.281 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.281 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:12.281 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.281 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.281 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.281 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 51/31 ECSubWrite(tid=95, reqid=client.4521.0:354, at_version=51'26, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:12.281 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.851558+0000", 2024-09-15T03:47:12.281 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.336737874000001, 2024-09-15T03:47:12.281 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0087097059999999994, 2024-09-15T03:47:12.281 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:12.281 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.281 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:12.282 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.282 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:12.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851558+0000", 2024-09-15T03:47:12.282 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.282 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.282 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:12.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851558+0000", 2024-09-15T03:47:12.282 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.282 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.282 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:12.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851559+0000", 2024-09-15T03:47:12.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7599999999999996e-07 2024-09-15T03:47:12.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.283 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.283 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:12.283 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851599+0000", 2024-09-15T03:47:12.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0472000000000001e-05 2024-09-15T03:47:12.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.283 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.283 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:12.283 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851614+0000", 2024-09-15T03:47:12.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4640999999999999e-05 2024-09-15T03:47:12.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.284 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.284 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.284 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851620+0000", 2024-09-15T03:47:12.284 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3430000000000001e-06 2024-09-15T03:47:12.284 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.284 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.284 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.284 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851680+0000", 2024-09-15T03:47:12.284 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9648000000000002e-05 2024-09-15T03:47:12.284 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.284 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.284 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.284 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851691+0000", 2024-09-15T03:47:12.285 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1012999999999999e-05 2024-09-15T03:47:12.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.285 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.285 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.285 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851803+0000", 2024-09-15T03:47:12.285 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112089 2024-09-15T03:47:12.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.285 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.285 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.285 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853234+0000", 2024-09-15T03:47:12.285 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0014310810000000001 2024-09-15T03:47:12.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.285 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.286 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.286 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853304+0000", 2024-09-15T03:47:12.286 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0656999999999998e-05 2024-09-15T03:47:12.286 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.286 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.286 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.286 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853317+0000", 2024-09-15T03:47:12.286 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2186e-05 2024-09-15T03:47:12.286 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.286 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.286 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:12.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.860267+0000", 2024-09-15T03:47:12.287 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069506999999999998 2024-09-15T03:47:12.287 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.287 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:12.287 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.287 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.287 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:424 3.as0 3:5515acd6:::smithi00132172-13:head [rollback 17] snapc 1a=[1a,18,17,16,15,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:47:12.287 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.898858+0000", 2024-09-15T03:47:12.287 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.2894376730000001, 2024-09-15T03:47:12.287 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013029552999999999, 2024-09-15T03:47:12.287 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:12.287 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:12.287 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:12.288 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:12.288 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:12.288 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 424 2024-09-15T03:47:12.288 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.288 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:12.288 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.288 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:12.288 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898858+0000", 2024-09-15T03:47:12.288 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.288 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.288 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.288 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:12.288 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898858+0000", 2024-09-15T03:47:12.289 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.289 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.289 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:12.289 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898858+0000", 2024-09-15T03:47:12.289 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2300000000000002e-07 2024-09-15T03:47:12.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.289 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.289 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:12.289 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898860+0000", 2024-09-15T03:47:12.289 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.623e-06 2024-09-15T03:47:12.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.289 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.289 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:12.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898863+0000", 2024-09-15T03:47:12.290 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5870000000000001e-06 2024-09-15T03:47:12.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.290 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.290 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898865+0000", 2024-09-15T03:47:12.290 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.075e-06 2024-09-15T03:47:12.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.290 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.290 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.899786+0000", 2024-09-15T03:47:12.290 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00092033499999999999 2024-09-15T03:47:12.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.291 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.291 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:12.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.899886+0000", 2024-09-15T03:47:12.291 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000100516 2024-09-15T03:47:12.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.291 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.291 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900376+0000", 2024-09-15T03:47:12.291 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00048932700000000001 2024-09-15T03:47:12.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.291 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.291 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900486+0000", 2024-09-15T03:47:12.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011024 2024-09-15T03:47:12.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.292 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.292 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.292 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900492+0000", 2024-09-15T03:47:12.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.908e-06 2024-09-15T03:47:12.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.292 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.292 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.292 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900724+0000", 2024-09-15T03:47:12.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000232382 2024-09-15T03:47:12.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.292 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.292 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.293 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901279+0000", 2024-09-15T03:47:12.293 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00055503599999999996 2024-09-15T03:47:12.293 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.293 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.293 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:12.293 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.902348+0000", 2024-09-15T03:47:12.293 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010692130000000001 2024-09-15T03:47:12.293 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.293 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.293 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:12.293 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.911887+0000", 2024-09-15T03:47:12.293 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.009538988 2024-09-15T03:47:12.294 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.294 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:12.294 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.294 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.294 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:425 3.as0 3:5515acd6:::smithi00132172-13:head [append 0~0] snapc 1a=[1a,18,17,16,15,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:47:12.294 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.898884+0000", 2024-09-15T03:47:12.294 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.2894112829999997, 2024-09-15T03:47:12.294 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013107466999999999, 2024-09-15T03:47:12.294 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:12.294 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:12.294 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:12.294 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:12.295 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:12.295 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 425 2024-09-15T03:47:12.295 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.295 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:12.295 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.295 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:12.295 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898884+0000", 2024-09-15T03:47:12.295 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.295 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.295 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.295 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:12.295 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898884+0000", 2024-09-15T03:47:12.295 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.295 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.296 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.296 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:12.296 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898884+0000", 2024-09-15T03:47:12.296 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4499999999999998e-07 2024-09-15T03:47:12.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.296 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.296 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:12.296 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898886+0000", 2024-09-15T03:47:12.296 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0140000000000001e-06 2024-09-15T03:47:12.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.296 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.296 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:12.296 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898890+0000", 2024-09-15T03:47:12.297 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9179999999999996e-06 2024-09-15T03:47:12.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.297 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.297 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.297 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898893+0000", 2024-09-15T03:47:12.297 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2900000000000001e-06 2024-09-15T03:47:12.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.297 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.297 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.297 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900560+0000", 2024-09-15T03:47:12.297 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00166749 2024-09-15T03:47:12.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.297 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.298 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:12.298 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900717+0000", 2024-09-15T03:47:12.298 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000157174 2024-09-15T03:47:12.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.298 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.298 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.298 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901127+0000", 2024-09-15T03:47:12.298 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00040965100000000003 2024-09-15T03:47:12.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.298 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.298 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.298 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901239+0000", 2024-09-15T03:47:12.298 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112324 2024-09-15T03:47:12.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.299 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.299 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.299 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901246+0000", 2024-09-15T03:47:12.299 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8730000000000001e-06 2024-09-15T03:47:12.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.299 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.299 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.299 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903551+0000", 2024-09-15T03:47:12.299 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023044459999999999 2024-09-15T03:47:12.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.299 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.299 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.299 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903565+0000", 2024-09-15T03:47:12.300 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4066999999999999e-05 2024-09-15T03:47:12.300 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.300 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.300 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:12.300 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903748+0000", 2024-09-15T03:47:12.300 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018303899999999999 2024-09-15T03:47:12.300 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.300 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.300 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:12.300 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.911992+0000", 2024-09-15T03:47:12.300 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082439360000000003 2024-09-15T03:47:12.300 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.300 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:12.301 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.301 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.301 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.301 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 60/31 ECSubWrite(tid=129, reqid=client.4521.0:422, at_version=60'41, trim_to=0'0, roll_forward_to=60'39))", 2024-09-15T03:47:12.301 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.901153+0000", 2024-09-15T03:47:12.301 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.2871420569999996, 2024-09-15T03:47:12.301 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017516662999999998, 2024-09-15T03:47:12.301 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:12.301 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.301 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:12.301 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.301 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:12.302 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901153+0000", 2024-09-15T03:47:12.302 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.302 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.302 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:12.302 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901153+0000", 2024-09-15T03:47:12.302 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.302 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.302 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:12.302 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901154+0000", 2024-09-15T03:47:12.302 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3000000000000001e-07 2024-09-15T03:47:12.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.302 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.303 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:12.303 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901593+0000", 2024-09-15T03:47:12.303 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043868200000000003 2024-09-15T03:47:12.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.303 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.303 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:12.303 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901605+0000", 2024-09-15T03:47:12.303 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1987000000000001e-05 2024-09-15T03:47:12.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.303 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.303 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.303 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901611+0000", 2024-09-15T03:47:12.303 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1720000000000004e-06 2024-09-15T03:47:12.304 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.304 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.304 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.304 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901648+0000", 2024-09-15T03:47:12.304 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7734999999999997e-05 2024-09-15T03:47:12.304 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.304 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.304 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.304 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901657+0000", 2024-09-15T03:47:12.304 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.242e-06 2024-09-15T03:47:12.304 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.304 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.304 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.305 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903437+0000", 2024-09-15T03:47:12.305 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001780404 2024-09-15T03:47:12.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.305 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.305 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.305 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903553+0000", 2024-09-15T03:47:12.305 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000115927 2024-09-15T03:47:12.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.305 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.305 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.305 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906054+0000", 2024-09-15T03:47:12.305 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002500604 2024-09-15T03:47:12.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.305 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.306 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.306 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906067+0000", 2024-09-15T03:47:12.306 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3699e-05 2024-09-15T03:47:12.306 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.306 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.306 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:12.306 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.918670+0000", 2024-09-15T03:47:12.306 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012602781 2024-09-15T03:47:12.306 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.306 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:12.306 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.306 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.306 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.307 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 60/31 ECSubWrite(tid=132, reqid=client.4521.0:422, at_version=0'0, trim_to=0'0, roll_forward_to=60'41))", 2024-09-15T03:47:12.307 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.906920+0000", 2024-09-15T03:47:12.307 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.2813752320000003, 2024-09-15T03:47:12.307 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011799888, 2024-09-15T03:47:12.307 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:12.307 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.307 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:12.307 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.307 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:12.307 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906920+0000", 2024-09-15T03:47:12.307 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.307 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.308 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:12.308 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906920+0000", 2024-09-15T03:47:12.308 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.308 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.308 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.308 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:12.308 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906921+0000", 2024-09-15T03:47:12.308 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4600000000000005e-07 2024-09-15T03:47:12.308 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.308 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.308 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:12.308 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906929+0000", 2024-09-15T03:47:12.308 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7840000000000004e-06 2024-09-15T03:47:12.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.309 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.309 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:12.309 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906938+0000", 2024-09-15T03:47:12.309 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4299999999999995e-06 2024-09-15T03:47:12.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.309 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.309 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.309 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906943+0000", 2024-09-15T03:47:12.309 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1209999999999998e-06 2024-09-15T03:47:12.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.309 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.309 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.310 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906988+0000", 2024-09-15T03:47:12.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5086999999999997e-05 2024-09-15T03:47:12.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.310 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.310 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.310 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906998+0000", 2024-09-15T03:47:12.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5610000000000004e-06 2024-09-15T03:47:12.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.310 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.310 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.310 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.907078+0000", 2024-09-15T03:47:12.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9989999999999998e-05 2024-09-15T03:47:12.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.310 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.907087+0000", 2024-09-15T03:47:12.311 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4110000000000002e-06 2024-09-15T03:47:12.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.311 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.908905+0000", 2024-09-15T03:47:12.311 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0018175229999999999 2024-09-15T03:47:12.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.311 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.908914+0000", 2024-09-15T03:47:12.311 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5829999999999996e-06 2024-09-15T03:47:12.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.312 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.312 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:12.312 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.918720+0000", 2024-09-15T03:47:12.312 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098058520000000003 2024-09-15T03:47:12.312 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.312 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:12.312 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.312 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.312 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 65/64 ECSubWrite(tid=110, reqid=client.4521.0:474, at_version=65'72, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:12.312 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:11.849490+0000", 2024-09-15T03:47:12.312 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.33880558100000002, 2024-09-15T03:47:12.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0099941470000000001, 2024-09-15T03:47:12.313 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:12.313 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.313 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:12.313 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.313 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:12.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.849490+0000", 2024-09-15T03:47:12.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.313 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.313 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:12.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.849490+0000", 2024-09-15T03:47:12.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:12.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.314 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.314 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:12.314 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.849490+0000", 2024-09-15T03:47:12.314 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8099999999999998e-07 2024-09-15T03:47:12.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.314 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.314 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:12.314 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.849594+0000", 2024-09-15T03:47:12.314 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000103806 2024-09-15T03:47:12.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.314 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.314 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:12.314 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.849602+0000", 2024-09-15T03:47:12.314 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6159999999999996e-06 2024-09-15T03:47:12.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.315 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.315 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.315 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.849607+0000", 2024-09-15T03:47:12.315 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.463e-06 2024-09-15T03:47:12.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.315 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.315 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:12.315 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.849732+0000", 2024-09-15T03:47:12.315 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012485900000000001 2024-09-15T03:47:12.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.315 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.315 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.315 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.849740+0000", 2024-09-15T03:47:12.315 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3180000000000007e-06 2024-09-15T03:47:12.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.316 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.316 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.850040+0000", 2024-09-15T03:47:12.316 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000299141 2024-09-15T03:47:12.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.316 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.316 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.850085+0000", 2024-09-15T03:47:12.316 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.566e-05 2024-09-15T03:47:12.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.316 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.316 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.850683+0000", 2024-09-15T03:47:12.317 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00059757400000000002 2024-09-15T03:47:12.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.317 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.317 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.850729+0000", 2024-09-15T03:47:12.317 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6548999999999998e-05 2024-09-15T03:47:12.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:12.317 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:12.317 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:12.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.859484+0000", 2024-09-15T03:47:12.317 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0087547800000000002 2024-09-15T03:47:12.317 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.318 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:12.318 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.318 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:12.318 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:12.318 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:12.319 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:47:12.320 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:12.318+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:12.420 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:12.419+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:12.430 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:12.430 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:12.430 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-15T03:47:12.430 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:12.439 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:47:12.521 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:12.520+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:12.541 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:12.541 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:12.541 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-15T03:47:12.541 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-15T03:47:12.541 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:12.549 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:47:12.622 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:12.621+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:12.651 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:12.651 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-15T03:47:12.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-15T03:47:12.652 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-15T03:47:12.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.652 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.4s4 39/31 ECSubWrite(tid=41, reqid=osd.6.0:41, at_version=39'9, trim_to=0'0, roll_forward_to=38'7))", 2024-09-15T03:47:12.652 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:46.146606+0000", 2024-09-15T03:47:12.652 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 26.502766803, 2024-09-15T03:47:12.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0048345970000000004, 2024-09-15T03:47:12.652 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:12.652 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.652 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:12.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:12.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.146606+0000", 2024-09-15T03:47:12.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:12.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.146606+0000", 2024-09-15T03:47:12.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:12.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.146606+0000", 2024-09-15T03:47:12.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3200000000000005e-07 2024-09-15T03:47:12.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:12.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.146636+0000", 2024-09-15T03:47:12.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9434e-05 2024-09-15T03:47:12.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:12.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.146666+0000", 2024-09-15T03:47:12.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9733000000000001e-05 2024-09-15T03:47:12.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.146674+0000", 2024-09-15T03:47:12.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2919999999999996e-06 2024-09-15T03:47:12.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:12.655 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150070+0000", 2024-09-15T03:47:12.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0033961590000000002 2024-09-15T03:47:12.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150080+0000", 2024-09-15T03:47:12.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0181e-05 2024-09-15T03:47:12.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150334+0000", 2024-09-15T03:47:12.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00025387100000000002 2024-09-15T03:47:12.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150370+0000", 2024-09-15T03:47:12.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6269999999999997e-05 2024-09-15T03:47:12.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150430+0000", 2024-09-15T03:47:12.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9404000000000002e-05 2024-09-15T03:47:12.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.150480+0000", 2024-09-15T03:47:12.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0541000000000003e-05 2024-09-15T03:47:12.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:12.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:46.151441+0000", 2024-09-15T03:47:12.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00096018000000000004 2024-09-15T03:47:12.658 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.658 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:12.658 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.658 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs2 60/31 ECSubWrite(tid=70, reqid=client.4521.0:421, at_version=60'22, trim_to=0'0, roll_forward_to=47'20))", 2024-09-15T03:47:12.659 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.837529+0000", 2024-09-15T03:47:12.659 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.8118439300000002, 2024-09-15T03:47:12.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0088963150000000001, 2024-09-15T03:47:12.659 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:12.659 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.659 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:12.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:12.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837529+0000", 2024-09-15T03:47:12.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:12.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837529+0000", 2024-09-15T03:47:12.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:12.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837529+0000", 2024-09-15T03:47:12.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8199999999999999e-07 2024-09-15T03:47:12.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:12.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837562+0000", 2024-09-15T03:47:12.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3080000000000002e-05 2024-09-15T03:47:12.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:12.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837572+0000", 2024-09-15T03:47:12.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7219999999999994e-06 2024-09-15T03:47:12.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837580+0000", 2024-09-15T03:47:12.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4470000000000006e-06 2024-09-15T03:47:12.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:12.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837727+0000", 2024-09-15T03:47:12.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014676199999999999 2024-09-15T03:47:12.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837738+0000", 2024-09-15T03:47:12.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0946e-05 2024-09-15T03:47:12.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.838091+0000", 2024-09-15T03:47:12.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00035338600000000002 2024-09-15T03:47:12.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.663 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.839733+0000", 2024-09-15T03:47:12.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016420409999999999 2024-09-15T03:47:12.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.663 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.663 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.839815+0000", 2024-09-15T03:47:12.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1433000000000005e-05 2024-09-15T03:47:12.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.664 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.664 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.839841+0000", 2024-09-15T03:47:12.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6529e-05 2024-09-15T03:47:12.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:12.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.846425+0000", 2024-09-15T03:47:12.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0065837869999999998 2024-09-15T03:47:12.665 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.665 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:12.665 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.666 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.ds4 60/31 ECSubWrite(tid=122, reqid=osd.8.0:122, at_version=60'16, trim_to=0'0, roll_forward_to=59'14))", 2024-09-15T03:47:12.666 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.840427+0000", 2024-09-15T03:47:12.666 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.8089454299999996, 2024-09-15T03:47:12.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0064843990000000001, 2024-09-15T03:47:12.667 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:12.667 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.667 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:12.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:12.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840427+0000", 2024-09-15T03:47:12.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:12.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840427+0000", 2024-09-15T03:47:12.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.668 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:12.668 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840428+0000", 2024-09-15T03:47:12.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1099999999999996e-07 2024-09-15T03:47:12.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.668 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:12.668 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840460+0000", 2024-09-15T03:47:12.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1869999999999998e-05 2024-09-15T03:47:12.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:12.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840487+0000", 2024-09-15T03:47:12.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7778999999999999e-05 2024-09-15T03:47:12.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840496+0000", 2024-09-15T03:47:12.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9660000000000002e-06 2024-09-15T03:47:12.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:12.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840565+0000", 2024-09-15T03:47:12.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8464e-05 2024-09-15T03:47:12.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840577+0000", 2024-09-15T03:47:12.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1789e-05 2024-09-15T03:47:12.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840853+0000", 2024-09-15T03:47:12.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00027612699999999998 2024-09-15T03:47:12.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840900+0000", 2024-09-15T03:47:12.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6801000000000001e-05 2024-09-15T03:47:12.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840948+0000", 2024-09-15T03:47:12.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8406999999999999e-05 2024-09-15T03:47:12.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840972+0000", 2024-09-15T03:47:12.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4377999999999999e-05 2024-09-15T03:47:12.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:12.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.846912+0000", 2024-09-15T03:47:12.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0059393069999999996 2024-09-15T03:47:12.672 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.672 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:12.672 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.673 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.5s7 60/31 ECSubWrite(tid=78, reqid=osd.1.0:78, at_version=60'33, trim_to=0'0, roll_forward_to=59'31))", 2024-09-15T03:47:12.673 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.842455+0000", 2024-09-15T03:47:12.673 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.8069177610000002, 2024-09-15T03:47:12.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0049164439999999998, 2024-09-15T03:47:12.673 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:12.673 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.673 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:12.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.673 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:12.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.842455+0000", 2024-09-15T03:47:12.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:12.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.842455+0000", 2024-09-15T03:47:12.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:12.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.842455+0000", 2024-09-15T03:47:12.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9800000000000004e-07 2024-09-15T03:47:12.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:12.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.842486+0000", 2024-09-15T03:47:12.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0309000000000001e-05 2024-09-15T03:47:12.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:12.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.842498+0000", 2024-09-15T03:47:12.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2221e-05 2024-09-15T03:47:12.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.675 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.842506+0000", 2024-09-15T03:47:12.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9310000000000008e-06 2024-09-15T03:47:12.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:12.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.842578+0000", 2024-09-15T03:47:12.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2080000000000001e-05 2024-09-15T03:47:12.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.842590+0000", 2024-09-15T03:47:12.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1646e-05 2024-09-15T03:47:12.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.842906+0000", 2024-09-15T03:47:12.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031659899999999998 2024-09-15T03:47:12.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.842943+0000", 2024-09-15T03:47:12.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7061000000000003e-05 2024-09-15T03:47:12.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.842994+0000", 2024-09-15T03:47:12.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0213999999999997e-05 2024-09-15T03:47:12.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.678 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.678 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843010+0000", 2024-09-15T03:47:12.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6716999999999999e-05 2024-09-15T03:47:12.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.678 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:12.678 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.847371+0000", 2024-09-15T03:47:12.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0043611680000000003 2024-09-15T03:47:12.678 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.678 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:12.678 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.679 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs2 60/31 ECSubWrite(tid=71, reqid=client.4521.0:421, at_version=0'0, trim_to=0'0, roll_forward_to=60'22))", 2024-09-15T03:47:12.679 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.843064+0000", 2024-09-15T03:47:12.679 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.8063083219999996, 2024-09-15T03:47:12.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0066467000000000002, 2024-09-15T03:47:12.679 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:12.679 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.679 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:12.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:12.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843064+0000", 2024-09-15T03:47:12.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:12.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843064+0000", 2024-09-15T03:47:12.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:12.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843065+0000", 2024-09-15T03:47:12.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5600000000000004e-07 2024-09-15T03:47:12.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:12.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843077+0000", 2024-09-15T03:47:12.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1336e-05 2024-09-15T03:47:12.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:12.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843089+0000", 2024-09-15T03:47:12.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2869e-05 2024-09-15T03:47:12.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843097+0000", 2024-09-15T03:47:12.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.503e-06 2024-09-15T03:47:12.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:12.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843157+0000", 2024-09-15T03:47:12.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9580999999999998e-05 2024-09-15T03:47:12.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843167+0000", 2024-09-15T03:47:12.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0406e-05 2024-09-15T03:47:12.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.683 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.683 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843279+0000", 2024-09-15T03:47:12.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112427 2024-09-15T03:47:12.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.683 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.683 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843339+0000", 2024-09-15T03:47:12.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0031000000000002e-05 2024-09-15T03:47:12.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.683 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843380+0000", 2024-09-15T03:47:12.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0683000000000003e-05 2024-09-15T03:47:12.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843393+0000", 2024-09-15T03:47:12.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3334e-05 2024-09-15T03:47:12.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:12.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.849711+0000", 2024-09-15T03:47:12.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0063176739999999997 2024-09-15T03:47:12.684 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.685 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:12.685 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.685 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.3s7 60/31 ECSubWrite(tid=101, reqid=osd.6.0:101, at_version=60'56, trim_to=0'0, roll_forward_to=59'52))", 2024-09-15T03:47:12.685 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.844152+0000", 2024-09-15T03:47:12.685 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.8052206110000002, 2024-09-15T03:47:12.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0052846519999999999, 2024-09-15T03:47:12.685 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:12.685 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.685 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:12.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:12.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844152+0000", 2024-09-15T03:47:12.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:12.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844152+0000", 2024-09-15T03:47:12.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:12.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844165+0000", 2024-09-15T03:47:12.687 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2847000000000001e-05 2024-09-15T03:47:12.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.687 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:12.687 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844199+0000", 2024-09-15T03:47:12.687 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4418000000000001e-05 2024-09-15T03:47:12.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.687 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:12.687 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844226+0000", 2024-09-15T03:47:12.687 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6665999999999998e-05 2024-09-15T03:47:12.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.688 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.688 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844234+0000", 2024-09-15T03:47:12.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5909999999999998e-06 2024-09-15T03:47:12.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.688 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:12.688 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844284+0000", 2024-09-15T03:47:12.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0183999999999999e-05 2024-09-15T03:47:12.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.688 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844293+0000", 2024-09-15T03:47:12.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.448e-06 2024-09-15T03:47:12.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844548+0000", 2024-09-15T03:47:12.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00025473999999999999 2024-09-15T03:47:12.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844558+0000", 2024-09-15T03:47:12.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0373999999999999e-05 2024-09-15T03:47:12.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844651+0000", 2024-09-15T03:47:12.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2225000000000006e-05 2024-09-15T03:47:12.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844683+0000", 2024-09-15T03:47:12.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2032000000000001e-05 2024-09-15T03:47:12.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:12.690 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.849437+0000", 2024-09-15T03:47:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0047541270000000004 2024-09-15T03:47:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.8s6 60/31 ECSubWrite(tid=47, reqid=osd.3.0:47, at_version=60'10, trim_to=0'0, roll_forward_to=58'8))", 2024-09-15T03:47:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.844673+0000", 2024-09-15T03:47:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.804699491, 2024-09-15T03:47:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0078207390000000002, 2024-09-15T03:47:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.691 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844673+0000", 2024-09-15T03:47:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844673+0000", 2024-09-15T03:47:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844674+0000", 2024-09-15T03:47:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4299999999999999e-07 2024-09-15T03:47:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844699+0000", 2024-09-15T03:47:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5003000000000001e-05 2024-09-15T03:47:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844708+0000", 2024-09-15T03:47:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8109999999999997e-06 2024-09-15T03:47:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844715+0000", 2024-09-15T03:47:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.127e-06 2024-09-15T03:47:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.848363+0000", 2024-09-15T03:47:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0036482390000000002 2024-09-15T03:47:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.848374+0000", 2024-09-15T03:47:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1137e-05 2024-09-15T03:47:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.848615+0000", 2024-09-15T03:47:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024119899999999999 2024-09-15T03:47:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.851231+0000", 2024-09-15T03:47:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0026161079999999998 2024-09-15T03:47:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.851513+0000", 2024-09-15T03:47:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000282102 2024-09-15T03:47:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.851556+0000", 2024-09-15T03:47:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2960999999999997e-05 2024-09-15T03:47:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.852494+0000", 2024-09-15T03:47:12.696 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000937709 2024-09-15T03:47:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs2 65/64 ECSubWrite(tid=103, reqid=client.4521.0:463, at_version=65'24, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.848275+0000", 2024-09-15T03:47:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.80109818099999996, 2024-09-15T03:47:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0067122830000000003, 2024-09-15T03:47:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:12.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848275+0000", 2024-09-15T03:47:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848275+0000", 2024-09-15T03:47:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848275+0000", 2024-09-15T03:47:12.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7899999999999999e-07 2024-09-15T03:47:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848319+0000", 2024-09-15T03:47:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4360000000000002e-05 2024-09-15T03:47:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848336+0000", 2024-09-15T03:47:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6606e-05 2024-09-15T03:47:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848344+0000", 2024-09-15T03:47:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0930000000000003e-06 2024-09-15T03:47:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848719+0000", 2024-09-15T03:47:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00037497000000000002 2024-09-15T03:47:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848729+0000", 2024-09-15T03:47:12.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9860000000000006e-06 2024-09-15T03:47:12.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.701 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.701 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.849013+0000", 2024-09-15T03:47:12.701 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00028383400000000001 2024-09-15T03:47:12.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.701 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.701 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.849048+0000", 2024-09-15T03:47:12.701 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5474e-05 2024-09-15T03:47:12.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.702 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.702 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.850459+0000", 2024-09-15T03:47:12.702 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001410325 2024-09-15T03:47:12.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.702 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.702 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.850515+0000", 2024-09-15T03:47:12.702 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6801e-05 2024-09-15T03:47:12.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.702 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:12.702 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854987+0000", 2024-09-15T03:47:12.703 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0044713549999999998 2024-09-15T03:47:12.703 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.703 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:12.703 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.703 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.3s7 65/64 ECSubWrite(tid=104, reqid=osd.11.0:104, at_version=65'63, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:12.703 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.848352+0000", 2024-09-15T03:47:12.703 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.80102115200000001, 2024-09-15T03:47:12.703 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0052704869999999999, 2024-09-15T03:47:12.703 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:12.703 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.703 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848352+0000", 2024-09-15T03:47:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848352+0000", 2024-09-15T03:47:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:12.704 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848352+0000", 2024-09-15T03:47:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4799999999999999e-07 2024-09-15T03:47:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848377+0000", 2024-09-15T03:47:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5273999999999999e-05 2024-09-15T03:47:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848391+0000", 2024-09-15T03:47:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3314e-05 2024-09-15T03:47:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.705 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.706 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848396+0000", 2024-09-15T03:47:12.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7400000000000001e-06 2024-09-15T03:47:12.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.706 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:12.706 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.850048+0000", 2024-09-15T03:47:12.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001651406 2024-09-15T03:47:12.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.706 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.706 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.850057+0000", 2024-09-15T03:47:12.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5880000000000004e-06 2024-09-15T03:47:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.850298+0000", 2024-09-15T03:47:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024073899999999999 2024-09-15T03:47:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.850464+0000", 2024-09-15T03:47:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016564999999999999 2024-09-15T03:47:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.707 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.850548+0000", 2024-09-15T03:47:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4427999999999999e-05 2024-09-15T03:47:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851528+0000", 2024-09-15T03:47:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00097984599999999993 2024-09-15T03:47:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.853622+0000", 2024-09-15T03:47:12.708 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020940540000000001 2024-09-15T03:47:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.3s7 65/64 ECSubWrite(tid=105, reqid=osd.11.0:105, at_version=65'65, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.848406+0000", 2024-09-15T03:47:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.80096628199999997, 2024-09-15T03:47:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.005664807, 2024-09-15T03:47:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:12.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848406+0000", 2024-09-15T03:47:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848406+0000", 2024-09-15T03:47:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848407+0000", 2024-09-15T03:47:12.710 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9900000000000002e-07 2024-09-15T03:47:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848427+0000", 2024-09-15T03:47:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0466e-05 2024-09-15T03:47:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848435+0000", 2024-09-15T03:47:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0320000000000003e-06 2024-09-15T03:47:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.711 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.712 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848441+0000", 2024-09-15T03:47:12.712 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5249999999999996e-06 2024-09-15T03:47:12.712 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.712 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:12.712 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.850341+0000", 2024-09-15T03:47:12.712 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001899962 2024-09-15T03:47:12.712 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.712 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.712 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.850349+0000", 2024-09-15T03:47:12.712 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6109999999999994e-06 2024-09-15T03:47:12.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.713 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.713 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.850576+0000", 2024-09-15T03:47:12.713 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022618200000000001 2024-09-15T03:47:12.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.713 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.713 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.850605+0000", 2024-09-15T03:47:12.713 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9754999999999999e-05 2024-09-15T03:47:12.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.713 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.713 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851885+0000", 2024-09-15T03:47:12.714 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001279975 2024-09-15T03:47:12.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.714 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.714 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.714 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851936+0000", 2024-09-15T03:47:12.714 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0256000000000001e-05 2024-09-15T03:47:12.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.714 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.714 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:12.714 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854071+0000", 2024-09-15T03:47:12.714 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021357440000000002 2024-09-15T03:47:12.714 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.715 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:12.715 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.715 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.715 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.715 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs2 65/64 ECSubWrite(tid=106, reqid=client.4521.0:464, at_version=65'25, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:12.715 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.851121+0000", 2024-09-15T03:47:12.715 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.79825181700000003, 2024-09-15T03:47:12.715 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0055986910000000003, 2024-09-15T03:47:12.715 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:12.715 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.715 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:12.715 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.716 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:12.716 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851121+0000", 2024-09-15T03:47:12.716 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.716 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.716 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.716 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:12.716 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851121+0000", 2024-09-15T03:47:12.716 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.716 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.716 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.716 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:12.716 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851122+0000", 2024-09-15T03:47:12.717 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1459999999999999e-06 2024-09-15T03:47:12.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.717 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.717 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:12.717 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851143+0000", 2024-09-15T03:47:12.717 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.084e-05 2024-09-15T03:47:12.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.717 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.717 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:12.717 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851162+0000", 2024-09-15T03:47:12.717 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9318999999999999e-05 2024-09-15T03:47:12.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.717 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.718 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.718 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851170+0000", 2024-09-15T03:47:12.718 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5700000000000004e-06 2024-09-15T03:47:12.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.718 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.718 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:12.718 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851217+0000", 2024-09-15T03:47:12.718 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7287000000000003e-05 2024-09-15T03:47:12.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.718 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.719 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.719 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851228+0000", 2024-09-15T03:47:12.719 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0448000000000001e-05 2024-09-15T03:47:12.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.719 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.719 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.719 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851332+0000", 2024-09-15T03:47:12.719 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010485200000000001 2024-09-15T03:47:12.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.719 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.719 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.719 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851892+0000", 2024-09-15T03:47:12.720 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00055992499999999996 2024-09-15T03:47:12.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.720 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.720 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.720 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851986+0000", 2024-09-15T03:47:12.720 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4140000000000003e-05 2024-09-15T03:47:12.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.720 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.720 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.720 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852012+0000", 2024-09-15T03:47:12.720 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5244000000000001e-05 2024-09-15T03:47:12.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.721 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.721 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:12.721 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.856720+0000", 2024-09-15T03:47:12.721 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00470792 2024-09-15T03:47:12.721 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.721 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:12.721 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.721 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.721 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.721 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs2 65/64 ECSubWrite(tid=107, reqid=client.4521.0:465, at_version=65'26, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:12.721 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.851183+0000", 2024-09-15T03:47:12.721 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.79819003200000005, 2024-09-15T03:47:12.721 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0066932240000000002, 2024-09-15T03:47:12.721 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:12.722 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.722 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:12.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.722 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:12.722 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851183+0000", 2024-09-15T03:47:12.722 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.722 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:12.722 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851183+0000", 2024-09-15T03:47:12.722 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.723 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:12.723 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851183+0000", 2024-09-15T03:47:12.723 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5900000000000002e-07 2024-09-15T03:47:12.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.723 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:12.723 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851210+0000", 2024-09-15T03:47:12.723 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6398e-05 2024-09-15T03:47:12.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.723 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:12.724 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851219+0000", 2024-09-15T03:47:12.724 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3789999999999995e-06 2024-09-15T03:47:12.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.724 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.724 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851225+0000", 2024-09-15T03:47:12.724 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.472e-06 2024-09-15T03:47:12.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.724 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:12.725 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851369+0000", 2024-09-15T03:47:12.725 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000143559 2024-09-15T03:47:12.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.725 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.725 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.725 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851378+0000", 2024-09-15T03:47:12.725 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0310000000000004e-06 2024-09-15T03:47:12.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.725 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.725 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.725 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851467+0000", 2024-09-15T03:47:12.725 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9300000000000002e-05 2024-09-15T03:47:12.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.726 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.726 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.726 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851896+0000", 2024-09-15T03:47:12.726 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00042909399999999998 2024-09-15T03:47:12.726 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.726 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.726 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.726 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851990+0000", 2024-09-15T03:47:12.726 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3428000000000001e-05 2024-09-15T03:47:12.726 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.726 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.726 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.726 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852035+0000", 2024-09-15T03:47:12.726 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4960000000000003e-05 2024-09-15T03:47:12.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.727 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:12.727 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.857876+0000", 2024-09-15T03:47:12.727 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0058411440000000004 2024-09-15T03:47:12.727 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.727 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:12.727 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.727 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.2s5 65/64 ECSubWrite(tid=155, reqid=osd.4.0:155, at_version=65'27, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:12.727 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.851690+0000", 2024-09-15T03:47:12.728 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.79768294299999998, 2024-09-15T03:47:12.728 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0053597849999999997, 2024-09-15T03:47:12.728 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:12.728 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.728 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:12.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.728 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:12.728 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851690+0000", 2024-09-15T03:47:12.728 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.728 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:12.728 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851690+0000", 2024-09-15T03:47:12.729 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.729 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:12.729 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851690+0000", 2024-09-15T03:47:12.729 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3399999999999999e-07 2024-09-15T03:47:12.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.729 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:12.729 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851712+0000", 2024-09-15T03:47:12.729 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2039e-05 2024-09-15T03:47:12.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.730 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:12.730 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851725+0000", 2024-09-15T03:47:12.730 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2081e-05 2024-09-15T03:47:12.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.730 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.730 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851732+0000", 2024-09-15T03:47:12.730 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3490000000000003e-06 2024-09-15T03:47:12.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.730 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:12.730 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851780+0000", 2024-09-15T03:47:12.730 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8568e-05 2024-09-15T03:47:12.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.731 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.731 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851792+0000", 2024-09-15T03:47:12.731 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1311999999999999e-05 2024-09-15T03:47:12.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.731 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.731 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852082+0000", 2024-09-15T03:47:12.731 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00028999599999999999 2024-09-15T03:47:12.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.731 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.732 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852118+0000", 2024-09-15T03:47:12.732 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5932000000000001e-05 2024-09-15T03:47:12.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.732 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.732 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852161+0000", 2024-09-15T03:47:12.732 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3469e-05 2024-09-15T03:47:12.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.732 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.732 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852196+0000", 2024-09-15T03:47:12.732 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4440999999999998e-05 2024-09-15T03:47:12.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.733 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:12.733 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.857050+0000", 2024-09-15T03:47:12.733 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0048539639999999997 2024-09-15T03:47:12.733 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.733 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:12.733 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.733 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.4s4 65/64 ECSubWrite(tid=68, reqid=osd.3.0:68, at_version=65'26, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:12.733 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.852456+0000", 2024-09-15T03:47:12.733 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.79691690000000004, 2024-09-15T03:47:12.734 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0053921139999999999, 2024-09-15T03:47:12.734 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:12.734 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.734 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:12.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.734 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:12.734 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852456+0000", 2024-09-15T03:47:12.734 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.734 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.734 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:12.734 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852456+0000", 2024-09-15T03:47:12.734 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.735 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:12.735 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852456+0000", 2024-09-15T03:47:12.735 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5400000000000002e-07 2024-09-15T03:47:12.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.735 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:12.735 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852495+0000", 2024-09-15T03:47:12.735 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8866999999999999e-05 2024-09-15T03:47:12.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.735 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:12.736 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852506+0000", 2024-09-15T03:47:12.736 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0778000000000001e-05 2024-09-15T03:47:12.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.736 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.736 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852512+0000", 2024-09-15T03:47:12.736 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4690000000000002e-06 2024-09-15T03:47:12.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.736 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:12.736 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852562+0000", 2024-09-15T03:47:12.736 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9385999999999998e-05 2024-09-15T03:47:12.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.737 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.737 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852571+0000", 2024-09-15T03:47:12.737 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4490000000000005e-06 2024-09-15T03:47:12.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.737 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.737 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.853581+0000", 2024-09-15T03:47:12.737 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010099880000000001 2024-09-15T03:47:12.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.737 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.738 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854185+0000", 2024-09-15T03:47:12.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00060344900000000004 2024-09-15T03:47:12.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.738 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.738 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854311+0000", 2024-09-15T03:47:12.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012594700000000001 2024-09-15T03:47:12.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.738 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.738 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854372+0000", 2024-09-15T03:47:12.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1711000000000006e-05 2024-09-15T03:47:12.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.739 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:12.739 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.857848+0000", 2024-09-15T03:47:12.739 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0034756159999999999 2024-09-15T03:47:12.739 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.739 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:12.739 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.739 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s2 65/64 ECSubWrite(tid=69, reqid=client.4521.0:458, at_version=65'22, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:12.739 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.863447+0000", 2024-09-15T03:47:12.739 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.78592559200000001, 2024-09-15T03:47:12.739 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0078021109999999996, 2024-09-15T03:47:12.739 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:12.740 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.740 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:12.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.740 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:12.740 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.863447+0000", 2024-09-15T03:47:12.740 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.740 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:12.740 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.863447+0000", 2024-09-15T03:47:12.740 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.741 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:12.741 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.863448+0000", 2024-09-15T03:47:12.741 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7399999999999999e-07 2024-09-15T03:47:12.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.741 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:12.741 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.863537+0000", 2024-09-15T03:47:12.741 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9034000000000002e-05 2024-09-15T03:47:12.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.741 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:12.741 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.863549+0000", 2024-09-15T03:47:12.741 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2554e-05 2024-09-15T03:47:12.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.742 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.742 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.863556+0000", 2024-09-15T03:47:12.742 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4049999999999997e-06 2024-09-15T03:47:12.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.742 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:12.742 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.863611+0000", 2024-09-15T03:47:12.742 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5494000000000001e-05 2024-09-15T03:47:12.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.742 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.743 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.863623+0000", 2024-09-15T03:47:12.743 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2265e-05 2024-09-15T03:47:12.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.743 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.743 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.864398+0000", 2024-09-15T03:47:12.743 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00077489300000000002 2024-09-15T03:47:12.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.743 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.743 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.864411+0000", 2024-09-15T03:47:12.743 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2529e-05 2024-09-15T03:47:12.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.744 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.744 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.864573+0000", 2024-09-15T03:47:12.744 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016181600000000001 2024-09-15T03:47:12.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.744 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.744 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.864588+0000", 2024-09-15T03:47:12.744 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5526999999999999e-05 2024-09-15T03:47:12.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.744 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:12.744 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871249+0000", 2024-09-15T03:47:12.745 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0066612199999999998 2024-09-15T03:47:12.745 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.745 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:12.745 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.745 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.3s7 65/64 ECSubWrite(tid=114, reqid=osd.11.0:105, at_version=0'0, trim_to=0'0, roll_forward_to=65'65))", 2024-09-15T03:47:12.745 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.866122+0000", 2024-09-15T03:47:12.745 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.78325085100000003, 2024-09-15T03:47:12.745 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0054563900000000002, 2024-09-15T03:47:12.745 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:12.745 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.746 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:12.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.746 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:12.746 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866122+0000", 2024-09-15T03:47:12.746 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.746 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:12.746 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866122+0000", 2024-09-15T03:47:12.746 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.746 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:12.746 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866123+0000", 2024-09-15T03:47:12.747 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1069999999999999e-06 2024-09-15T03:47:12.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.747 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:12.747 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866136+0000", 2024-09-15T03:47:12.747 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3125000000000001e-05 2024-09-15T03:47:12.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.747 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:12.747 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866162+0000", 2024-09-15T03:47:12.747 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6333999999999999e-05 2024-09-15T03:47:12.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.747 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.747 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866172+0000", 2024-09-15T03:47:12.748 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4590000000000004e-06 2024-09-15T03:47:12.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.748 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:12.748 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866277+0000", 2024-09-15T03:47:12.748 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010503200000000001 2024-09-15T03:47:12.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.748 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.748 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866286+0000", 2024-09-15T03:47:12.748 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1789999999999997e-06 2024-09-15T03:47:12.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.748 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.749 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866439+0000", 2024-09-15T03:47:12.749 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00015331200000000001 2024-09-15T03:47:12.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.749 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.749 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866805+0000", 2024-09-15T03:47:12.749 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00036521699999999998 2024-09-15T03:47:12.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.749 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.749 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866992+0000", 2024-09-15T03:47:12.749 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018734200000000001 2024-09-15T03:47:12.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.750 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.750 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867029+0000", 2024-09-15T03:47:12.750 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7382000000000002e-05 2024-09-15T03:47:12.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.750 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:12.750 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871578+0000", 2024-09-15T03:47:12.750 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0045489010000000002 2024-09-15T03:47:12.750 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.750 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:12.750 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.3s7 65/64 ECSubWrite(tid=118, reqid=osd.11.0:118, at_version=65'67, trim_to=0'0, roll_forward_to=65'65))", 2024-09-15T03:47:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.867685+0000", 2024-09-15T03:47:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.78168744700000004, 2024-09-15T03:47:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0050971300000000001, 2024-09-15T03:47:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867685+0000", 2024-09-15T03:47:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.752 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:12.752 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867685+0000", 2024-09-15T03:47:12.752 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.752 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.752 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:12.752 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867686+0000", 2024-09-15T03:47:12.752 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2600000000000001e-07 2024-09-15T03:47:12.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.752 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.752 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:12.752 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867716+0000", 2024-09-15T03:47:12.752 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0049e-05 2024-09-15T03:47:12.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.753 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.753 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:12.753 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867729+0000", 2024-09-15T03:47:12.753 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2794e-05 2024-09-15T03:47:12.753 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.753 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.753 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.753 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867737+0000", 2024-09-15T03:47:12.753 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.627e-06 2024-09-15T03:47:12.753 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.753 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.753 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:12.753 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868289+0000", 2024-09-15T03:47:12.754 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00055240599999999997 2024-09-15T03:47:12.754 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.754 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.754 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.754 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868300+0000", 2024-09-15T03:47:12.754 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0845999999999999e-05 2024-09-15T03:47:12.754 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.754 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.754 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.754 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868660+0000", 2024-09-15T03:47:12.754 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00036019999999999997 2024-09-15T03:47:12.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.755 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.755 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868976+0000", 2024-09-15T03:47:12.755 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031557699999999999 2024-09-15T03:47:12.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.755 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.755 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869822+0000", 2024-09-15T03:47:12.755 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00084606700000000004 2024-09-15T03:47:12.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.756 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.756 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869877+0000", 2024-09-15T03:47:12.756 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5310999999999999e-05 2024-09-15T03:47:12.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.756 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:12.756 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.872782+0000", 2024-09-15T03:47:12.756 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0029054269999999999 2024-09-15T03:47:12.756 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.756 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:12.756 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.757 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.757 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.es1 65/28 ECSubWrite(tid=103, reqid=client.4521.0:487, at_version=65'67, trim_to=0'0, roll_forward_to=64'65))", 2024-09-15T03:47:12.757 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.869892+0000", 2024-09-15T03:47:12.757 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.77948106100000003, 2024-09-15T03:47:12.757 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0051626859999999997, 2024-09-15T03:47:12.757 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:12.757 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.757 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:12.757 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.757 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:12.758 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869892+0000", 2024-09-15T03:47:12.758 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.758 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:12.758 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869892+0000", 2024-09-15T03:47:12.758 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.758 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:12.758 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869893+0000", 2024-09-15T03:47:12.758 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.16e-07 2024-09-15T03:47:12.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.759 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:12.759 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869916+0000", 2024-09-15T03:47:12.759 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3108e-05 2024-09-15T03:47:12.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.759 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:12.759 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869930+0000", 2024-09-15T03:47:12.759 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4292e-05 2024-09-15T03:47:12.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.759 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.759 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869943+0000", 2024-09-15T03:47:12.759 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3001e-05 2024-09-15T03:47:12.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.760 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.760 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:12.760 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871493+0000", 2024-09-15T03:47:12.760 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001550042 2024-09-15T03:47:12.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.760 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.760 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.760 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871504+0000", 2024-09-15T03:47:12.760 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0597999999999999e-05 2024-09-15T03:47:12.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.760 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.760 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.761 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871756+0000", 2024-09-15T03:47:12.761 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00025288200000000001 2024-09-15T03:47:12.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.761 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.761 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871940+0000", 2024-09-15T03:47:12.761 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018375100000000001 2024-09-15T03:47:12.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.761 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.761 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.872610+0000", 2024-09-15T03:47:12.761 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00066978800000000005 2024-09-15T03:47:12.762 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.762 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.762 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.872629+0000", 2024-09-15T03:47:12.762 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8732e-05 2024-09-15T03:47:12.762 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.762 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:12.762 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875054+0000", 2024-09-15T03:47:12.762 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024256759999999999 2024-09-15T03:47:12.762 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.762 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:12.763 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.763 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.763 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.763 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.es1 65/28 ECSubWrite(tid=104, reqid=client.4521.0:488, at_version=65'68, trim_to=0'0, roll_forward_to=64'65))", 2024-09-15T03:47:12.763 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.869959+0000", 2024-09-15T03:47:12.763 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.77941422699999996, 2024-09-15T03:47:12.763 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0085038979999999993, 2024-09-15T03:47:12.763 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:12.763 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.763 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:12.763 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.763 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:12.764 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869959+0000", 2024-09-15T03:47:12.764 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.764 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.764 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:12.764 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869959+0000", 2024-09-15T03:47:12.764 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.764 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.764 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:12.764 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869959+0000", 2024-09-15T03:47:12.764 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4900000000000001e-07 2024-09-15T03:47:12.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.765 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:12.765 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869989+0000", 2024-09-15T03:47:12.765 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0179e-05 2024-09-15T03:47:12.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.765 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:12.765 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870000+0000", 2024-09-15T03:47:12.765 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0414000000000001e-05 2024-09-15T03:47:12.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.765 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.766 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870005+0000", 2024-09-15T03:47:12.766 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3870000000000003e-06 2024-09-15T03:47:12.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.766 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:12.766 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871808+0000", 2024-09-15T03:47:12.766 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001803453 2024-09-15T03:47:12.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.766 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.766 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871817+0000", 2024-09-15T03:47:12.766 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9309999999999997e-06 2024-09-15T03:47:12.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.767 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.767 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871919+0000", 2024-09-15T03:47:12.767 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000101195 2024-09-15T03:47:12.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.767 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.767 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871941+0000", 2024-09-15T03:47:12.767 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2693000000000001e-05 2024-09-15T03:47:12.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.768 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.768 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.872611+0000", 2024-09-15T03:47:12.768 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00067015100000000001 2024-09-15T03:47:12.768 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.768 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.768 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.768 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.872799+0000", 2024-09-15T03:47:12.768 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000188116 2024-09-15T03:47:12.768 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.768 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.768 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:12.769 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.878462+0000", 2024-09-15T03:47:12.769 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00566293 2024-09-15T03:47:12.769 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.769 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:12.769 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.769 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.769 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.769 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.es1 65/28 ECSubWrite(tid=105, reqid=client.4521.0:489, at_version=65'69, trim_to=0'0, roll_forward_to=64'65))", 2024-09-15T03:47:12.769 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.870601+0000", 2024-09-15T03:47:12.769 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 0.77877204499999997, 2024-09-15T03:47:12.769 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0085606629999999996, 2024-09-15T03:47:12.769 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:12.769 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:12.770 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:12.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.770 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:12.770 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870601+0000", 2024-09-15T03:47:12.770 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.770 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:12.770 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870601+0000", 2024-09-15T03:47:12.770 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:12.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.771 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:12.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870601+0000", 2024-09-15T03:47:12.771 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.23e-07 2024-09-15T03:47:12.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.771 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:12.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870622+0000", 2024-09-15T03:47:12.771 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0511000000000001e-05 2024-09-15T03:47:12.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.772 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:12.772 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870640+0000", 2024-09-15T03:47:12.772 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8070000000000001e-05 2024-09-15T03:47:12.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.772 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:12.772 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870647+0000", 2024-09-15T03:47:12.772 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0770000000000002e-06 2024-09-15T03:47:12.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.773 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:12.773 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875542+0000", 2024-09-15T03:47:12.773 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0048952650000000002 2024-09-15T03:47:12.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.773 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.773 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:12.773 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875553+0000", 2024-09-15T03:47:12.773 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0163000000000001e-05 2024-09-15T03:47:12.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.773 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.773 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:12.773 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875674+0000", 2024-09-15T03:47:12.774 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000121618 2024-09-15T03:47:12.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.774 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:12.774 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876678+0000", 2024-09-15T03:47:12.774 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010037189999999999 2024-09-15T03:47:12.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.774 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:12.774 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876902+0000", 2024-09-15T03:47:12.774 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000224279 2024-09-15T03:47:12.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.775 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.775 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:12.775 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.879132+0000", 2024-09-15T03:47:12.775 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022297039999999999 2024-09-15T03:47:12.775 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:12.775 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:12.775 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:12.775 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.879161+0000", 2024-09-15T03:47:12.775 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9533999999999999e-05 2024-09-15T03:47:12.775 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.775 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:12.776 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.776 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:12.776 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:12.776 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:12.778 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:47:12.778 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:12.776+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:12.878 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:12.877+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:12.899 INFO:tasks.rados.rados.0.smithi001.stdout:191: done (5 left) 2024-09-15T03:47:12.899 INFO:tasks.rados.rados.0.smithi001.stdout:192: done (4 left) 2024-09-15T03:47:12.899 INFO:tasks.rados.rados.0.smithi001.stdout:193: done (3 left) 2024-09-15T03:47:12.899 INFO:tasks.rados.rados.0.smithi001.stdout:194: done (2 left) 2024-09-15T03:47:12.899 INFO:tasks.rados.rados.0.smithi001.stdout:195: done (1 left) 2024-09-15T03:47:12.899 INFO:tasks.rados.rados.0.smithi001.stdout:196: done (0 left) 2024-09-15T03:47:12.899 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 21 current snap is 22 2024-09-15T03:47:12.899 INFO:tasks.rados.rados.0.smithi001.stdout:197: seq_num 97 ranges {1769472=475136,2244608=442368,2686976=442368,3129344=278528} 2024-09-15T03:47:12.912 INFO:tasks.rados.rados.0.smithi001.stdout:197: writing smithi00132172-21 from 1769472 to 2244608 tid 1 2024-09-15T03:47:12.914 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [10, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-15T03:47:12.914 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T03:47:12.915 INFO:tasks.thrashosds.thrasher:Adding osd 10 2024-09-15T03:47:12.915 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 10 2024-09-15T03:47:12.916 INFO:tasks.rados.rados.0.smithi001.stdout:197: writing smithi00132172-21 from 2244608 to 2686976 tid 2 2024-09-15T03:47:12.920 INFO:tasks.rados.rados.0.smithi001.stdout:197: writing smithi00132172-21 from 2686976 to 3129344 tid 3 2024-09-15T03:47:12.922 INFO:tasks.rados.rados.0.smithi001.stdout:197: writing smithi00132172-21 from 3129344 to 3407872 tid 4 2024-09-15T03:47:12.922 INFO:tasks.rados.rados.0.smithi001.stdout:198: snap_create 2024-09-15T03:47:12.922 INFO:tasks.rados.rados.0.smithi001.stdout:197: finishing write tid 1 to smithi00132172-21 2024-09-15T03:47:12.923 INFO:tasks.rados.rados.0.smithi001.stdout:197: finishing write tid 2 to smithi00132172-21 2024-09-15T03:47:12.925 INFO:tasks.rados.rados.0.smithi001.stdout:197: finishing write tid 3 to smithi00132172-21 2024-09-15T03:47:12.925 INFO:tasks.rados.rados.0.smithi001.stdout:197: finishing write tid 4 to smithi00132172-21 2024-09-15T03:47:12.926 INFO:tasks.rados.rados.0.smithi001.stdout:197: finishing write tid 5 to smithi00132172-21 2024-09-15T03:47:12.926 INFO:tasks.rados.rados.0.smithi001.stdout:197: finishing write tid 6 to smithi00132172-21 2024-09-15T03:47:12.926 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 21 v 26 (ObjNum 97 snap 22 seq_num 97) dirty exists 2024-09-15T03:47:12.926 INFO:tasks.rados.rados.0.smithi001.stdout:197: left oid 21 (ObjNum 97 snap 22 seq_num 97) 2024-09-15T03:47:12.932 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:12.932 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:12.932 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-15T03:47:12.932 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:12.941 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:47:12.982 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:12.981+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:13.032 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:47:13.079 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:13.079+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:13.087 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:13.097 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:13.097 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-15T03:47:13.097 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-15T03:47:13.097 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:13.098 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:47:13.152 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:13.153 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:13.167 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:13.167 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:13.180 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:13.180+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:13.182 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:13.182 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:13.197 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:13.197 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:13.201 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:13.201 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:13.208 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-09-15T03:47:13.207+0000 7ff94fdf1640 -1 mon.a@0(leader).osd e66 definitely_dead 0 2024-09-15T03:47:13.210 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:47:13.211 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:13.211 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:13.226 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:13.226 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:13.242 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:13.242 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:13.257 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:13.257 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:13.272 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:13.272 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:13.281 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:13.281+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:13.287 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:13.302 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:47:13.302 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:47:13.312 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:13.312 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:13.317 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:13.317 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:13.321 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:47:13.413 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:13.412+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:13.469 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:13.469 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:13.477 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:47:13.484 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:13.484+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:13.584 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:13.584+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:13.618 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:13.618 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:13.627 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:47:13.685 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:13.685+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:13.768 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:13.768 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:13.776 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:47:13.820 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:13.819+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:13.877 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:13.877 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:13.884 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T03:47:13.886 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:47:13.902 INFO:teuthology.orchestra.run.smithi001.stderr:marked in osd.10. 2024-09-15T03:47:13.903 INFO:tasks.rados.rados.0.smithi001.stdout:197: done (1 left) 2024-09-15T03:47:13.903 INFO:tasks.rados.rados.0.smithi001.stdout:198: done (0 left) 2024-09-15T03:47:13.903 INFO:tasks.rados.rados.0.smithi001.stdout:199: delete oid 37 current snap is 23 2024-09-15T03:47:13.904 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:13.903+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:13.915 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:13.914+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:13.919 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:13.919+0000 7f63a77b2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:13.928 INFO:tasks.thrashosds.thrasher:Added osd 10 2024-09-15T03:47:13.989 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:13.987+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:14.014 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:14.014 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:14.023 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:47:14.089 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:14.089+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:14.136 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:14.136 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:14.145 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:47:14.230 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:14.229+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:14.288 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:14.288 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:14.291 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:14.290+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:14.296 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:47:14.392 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:14.392+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:14.437 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:14.438 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:14.446 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:47:14.494 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:14.493+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:14.548 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:14.548 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:14.556 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:47:14.600 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:14.600+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:14.657 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:14.658 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:14.666 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:47:14.695 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:14.694+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:14.797 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:14.795+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:14.807 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:14.807 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:14.815 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:47:14.833 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-15T03:47:14.844 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:14.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:14.848 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:14.848+0000 7f63ab7ba640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:14.898 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:14.897+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:14.922 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:14.923 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:14.931 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:47:15.014 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:15.014+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:15.071 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:15.071 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:15.079 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:47:15.099 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:15.097+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:15.201 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:15.200+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:15.220 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:15.220 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:15.229 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:47:15.302 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:15.301+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:15.328 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:47:15.370 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:15.370 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:15.378 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:47:15.402 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:15.402+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:15.457 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:15.458 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:15.472 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:15.472 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:15.478 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:15.478 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:15.486 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:47:15.487 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:15.487 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:15.501 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:15.501 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:15.503 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:15.502+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:15.515 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:15.516 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:15.530 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:15.530 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:15.544 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:15.545 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:15.559 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:15.559 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:15.575 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:15.575 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:15.590 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:15.591 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:15.591 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:15.591 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:15.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:15.591 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:15.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.592 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs3 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:15.592 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108456+0000", 2024-09-15T03:47:15.592 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.479976076, 2024-09-15T03:47:15.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015075257, 2024-09-15T03:47:15.592 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:15.592 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:15.592 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:15.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:15.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108456+0000", 2024-09-15T03:47:15.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:15.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108456+0000", 2024-09-15T03:47:15.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:15.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108457+0000", 2024-09-15T03:47:15.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9999999999999997e-07 2024-09-15T03:47:15.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:15.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108477+0000", 2024-09-15T03:47:15.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0344e-05 2024-09-15T03:47:15.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:15.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108497+0000", 2024-09-15T03:47:15.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0143999999999999e-05 2024-09-15T03:47:15.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:15.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108504+0000", 2024-09-15T03:47:15.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8499999999999996e-06 2024-09-15T03:47:15.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108554+0000", 2024-09-15T03:47:15.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9567999999999997e-05 2024-09-15T03:47:15.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:15.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108568+0000", 2024-09-15T03:47:15.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4326e-05 2024-09-15T03:47:15.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:15.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108654+0000", 2024-09-15T03:47:15.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6193999999999995e-05 2024-09-15T03:47:15.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:15.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108663+0000", 2024-09-15T03:47:15.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7479999999999999e-06 2024-09-15T03:47:15.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:15.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111234+0000", 2024-09-15T03:47:15.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025709079999999998 2024-09-15T03:47:15.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:15.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111259+0000", 2024-09-15T03:47:15.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4797999999999999e-05 2024-09-15T03:47:15.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:15.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123531+0000", 2024-09-15T03:47:15.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012272777 2024-09-15T03:47:15.597 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.597 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:15.597 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.598 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as2 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:15.598 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:47:15.598 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.479747422000003, 2024-09-15T03:47:15.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0149108, 2024-09-15T03:47:15.598 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:15.598 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:15.598 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:15.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:15.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:47:15.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:15.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:47:15.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:15.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:47:15.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1699999999999999e-07 2024-09-15T03:47:15.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:15.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108710+0000", 2024-09-15T03:47:15.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5032999999999999e-05 2024-09-15T03:47:15.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:15.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108725+0000", 2024-09-15T03:47:15.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4788e-05 2024-09-15T03:47:15.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:15.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108733+0000", 2024-09-15T03:47:15.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4610000000000008e-06 2024-09-15T03:47:15.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108778+0000", 2024-09-15T03:47:15.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4224999999999998e-05 2024-09-15T03:47:15.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:15.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108800+0000", 2024-09-15T03:47:15.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2189999999999999e-05 2024-09-15T03:47:15.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:15.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108884+0000", 2024-09-15T03:47:15.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4569999999999998e-05 2024-09-15T03:47:15.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:15.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111237+0000", 2024-09-15T03:47:15.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002352313 2024-09-15T03:47:15.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:15.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114139+0000", 2024-09-15T03:47:15.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002902057 2024-09-15T03:47:15.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:15.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114173+0000", 2024-09-15T03:47:15.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4461000000000001e-05 2024-09-15T03:47:15.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:15.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123596+0000", 2024-09-15T03:47:15.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094222850000000007 2024-09-15T03:47:15.603 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.603 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:15.603 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.603 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs3 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:15.603 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:47:15.603 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.479688144000001, 2024-09-15T03:47:15.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014886043, 2024-09-15T03:47:15.603 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:15.603 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:15.603 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:15.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:15.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:47:15.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:15.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:47:15.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:15.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:47:15.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5100000000000001e-07 2024-09-15T03:47:15.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:15.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108759+0000", 2024-09-15T03:47:15.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4945e-05 2024-09-15T03:47:15.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:15.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108768+0000", 2024-09-15T03:47:15.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2309999999999996e-06 2024-09-15T03:47:15.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:15.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108795+0000", 2024-09-15T03:47:15.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7829000000000001e-05 2024-09-15T03:47:15.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108841+0000", 2024-09-15T03:47:15.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5330999999999997e-05 2024-09-15T03:47:15.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:15.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108850+0000", 2024-09-15T03:47:15.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9959999999999999e-06 2024-09-15T03:47:15.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:15.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108940+0000", 2024-09-15T03:47:15.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9823999999999999e-05 2024-09-15T03:47:15.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:15.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111237+0000", 2024-09-15T03:47:15.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002297927 2024-09-15T03:47:15.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:15.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114139+0000", 2024-09-15T03:47:15.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029020029999999998 2024-09-15T03:47:15.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:15.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114212+0000", 2024-09-15T03:47:15.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2177999999999999e-05 2024-09-15T03:47:15.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:15.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123630+0000", 2024-09-15T03:47:15.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094184279999999995 2024-09-15T03:47:15.608 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.609 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:15.609 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.609 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as2 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:15.609 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.109397+0000", 2024-09-15T03:47:15.609 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.479035086000003, 2024-09-15T03:47:15.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014322056, 2024-09-15T03:47:15.609 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:15.609 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:15.609 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:15.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:15.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109397+0000", 2024-09-15T03:47:15.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:15.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109397+0000", 2024-09-15T03:47:15.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:15.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109398+0000", 2024-09-15T03:47:15.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1600000000000001e-07 2024-09-15T03:47:15.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:15.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109410+0000", 2024-09-15T03:47:15.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2294e-05 2024-09-15T03:47:15.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:15.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109420+0000", 2024-09-15T03:47:15.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.764e-06 2024-09-15T03:47:15.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:15.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109426+0000", 2024-09-15T03:47:15.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9309999999999996e-06 2024-09-15T03:47:15.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109457+0000", 2024-09-15T03:47:15.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1047000000000003e-05 2024-09-15T03:47:15.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:15.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109462+0000", 2024-09-15T03:47:15.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1499999999999998e-06 2024-09-15T03:47:15.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:15.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109513+0000", 2024-09-15T03:47:15.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0722e-05 2024-09-15T03:47:15.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:15.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111238+0000", 2024-09-15T03:47:15.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017253380000000001 2024-09-15T03:47:15.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:15.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114140+0000", 2024-09-15T03:47:15.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029017370000000002 2024-09-15T03:47:15.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:15.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114201+0000", 2024-09-15T03:47:15.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1047e-05 2024-09-15T03:47:15.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:15.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123719+0000", 2024-09-15T03:47:15.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095184099999999997 2024-09-15T03:47:15.614 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.614 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:15.614 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.614 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:197 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [delete] snapc 3=[3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e34)", 2024-09-15T03:47:15.614 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:15.615 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 34.476166910000003, 2024-09-15T03:47:15.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021934704999999999, 2024-09-15T03:47:15.615 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:15.615 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:15.615 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:15.615 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:15.615 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:15.615 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197 2024-09-15T03:47:15.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.615 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:15.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:15.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:15.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:15.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:15.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:15.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112266+0000", 2024-09-15T03:47:15.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5000000000000002e-07 2024-09-15T03:47:15.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:15.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112269+0000", 2024-09-15T03:47:15.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.066e-06 2024-09-15T03:47:15.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:15.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112276+0000", 2024-09-15T03:47:15.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-09-15T03:47:15.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:15.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112281+0000", 2024-09-15T03:47:15.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3970000000000004e-06 2024-09-15T03:47:15.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123281+0000", 2024-09-15T03:47:15.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011000575 2024-09-15T03:47:15.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:15.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123395+0000", 2024-09-15T03:47:15.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113398 2024-09-15T03:47:15.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:15.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123963+0000", 2024-09-15T03:47:15.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000568101 2024-09-15T03:47:15.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:15.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124220+0000", 2024-09-15T03:47:15.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025704900000000003 2024-09-15T03:47:15.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:15.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124229+0000", 2024-09-15T03:47:15.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4860000000000003e-06 2024-09-15T03:47:15.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:15.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124277+0000", 2024-09-15T03:47:15.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7574000000000002e-05 2024-09-15T03:47:15.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:15.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124312+0000", 2024-09-15T03:47:15.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5256000000000003e-05 2024-09-15T03:47:15.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:15.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125716+0000", 2024-09-15T03:47:15.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014043720000000001 2024-09-15T03:47:15.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:15.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.134200+0000", 2024-09-15T03:47:15.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084835120000000003 2024-09-15T03:47:15.621 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.621 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:15.621 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.622 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es4 34/28 ECSubWrite(tid=33, reqid=client.4521.0:195, at_version=34'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-15T03:47:15.622 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:47:15.622 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 34.470339551000002, 2024-09-15T03:47:15.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015907482000000001, 2024-09-15T03:47:15.622 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:15.622 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:15.622 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:15.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:15.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:47:15.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:15.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:47:15.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:15.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:47:15.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9400000000000004e-07 2024-09-15T03:47:15.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:15.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118299+0000", 2024-09-15T03:47:15.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000205677 2024-09-15T03:47:15.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:15.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118315+0000", 2024-09-15T03:47:15.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5429000000000001e-05 2024-09-15T03:47:15.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:15.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118322+0000", 2024-09-15T03:47:15.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8290000000000008e-06 2024-09-15T03:47:15.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118416+0000", 2024-09-15T03:47:15.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3140000000000006e-05 2024-09-15T03:47:15.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:15.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118431+0000", 2024-09-15T03:47:15.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5648e-05 2024-09-15T03:47:15.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:15.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118678+0000", 2024-09-15T03:47:15.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024701200000000001 2024-09-15T03:47:15.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:15.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118691+0000", 2024-09-15T03:47:15.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2653000000000001e-05 2024-09-15T03:47:15.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:15.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.119431+0000", 2024-09-15T03:47:15.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00074060399999999996 2024-09-15T03:47:15.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:15.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124280+0000", 2024-09-15T03:47:15.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0048484410000000002 2024-09-15T03:47:15.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:15.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.134000+0000", 2024-09-15T03:47:15.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0097201549999999994 2024-09-15T03:47:15.627 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.627 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:15.627 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.627 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:220 3.bs0 3:d99ddedc:::smithi00132172-44:head [copy-from ver 22] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:15.627 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:15.627 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.460787239999998, 2024-09-15T03:47:15.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024273429999999999, 2024-09-15T03:47:15.628 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:15.628 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:15.628 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:15.628 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:15.628 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:15.628 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 220 2024-09-15T03:47:15.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.628 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:15.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:15.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:15.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:15.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:15.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:15.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127647+0000", 2024-09-15T03:47:15.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1799999999999999e-06 2024-09-15T03:47:15.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:15.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127656+0000", 2024-09-15T03:47:15.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3759999999999997e-06 2024-09-15T03:47:15.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:15.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127674+0000", 2024-09-15T03:47:15.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7374e-05 2024-09-15T03:47:15.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:15.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127680+0000", 2024-09-15T03:47:15.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6949999999999997e-06 2024-09-15T03:47:15.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127736+0000", 2024-09-15T03:47:15.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6011999999999998e-05 2024-09-15T03:47:15.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:15.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127744+0000", 2024-09-15T03:47:15.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2870000000000004e-06 2024-09-15T03:47:15.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130338+0000", 2024-09-15T03:47:15.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025940249999999998 2024-09-15T03:47:15.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:15.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130423+0000", 2024-09-15T03:47:15.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5303000000000006e-05 2024-09-15T03:47:15.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:15.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.147840+0000", 2024-09-15T03:47:15.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017416580000000001 2024-09-15T03:47:15.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:15.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148318+0000", 2024-09-15T03:47:15.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047822899999999999 2024-09-15T03:47:15.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:15.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148329+0000", 2024-09-15T03:47:15.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0611000000000001e-05 2024-09-15T03:47:15.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:15.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148488+0000", 2024-09-15T03:47:15.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015938399999999999 2024-09-15T03:47:15.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:15.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148870+0000", 2024-09-15T03:47:15.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000382405 2024-09-15T03:47:15.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:15.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.151092+0000", 2024-09-15T03:47:15.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022212529999999999 2024-09-15T03:47:15.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:15.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.151918+0000", 2024-09-15T03:47:15.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082671600000000002 2024-09-15T03:47:15.635 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.635 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:15.635 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.635 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:221 3.bs0 3:d99ddedc:::smithi00132172-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:15.635 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:15.635 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.460743085000001, 2024-09-15T03:47:15.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021194474000000001, 2024-09-15T03:47:15.635 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:15.635 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:15.635 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:15.636 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:15.636 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:15.636 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 221 2024-09-15T03:47:15.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.636 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:15.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:15.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:15.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:15.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:15.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:15.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127690+0000", 2024-09-15T03:47:15.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:47:15.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:15.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127692+0000", 2024-09-15T03:47:15.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6589999999999999e-06 2024-09-15T03:47:15.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:15.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127700+0000", 2024-09-15T03:47:15.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4769999999999997e-06 2024-09-15T03:47:15.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:15.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127704+0000", 2024-09-15T03:47:15.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3819999999999997e-06 2024-09-15T03:47:15.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127805+0000", 2024-09-15T03:47:15.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100639 2024-09-15T03:47:15.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:15.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127809+0000", 2024-09-15T03:47:15.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8870000000000002e-06 2024-09-15T03:47:15.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130527+0000", 2024-09-15T03:47:15.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002718579 2024-09-15T03:47:15.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:15.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130558+0000", 2024-09-15T03:47:15.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0827999999999997e-05 2024-09-15T03:47:15.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148546+0000", 2024-09-15T03:47:15.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017988356 2024-09-15T03:47:15.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:15.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148683+0000", 2024-09-15T03:47:15.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013650899999999999 2024-09-15T03:47:15.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:15.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148884+0000", 2024-09-15T03:47:15.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020061499999999999 2024-09-15T03:47:15.641 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.641 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:15.641 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.641 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:252 3.bs0 3:d6f79b7b:::smithi00132172-29:head [copy-from ver 6] snapc 7=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:47:15.641 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.169146+0000", 2024-09-15T03:47:15.641 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 29.419285815999999, 2024-09-15T03:47:15.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016731068000000002, 2024-09-15T03:47:15.642 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:15.642 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:15.642 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:15.642 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:15.642 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:15.642 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 252 2024-09-15T03:47:15.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.642 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:15.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:15.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169146+0000", 2024-09-15T03:47:15.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:15.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169146+0000", 2024-09-15T03:47:15.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:15.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169147+0000", 2024-09-15T03:47:15.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1400000000000001e-07 2024-09-15T03:47:15.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:15.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169149+0000", 2024-09-15T03:47:15.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0279999999999999e-06 2024-09-15T03:47:15.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:15.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169152+0000", 2024-09-15T03:47:15.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0800000000000002e-06 2024-09-15T03:47:15.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:15.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169154+0000", 2024-09-15T03:47:15.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4679999999999999e-06 2024-09-15T03:47:15.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.174427+0000", 2024-09-15T03:47:15.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052724349999999998 2024-09-15T03:47:15.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:15.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.174543+0000", 2024-09-15T03:47:15.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116803 2024-09-15T03:47:15.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:15.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.178297+0000", 2024-09-15T03:47:15.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037531539999999999 2024-09-15T03:47:15.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:15.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179112+0000", 2024-09-15T03:47:15.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081548600000000003 2024-09-15T03:47:15.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:15.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179151+0000", 2024-09-15T03:47:15.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9158e-05 2024-09-15T03:47:15.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:15.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179272+0000", 2024-09-15T03:47:15.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120802 2024-09-15T03:47:15.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:15.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179531+0000", 2024-09-15T03:47:15.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000259351 2024-09-15T03:47:15.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:15.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179723+0000", 2024-09-15T03:47:15.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000191447 2024-09-15T03:47:15.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:15.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.185877+0000", 2024-09-15T03:47:15.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061546420000000001 2024-09-15T03:47:15.648 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.648 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:15.648 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.648 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:357 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:15.648 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:15.648 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.739391551000001, 2024-09-15T03:47:15.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.02017917, 2024-09-15T03:47:15.648 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:15.648 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:15.649 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:15.649 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:15.649 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:15.649 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 357 2024-09-15T03:47:15.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.649 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:15.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:15.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:15.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:15.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:15.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:15.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849043+0000", 2024-09-15T03:47:15.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.295e-06 2024-09-15T03:47:15.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:15.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849051+0000", 2024-09-15T03:47:15.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4079999999999998e-06 2024-09-15T03:47:15.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:15.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849069+0000", 2024-09-15T03:47:15.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7462999999999999e-05 2024-09-15T03:47:15.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:15.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849077+0000", 2024-09-15T03:47:15.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8869999999999992e-06 2024-09-15T03:47:15.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849194+0000", 2024-09-15T03:47:15.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011723899999999999 2024-09-15T03:47:15.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:15.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849296+0000", 2024-09-15T03:47:15.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102011 2024-09-15T03:47:15.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861106+0000", 2024-09-15T03:47:15.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0118097610000001 2024-09-15T03:47:15.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:15.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861222+0000", 2024-09-15T03:47:15.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116561 2024-09-15T03:47:15.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:15.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.869220+0000", 2024-09-15T03:47:15.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007997545 2024-09-15T03:47:15.653 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.653 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:15.653 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.653 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:358 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:15.654 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:15.654 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.739336604999998, 2024-09-15T03:47:15.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0228638050000001, 2024-09-15T03:47:15.654 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:15.654 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:15.654 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:15.654 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:15.654 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:15.654 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 358 2024-09-15T03:47:15.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.654 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:15.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:15.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:15.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:15.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:15.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:15.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:15.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4099999999999997e-07 2024-09-15T03:47:15.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:15.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849101+0000", 2024-09-15T03:47:15.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8180000000000003e-06 2024-09-15T03:47:15.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:15.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849110+0000", 2024-09-15T03:47:15.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1409999999999994e-06 2024-09-15T03:47:15.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:15.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849116+0000", 2024-09-15T03:47:15.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2390000000000002e-06 2024-09-15T03:47:15.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849326+0000", 2024-09-15T03:47:15.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000210029 2024-09-15T03:47:15.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:15.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849421+0000", 2024-09-15T03:47:15.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5775999999999999e-05 2024-09-15T03:47:15.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861647+0000", 2024-09-15T03:47:15.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0122257139999999 2024-09-15T03:47:15.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:15.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861777+0000", 2024-09-15T03:47:15.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130087 2024-09-15T03:47:15.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:15.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.871959+0000", 2024-09-15T03:47:15.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010182160000000001 2024-09-15T03:47:15.658 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.659 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:15.659 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.659 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:359 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:15.659 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:15.659 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.739311452999999, 2024-09-15T03:47:15.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0240468149999999, 2024-09-15T03:47:15.659 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:15.659 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:15.659 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:15.659 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:15.659 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:15.660 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 359 2024-09-15T03:47:15.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.660 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:15.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:15.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:15.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:15.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:15.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:15.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:15.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:47:15.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:15.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849124+0000", 2024-09-15T03:47:15.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2679999999999999e-06 2024-09-15T03:47:15.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:15.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849132+0000", 2024-09-15T03:47:15.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4030000000000001e-06 2024-09-15T03:47:15.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:15.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849136+0000", 2024-09-15T03:47:15.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2289999999999997e-06 2024-09-15T03:47:15.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849446+0000", 2024-09-15T03:47:15.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031043299999999999 2024-09-15T03:47:15.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:15.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849553+0000", 2024-09-15T03:47:15.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106415 2024-09-15T03:47:15.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.863404+0000", 2024-09-15T03:47:15.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0138511029999999 2024-09-15T03:47:15.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:15.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.863475+0000", 2024-09-15T03:47:15.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0814000000000003e-05 2024-09-15T03:47:15.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:15.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.873168+0000", 2024-09-15T03:47:15.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00969274 2024-09-15T03:47:15.664 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.664 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:15.664 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.664 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:474 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~786432] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:15.664 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:15.664 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7133867780000003, 2024-09-15T03:47:15.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98604943700000003, 2024-09-15T03:47:15.665 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:15.665 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:15.665 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:15.665 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:15.665 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:15.665 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 474 2024-09-15T03:47:15.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.665 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:15.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:15.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:15.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:15.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:15.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:15.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875048+0000", 2024-09-15T03:47:15.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1600000000000001e-06 2024-09-15T03:47:15.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:15.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875844+0000", 2024-09-15T03:47:15.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079686299999999998 2024-09-15T03:47:15.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:15.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875857+0000", 2024-09-15T03:47:15.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2889e-05 2024-09-15T03:47:15.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:15.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875864+0000", 2024-09-15T03:47:15.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6139999999999997e-06 2024-09-15T03:47:15.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875916+0000", 2024-09-15T03:47:15.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1992e-05 2024-09-15T03:47:15.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:15.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875918+0000", 2024-09-15T03:47:15.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3300000000000001e-06 2024-09-15T03:47:15.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.845918+0000", 2024-09-15T03:47:15.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96999959199999997 2024-09-15T03:47:15.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:15.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.846229+0000", 2024-09-15T03:47:15.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000311601 2024-09-15T03:47:15.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:15.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848445+0000", 2024-09-15T03:47:15.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022160499999999998 2024-09-15T03:47:15.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:15.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848880+0000", 2024-09-15T03:47:15.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043432600000000002 2024-09-15T03:47:15.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:15.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848952+0000", 2024-09-15T03:47:15.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2509000000000007e-05 2024-09-15T03:47:15.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:15.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.849311+0000", 2024-09-15T03:47:15.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000358611 2024-09-15T03:47:15.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:15.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851131+0000", 2024-09-15T03:47:15.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001820115 2024-09-15T03:47:15.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:15.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861048+0000", 2024-09-15T03:47:15.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099164830000000002 2024-09-15T03:47:15.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:15.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861095+0000", 2024-09-15T03:47:15.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7302000000000002e-05 2024-09-15T03:47:15.671 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.671 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:15.672 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.672 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:475 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~491520] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:15.672 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:15.672 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7106203070000001, 2024-09-15T03:47:15.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98407319000000004, 2024-09-15T03:47:15.672 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:15.672 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:15.672 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:15.672 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:15.672 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:15.673 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 475 2024-09-15T03:47:15.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.673 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:15.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:15.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:15.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:15.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:15.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:15.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877813+0000", 2024-09-15T03:47:15.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8800000000000002e-07 2024-09-15T03:47:15.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:15.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878260+0000", 2024-09-15T03:47:15.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044732099999999998 2024-09-15T03:47:15.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:15.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878270+0000", 2024-09-15T03:47:15.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9799999999999993e-06 2024-09-15T03:47:15.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:15.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878276+0000", 2024-09-15T03:47:15.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2269999999999998e-06 2024-09-15T03:47:15.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878313+0000", 2024-09-15T03:47:15.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6570999999999999e-05 2024-09-15T03:47:15.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:15.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878315+0000", 2024-09-15T03:47:15.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8729999999999999e-06 2024-09-15T03:47:15.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851316+0000", 2024-09-15T03:47:15.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97300154800000005 2024-09-15T03:47:15.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:15.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851440+0000", 2024-09-15T03:47:15.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124287 2024-09-15T03:47:15.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:15.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852732+0000", 2024-09-15T03:47:15.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001291174 2024-09-15T03:47:15.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:15.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852991+0000", 2024-09-15T03:47:15.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025905299999999997 2024-09-15T03:47:15.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:15.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853003+0000", 2024-09-15T03:47:15.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2908000000000001e-05 2024-09-15T03:47:15.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:15.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853104+0000", 2024-09-15T03:47:15.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010011800000000001 2024-09-15T03:47:15.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:15.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854342+0000", 2024-09-15T03:47:15.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012380189999999999 2024-09-15T03:47:15.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:15.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861206+0000", 2024-09-15T03:47:15.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068648499999999996 2024-09-15T03:47:15.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:15.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861885+0000", 2024-09-15T03:47:15.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00067857300000000002 2024-09-15T03:47:15.681 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.681 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:15.681 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.681 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:476 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~278528] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:15.681 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:15.681 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7089932939999999, 2024-09-15T03:47:15.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98268067299999995, 2024-09-15T03:47:15.681 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:15.681 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:15.682 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:15.682 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:15.682 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:15.682 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 476 2024-09-15T03:47:15.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.682 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:15.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:15.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:15.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:15.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:15.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:15.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:15.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.13e-07 2024-09-15T03:47:15.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:15.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879706+0000", 2024-09-15T03:47:15.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026656100000000001 2024-09-15T03:47:15.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:15.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879715+0000", 2024-09-15T03:47:15.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9330000000000007e-06 2024-09-15T03:47:15.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:15.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879720+0000", 2024-09-15T03:47:15.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0880000000000002e-06 2024-09-15T03:47:15.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879743+0000", 2024-09-15T03:47:15.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2654e-05 2024-09-15T03:47:15.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:15.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879743+0000", 2024-09-15T03:47:15.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.99e-07 2024-09-15T03:47:15.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853170+0000", 2024-09-15T03:47:15.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97342687900000002 2024-09-15T03:47:15.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:15.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853297+0000", 2024-09-15T03:47:15.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127157 2024-09-15T03:47:15.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:15.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854150+0000", 2024-09-15T03:47:15.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085288900000000001 2024-09-15T03:47:15.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:15.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854292+0000", 2024-09-15T03:47:15.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014199300000000001 2024-09-15T03:47:15.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:15.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854559+0000", 2024-09-15T03:47:15.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026668999999999998 2024-09-15T03:47:15.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:15.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855998+0000", 2024-09-15T03:47:15.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001438714 2024-09-15T03:47:15.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:15.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856547+0000", 2024-09-15T03:47:15.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054880399999999996 2024-09-15T03:47:15.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:15.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861858+0000", 2024-09-15T03:47:15.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053110320000000003 2024-09-15T03:47:15.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:15.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.862120+0000", 2024-09-15T03:47:15.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026196699999999999 2024-09-15T03:47:15.689 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.689 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:15.689 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.689 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:477 3.bs0 3:d52faf55:::smithi00132172-40:head [setxattr _header (57) in=7b] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:15.689 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:15.689 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7087053570000004, 2024-09-15T03:47:15.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98562198000000001, 2024-09-15T03:47:15.689 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:15.690 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:15.690 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:15.690 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:15.690 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:15.690 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 477 2024-09-15T03:47:15.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.690 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:15.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:15.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:15.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:15.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:15.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:15.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:15.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2800000000000002e-07 2024-09-15T03:47:15.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:15.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879730+0000", 2024-09-15T03:47:15.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0589999999999998e-06 2024-09-15T03:47:15.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:15.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879737+0000", 2024-09-15T03:47:15.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0419999999999997e-06 2024-09-15T03:47:15.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:15.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879741+0000", 2024-09-15T03:47:15.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9960000000000004e-06 2024-09-15T03:47:15.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879756+0000", 2024-09-15T03:47:15.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4681999999999999e-05 2024-09-15T03:47:15.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:15.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879756+0000", 2024-09-15T03:47:15.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.22e-07 2024-09-15T03:47:15.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854590+0000", 2024-09-15T03:47:15.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97483319000000002 2024-09-15T03:47:15.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:15.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854718+0000", 2024-09-15T03:47:15.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012842100000000001 2024-09-15T03:47:15.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:15.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855416+0000", 2024-09-15T03:47:15.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069759200000000003 2024-09-15T03:47:15.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:15.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855551+0000", 2024-09-15T03:47:15.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013517600000000001 2024-09-15T03:47:15.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:15.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856002+0000", 2024-09-15T03:47:15.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045169399999999999 2024-09-15T03:47:15.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:15.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856065+0000", 2024-09-15T03:47:15.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3028999999999996e-05 2024-09-15T03:47:15.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:15.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856554+0000", 2024-09-15T03:47:15.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000488486 2024-09-15T03:47:15.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:15.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865278+0000", 2024-09-15T03:47:15.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087241929999999999 2024-09-15T03:47:15.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:15.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865349+0000", 2024-09-15T03:47:15.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0669999999999999e-05 2024-09-15T03:47:15.697 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.697 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:15.697 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.697 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:478 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:15.697 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:15.697 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7086865620000005, 2024-09-15T03:47:15.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99471090500000003, 2024-09-15T03:47:15.697 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:15.697 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:15.697 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:15.697 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:15.698 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:15.698 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 478 2024-09-15T03:47:15.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.698 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:15.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:15.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:15.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:15.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:15.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:15.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:15.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2599999999999998e-07 2024-09-15T03:47:15.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:15.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879748+0000", 2024-09-15T03:47:15.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9609999999999999e-06 2024-09-15T03:47:15.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:15.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879755+0000", 2024-09-15T03:47:15.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1879999999999996e-06 2024-09-15T03:47:15.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:15.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879760+0000", 2024-09-15T03:47:15.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2560000000000002e-06 2024-09-15T03:47:15.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879797+0000", 2024-09-15T03:47:15.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6452999999999998e-05 2024-09-15T03:47:15.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:15.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879799+0000", 2024-09-15T03:47:15.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6619999999999999e-06 2024-09-15T03:47:15.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855637+0000", 2024-09-15T03:47:15.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97583850999999999 2024-09-15T03:47:15.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:15.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855763+0000", 2024-09-15T03:47:15.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012612699999999999 2024-09-15T03:47:15.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865707+0000", 2024-09-15T03:47:15.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099439660000000003 2024-09-15T03:47:15.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:15.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865803+0000", 2024-09-15T03:47:15.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5946999999999995e-05 2024-09-15T03:47:15.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:15.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.874456+0000", 2024-09-15T03:47:15.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0086534090000000008 2024-09-15T03:47:15.703 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.703 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:15.703 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.703 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:486 3.bs0 3:df8b18a3:::smithi00132172-17:head [delete] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e64)", 2024-09-15T03:47:15.703 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:15.703 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7058740830000003, 2024-09-15T03:47:15.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98362617100000005, 2024-09-15T03:47:15.703 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:15.703 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:15.703 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:15.704 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:15.704 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:15.704 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 486 2024-09-15T03:47:15.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.704 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:15.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:15.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:15.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:15.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:15.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:15.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882559+0000", 2024-09-15T03:47:15.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0359999999999999e-06 2024-09-15T03:47:15.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:15.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882564+0000", 2024-09-15T03:47:15.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6530000000000002e-06 2024-09-15T03:47:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882574+0000", 2024-09-15T03:47:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0464e-05 2024-09-15T03:47:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882578+0000", 2024-09-15T03:47:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2459999999999997e-06 2024-09-15T03:47:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882624+0000", 2024-09-15T03:47:15.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5698000000000002e-05 2024-09-15T03:47:15.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:15.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882627+0000", 2024-09-15T03:47:15.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3350000000000001e-06 2024-09-15T03:47:15.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856574+0000", 2024-09-15T03:47:15.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97394739399999997 2024-09-15T03:47:15.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:15.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856853+0000", 2024-09-15T03:47:15.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027918399999999998 2024-09-15T03:47:15.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:15.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.866184+0000", 2024-09-15T03:47:15.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093311609999999993 2024-09-15T03:47:15.708 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.708 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:15.708 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.709 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:495 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [copy-from ver 51] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-15T03:47:15.709 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:15.709 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.6999730340000001, 2024-09-15T03:47:15.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023555178, 2024-09-15T03:47:15.709 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:15.709 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:15.709 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:15.709 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:15.709 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:15.709 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 495 2024-09-15T03:47:15.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.709 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:15.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:15.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:15.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:15.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:15.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:15.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888461+0000", 2024-09-15T03:47:15.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.719e-06 2024-09-15T03:47:15.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:15.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888471+0000", 2024-09-15T03:47:15.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0161e-05 2024-09-15T03:47:15.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:15.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888500+0000", 2024-09-15T03:47:15.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8989e-05 2024-09-15T03:47:15.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:15.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888508+0000", 2024-09-15T03:47:15.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5299999999999999e-06 2024-09-15T03:47:15.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888554+0000", 2024-09-15T03:47:15.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6489000000000002e-05 2024-09-15T03:47:15.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:15.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888809+0000", 2024-09-15T03:47:15.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000255467 2024-09-15T03:47:15.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:15.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.905724+0000", 2024-09-15T03:47:15.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016914348999999999 2024-09-15T03:47:15.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:15.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906244+0000", 2024-09-15T03:47:15.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052054900000000003 2024-09-15T03:47:15.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:15.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906262+0000", 2024-09-15T03:47:15.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7499999999999998e-05 2024-09-15T03:47:15.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:15.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.907367+0000", 2024-09-15T03:47:15.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011049460000000001 2024-09-15T03:47:15.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:15.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.907384+0000", 2024-09-15T03:47:15.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7000999999999999e-05 2024-09-15T03:47:15.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:15.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.911965+0000", 2024-09-15T03:47:15.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045809470000000001 2024-09-15T03:47:15.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:15.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.912014+0000", 2024-09-15T03:47:15.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9530999999999997e-05 2024-09-15T03:47:15.715 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.715 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:15.715 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.715 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:496 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-15T03:47:15.715 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.888524+0000", 2024-09-15T03:47:15.715 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.6999079419999998, 2024-09-15T03:47:15.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01823953, 2024-09-15T03:47:15.716 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:15.716 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:15.716 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:15.716 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:15.716 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:15.716 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 496 2024-09-15T03:47:15.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.716 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:15.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:15.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888524+0000", 2024-09-15T03:47:15.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:15.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888524+0000", 2024-09-15T03:47:15.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:15.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:15.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888525+0000", 2024-09-15T03:47:15.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2699999999999999e-07 2024-09-15T03:47:15.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:15.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888529+0000", 2024-09-15T03:47:15.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7840000000000001e-06 2024-09-15T03:47:15.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:15.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888540+0000", 2024-09-15T03:47:15.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1477e-05 2024-09-15T03:47:15.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:15.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888546+0000", 2024-09-15T03:47:15.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4990000000000002e-06 2024-09-15T03:47:15.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888971+0000", 2024-09-15T03:47:15.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042491300000000002 2024-09-15T03:47:15.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:15.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.889024+0000", 2024-09-15T03:47:15.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3046e-05 2024-09-15T03:47:15.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:15.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906517+0000", 2024-09-15T03:47:15.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017492909000000001 2024-09-15T03:47:15.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:15.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906638+0000", 2024-09-15T03:47:15.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012138000000000001 2024-09-15T03:47:15.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:15.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:15.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:15.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906764+0000", 2024-09-15T03:47:15.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125795 2024-09-15T03:47:15.720 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.720 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:15.720 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.720 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:15.720 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:15.720 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:15.722 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:47:15.722 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:47:15.723 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:15.723 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:47:15.724 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:15.724 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:15.721+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:15.823 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:15.822+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:15.832 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:15.832 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:15.832 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:15.832 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:15.837 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.836+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.837 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.836+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.837 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.836+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.837 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.836+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.838 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.836+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.838 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.836+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.838 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.837+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.838 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.837+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.838 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.837+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.838 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.837+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.838 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.837+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.838 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.837+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.840 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.838+0000 7f63ab7ba640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.840 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.838+0000 7f63ab7ba640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.840 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.838+0000 7f63ab7ba640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.840 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.838+0000 7f63ab7ba640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.840 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.838+0000 7f63ab7ba640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.840 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.838+0000 7f63ab7ba640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.841 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.838+0000 7f63ab7ba640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.841 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.838+0000 7f63ab7ba640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.841 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.838+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.841 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.838+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.841 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.838+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.841 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.838+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.841 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.838+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.841 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.838+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.841 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.839+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.841 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.839+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.841 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.839+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.841 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.839+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.841 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.839+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.842 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.839+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.842 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.839+0000 7f63ab7ba640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.842 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.839+0000 7f63ab7ba640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.842 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.839+0000 7f63ab7ba640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.842 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.839+0000 7f63ab7ba640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.842 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.842 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.842 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.842 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.842 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.842 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.843 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.843 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.843 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.844 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.844 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.844 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.844 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.840+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.844 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.840+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.844 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.840+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.844 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.840+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.844 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.840+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.844 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.840+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.844 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.844 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.844 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.844 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.845 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.845 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:15.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:15.847 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:47:15.924 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:15.923+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:15.947 INFO:tasks.rados.rados.0.smithi001.stdout:199: done (0 left) 2024-09-15T03:47:15.947 INFO:tasks.rados.rados.0.smithi001.stdout:200: read oid 10 snap -1 2024-09-15T03:47:15.947 INFO:tasks.rados.rados.0.smithi001.stdout:200: expect (ObjNum 96 snap 21 seq_num 96) 2024-09-15T03:47:15.947 INFO:tasks.rados.rados.0.smithi001.stdout:201: read oid 28 snap -1 2024-09-15T03:47:15.947 INFO:tasks.rados.rados.0.smithi001.stdout:201: expect deleted 2024-09-15T03:47:15.947 INFO:tasks.rados.rados.0.smithi001.stdout:202: read oid 32 snap -1 2024-09-15T03:47:15.947 INFO:tasks.rados.rados.0.smithi001.stdout:202: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-15T03:47:15.947 INFO:tasks.rados.rados.0.smithi001.stdout:203: setattr oid 18 current snap is 23 2024-09-15T03:47:15.948 INFO:tasks.rados.rados.0.smithi001.stdout:201: done (3 left) 2024-09-15T03:47:15.948 INFO:tasks.rados.rados.0.smithi001.stdout:204: rmattr oid 5 current snap is 23 2024-09-15T03:47:15.948 INFO:tasks.rados.rados.0.smithi001.stdout:204: done (3 left) 2024-09-15T03:47:15.949 INFO:tasks.rados.rados.0.smithi001.stdout:205: copy_from oid 41 from oid 3 current snap is 23 2024-09-15T03:47:15.949 INFO:tasks.rados.rados.0.smithi001.stdout:206: snap_create 2024-09-15T03:47:15.953 INFO:tasks.rados.rados.0.smithi001.stdout:202: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-15T03:47:15.959 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:15.959 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:15.959 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:15.959 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:15.960 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:15.969 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 18 v 27 (ObjNum 90 snap 21 seq_num 90) dirty exists 2024-09-15T03:47:15.972 INFO:tasks.rados.rados.0.smithi001.stdout:200: expect (ObjNum 96 snap 21 seq_num 96) 2024-09-15T03:47:15.977 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:47:16.024 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:16.024+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:16.095 INFO:tasks.rados.rados.0.smithi001.stdout:205: finishing copy_from racing read to smithi00132172-41 2024-09-15T03:47:16.095 INFO:tasks.rados.rados.0.smithi001.stdout:205: finishing copy_from to smithi00132172-41 2024-09-15T03:47:16.095 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 41 v 51 (ObjNum 81 snap 16 seq_num 81) dirty exists 2024-09-15T03:47:16.097 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:16.097 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:16.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:16.098 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:47:16.098 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.098 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:47:16.098 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:16.098 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 58.193449350000002, 2024-09-15T03:47:16.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10988437500000001, 2024-09-15T03:47:16.098 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.098 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:16.098 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:16.098 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:47:16.098 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:47:16.099 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1 2024-09-15T03:47:16.099 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.099 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.099 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:16.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.099 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.099 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:16.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.100 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.100 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902355+0000", 2024-09-15T03:47:16.100 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.429e-06 2024-09-15T03:47:16.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.100 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.100 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902362+0000", 2024-09-15T03:47:16.100 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7290000000000003e-06 2024-09-15T03:47:16.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902373+0000", 2024-09-15T03:47:16.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0533999999999999e-05 2024-09-15T03:47:16.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.101 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902379+0000", 2024-09-15T03:47:16.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4350000000000002e-06 2024-09-15T03:47:16.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.101 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902428+0000", 2024-09-15T03:47:16.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8571e-05 2024-09-15T03:47:16.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:47:16.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902434+0000", 2024-09-15T03:47:16.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5610000000000004e-06 2024-09-15T03:47:16.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011307+0000", 2024-09-15T03:47:16.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.108872306 2024-09-15T03:47:16.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:16.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011448+0000", 2024-09-15T03:47:16.103 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014137199999999999 2024-09-15T03:47:16.103 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.103 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.103 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:47:16.103 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011569+0000", 2024-09-15T03:47:16.103 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120925 2024-09-15T03:47:16.103 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.103 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.103 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.103 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011695+0000", 2024-09-15T03:47:16.103 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012586800000000001 2024-09-15T03:47:16.103 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.103 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.103 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.104 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011703+0000", 2024-09-15T03:47:16.104 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4190000000000002e-06 2024-09-15T03:47:16.104 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.104 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.104 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011739+0000", 2024-09-15T03:47:16.104 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5474e-05 2024-09-15T03:47:16.104 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.104 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:47:16.104 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011940+0000", 2024-09-15T03:47:16.104 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020108699999999999 2024-09-15T03:47:16.104 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:47:16.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012179+0000", 2024-09-15T03:47:16.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023950299999999999 2024-09-15T03:47:16.105 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:16.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012215+0000", 2024-09-15T03:47:16.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5332e-05 2024-09-15T03:47:16.105 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012238+0000", 2024-09-15T03:47:16.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3830000000000001e-05 2024-09-15T03:47:16.106 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.106 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.106 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.106 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.106 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.106 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4263.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2024-09-15T03:47:16.106 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:16.106 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 52.052052664000001, 2024-09-15T03:47:16.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019495788999999999, 2024-09-15T03:47:16.106 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.106 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.106 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:16.106 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4263", 2024-09-15T03:47:16.106 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/646873808", 2024-09-15T03:47:16.107 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2 2024-09-15T03:47:16.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.107 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:16.107 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:16.107 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.108 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043752+0000", 2024-09-15T03:47:16.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.815e-06 2024-09-15T03:47:16.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.108 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.108 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043757+0000", 2024-09-15T03:47:16.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4240000000000003e-06 2024-09-15T03:47:16.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.108 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.108 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043766+0000", 2024-09-15T03:47:16.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8629999999999997e-06 2024-09-15T03:47:16.109 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.109 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.109 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.109 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043773+0000", 2024-09-15T03:47:16.109 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0099999999999998e-06 2024-09-15T03:47:16.109 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.109 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.109 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.109 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043830+0000", 2024-09-15T03:47:16.109 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7089999999999997e-05 2024-09-15T03:47:16.109 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.109 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.109 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:47:16.109 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043837+0000", 2024-09-15T03:47:16.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.08e-06 2024-09-15T03:47:16.110 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.110 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.110 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063103+0000", 2024-09-15T03:47:16.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019265619000000001 2024-09-15T03:47:16.110 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.110 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.110 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063246+0000", 2024-09-15T03:47:16.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000143888 2024-09-15T03:47:16.110 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.110 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.110 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.110 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.111 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.fs5 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:16.111 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:47:16.111 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 35.988204758000002, 2024-09-15T03:47:16.111 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098369180000000001, 2024-09-15T03:47:16.111 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.111 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.111 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.111 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:47:16.111 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.111 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:47:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107599+0000", 2024-09-15T03:47:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4499999999999998e-07 2024-09-15T03:47:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107630+0000", 2024-09-15T03:47:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0877999999999998e-05 2024-09-15T03:47:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107642+0000", 2024-09-15T03:47:16.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2567e-05 2024-09-15T03:47:16.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107648+0000", 2024-09-15T03:47:16.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8690000000000001e-06 2024-09-15T03:47:16.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107894+0000", 2024-09-15T03:47:16.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024540400000000001 2024-09-15T03:47:16.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107900+0000", 2024-09-15T03:47:16.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5710000000000003e-06 2024-09-15T03:47:16.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107968+0000", 2024-09-15T03:47:16.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7957e-05 2024-09-15T03:47:16.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108274+0000", 2024-09-15T03:47:16.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030591700000000001 2024-09-15T03:47:16.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108504+0000", 2024-09-15T03:47:16.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022952199999999999 2024-09-15T03:47:16.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109438+0000", 2024-09-15T03:47:16.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00093439100000000002 2024-09-15T03:47:16.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.116 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117435+0000", 2024-09-15T03:47:16.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0079974969999999992 2024-09-15T03:47:16.116 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.116 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.116 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.116 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:16.116 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:47:16.116 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 35.988147505999997, 2024-09-15T03:47:16.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084894540000000004, 2024-09-15T03:47:16.117 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.117 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.117 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.117 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:47:16.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.117 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:47:16.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.118 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.118 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107656+0000", 2024-09-15T03:47:16.118 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4700000000000002e-07 2024-09-15T03:47:16.118 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.118 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.118 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.118 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107670+0000", 2024-09-15T03:47:16.118 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3892e-05 2024-09-15T03:47:16.118 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.119 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107678+0000", 2024-09-15T03:47:16.119 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9589999999999995e-06 2024-09-15T03:47:16.119 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.119 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107683+0000", 2024-09-15T03:47:16.119 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7199999999999997e-06 2024-09-15T03:47:16.119 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.119 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107768+0000", 2024-09-15T03:47:16.120 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4968000000000004e-05 2024-09-15T03:47:16.120 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.120 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.120 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.120 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107776+0000", 2024-09-15T03:47:16.120 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9109999999999993e-06 2024-09-15T03:47:16.120 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.120 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.120 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.120 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107858+0000", 2024-09-15T03:47:16.121 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2349999999999996e-05 2024-09-15T03:47:16.121 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.121 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.121 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.121 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108273+0000", 2024-09-15T03:47:16.121 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00041525600000000002 2024-09-15T03:47:16.121 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.121 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.121 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.121 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108503+0000", 2024-09-15T03:47:16.121 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022990599999999999 2024-09-15T03:47:16.121 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.121 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.121 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.122 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108705+0000", 2024-09-15T03:47:16.122 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020173000000000001 2024-09-15T03:47:16.122 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.122 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.122 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.122 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.116145+0000", 2024-09-15T03:47:16.122 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0074404149999999997 2024-09-15T03:47:16.122 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.122 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.122 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.122 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.122 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.122 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/31 ECSubWrite(tid=38, reqid=client.4521.0:223, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:47:16.123 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:47:16.123 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 30.948596327000001, 2024-09-15T03:47:16.123 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010504493, 2024-09-15T03:47:16.123 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.123 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.123 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.123 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.123 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.123 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:47:16.123 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.123 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.123 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.123 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.123 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:47:16.124 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.124 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.124 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.124 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.124 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:47:16.124 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4999999999999999e-07 2024-09-15T03:47:16.124 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.124 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.124 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.124 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147648+0000", 2024-09-15T03:47:16.124 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00044030399999999998 2024-09-15T03:47:16.124 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.125 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.125 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.125 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147660+0000", 2024-09-15T03:47:16.125 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2517000000000001e-05 2024-09-15T03:47:16.125 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.125 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.125 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.125 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147667+0000", 2024-09-15T03:47:16.125 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.737e-06 2024-09-15T03:47:16.125 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.125 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.125 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.125 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147713+0000", 2024-09-15T03:47:16.126 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6405e-05 2024-09-15T03:47:16.126 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.126 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.126 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.126 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147719+0000", 2024-09-15T03:47:16.126 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0580000000000003e-06 2024-09-15T03:47:16.126 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.126 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.126 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.126 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147905+0000", 2024-09-15T03:47:16.126 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018574600000000001 2024-09-15T03:47:16.126 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.126 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.127 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.127 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147917+0000", 2024-09-15T03:47:16.127 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1749999999999999e-05 2024-09-15T03:47:16.127 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.127 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.127 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.127 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148022+0000", 2024-09-15T03:47:16.127 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104863 2024-09-15T03:47:16.127 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.127 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.127 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.127 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148037+0000", 2024-09-15T03:47:16.127 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5302999999999999e-05 2024-09-15T03:47:16.128 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.128 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.128 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.128 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157711+0000", 2024-09-15T03:47:16.128 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0096745600000000005 2024-09-15T03:47:16.128 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.128 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.128 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.128 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.130 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.130 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 39/28 ECSubWrite(tid=42, reqid=client.4521.0:255, at_version=39'19, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:47:16.130 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.170708+0000", 2024-09-15T03:47:16.130 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 29.925095540000001, 2024-09-15T03:47:16.130 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0093655969999999998, 2024-09-15T03:47:16.130 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.130 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.130 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.131 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.131 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.131 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170708+0000", 2024-09-15T03:47:16.131 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.131 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.131 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.131 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.131 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170708+0000", 2024-09-15T03:47:16.131 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.131 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.131 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.131 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.131 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170708+0000", 2024-09-15T03:47:16.131 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8899999999999999e-07 2024-09-15T03:47:16.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.132 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.132 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170728+0000", 2024-09-15T03:47:16.132 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9184000000000001e-05 2024-09-15T03:47:16.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.132 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.132 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170741+0000", 2024-09-15T03:47:16.132 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3696999999999999e-05 2024-09-15T03:47:16.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.132 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170747+0000", 2024-09-15T03:47:16.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1569999999999997e-06 2024-09-15T03:47:16.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.133 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.133 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170803+0000", 2024-09-15T03:47:16.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5974000000000003e-05 2024-09-15T03:47:16.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.133 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.133 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170811+0000", 2024-09-15T03:47:16.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8259999999999993e-06 2024-09-15T03:47:16.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170902+0000", 2024-09-15T03:47:16.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1116e-05 2024-09-15T03:47:16.134 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170909+0000", 2024-09-15T03:47:16.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3489999999999997e-06 2024-09-15T03:47:16.134 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173637+0000", 2024-09-15T03:47:16.135 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027282270000000002 2024-09-15T03:47:16.135 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.135 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.135 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173649+0000", 2024-09-15T03:47:16.135 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2561000000000001e-05 2024-09-15T03:47:16.135 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.135 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.135 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180073+0000", 2024-09-15T03:47:16.135 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064239170000000003 2024-09-15T03:47:16.135 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.135 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.135 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.136 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.136 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.136 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:47:16.136 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:16.136 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 29.924979274999998, 2024-09-15T03:47:16.136 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011878227, 2024-09-15T03:47:16.136 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.136 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.136 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.136 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.136 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.136 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:16.137 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.137 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.137 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:16.137 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.137 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.137 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170825+0000", 2024-09-15T03:47:16.137 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9599999999999999e-07 2024-09-15T03:47:16.137 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.138 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.138 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171100+0000", 2024-09-15T03:47:16.138 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027548500000000001 2024-09-15T03:47:16.138 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.138 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.138 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.138 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171112+0000", 2024-09-15T03:47:16.138 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2152e-05 2024-09-15T03:47:16.138 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.138 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.138 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.138 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171119+0000", 2024-09-15T03:47:16.138 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5050000000000004e-06 2024-09-15T03:47:16.138 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.139 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.139 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.139 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171162+0000", 2024-09-15T03:47:16.139 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2908000000000003e-05 2024-09-15T03:47:16.139 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.139 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.139 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.139 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171169+0000", 2024-09-15T03:47:16.139 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9739999999999996e-06 2024-09-15T03:47:16.139 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.139 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.139 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.139 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171398+0000", 2024-09-15T03:47:16.139 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022954799999999999 2024-09-15T03:47:16.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.140 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.140 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.140 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173640+0000", 2024-09-15T03:47:16.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022417000000000001 2024-09-15T03:47:16.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.140 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.140 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.140 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173705+0000", 2024-09-15T03:47:16.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4907999999999996e-05 2024-09-15T03:47:16.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.140 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.140 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.140 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173742+0000", 2024-09-15T03:47:16.141 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6940999999999997e-05 2024-09-15T03:47:16.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.141 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.141 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.141 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.182702+0000", 2024-09-15T03:47:16.141 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0089604100000000002 2024-09-15T03:47:16.141 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.141 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.141 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.141 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.141 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 39/28 ECSubWrite(tid=43, reqid=client.4521.0:256, at_version=39'20, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:47:16.141 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.171196+0000", 2024-09-15T03:47:16.141 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 29.92460719, 2024-09-15T03:47:16.142 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0090205690000000009, 2024-09-15T03:47:16.142 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.142 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.142 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.142 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.142 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.142 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171196+0000", 2024-09-15T03:47:16.142 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.142 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.142 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.142 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.142 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171196+0000", 2024-09-15T03:47:16.142 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.143 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.143 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.143 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.143 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171197+0000", 2024-09-15T03:47:16.143 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3e-07 2024-09-15T03:47:16.143 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.143 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.143 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.143 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171218+0000", 2024-09-15T03:47:16.143 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0775e-05 2024-09-15T03:47:16.143 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.143 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.143 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.144 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171241+0000", 2024-09-15T03:47:16.144 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3538999999999999e-05 2024-09-15T03:47:16.144 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.144 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.144 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.144 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171248+0000", 2024-09-15T03:47:16.144 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6220000000000003e-06 2024-09-15T03:47:16.144 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.144 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.144 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.144 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171291+0000", 2024-09-15T03:47:16.144 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3115000000000003e-05 2024-09-15T03:47:16.144 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.145 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.145 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.145 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171301+0000", 2024-09-15T03:47:16.145 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.662e-06 2024-09-15T03:47:16.145 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.145 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.145 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.145 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171395+0000", 2024-09-15T03:47:16.145 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4368000000000002e-05 2024-09-15T03:47:16.145 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.145 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.145 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.145 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173639+0000", 2024-09-15T03:47:16.145 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022444040000000002 2024-09-15T03:47:16.146 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.146 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.146 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.146 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173704+0000", 2024-09-15T03:47:16.146 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5031999999999997e-05 2024-09-15T03:47:16.146 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.146 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.146 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.146 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173718+0000", 2024-09-15T03:47:16.146 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3159000000000001e-05 2024-09-15T03:47:16.146 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.146 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.146 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.147 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180217+0000", 2024-09-15T03:47:16.147 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064991629999999996 2024-09-15T03:47:16.147 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.147 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.147 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.147 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.147 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.147 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 39/31 ECSubWrite(tid=43, reqid=client.4521.0:252, at_version=39'27, trim_to=0'0, roll_forward_to=39'25))", 2024-09-15T03:47:16.147 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:16.147 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 29.917261957000001, 2024-09-15T03:47:16.147 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011286757, 2024-09-15T03:47:16.147 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.147 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.147 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.148 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.148 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.148 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:16.148 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.148 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.148 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.148 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.148 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:16.148 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.148 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.148 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.148 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.148 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178542+0000", 2024-09-15T03:47:16.148 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9400000000000004e-07 2024-09-15T03:47:16.149 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.149 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.149 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.149 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178754+0000", 2024-09-15T03:47:16.149 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021228800000000001 2024-09-15T03:47:16.149 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.149 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.149 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.149 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178765+0000", 2024-09-15T03:47:16.149 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0176e-05 2024-09-15T03:47:16.149 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.149 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.149 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.150 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178769+0000", 2024-09-15T03:47:16.150 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1359999999999999e-06 2024-09-15T03:47:16.150 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.150 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.150 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.150 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178810+0000", 2024-09-15T03:47:16.150 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0825999999999998e-05 2024-09-15T03:47:16.150 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.150 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.150 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.150 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178820+0000", 2024-09-15T03:47:16.150 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0237999999999999e-05 2024-09-15T03:47:16.150 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.151 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.151 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.151 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179033+0000", 2024-09-15T03:47:16.151 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021361500000000001 2024-09-15T03:47:16.151 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.151 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.151 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.151 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179040+0000", 2024-09-15T03:47:16.151 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0509999999999999e-06 2024-09-15T03:47:16.151 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.151 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.151 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.151 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179102+0000", 2024-09-15T03:47:16.151 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1977000000000006e-05 2024-09-15T03:47:16.152 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.152 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.152 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.152 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179114+0000", 2024-09-15T03:47:16.152 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1223000000000001e-05 2024-09-15T03:47:16.152 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.152 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.152 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.152 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.189828+0000", 2024-09-15T03:47:16.152 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010714433000000001 2024-09-15T03:47:16.152 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.152 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.152 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.152 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.153 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.153 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 39/31 ECSubWrite(tid=44, reqid=client.4521.0:252, at_version=0'0, trim_to=0'0, roll_forward_to=39'27))", 2024-09-15T03:47:16.153 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.179948+0000", 2024-09-15T03:47:16.153 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 29.915854857999999, 2024-09-15T03:47:16.153 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010068739, 2024-09-15T03:47:16.153 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.153 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.153 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.153 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.153 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.153 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179948+0000", 2024-09-15T03:47:16.153 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.153 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.154 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.154 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.154 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179948+0000", 2024-09-15T03:47:16.154 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.154 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.154 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.154 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.154 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179949+0000", 2024-09-15T03:47:16.154 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6199999999999998e-07 2024-09-15T03:47:16.154 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.154 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.154 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.154 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179960+0000", 2024-09-15T03:47:16.154 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1216000000000001e-05 2024-09-15T03:47:16.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.155 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.155 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.155 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179970+0000", 2024-09-15T03:47:16.155 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8980000000000008e-06 2024-09-15T03:47:16.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.155 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.155 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.155 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179975+0000", 2024-09-15T03:47:16.155 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5650000000000003e-06 2024-09-15T03:47:16.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.156 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.156 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180004+0000", 2024-09-15T03:47:16.156 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9224999999999999e-05 2024-09-15T03:47:16.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.156 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.156 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180009+0000", 2024-09-15T03:47:16.156 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9250000000000003e-06 2024-09-15T03:47:16.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.156 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.156 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180061+0000", 2024-09-15T03:47:16.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1758000000000001e-05 2024-09-15T03:47:16.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180067+0000", 2024-09-15T03:47:16.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3269999999999997e-06 2024-09-15T03:47:16.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180101+0000", 2024-09-15T03:47:16.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3701000000000001e-05 2024-09-15T03:47:16.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.158 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.158 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.158 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180112+0000", 2024-09-15T03:47:16.158 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1287000000000001e-05 2024-09-15T03:47:16.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.158 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.158 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.158 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.190017+0000", 2024-09-15T03:47:16.158 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0099052749999999998 2024-09-15T03:47:16.158 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.158 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.158 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.159 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.159 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/31 ECSubWrite(tid=90, reqid=client.4521.0:346, at_version=51'12, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:16.159 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.841846+0000", 2024-09-15T03:47:16.159 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 18.253957063000001, 2024-09-15T03:47:16.159 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0088888219999999993, 2024-09-15T03:47:16.159 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.159 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.159 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.159 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.159 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.159 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.841846+0000", 2024-09-15T03:47:16.159 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.159 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.160 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.160 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.841846+0000", 2024-09-15T03:47:16.160 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.160 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.160 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.160 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.160 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.841846+0000", 2024-09-15T03:47:16.160 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8300000000000001e-07 2024-09-15T03:47:16.160 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.160 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.160 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.160 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.841882+0000", 2024-09-15T03:47:16.160 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5574999999999997e-05 2024-09-15T03:47:16.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.161 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.161 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.161 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.841896+0000", 2024-09-15T03:47:16.161 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4100999999999999e-05 2024-09-15T03:47:16.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.161 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.161 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.161 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.841906+0000", 2024-09-15T03:47:16.161 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0356e-05 2024-09-15T03:47:16.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.161 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.161 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.162 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.841983+0000", 2024-09-15T03:47:16.162 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6477000000000007e-05 2024-09-15T03:47:16.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.162 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.162 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.162 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.841996+0000", 2024-09-15T03:47:16.162 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3360000000000001e-05 2024-09-15T03:47:16.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.162 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.162 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.162 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842321+0000", 2024-09-15T03:47:16.162 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00032513000000000003 2024-09-15T03:47:16.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.163 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842474+0000", 2024-09-15T03:47:16.163 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015222099999999999 2024-09-15T03:47:16.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.163 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842529+0000", 2024-09-15T03:47:16.163 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5262e-05 2024-09-15T03:47:16.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.163 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842553+0000", 2024-09-15T03:47:16.164 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3978999999999999e-05 2024-09-15T03:47:16.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.164 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.164 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850735+0000", 2024-09-15T03:47:16.164 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.008182178 2024-09-15T03:47:16.164 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.164 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.164 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.164 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 51/31 ECSubWrite(tid=66, reqid=client.4521.0:349, at_version=51'32, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:16.164 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.842006+0000", 2024-09-15T03:47:16.165 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 18.253797203000001, 2024-09-15T03:47:16.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095108069999999996, 2024-09-15T03:47:16.165 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.165 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.165 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.165 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.165 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842006+0000", 2024-09-15T03:47:16.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.165 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.165 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.165 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842006+0000", 2024-09-15T03:47:16.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.166 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.166 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842006+0000", 2024-09-15T03:47:16.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9000000000000001e-07 2024-09-15T03:47:16.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.166 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.166 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842021+0000", 2024-09-15T03:47:16.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5089000000000001e-05 2024-09-15T03:47:16.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.167 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842030+0000", 2024-09-15T03:47:16.167 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2709999999999992e-06 2024-09-15T03:47:16.167 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.167 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.167 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842034+0000", 2024-09-15T03:47:16.167 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5469999999999998e-06 2024-09-15T03:47:16.167 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.167 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.167 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842099+0000", 2024-09-15T03:47:16.167 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4911000000000002e-05 2024-09-15T03:47:16.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.168 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842111+0000", 2024-09-15T03:47:16.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1681e-05 2024-09-15T03:47:16.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.168 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842399+0000", 2024-09-15T03:47:16.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00028861199999999999 2024-09-15T03:47:16.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.168 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842475+0000", 2024-09-15T03:47:16.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5420999999999995e-05 2024-09-15T03:47:16.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842530+0000", 2024-09-15T03:47:16.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4863e-05 2024-09-15T03:47:16.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.842603+0000", 2024-09-15T03:47:16.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3615000000000006e-05 2024-09-15T03:47:16.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.170 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851517+0000", 2024-09-15T03:47:16.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0089136070000000005 2024-09-15T03:47:16.170 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.170 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.170 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.170 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.170 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 51/31 ECSubWrite(tid=67, reqid=client.4521.0:350, at_version=51'33, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:16.170 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:16.170 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 18.251961915999999, 2024-09-15T03:47:16.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.015569239, 2024-09-15T03:47:16.171 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.171 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.171 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.171 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.171 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:16.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.171 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.172 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:16.172 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.172 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.172 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.172 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843843+0000", 2024-09-15T03:47:16.172 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.195e-06 2024-09-15T03:47:16.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.172 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.172 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.173 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843880+0000", 2024-09-15T03:47:16.173 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7527999999999997e-05 2024-09-15T03:47:16.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.173 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.173 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.173 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843898+0000", 2024-09-15T03:47:16.173 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7696000000000001e-05 2024-09-15T03:47:16.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.173 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.173 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.173 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843907+0000", 2024-09-15T03:47:16.173 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8820000000000007e-06 2024-09-15T03:47:16.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.174 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.174 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.174 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843978+0000", 2024-09-15T03:47:16.174 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1537000000000003e-05 2024-09-15T03:47:16.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.174 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.174 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.174 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843989+0000", 2024-09-15T03:47:16.174 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0720000000000001e-05 2024-09-15T03:47:16.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844102+0000", 2024-09-15T03:47:16.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000113406 2024-09-15T03:47:16.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844131+0000", 2024-09-15T03:47:16.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8762e-05 2024-09-15T03:47:16.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844180+0000", 2024-09-15T03:47:16.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9329999999999997e-05 2024-09-15T03:47:16.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844201+0000", 2024-09-15T03:47:16.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0772999999999999e-05 2024-09-15T03:47:16.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.859411+0000", 2024-09-15T03:47:16.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01520941 2024-09-15T03:47:16.176 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.176 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.177 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.177 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:16.177 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:16.178 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 18.251083005000002, 2024-09-15T03:47:16.178 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012156452, 2024-09-15T03:47:16.178 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.178 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.178 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.178 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.178 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.178 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:16.178 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.179 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.179 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.179 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.179 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:16.179 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.179 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.179 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.179 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.179 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844721+0000", 2024-09-15T03:47:16.179 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1100000000000005e-07 2024-09-15T03:47:16.179 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.179 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.179 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844743+0000", 2024-09-15T03:47:16.180 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1999000000000001e-05 2024-09-15T03:47:16.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.180 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.180 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844758+0000", 2024-09-15T03:47:16.180 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4718999999999999e-05 2024-09-15T03:47:16.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.181 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.181 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844772+0000", 2024-09-15T03:47:16.181 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4637000000000001e-05 2024-09-15T03:47:16.181 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.181 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.181 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844842+0000", 2024-09-15T03:47:16.181 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9596000000000002e-05 2024-09-15T03:47:16.181 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.181 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.181 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844853+0000", 2024-09-15T03:47:16.182 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0912e-05 2024-09-15T03:47:16.182 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.182 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.182 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.182 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844966+0000", 2024-09-15T03:47:16.182 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011276 2024-09-15T03:47:16.182 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.182 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.182 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.182 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844977+0000", 2024-09-15T03:47:16.182 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1676e-05 2024-09-15T03:47:16.182 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.182 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.183 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.183 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847414+0000", 2024-09-15T03:47:16.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024363760000000001 2024-09-15T03:47:16.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.183 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.183 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.183 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847438+0000", 2024-09-15T03:47:16.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3737e-05 2024-09-15T03:47:16.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.183 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.183 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.183 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856877+0000", 2024-09-15T03:47:16.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094392290000000004 2024-09-15T03:47:16.184 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.184 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.184 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.184 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:16.184 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:16.184 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 18.251013610000001, 2024-09-15T03:47:16.184 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012258211, 2024-09-15T03:47:16.184 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.184 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.184 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.185 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:16.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.185 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.185 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:16.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.185 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.185 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:16.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.58e-07 2024-09-15T03:47:16.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.186 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.186 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.186 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844818+0000", 2024-09-15T03:47:16.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.796e-05 2024-09-15T03:47:16.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.186 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.186 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.186 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844829+0000", 2024-09-15T03:47:16.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0895e-05 2024-09-15T03:47:16.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.186 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.186 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.186 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844836+0000", 2024-09-15T03:47:16.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.618e-06 2024-09-15T03:47:16.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845023+0000", 2024-09-15T03:47:16.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018743500000000001 2024-09-15T03:47:16.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845035+0000", 2024-09-15T03:47:16.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1958e-05 2024-09-15T03:47:16.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845147+0000", 2024-09-15T03:47:16.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112281 2024-09-15T03:47:16.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847417+0000", 2024-09-15T03:47:16.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022693969999999998 2024-09-15T03:47:16.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847462+0000", 2024-09-15T03:47:16.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5466999999999997e-05 2024-09-15T03:47:16.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.189 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.189 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847503+0000", 2024-09-15T03:47:16.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1022999999999997e-05 2024-09-15T03:47:16.189 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.189 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.189 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.189 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.857048+0000", 2024-09-15T03:47:16.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095447190000000001 2024-09-15T03:47:16.189 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.189 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.189 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.189 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.189 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.189 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 51/31 ECSubWrite(tid=68, reqid=client.4521.0:351, at_version=51'34, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:16.190 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:16.190 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 18.250595491999999, 2024-09-15T03:47:16.190 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.014544793, 2024-09-15T03:47:16.190 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.190 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.190 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.190 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.190 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:16.190 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.190 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.190 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.190 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:16.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845209+0000", 2024-09-15T03:47:16.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5300000000000003e-07 2024-09-15T03:47:16.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845227+0000", 2024-09-15T03:47:16.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8759000000000001e-05 2024-09-15T03:47:16.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845241+0000", 2024-09-15T03:47:16.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3740000000000001e-05 2024-09-15T03:47:16.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845247+0000", 2024-09-15T03:47:16.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2779999999999999e-06 2024-09-15T03:47:16.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845308+0000", 2024-09-15T03:47:16.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0702000000000003e-05 2024-09-15T03:47:16.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845317+0000", 2024-09-15T03:47:16.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3770000000000002e-06 2024-09-15T03:47:16.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845413+0000", 2024-09-15T03:47:16.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5356e-05 2024-09-15T03:47:16.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.194 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847421+0000", 2024-09-15T03:47:16.194 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020077200000000002 2024-09-15T03:47:16.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.194 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847466+0000", 2024-09-15T03:47:16.194 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5754999999999998e-05 2024-09-15T03:47:16.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.194 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.195 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847572+0000", 2024-09-15T03:47:16.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010594000000000001 2024-09-15T03:47:16.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.195 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.195 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.195 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.859753+0000", 2024-09-15T03:47:16.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012180412999999999 2024-09-15T03:47:16.195 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.195 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.195 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.195 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.195 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 60/28 ECSubWrite(tid=73, reqid=client.4521.0:424, at_version=60'41, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:16.196 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:16.196 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.1953922380000002, 2024-09-15T03:47:16.196 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01137208, 2024-09-15T03:47:16.196 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.196 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.196 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.196 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.196 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:16.196 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.196 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:16.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900412+0000", 2024-09-15T03:47:16.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7599999999999996e-07 2024-09-15T03:47:16.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:16.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7986999999999999e-05 2024-09-15T03:47:16.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900445+0000", 2024-09-15T03:47:16.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.468e-05 2024-09-15T03:47:16.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900451+0000", 2024-09-15T03:47:16.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5799999999999997e-06 2024-09-15T03:47:16.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900507+0000", 2024-09-15T03:47:16.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5855e-05 2024-09-15T03:47:16.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.199 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900530+0000", 2024-09-15T03:47:16.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2929e-05 2024-09-15T03:47:16.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.199 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900640+0000", 2024-09-15T03:47:16.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000110536 2024-09-15T03:47:16.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.199 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900648+0000", 2024-09-15T03:47:16.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9470000000000003e-06 2024-09-15T03:47:16.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.200 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900764+0000", 2024-09-15T03:47:16.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011518099999999999 2024-09-15T03:47:16.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.200 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900803+0000", 2024-09-15T03:47:16.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9941999999999998e-05 2024-09-15T03:47:16.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.201 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.201 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.911783+0000", 2024-09-15T03:47:16.201 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010979667 2024-09-15T03:47:16.201 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.201 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.201 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.201 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.201 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.201 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 60/28 ECSubWrite(tid=74, reqid=client.4521.0:425, at_version=60'42, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:16.201 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.901183+0000", 2024-09-15T03:47:16.201 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.1946202400000008, 2024-09-15T03:47:16.201 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010721191999999999, 2024-09-15T03:47:16.201 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.201 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.202 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901183+0000", 2024-09-15T03:47:16.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901183+0000", 2024-09-15T03:47:16.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901184+0000", 2024-09-15T03:47:16.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6599999999999996e-07 2024-09-15T03:47:16.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901202+0000", 2024-09-15T03:47:16.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8658999999999999e-05 2024-09-15T03:47:16.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901217+0000", 2024-09-15T03:47:16.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4371e-05 2024-09-15T03:47:16.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901223+0000", 2024-09-15T03:47:16.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2489999999999998e-06 2024-09-15T03:47:16.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901274+0000", 2024-09-15T03:47:16.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0602000000000001e-05 2024-09-15T03:47:16.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901284+0000", 2024-09-15T03:47:16.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.995e-06 2024-09-15T03:47:16.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901372+0000", 2024-09-15T03:47:16.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8399000000000005e-05 2024-09-15T03:47:16.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901380+0000", 2024-09-15T03:47:16.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8040000000000002e-06 2024-09-15T03:47:16.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903553+0000", 2024-09-15T03:47:16.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021731580000000001 2024-09-15T03:47:16.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903570+0000", 2024-09-15T03:47:16.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7218999999999999e-05 2024-09-15T03:47:16.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.911904+0000", 2024-09-15T03:47:16.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0083340700000000007 2024-09-15T03:47:16.207 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.207 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.207 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.207 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es5 60/28 ECSubWrite(tid=89, reqid=client.4521.0:446, at_version=60'57, trim_to=0'0, roll_forward_to=60'55))", 2024-09-15T03:47:16.207 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.962207+0000", 2024-09-15T03:47:16.207 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.1335963210000006, 2024-09-15T03:47:16.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010047031, 2024-09-15T03:47:16.207 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.207 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.207 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.208 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.962207+0000", 2024-09-15T03:47:16.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.208 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.962207+0000", 2024-09-15T03:47:16.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.208 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.962208+0000", 2024-09-15T03:47:16.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6199999999999997e-07 2024-09-15T03:47:16.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.963166+0000", 2024-09-15T03:47:16.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00095856700000000001 2024-09-15T03:47:16.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.963182+0000", 2024-09-15T03:47:16.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5865000000000001e-05 2024-09-15T03:47:16.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.963189+0000", 2024-09-15T03:47:16.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9190000000000002e-06 2024-09-15T03:47:16.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.963238+0000", 2024-09-15T03:47:16.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8724999999999999e-05 2024-09-15T03:47:16.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.963245+0000", 2024-09-15T03:47:16.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3189999999999997e-06 2024-09-15T03:47:16.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.963529+0000", 2024-09-15T03:47:16.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00028405200000000002 2024-09-15T03:47:16.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.963537+0000", 2024-09-15T03:47:16.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3250000000000008e-06 2024-09-15T03:47:16.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.212 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.212 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.963687+0000", 2024-09-15T03:47:16.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015002599999999999 2024-09-15T03:47:16.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.212 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.212 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.963707+0000", 2024-09-15T03:47:16.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9241000000000001e-05 2024-09-15T03:47:16.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.213 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.213 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.972254+0000", 2024-09-15T03:47:16.213 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0085473300000000006 2024-09-15T03:47:16.213 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.213 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.213 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.213 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.213 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.fs5 65/64 ECSubWrite(tid=108, reqid=client.4521.0:466, at_version=65'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:16.213 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:11.854597+0000", 2024-09-15T03:47:16.213 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.2412067990000004, 2024-09-15T03:47:16.213 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0097150840000000006, 2024-09-15T03:47:16.213 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.214 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.214 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854597+0000", 2024-09-15T03:47:16.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854597+0000", 2024-09-15T03:47:16.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.215 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854597+0000", 2024-09-15T03:47:16.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9500000000000002e-07 2024-09-15T03:47:16.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.215 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.215 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854659+0000", 2024-09-15T03:47:16.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1938000000000002e-05 2024-09-15T03:47:16.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.215 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.215 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854674+0000", 2024-09-15T03:47:16.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5119999999999999e-05 2024-09-15T03:47:16.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854681+0000", 2024-09-15T03:47:16.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7469999999999999e-06 2024-09-15T03:47:16.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854735+0000", 2024-09-15T03:47:16.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3936e-05 2024-09-15T03:47:16.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854745+0000", 2024-09-15T03:47:16.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0482000000000001e-05 2024-09-15T03:47:16.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.855069+0000", 2024-09-15T03:47:16.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00032310599999999998 2024-09-15T03:47:16.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.855243+0000", 2024-09-15T03:47:16.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000174842 2024-09-15T03:47:16.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.218 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.218 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.855351+0000", 2024-09-15T03:47:16.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000107508 2024-09-15T03:47:16.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.218 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.218 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.855372+0000", 2024-09-15T03:47:16.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0815999999999999e-05 2024-09-15T03:47:16.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.218 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.218 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.864312+0000", 2024-09-15T03:47:16.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0089398940000000003 2024-09-15T03:47:16.218 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.219 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.219 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.219 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.219 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.219 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:16.220 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:47:16.221 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:16.219+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:16.320 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:16.320 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:16.320 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:16.320 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:16.322 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:16.321+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:16.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:47:16.422 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:16.421+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:16.424 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:16.424 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:16.424 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:16.424 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:16.424 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:16.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:47:16.523 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:16.521+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:16.530 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:16.531 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:16.531 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:16.531 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:47:16.531 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.531 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 39/28 ECSubWrite(tid=42, reqid=client.4521.0:255, at_version=39'19, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:47:16.531 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.170724+0000", 2024-09-15T03:47:16.531 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 30.358189003, 2024-09-15T03:47:16.531 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0086604270000000001, 2024-09-15T03:47:16.531 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.531 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.531 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.531 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.531 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170724+0000", 2024-09-15T03:47:16.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.532 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170724+0000", 2024-09-15T03:47:16.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.532 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170725+0000", 2024-09-15T03:47:16.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8999999999999995e-07 2024-09-15T03:47:16.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170742+0000", 2024-09-15T03:47:16.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7181e-05 2024-09-15T03:47:16.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170757+0000", 2024-09-15T03:47:16.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4693e-05 2024-09-15T03:47:16.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170763+0000", 2024-09-15T03:47:16.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4509999999999997e-06 2024-09-15T03:47:16.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.534 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170807+0000", 2024-09-15T03:47:16.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3574e-05 2024-09-15T03:47:16.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.534 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170815+0000", 2024-09-15T03:47:16.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3459999999999994e-06 2024-09-15T03:47:16.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.534 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.535 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170900+0000", 2024-09-15T03:47:16.535 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5436e-05 2024-09-15T03:47:16.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.535 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.535 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.535 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170906+0000", 2024-09-15T03:47:16.535 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9190000000000004e-06 2024-09-15T03:47:16.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.535 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.535 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.535 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173681+0000", 2024-09-15T03:47:16.535 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00277437 2024-09-15T03:47:16.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.536 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173695+0000", 2024-09-15T03:47:16.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4446e-05 2024-09-15T03:47:16.536 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.536 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179384+0000", 2024-09-15T03:47:16.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0056892210000000004 2024-09-15T03:47:16.536 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.537 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.537 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.537 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:47:16.537 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:16.537 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 30.357811168000001, 2024-09-15T03:47:16.537 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012183955999999999, 2024-09-15T03:47:16.537 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.537 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.537 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.537 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.537 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:16.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:16.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:16.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.17e-07 2024-09-15T03:47:16.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.539 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171454+0000", 2024-09-15T03:47:16.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00035226700000000002 2024-09-15T03:47:16.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.539 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171465+0000", 2024-09-15T03:47:16.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0943e-05 2024-09-15T03:47:16.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.539 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171471+0000", 2024-09-15T03:47:16.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7400000000000001e-06 2024-09-15T03:47:16.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171508+0000", 2024-09-15T03:47:16.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7147999999999997e-05 2024-09-15T03:47:16.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171517+0000", 2024-09-15T03:47:16.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2560000000000008e-06 2024-09-15T03:47:16.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171716+0000", 2024-09-15T03:47:16.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019899699999999999 2024-09-15T03:47:16.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.541 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173686+0000", 2024-09-15T03:47:16.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019696829999999999 2024-09-15T03:47:16.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.541 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173777+0000", 2024-09-15T03:47:16.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0637999999999995e-05 2024-09-15T03:47:16.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.542 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173814+0000", 2024-09-15T03:47:16.542 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7499000000000001e-05 2024-09-15T03:47:16.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.542 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.183286+0000", 2024-09-15T03:47:16.542 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094716680000000008 2024-09-15T03:47:16.542 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.542 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.542 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.542 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 39/28 ECSubWrite(tid=43, reqid=client.4521.0:256, at_version=39'20, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:47:16.543 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.171211+0000", 2024-09-15T03:47:16.543 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 30.357702074999999, 2024-09-15T03:47:16.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082752069999999997, 2024-09-15T03:47:16.543 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.543 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.543 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.543 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.543 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.543 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171211+0000", 2024-09-15T03:47:16.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171211+0000", 2024-09-15T03:47:16.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171211+0000", 2024-09-15T03:47:16.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.39e-07 2024-09-15T03:47:16.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.545 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.545 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171228+0000", 2024-09-15T03:47:16.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6254999999999999e-05 2024-09-15T03:47:16.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.545 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.545 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171241+0000", 2024-09-15T03:47:16.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3482999999999999e-05 2024-09-15T03:47:16.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.546 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.546 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171247+0000", 2024-09-15T03:47:16.546 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.088e-06 2024-09-15T03:47:16.546 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.546 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.546 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171292+0000", 2024-09-15T03:47:16.546 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5062000000000003e-05 2024-09-15T03:47:16.546 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.546 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.547 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171302+0000", 2024-09-15T03:47:16.547 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.302e-06 2024-09-15T03:47:16.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.547 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.547 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.547 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171382+0000", 2024-09-15T03:47:16.547 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0717999999999995e-05 2024-09-15T03:47:16.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.547 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.547 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.547 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173686+0000", 2024-09-15T03:47:16.547 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023032320000000001 2024-09-15T03:47:16.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.548 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173776+0000", 2024-09-15T03:47:16.548 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0302000000000004e-05 2024-09-15T03:47:16.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.548 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173805+0000", 2024-09-15T03:47:16.548 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8979000000000002e-05 2024-09-15T03:47:16.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.549 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179486+0000", 2024-09-15T03:47:16.549 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0056813469999999998 2024-09-15T03:47:16.549 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.549 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.549 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.549 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.549 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 49/31 ECSubWrite(tid=87, reqid=client.4521.0:335, at_version=49'21, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:16.549 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.858786+0000", 2024-09-15T03:47:16.549 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 20.670127299000001, 2024-09-15T03:47:16.549 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0085806300000000005, 2024-09-15T03:47:16.549 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.549 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.550 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.550 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.550 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858786+0000", 2024-09-15T03:47:16.550 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.550 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.550 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858786+0000", 2024-09-15T03:47:16.550 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.550 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858786+0000", 2024-09-15T03:47:16.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3099999999999997e-07 2024-09-15T03:47:16.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858809+0000", 2024-09-15T03:47:16.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2331999999999999e-05 2024-09-15T03:47:16.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858819+0000", 2024-09-15T03:47:16.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0468e-05 2024-09-15T03:47:16.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858828+0000", 2024-09-15T03:47:16.552 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0499999999999997e-06 2024-09-15T03:47:16.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858864+0000", 2024-09-15T03:47:16.552 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6072000000000003e-05 2024-09-15T03:47:16.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858870+0000", 2024-09-15T03:47:16.552 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.896e-06 2024-09-15T03:47:16.553 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.553 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.553 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.553 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858938+0000", 2024-09-15T03:47:16.553 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7657000000000006e-05 2024-09-15T03:47:16.553 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.553 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.553 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.553 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858946+0000", 2024-09-15T03:47:16.553 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0609999999999996e-06 2024-09-15T03:47:16.553 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.553 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.553 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861287+0000", 2024-09-15T03:47:16.554 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023413779999999999 2024-09-15T03:47:16.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.554 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.554 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861301+0000", 2024-09-15T03:47:16.554 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3626e-05 2024-09-15T03:47:16.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.554 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.554 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.867366+0000", 2024-09-15T03:47:16.554 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0060654589999999996 2024-09-15T03:47:16.554 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.554 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.555 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.555 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.555 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.555 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 49/31 ECSubWrite(tid=88, reqid=client.4521.0:336, at_version=49'22, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:16.555 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.859239+0000", 2024-09-15T03:47:16.555 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 20.669674326999999, 2024-09-15T03:47:16.555 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082420800000000006, 2024-09-15T03:47:16.555 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.555 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.555 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.555 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.555 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859239+0000", 2024-09-15T03:47:16.556 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.556 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.556 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859239+0000", 2024-09-15T03:47:16.556 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.556 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.556 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859241+0000", 2024-09-15T03:47:16.556 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4880000000000001e-06 2024-09-15T03:47:16.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.556 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859256+0000", 2024-09-15T03:47:16.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5119999999999999e-05 2024-09-15T03:47:16.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859265+0000", 2024-09-15T03:47:16.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1500000000000005e-06 2024-09-15T03:47:16.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859274+0000", 2024-09-15T03:47:16.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2060000000000007e-06 2024-09-15T03:47:16.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859314+0000", 2024-09-15T03:47:16.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0250000000000003e-05 2024-09-15T03:47:16.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859324+0000", 2024-09-15T03:47:16.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0513000000000001e-05 2024-09-15T03:47:16.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859418+0000", 2024-09-15T03:47:16.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3397e-05 2024-09-15T03:47:16.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861290+0000", 2024-09-15T03:47:16.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0018723920000000001 2024-09-15T03:47:16.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861353+0000", 2024-09-15T03:47:16.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2706000000000006e-05 2024-09-15T03:47:16.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.560 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.560 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.861365+0000", 2024-09-15T03:47:16.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2316e-05 2024-09-15T03:47:16.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.560 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.560 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.867481+0000", 2024-09-15T03:47:16.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0061155419999999999 2024-09-15T03:47:16.560 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.561 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.561 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.561 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 51/31 ECSubWrite(tid=67, reqid=client.4521.0:350, at_version=51'33, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:16.561 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.843852+0000", 2024-09-15T03:47:16.561 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 18.685061207, 2024-09-15T03:47:16.561 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010895471, 2024-09-15T03:47:16.561 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.561 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.561 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.562 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.562 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843852+0000", 2024-09-15T03:47:16.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.562 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.562 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843852+0000", 2024-09-15T03:47:16.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.562 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.562 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843853+0000", 2024-09-15T03:47:16.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.359e-06 2024-09-15T03:47:16.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843882+0000", 2024-09-15T03:47:16.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8711999999999999e-05 2024-09-15T03:47:16.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843901+0000", 2024-09-15T03:47:16.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9477e-05 2024-09-15T03:47:16.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843911+0000", 2024-09-15T03:47:16.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2590000000000006e-06 2024-09-15T03:47:16.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843973+0000", 2024-09-15T03:47:16.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2310999999999993e-05 2024-09-15T03:47:16.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843984+0000", 2024-09-15T03:47:16.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.132e-05 2024-09-15T03:47:16.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844108+0000", 2024-09-15T03:47:16.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012384399999999999 2024-09-15T03:47:16.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844139+0000", 2024-09-15T03:47:16.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1148000000000001e-05 2024-09-15T03:47:16.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844185+0000", 2024-09-15T03:47:16.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5388e-05 2024-09-15T03:47:16.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844202+0000", 2024-09-15T03:47:16.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7708999999999999e-05 2024-09-15T03:47:16.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.854747+0000", 2024-09-15T03:47:16.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010544944000000001 2024-09-15T03:47:16.566 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.566 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.567 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.567 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.567 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s7 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:16.567 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844746+0000", 2024-09-15T03:47:16.567 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 18.684166609999998, 2024-09-15T03:47:16.567 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095535919999999996, 2024-09-15T03:47:16.567 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.567 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.567 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.567 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.567 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.567 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844746+0000", 2024-09-15T03:47:16.567 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.568 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844746+0000", 2024-09-15T03:47:16.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.568 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844747+0000", 2024-09-15T03:47:16.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6899999999999996e-07 2024-09-15T03:47:16.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.568 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.569 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844769+0000", 2024-09-15T03:47:16.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1787999999999999e-05 2024-09-15T03:47:16.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.569 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.569 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844785+0000", 2024-09-15T03:47:16.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6212999999999999e-05 2024-09-15T03:47:16.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.569 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.569 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844808+0000", 2024-09-15T03:47:16.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2167e-05 2024-09-15T03:47:16.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844875+0000", 2024-09-15T03:47:16.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7683999999999998e-05 2024-09-15T03:47:16.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844893+0000", 2024-09-15T03:47:16.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7331000000000001e-05 2024-09-15T03:47:16.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844978+0000", 2024-09-15T03:47:16.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5711999999999995e-05 2024-09-15T03:47:16.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844989+0000", 2024-09-15T03:47:16.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1026000000000001e-05 2024-09-15T03:47:16.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847475+0000", 2024-09-15T03:47:16.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024854830000000001 2024-09-15T03:47:16.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847492+0000", 2024-09-15T03:47:16.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7189000000000001e-05 2024-09-15T03:47:16.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.854300+0000", 2024-09-15T03:47:16.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0068080299999999996 2024-09-15T03:47:16.572 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.572 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.572 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.572 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s7 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:16.572 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844831+0000", 2024-09-15T03:47:16.572 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 18.684081566, 2024-09-15T03:47:16.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095609189999999993, 2024-09-15T03:47:16.573 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.573 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.573 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.573 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.573 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.573 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844831+0000", 2024-09-15T03:47:16.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.573 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.573 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.573 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844831+0000", 2024-09-15T03:47:16.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.574 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.574 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.574 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844832+0000", 2024-09-15T03:47:16.574 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1399999999999997e-07 2024-09-15T03:47:16.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.574 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.574 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.574 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844855+0000", 2024-09-15T03:47:16.574 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2904e-05 2024-09-15T03:47:16.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.574 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.574 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844866+0000", 2024-09-15T03:47:16.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0691e-05 2024-09-15T03:47:16.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.575 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844872+0000", 2024-09-15T03:47:16.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.409e-06 2024-09-15T03:47:16.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.575 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845030+0000", 2024-09-15T03:47:16.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015832800000000001 2024-09-15T03:47:16.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845037+0000", 2024-09-15T03:47:16.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1099999999999997e-06 2024-09-15T03:47:16.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845118+0000", 2024-09-15T03:47:16.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0840000000000005e-05 2024-09-15T03:47:16.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847478+0000", 2024-09-15T03:47:16.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023595869999999998 2024-09-15T03:47:16.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.577 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847529+0000", 2024-09-15T03:47:16.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1585000000000002e-05 2024-09-15T03:47:16.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.577 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847544+0000", 2024-09-15T03:47:16.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4365000000000001e-05 2024-09-15T03:47:16.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.578 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.854392+0000", 2024-09-15T03:47:16.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0068485860000000003 2024-09-15T03:47:16.578 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.578 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.578 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.578 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 51/31 ECSubWrite(tid=68, reqid=client.4521.0:351, at_version=51'34, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:16.578 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:16.578 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 18.683697385999999, 2024-09-15T03:47:16.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098406970000000007, 2024-09-15T03:47:16.579 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.579 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.579 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:16.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:16.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.580 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:16.580 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1299999999999999e-07 2024-09-15T03:47:16.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.580 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845232+0000", 2024-09-15T03:47:16.580 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5366000000000002e-05 2024-09-15T03:47:16.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.580 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845242+0000", 2024-09-15T03:47:16.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0403e-05 2024-09-15T03:47:16.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845247+0000", 2024-09-15T03:47:16.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4870000000000004e-06 2024-09-15T03:47:16.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845292+0000", 2024-09-15T03:47:16.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5762e-05 2024-09-15T03:47:16.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845302+0000", 2024-09-15T03:47:16.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5969999999999998e-06 2024-09-15T03:47:16.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.582 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845378+0000", 2024-09-15T03:47:16.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5628000000000002e-05 2024-09-15T03:47:16.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.582 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847479+0000", 2024-09-15T03:47:16.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021009380000000001 2024-09-15T03:47:16.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.583 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.583 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847530+0000", 2024-09-15T03:47:16.583 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1548000000000002e-05 2024-09-15T03:47:16.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.583 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.583 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847544+0000", 2024-09-15T03:47:16.583 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4355000000000001e-05 2024-09-15T03:47:16.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.583 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.584 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.855056+0000", 2024-09-15T03:47:16.584 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075119000000000002 2024-09-15T03:47:16.584 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.584 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.584 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.584 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.584 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 51/31 ECSubWrite(tid=93, reqid=client.4521.0:352, at_version=51'24, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:16.584 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:16.584 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 18.681633452, 2024-09-15T03:47:16.584 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0096136880000000004, 2024-09-15T03:47:16.584 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.584 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.585 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:16.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:16.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.586 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:16.586 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9100000000000003e-07 2024-09-15T03:47:16.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.586 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.586 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847669+0000", 2024-09-15T03:47:16.586 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00038855999999999998 2024-09-15T03:47:16.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.586 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.586 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847682+0000", 2024-09-15T03:47:16.586 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3112000000000001e-05 2024-09-15T03:47:16.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.587 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847687+0000", 2024-09-15T03:47:16.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0880000000000002e-06 2024-09-15T03:47:16.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.587 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.587 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847745+0000", 2024-09-15T03:47:16.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8400999999999999e-05 2024-09-15T03:47:16.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.587 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.587 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847756+0000", 2024-09-15T03:47:16.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0325e-05 2024-09-15T03:47:16.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.588 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.848092+0000", 2024-09-15T03:47:16.588 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033636099999999997 2024-09-15T03:47:16.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.588 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.848101+0000", 2024-09-15T03:47:16.588 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6240000000000008e-06 2024-09-15T03:47:16.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850132+0000", 2024-09-15T03:47:16.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020309239999999999 2024-09-15T03:47:16.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.589 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850174+0000", 2024-09-15T03:47:16.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2630999999999999e-05 2024-09-15T03:47:16.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.589 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856893+0000", 2024-09-15T03:47:16.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0067188710000000004 2024-09-15T03:47:16.589 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.590 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.590 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.590 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.590 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 57/31 ECSubWrite(tid=90, reqid=client.4521.0:410, at_version=57'46, trim_to=0'0, roll_forward_to=56'44))", 2024-09-15T03:47:16.590 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:03.883223+0000", 2024-09-15T03:47:16.590 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 12.645690288000001, 2024-09-15T03:47:16.590 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092188510000000001, 2024-09-15T03:47:16.590 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.590 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.590 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.590 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.883223+0000", 2024-09-15T03:47:16.591 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.883223+0000", 2024-09-15T03:47:16.591 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.883223+0000", 2024-09-15T03:47:16.591 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1600000000000001e-07 2024-09-15T03:47:16.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.592 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.592 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884196+0000", 2024-09-15T03:47:16.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00097307999999999997 2024-09-15T03:47:16.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.592 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.592 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884210+0000", 2024-09-15T03:47:16.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4459e-05 2024-09-15T03:47:16.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.593 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.593 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884217+0000", 2024-09-15T03:47:16.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2870000000000001e-06 2024-09-15T03:47:16.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.593 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.593 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884281+0000", 2024-09-15T03:47:16.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4079999999999996e-05 2024-09-15T03:47:16.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.593 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884293+0000", 2024-09-15T03:47:16.594 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2081e-05 2024-09-15T03:47:16.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884681+0000", 2024-09-15T03:47:16.594 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00038795600000000001 2024-09-15T03:47:16.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884695+0000", 2024-09-15T03:47:16.594 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4497e-05 2024-09-15T03:47:16.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884962+0000", 2024-09-15T03:47:16.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00026705199999999999 2024-09-15T03:47:16.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884982+0000", 2024-09-15T03:47:16.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9211999999999998e-05 2024-09-15T03:47:16.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.892442+0000", 2024-09-15T03:47:16.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0074600309999999998 2024-09-15T03:47:16.596 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.596 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.596 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.596 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.596 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 60/28 ECSubWrite(tid=73, reqid=client.4521.0:424, at_version=60'41, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:16.596 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.900505+0000", 2024-09-15T03:47:16.596 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.6284077890000006, 2024-09-15T03:47:16.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011912449, 2024-09-15T03:47:16.596 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.597 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.597 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.597 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.597 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.597 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900505+0000", 2024-09-15T03:47:16.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.597 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.597 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.597 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900505+0000", 2024-09-15T03:47:16.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.597 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.598 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.598 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900506+0000", 2024-09-15T03:47:16.598 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0800000000000003e-07 2024-09-15T03:47:16.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.598 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.598 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.598 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900527+0000", 2024-09-15T03:47:16.598 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0682999999999998e-05 2024-09-15T03:47:16.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.598 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.598 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.598 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900541+0000", 2024-09-15T03:47:16.598 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4217e-05 2024-09-15T03:47:16.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.599 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.599 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.599 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900547+0000", 2024-09-15T03:47:16.599 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3010000000000003e-06 2024-09-15T03:47:16.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.599 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.599 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.599 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900596+0000", 2024-09-15T03:47:16.599 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8612000000000001e-05 2024-09-15T03:47:16.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.599 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.599 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.600 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900607+0000", 2024-09-15T03:47:16.600 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.063e-05 2024-09-15T03:47:16.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.600 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.600 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.600 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900714+0000", 2024-09-15T03:47:16.600 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000107321 2024-09-15T03:47:16.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.600 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.600 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.600 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900721+0000", 2024-09-15T03:47:16.600 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1709999999999996e-06 2024-09-15T03:47:16.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.601 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.601 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.601 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900753+0000", 2024-09-15T03:47:16.601 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1565e-05 2024-09-15T03:47:16.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.601 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.601 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.601 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900766+0000", 2024-09-15T03:47:16.601 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3406e-05 2024-09-15T03:47:16.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.601 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.601 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.602 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.912418+0000", 2024-09-15T03:47:16.602 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011651635 2024-09-15T03:47:16.602 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.602 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.602 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.602 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.602 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.602 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 60/28 ECSubWrite(tid=74, reqid=client.4521.0:425, at_version=60'42, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:16.602 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.901156+0000", 2024-09-15T03:47:16.602 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.6277570370000003, 2024-09-15T03:47:16.602 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011338225, 2024-09-15T03:47:16.602 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.603 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.603 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.603 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.603 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.603 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901156+0000", 2024-09-15T03:47:16.603 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.603 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.603 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.603 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.603 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901156+0000", 2024-09-15T03:47:16.603 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.603 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.603 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.604 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.604 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901157+0000", 2024-09-15T03:47:16.604 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9400000000000005e-07 2024-09-15T03:47:16.604 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.604 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.604 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.604 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901168+0000", 2024-09-15T03:47:16.604 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1924e-05 2024-09-15T03:47:16.604 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.604 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.604 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.604 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901178+0000", 2024-09-15T03:47:16.605 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7040000000000006e-06 2024-09-15T03:47:16.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.605 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.605 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.605 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901182+0000", 2024-09-15T03:47:16.605 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3080000000000001e-06 2024-09-15T03:47:16.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.605 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.605 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.605 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901226+0000", 2024-09-15T03:47:16.605 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3436999999999998e-05 2024-09-15T03:47:16.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.605 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.606 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.606 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901235+0000", 2024-09-15T03:47:16.606 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9400000000000008e-06 2024-09-15T03:47:16.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.606 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.606 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.606 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901329+0000", 2024-09-15T03:47:16.606 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3686999999999999e-05 2024-09-15T03:47:16.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.606 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.606 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.606 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901337+0000", 2024-09-15T03:47:16.606 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0129999999999993e-06 2024-09-15T03:47:16.607 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.607 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.607 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903268+0000", 2024-09-15T03:47:16.607 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001931545 2024-09-15T03:47:16.607 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.607 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.607 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903279+0000", 2024-09-15T03:47:16.607 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0626e-05 2024-09-15T03:47:16.607 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.607 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.607 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.912494+0000", 2024-09-15T03:47:16.608 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092154469999999999 2024-09-15T03:47:16.608 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.608 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.608 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.608 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.608 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 60/31 ECSubWrite(tid=129, reqid=client.4521.0:422, at_version=60'41, trim_to=0'0, roll_forward_to=60'39))", 2024-09-15T03:47:16.608 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.901373+0000", 2024-09-15T03:47:16.608 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.6275398600000006, 2024-09-15T03:47:16.608 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013085485000000001, 2024-09-15T03:47:16.608 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.608 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.609 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.609 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.609 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.609 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901373+0000", 2024-09-15T03:47:16.609 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.609 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.609 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.609 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901373+0000", 2024-09-15T03:47:16.609 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.609 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.610 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901374+0000", 2024-09-15T03:47:16.610 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5600000000000005e-07 2024-09-15T03:47:16.610 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.610 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.610 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901713+0000", 2024-09-15T03:47:16.610 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033933299999999998 2024-09-15T03:47:16.610 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.610 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.610 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901726+0000", 2024-09-15T03:47:16.611 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2843999999999999e-05 2024-09-15T03:47:16.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.611 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.611 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.611 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901730+0000", 2024-09-15T03:47:16.611 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4660000000000001e-06 2024-09-15T03:47:16.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.611 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.611 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.611 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901773+0000", 2024-09-15T03:47:16.611 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2542e-05 2024-09-15T03:47:16.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.611 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.612 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.612 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901783+0000", 2024-09-15T03:47:16.612 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.041e-05 2024-09-15T03:47:16.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.612 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.612 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906331+0000", 2024-09-15T03:47:16.613 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0045479049999999997 2024-09-15T03:47:16.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.613 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.613 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906340+0000", 2024-09-15T03:47:16.613 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8920000000000006e-06 2024-09-15T03:47:16.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.613 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.613 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906403+0000", 2024-09-15T03:47:16.613 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2793999999999995e-05 2024-09-15T03:47:16.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.614 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.614 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.614 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906413+0000", 2024-09-15T03:47:16.614 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0370999999999999e-05 2024-09-15T03:47:16.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.614 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.614 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.614 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.914459+0000", 2024-09-15T03:47:16.614 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0080452719999999991 2024-09-15T03:47:16.614 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.614 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.614 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.615 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.615 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 60/28 ECSubWrite(tid=75, reqid=client.4521.0:425, at_version=0'0, trim_to=0'0, roll_forward_to=60'42))", 2024-09-15T03:47:16.615 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.903981+0000", 2024-09-15T03:47:16.615 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.6249316470000004, 2024-09-15T03:47:16.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0086389139999999993, 2024-09-15T03:47:16.615 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.615 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.615 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.615 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.615 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.615 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903981+0000", 2024-09-15T03:47:16.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903981+0000", 2024-09-15T03:47:16.616 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903982+0000", 2024-09-15T03:47:16.616 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9699999999999996e-07 2024-09-15T03:47:16.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903990+0000", 2024-09-15T03:47:16.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6660000000000008e-06 2024-09-15T03:47:16.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.904001+0000", 2024-09-15T03:47:16.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1083e-05 2024-09-15T03:47:16.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.904005+0000", 2024-09-15T03:47:16.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1309999999999999e-06 2024-09-15T03:47:16.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.904044+0000", 2024-09-15T03:47:16.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8736999999999999e-05 2024-09-15T03:47:16.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.618 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.904055+0000", 2024-09-15T03:47:16.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1858999999999999e-05 2024-09-15T03:47:16.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.618 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.904181+0000", 2024-09-15T03:47:16.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000125284 2024-09-15T03:47:16.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.619 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.619 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.904187+0000", 2024-09-15T03:47:16.619 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3149999999999997e-06 2024-09-15T03:47:16.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.619 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.619 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906069+0000", 2024-09-15T03:47:16.619 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001882416 2024-09-15T03:47:16.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.619 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.619 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906084+0000", 2024-09-15T03:47:16.619 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4620999999999999e-05 2024-09-15T03:47:16.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.912620+0000", 2024-09-15T03:47:16.620 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0065362049999999998 2024-09-15T03:47:16.620 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.620 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.620 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.620 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.fs7 65/64 ECSubWrite(tid=108, reqid=client.4521.0:466, at_version=65'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:16.620 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:11.853993+0000", 2024-09-15T03:47:16.620 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.6749196719999997, 2024-09-15T03:47:16.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013335351, 2024-09-15T03:47:16.621 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.621 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.621 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.621 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.853993+0000", 2024-09-15T03:47:16.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.621 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.853993+0000", 2024-09-15T03:47:16.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.853994+0000", 2024-09-15T03:47:16.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9999999999999997e-07 2024-09-15T03:47:16.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854199+0000", 2024-09-15T03:47:16.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020490999999999999 2024-09-15T03:47:16.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854210+0000", 2024-09-15T03:47:16.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0813e-05 2024-09-15T03:47:16.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854215+0000", 2024-09-15T03:47:16.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8270000000000003e-06 2024-09-15T03:47:16.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854264+0000", 2024-09-15T03:47:16.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8477999999999999e-05 2024-09-15T03:47:16.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854274+0000", 2024-09-15T03:47:16.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5729999999999997e-06 2024-09-15T03:47:16.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854563+0000", 2024-09-15T03:47:16.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000289208 2024-09-15T03:47:16.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.855280+0000", 2024-09-15T03:47:16.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00071722500000000005 2024-09-15T03:47:16.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.855441+0000", 2024-09-15T03:47:16.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016084199999999999 2024-09-15T03:47:16.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.855456+0000", 2024-09-15T03:47:16.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5123000000000001e-05 2024-09-15T03:47:16.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.867329+0000", 2024-09-15T03:47:16.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011872752 2024-09-15T03:47:16.626 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.626 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.626 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.626 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.fs7 65/64 ECSubWrite(tid=111, reqid=client.4521.0:467, at_version=65'29, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:16.626 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:11.858337+0000", 2024-09-15T03:47:16.626 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.6705763500000002, 2024-09-15T03:47:16.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0091986949999999998, 2024-09-15T03:47:16.626 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.627 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.627 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.627 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.627 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.627 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858337+0000", 2024-09-15T03:47:16.627 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.627 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.627 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.627 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.627 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858337+0000", 2024-09-15T03:47:16.627 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.627 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.627 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.628 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.628 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858337+0000", 2024-09-15T03:47:16.628 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5899999999999995e-07 2024-09-15T03:47:16.628 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.628 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.628 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.628 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858383+0000", 2024-09-15T03:47:16.628 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5358999999999997e-05 2024-09-15T03:47:16.628 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.628 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.628 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.628 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858399+0000", 2024-09-15T03:47:16.628 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6047999999999999e-05 2024-09-15T03:47:16.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.629 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.629 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858408+0000", 2024-09-15T03:47:16.629 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7530000000000007e-06 2024-09-15T03:47:16.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.629 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.629 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858456+0000", 2024-09-15T03:47:16.629 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8081000000000003e-05 2024-09-15T03:47:16.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.629 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858465+0000", 2024-09-15T03:47:16.630 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0769999999999997e-06 2024-09-15T03:47:16.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858579+0000", 2024-09-15T03:47:16.630 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000114318 2024-09-15T03:47:16.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858648+0000", 2024-09-15T03:47:16.630 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9244000000000003e-05 2024-09-15T03:47:16.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.631 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.631 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.631 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858811+0000", 2024-09-15T03:47:16.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016291499999999999 2024-09-15T03:47:16.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.631 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.631 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.631 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858950+0000", 2024-09-15T03:47:16.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000139257 2024-09-15T03:47:16.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.631 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.631 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.631 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.867535+0000", 2024-09-15T03:47:16.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0085848839999999992 2024-09-15T03:47:16.632 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.632 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.632 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.632 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.fs7 65/64 ECSubWrite(tid=112, reqid=client.4521.0:468, at_version=65'30, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:16.632 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:11.858610+0000", 2024-09-15T03:47:16.632 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.6703031490000004, 2024-09-15T03:47:16.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092864230000000002, 2024-09-15T03:47:16.632 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.632 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.632 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.633 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858610+0000", 2024-09-15T03:47:16.633 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.633 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858610+0000", 2024-09-15T03:47:16.633 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.633 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858610+0000", 2024-09-15T03:47:16.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0999999999999999e-07 2024-09-15T03:47:16.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858645+0000", 2024-09-15T03:47:16.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4848000000000002e-05 2024-09-15T03:47:16.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858654+0000", 2024-09-15T03:47:16.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2909999999999996e-06 2024-09-15T03:47:16.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858660+0000", 2024-09-15T03:47:16.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7119999999999997e-06 2024-09-15T03:47:16.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.635 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.635 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858699+0000", 2024-09-15T03:47:16.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8949000000000003e-05 2024-09-15T03:47:16.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.635 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.635 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858711+0000", 2024-09-15T03:47:16.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1669e-05 2024-09-15T03:47:16.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.636 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.636 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.636 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858830+0000", 2024-09-15T03:47:16.636 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011890700000000001 2024-09-15T03:47:16.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.636 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.636 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.636 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858838+0000", 2024-09-15T03:47:16.636 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0479999999999999e-06 2024-09-15T03:47:16.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.636 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.636 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.636 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858903+0000", 2024-09-15T03:47:16.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4788000000000004e-05 2024-09-15T03:47:16.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.637 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.637 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.637 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858975+0000", 2024-09-15T03:47:16.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2336000000000007e-05 2024-09-15T03:47:16.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.637 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.637 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.637 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.867896+0000", 2024-09-15T03:47:16.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0089213650000000005 2024-09-15T03:47:16.637 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.637 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.638 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.638 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.fs7 65/64 ECSubWrite(tid=113, reqid=client.4521.0:469, at_version=65'31, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:16.638 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:11.859692+0000", 2024-09-15T03:47:16.638 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.6692214139999999, 2024-09-15T03:47:16.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0085184000000000006, 2024-09-15T03:47:16.638 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.638 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.638 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.638 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.859692+0000", 2024-09-15T03:47:16.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.859692+0000", 2024-09-15T03:47:16.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.859692+0000", 2024-09-15T03:47:16.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7800000000000001e-07 2024-09-15T03:47:16.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.859714+0000", 2024-09-15T03:47:16.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1667000000000001e-05 2024-09-15T03:47:16.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.859731+0000", 2024-09-15T03:47:16.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6518999999999999e-05 2024-09-15T03:47:16.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.859737+0000", 2024-09-15T03:47:16.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8399999999999997e-06 2024-09-15T03:47:16.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.859782+0000", 2024-09-15T03:47:16.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4631e-05 2024-09-15T03:47:16.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.859798+0000", 2024-09-15T03:47:16.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5630000000000001e-05 2024-09-15T03:47:16.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.859887+0000", 2024-09-15T03:47:16.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9025999999999998e-05 2024-09-15T03:47:16.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.859910+0000", 2024-09-15T03:47:16.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3751999999999999e-05 2024-09-15T03:47:16.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.859948+0000", 2024-09-15T03:47:16.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7286000000000002e-05 2024-09-15T03:47:16.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.859969+0000", 2024-09-15T03:47:16.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0968999999999999e-05 2024-09-15T03:47:16.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.868210+0000", 2024-09-15T03:47:16.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082413020000000007 2024-09-15T03:47:16.643 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.643 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.643 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.644 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.ds6 66/31 ECSubWrite(tid=152, reqid=client.4521.0:509, at_version=66'22, trim_to=0'0, roll_forward_to=63'20))", 2024-09-15T03:47:16.644 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:12.916387+0000", 2024-09-15T03:47:16.644 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 3.6125255210000002, 2024-09-15T03:47:16.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0086842389999999998, 2024-09-15T03:47:16.644 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:16.644 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:16.644 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:16.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:16.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.916387+0000", 2024-09-15T03:47:16.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:16.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.916387+0000", 2024-09-15T03:47:16.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:16.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:16.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.916388+0000", 2024-09-15T03:47:16.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3300000000000001e-07 2024-09-15T03:47:16.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:16.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.916450+0000", 2024-09-15T03:47:16.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1907000000000002e-05 2024-09-15T03:47:16.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:16.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.916461+0000", 2024-09-15T03:47:16.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1260000000000001e-05 2024-09-15T03:47:16.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:16.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.916474+0000", 2024-09-15T03:47:16.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3135e-05 2024-09-15T03:47:16.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.647 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:16.647 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.916547+0000", 2024-09-15T03:47:16.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2627999999999996e-05 2024-09-15T03:47:16.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.647 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:16.647 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.916563+0000", 2024-09-15T03:47:16.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6123000000000001e-05 2024-09-15T03:47:16.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.647 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:16.647 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.916886+0000", 2024-09-15T03:47:16.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00032349100000000002 2024-09-15T03:47:16.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:16.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.916896+0000", 2024-09-15T03:47:16.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6649999999999998e-06 2024-09-15T03:47:16.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:16.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.917092+0000", 2024-09-15T03:47:16.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000196059 2024-09-15T03:47:16.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:16.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.917533+0000", 2024-09-15T03:47:16.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00044085700000000001 2024-09-15T03:47:16.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:16.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:16.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:16.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.925072+0000", 2024-09-15T03:47:16.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075388809999999999 2024-09-15T03:47:16.649 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.649 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.649 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.649 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:16.649 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:16.649 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:16.651 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:47:16.651 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:16.650+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:16.752 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:16.751+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:16.788 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:16.788 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:16.788 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:16.789 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:16.796 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:47:16.852 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:16.851+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:16.861 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T03:47:16.892 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:16.893 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:16.893 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:16.893 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:16.893 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:16.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:47:16.922 INFO:tasks.rados.rados.0.smithi001.stdout:200: done (4 left) 2024-09-15T03:47:16.922 INFO:tasks.rados.rados.0.smithi001.stdout:202: done (3 left) 2024-09-15T03:47:16.922 INFO:tasks.rados.rados.0.smithi001.stdout:203: done (2 left) 2024-09-15T03:47:16.922 INFO:tasks.rados.rados.0.smithi001.stdout:205: done (1 left) 2024-09-15T03:47:16.922 INFO:tasks.rados.rados.0.smithi001.stdout:206: done (0 left) 2024-09-15T03:47:16.922 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 20 current snap is 24 2024-09-15T03:47:16.922 INFO:tasks.rados.rados.0.smithi001.stdout:207: seq_num 99 ranges {1818624=507904,2326528=606208,2932736=557056} 2024-09-15T03:47:16.934 INFO:tasks.rados.rados.0.smithi001.stdout:207: writing smithi00132172-20 from 1818624 to 2326528 tid 1 2024-09-15T03:47:16.937 INFO:tasks.rados.rados.0.smithi001.stdout:207: writing smithi00132172-20 from 2326528 to 2932736 tid 2 2024-09-15T03:47:16.941 INFO:tasks.rados.rados.0.smithi001.stdout:207: writing smithi00132172-20 from 2932736 to 3489792 tid 3 2024-09-15T03:47:16.941 INFO:tasks.rados.rados.0.smithi001.stdout:208: snap_create 2024-09-15T03:47:16.953 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:16.952+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:17.006 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:17.007 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:17.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:17.007 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:47:17.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.007 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "pg_update_log_missing(3.fs7 epoch 69/68 rep_tid 131 entries 69'34 (0'0) error 3:f7b86273:::smithi00132172-37:head by client.4521.0:516 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-09-15T03:47:17.007 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.898080+0000", 2024-09-15T03:47:17.007 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.107036599, 2024-09-15T03:47:17.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.047982374000000001, 2024-09-15T03:47:17.007 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.007 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.007 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.008 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898080+0000", 2024-09-15T03:47:17.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.008 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898080+0000", 2024-09-15T03:47:17.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898080+0000", 2024-09-15T03:47:17.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2399999999999999e-07 2024-09-15T03:47:17.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898086+0000", 2024-09-15T03:47:17.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8499999999999999e-06 2024-09-15T03:47:17.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898102+0000", 2024-09-15T03:47:17.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5233e-05 2024-09-15T03:47:17.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898111+0000", 2024-09-15T03:47:17.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3579999999999992e-06 2024-09-15T03:47:17.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898169+0000", 2024-09-15T03:47:17.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8563999999999997e-05 2024-09-15T03:47:17.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for flush", 2024-09-15T03:47:17.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898205+0000", 2024-09-15T03:47:17.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5833000000000001e-05 2024-09-15T03:47:17.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.945919+0000", 2024-09-15T03:47:17.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.047713658999999999 2024-09-15T03:47:17.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.946063+0000", 2024-09-15T03:47:17.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014365299999999999 2024-09-15T03:47:17.011 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.011 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.012 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.012 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.012 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.012 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(3.2s2 69/68 [PushOp(3:4cab1a9a:::smithi00132172-23:head, version: 32'6, data_included: [0~94208], data_size: 94208, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:4cab1a9a:::smithi00132172-23:head@32'6, size: 376832, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:376832, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T03:47:17.012 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.898323+0000", 2024-09-15T03:47:17.012 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.1067937299999999, 2024-09-15T03:47:17.012 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.027476691000000001, 2024-09-15T03:47:17.012 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.012 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.012 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.012 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.012 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.012 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898323+0000", 2024-09-15T03:47:17.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.013 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.013 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.013 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898323+0000", 2024-09-15T03:47:17.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.013 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.013 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.013 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898323+0000", 2024-09-15T03:47:17.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2800000000000003e-07 2024-09-15T03:47:17.013 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.014 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.014 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.014 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898437+0000", 2024-09-15T03:47:17.014 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000113899 2024-09-15T03:47:17.014 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.014 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.014 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.014 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898458+0000", 2024-09-15T03:47:17.014 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0738e-05 2024-09-15T03:47:17.014 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.014 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.014 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.014 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898469+0000", 2024-09-15T03:47:17.014 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0446e-05 2024-09-15T03:47:17.015 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.015 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.015 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.015 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898665+0000", 2024-09-15T03:47:17.015 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000196665 2024-09-15T03:47:17.015 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.015 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.016 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for flush", 2024-09-15T03:47:17.016 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898674+0000", 2024-09-15T03:47:17.016 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1819999999999995e-06 2024-09-15T03:47:17.016 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.016 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.016 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.016 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.925633+0000", 2024-09-15T03:47:17.016 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.026958947 2024-09-15T03:47:17.016 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.016 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.016 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.016 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.925800+0000", 2024-09-15T03:47:17.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000166486 2024-09-15T03:47:17.017 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.017 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.017 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.017 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.017 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(3.bs5 69/68 [PushOp(3:dbbb10d0:::smithi00132172-15:head, version: 32'4, data_included: [0~61440], data_size: 61440, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:dbbb10d0:::smithi00132172-15:head@32'4, size: 245760, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:245760, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T03:47:17.017 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.899036+0000", 2024-09-15T03:47:17.017 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.106081093, 2024-09-15T03:47:17.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.043768267999999999, 2024-09-15T03:47:17.017 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.017 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.017 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899036+0000", 2024-09-15T03:47:17.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.018 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899036+0000", 2024-09-15T03:47:17.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.018 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899036+0000", 2024-09-15T03:47:17.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8500000000000001e-07 2024-09-15T03:47:17.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.019 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899149+0000", 2024-09-15T03:47:17.019 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112902 2024-09-15T03:47:17.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.020 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.020 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899166+0000", 2024-09-15T03:47:17.020 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6939000000000002e-05 2024-09-15T03:47:17.020 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.020 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.020 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.020 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899176+0000", 2024-09-15T03:47:17.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8300000000000008e-06 2024-09-15T03:47:17.021 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.021 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899248+0000", 2024-09-15T03:47:17.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2231999999999996e-05 2024-09-15T03:47:17.021 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.021 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for flush", 2024-09-15T03:47:17.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899258+0000", 2024-09-15T03:47:17.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7690000000000008e-06 2024-09-15T03:47:17.021 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.022 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.022 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.942666+0000", 2024-09-15T03:47:17.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.043408407000000003 2024-09-15T03:47:17.023 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.942804+0000", 2024-09-15T03:47:17.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013760399999999999 2024-09-15T03:47:17.024 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.024 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.024 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.024 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(3.7s7 e69,68 3:e4b8e19d:::smithi00132172-41:head,32'13)", 2024-09-15T03:47:17.024 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.899363+0000", 2024-09-15T03:47:17.024 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.1057537660000001, 2024-09-15T03:47:17.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.046423364000000002, 2024-09-15T03:47:17.025 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.025 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:17.025 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899363+0000", 2024-09-15T03:47:17.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899363+0000", 2024-09-15T03:47:17.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.026 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.026 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899364+0000", 2024-09-15T03:47:17.026 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5899999999999996e-07 2024-09-15T03:47:17.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.026 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.026 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899369+0000", 2024-09-15T03:47:17.026 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0069999999999997e-06 2024-09-15T03:47:17.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.026 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.026 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899383+0000", 2024-09-15T03:47:17.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3766e-05 2024-09-15T03:47:17.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.027 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.027 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899389+0000", 2024-09-15T03:47:17.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.511e-06 2024-09-15T03:47:17.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.027 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.027 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899439+0000", 2024-09-15T03:47:17.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9910999999999997e-05 2024-09-15T03:47:17.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.028 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for flush", 2024-09-15T03:47:17.028 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899447+0000", 2024-09-15T03:47:17.028 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1070000000000005e-06 2024-09-15T03:47:17.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.028 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.028 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.945757+0000", 2024-09-15T03:47:17.028 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.046310285 2024-09-15T03:47:17.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.028 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:17.028 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.945764+0000", 2024-09-15T03:47:17.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6069999999999996e-06 2024-09-15T03:47:17.029 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.029 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.029 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.945786+0000", 2024-09-15T03:47:17.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2510999999999998e-05 2024-09-15T03:47:17.029 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.029 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.029 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.029 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.029 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(3.6s7 69/68 [PushOp(3:68743546:::smithi00132172-8:head, version: 32'2, data_included: [0~32768], data_size: 32768, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:68743546:::smithi00132172-8:head@32'2, size: 131072, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131072, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T03:47:17.030 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.900159+0000", 2024-09-15T03:47:17.030 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.1049580510000001, 2024-09-15T03:47:17.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.042041885000000001, 2024-09-15T03:47:17.030 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.030 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.030 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900159+0000", 2024-09-15T03:47:17.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.030 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900159+0000", 2024-09-15T03:47:17.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900160+0000", 2024-09-15T03:47:17.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0299999999999998e-07 2024-09-15T03:47:17.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900200+0000", 2024-09-15T03:47:17.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0435000000000002e-05 2024-09-15T03:47:17.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900220+0000", 2024-09-15T03:47:17.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9752e-05 2024-09-15T03:47:17.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900230+0000", 2024-09-15T03:47:17.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0368e-05 2024-09-15T03:47:17.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900298+0000", 2024-09-15T03:47:17.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8399999999999996e-05 2024-09-15T03:47:17.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.033 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for flush", 2024-09-15T03:47:17.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900307+0000", 2024-09-15T03:47:17.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7499999999999992e-06 2024-09-15T03:47:17.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.033 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.942080+0000", 2024-09-15T03:47:17.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.041773235999999998 2024-09-15T03:47:17.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.033 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.034 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.942201+0000", 2024-09-15T03:47:17.034 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120241 2024-09-15T03:47:17.034 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.034 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.034 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.034 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.034 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.034 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(3.3s0 69/68 [PushOp(3:c325496f:::smithi00132172-2:head, version: 32'2, data_included: [0~8192], data_size: 8192, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:c325496f:::smithi00132172-2:head@32'2, size: 32768, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:32768, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T03:47:17.034 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.900588+0000", 2024-09-15T03:47:17.034 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.104529189, 2024-09-15T03:47:17.034 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.046022875999999997, 2024-09-15T03:47:17.034 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.034 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.035 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.035 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.035 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.035 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900588+0000", 2024-09-15T03:47:17.035 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.035 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.035 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.035 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.035 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900588+0000", 2024-09-15T03:47:17.035 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.035 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.035 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.035 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.035 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900588+0000", 2024-09-15T03:47:17.036 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9399999999999999e-07 2024-09-15T03:47:17.036 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.036 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.036 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.036 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900618+0000", 2024-09-15T03:47:17.036 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9828999999999999e-05 2024-09-15T03:47:17.036 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.036 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.036 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.036 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900635+0000", 2024-09-15T03:47:17.036 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7402999999999999e-05 2024-09-15T03:47:17.036 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.037 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900648+0000", 2024-09-15T03:47:17.037 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2435999999999999e-05 2024-09-15T03:47:17.037 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.037 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900706+0000", 2024-09-15T03:47:17.037 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8322000000000002e-05 2024-09-15T03:47:17.037 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for flush", 2024-09-15T03:47:17.038 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900716+0000", 2024-09-15T03:47:17.038 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7030000000000001e-06 2024-09-15T03:47:17.038 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.038 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.038 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.038 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.946484+0000", 2024-09-15T03:47:17.038 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.045768640999999999 2024-09-15T03:47:17.038 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.038 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.038 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.038 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.946611+0000", 2024-09-15T03:47:17.038 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000126348 2024-09-15T03:47:17.038 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.038 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.039 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.039 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.039 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.039 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(3.ds4 69/68 [PushOp(3:b166213b:::smithi00132172-21:19, version: 62'17, data_included: [0~86016], data_size: 86016, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:b166213b:::smithi00132172-21:19@62'17, size: 344064, copy_subset: [], clone_subset: {}, snapset: 22=[]:{19=[18,17,16,12,11,c,3,2],22=[22,21,20]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:344064, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T03:47:17.039 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.900892+0000", 2024-09-15T03:47:17.039 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.1042246769999999, 2024-09-15T03:47:17.039 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.045623043000000002, 2024-09-15T03:47:17.039 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.039 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.039 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.039 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.039 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.039 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900892+0000", 2024-09-15T03:47:17.040 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.040 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.040 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.040 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.040 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900892+0000", 2024-09-15T03:47:17.040 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.040 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.040 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.040 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.040 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900893+0000", 2024-09-15T03:47:17.040 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5599999999999995e-07 2024-09-15T03:47:17.040 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.040 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.041 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.041 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900960+0000", 2024-09-15T03:47:17.041 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7058999999999996e-05 2024-09-15T03:47:17.041 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.041 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.041 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.041 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900978+0000", 2024-09-15T03:47:17.041 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8167000000000001e-05 2024-09-15T03:47:17.041 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.041 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.041 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.041 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900990+0000", 2024-09-15T03:47:17.041 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1625e-05 2024-09-15T03:47:17.042 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.042 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.042 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.042 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.901037+0000", 2024-09-15T03:47:17.042 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7407000000000002e-05 2024-09-15T03:47:17.042 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.042 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.042 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for flush", 2024-09-15T03:47:17.042 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.901043+0000", 2024-09-15T03:47:17.042 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5459999999999999e-06 2024-09-15T03:47:17.042 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.042 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.043 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.043 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.946208+0000", 2024-09-15T03:47:17.043 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.045165023999999998 2024-09-15T03:47:17.043 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.043 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.043 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.043 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.946515+0000", 2024-09-15T03:47:17.043 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000307659 2024-09-15T03:47:17.043 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.043 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.043 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.043 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.043 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.044 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.9s2 69/68 ECSubWrite(tid=77, reqid=client.4521.0:526, at_version=69'27, trim_to=0'0, roll_forward_to=0'0, backfill_or_async_recovery))", 2024-09-15T03:47:17.044 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.951450+0000", 2024-09-15T03:47:17.044 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.0536664650000001, 2024-09-15T03:47:17.044 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001691391, 2024-09-15T03:47:17.044 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.044 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.044 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.044 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.044 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.044 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.951450+0000", 2024-09-15T03:47:17.044 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.044 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.044 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.044 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.045 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.951450+0000", 2024-09-15T03:47:17.045 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.045 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.045 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.045 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.045 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.951451+0000", 2024-09-15T03:47:17.045 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1899999999999999e-07 2024-09-15T03:47:17.045 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.045 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.045 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.045 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.951482+0000", 2024-09-15T03:47:17.045 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1579999999999999e-05 2024-09-15T03:47:17.046 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.046 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.046 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.046 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.951495+0000", 2024-09-15T03:47:17.046 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2655e-05 2024-09-15T03:47:17.046 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.046 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.046 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.046 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.951502+0000", 2024-09-15T03:47:17.046 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1219999999999997e-06 2024-09-15T03:47:17.046 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.046 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.046 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.046 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.951550+0000", 2024-09-15T03:47:17.047 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7951000000000002e-05 2024-09-15T03:47:17.047 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.047 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.047 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:17.047 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.951564+0000", 2024-09-15T03:47:17.047 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3981e-05 2024-09-15T03:47:17.047 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.047 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.047 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:17.047 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.951749+0000", 2024-09-15T03:47:17.047 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018535599999999999 2024-09-15T03:47:17.047 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.047 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.048 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:17.048 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.951758+0000", 2024-09-15T03:47:17.048 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2789999999999998e-06 2024-09-15T03:47:17.048 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.048 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.048 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:17.048 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.951800+0000", 2024-09-15T03:47:17.048 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2456000000000001e-05 2024-09-15T03:47:17.048 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.048 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.048 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:17.048 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.951814+0000", 2024-09-15T03:47:17.048 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4452e-05 2024-09-15T03:47:17.049 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.049 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.049 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.049 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953142+0000", 2024-09-15T03:47:17.049 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00132734 2024-09-15T03:47:17.049 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.049 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.049 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.049 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.049 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.049 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(3.6s7 69/68 [PushOp(3:6f2f5edd:::smithi00132172-26:7, version: 47'13, data_included: [0~106496], data_size: 106496, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:6f2f5edd:::smithi00132172-26:7@47'13, size: 425984, copy_subset: [], clone_subset: {}, snapset: c=[]:{7=[3,2],c=[c]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:425984, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T03:47:17.049 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.956477+0000", 2024-09-15T03:47:17.049 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.0486399150000001, 2024-09-15T03:47:17.050 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000984595, 2024-09-15T03:47:17.050 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.050 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.050 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.050 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.050 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.050 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956477+0000", 2024-09-15T03:47:17.050 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.050 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.050 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.050 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.050 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956477+0000", 2024-09-15T03:47:17.051 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.051 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.051 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.051 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.051 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956478+0000", 2024-09-15T03:47:17.051 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8400000000000003e-07 2024-09-15T03:47:17.051 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.051 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.051 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.051 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956872+0000", 2024-09-15T03:47:17.051 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039428899999999998 2024-09-15T03:47:17.051 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.051 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.052 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.052 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956885+0000", 2024-09-15T03:47:17.052 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3243e-05 2024-09-15T03:47:17.052 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.052 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.052 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.052 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956895+0000", 2024-09-15T03:47:17.052 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7780000000000002e-06 2024-09-15T03:47:17.052 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.052 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.052 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.052 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956983+0000", 2024-09-15T03:47:17.052 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7687000000000003e-05 2024-09-15T03:47:17.053 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.053 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.053 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.053 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957462+0000", 2024-09-15T03:47:17.053 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047871399999999998 2024-09-15T03:47:17.053 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.053 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.053 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.053 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.053 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.053 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(3.7s7 69/68 [PushOp(3:e40d1694:::smithi00132172-50:12, version: 58'32, data_included: [0~208896], data_size: 208896, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:e40d1694:::smithi00132172-50:12@58'32, size: 835584, copy_subset: [], clone_subset: {}, snapset: 1a=[]:{12=[12,11,c,3,2],1a=[18,17,16]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:835584, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T03:47:17.053 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.957048+0000", 2024-09-15T03:47:17.054 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.04806906, 2024-09-15T03:47:17.054 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00099665299999999991, 2024-09-15T03:47:17.054 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.054 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.054 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.054 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.054 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.054 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957048+0000", 2024-09-15T03:47:17.054 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.054 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.054 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.054 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.054 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957048+0000", 2024-09-15T03:47:17.055 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.055 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.055 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.055 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.055 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957048+0000", 2024-09-15T03:47:17.055 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0400000000000002e-07 2024-09-15T03:47:17.055 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.055 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.055 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.055 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957521+0000", 2024-09-15T03:47:17.055 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047267000000000001 2024-09-15T03:47:17.055 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.055 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.056 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.056 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957538+0000", 2024-09-15T03:47:17.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7278e-05 2024-09-15T03:47:17.056 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.056 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.056 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.056 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957548+0000", 2024-09-15T03:47:17.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0179e-05 2024-09-15T03:47:17.056 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.056 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.056 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.056 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957612+0000", 2024-09-15T03:47:17.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4158000000000005e-05 2024-09-15T03:47:17.057 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.057 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.057 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.057 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.958044+0000", 2024-09-15T03:47:17.057 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043196400000000001 2024-09-15T03:47:17.057 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.057 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.057 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.057 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.057 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.057 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(3.bs5 69/68 [PushOp(3:d6f79b7b:::smithi00132172-29:7, version: 47'51, data_included: [0~118784], data_size: 118784, omap_header_size: 0, omap_entries_size: 0, attrset_size: 23, recovery_info: ObjectRecoveryInfo(3:d6f79b7b:::smithi00132172-29:7@47'51, size: 475136, copy_subset: [], clone_subset: {}, snapset: 1a=[]:{7=[3,2],e=[c],1a=[18,17,16,12,11]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:475136, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T03:47:17.057 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.970073+0000", 2024-09-15T03:47:17.057 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.035044224, 2024-09-15T03:47:17.058 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010603380000000001, 2024-09-15T03:47:17.058 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.058 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.058 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.058 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.058 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.058 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.970073+0000", 2024-09-15T03:47:17.058 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.058 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.058 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.058 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.058 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.970073+0000", 2024-09-15T03:47:17.058 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.059 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.059 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.059 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.059 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.970073+0000", 2024-09-15T03:47:17.059 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5099999999999999e-07 2024-09-15T03:47:17.059 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.059 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.059 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.059 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.970372+0000", 2024-09-15T03:47:17.059 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000298521 2024-09-15T03:47:17.059 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.059 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.059 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.059 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.970389+0000", 2024-09-15T03:47:17.060 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7405e-05 2024-09-15T03:47:17.060 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.060 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.060 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.060 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.970400+0000", 2024-09-15T03:47:17.060 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0563e-05 2024-09-15T03:47:17.060 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.060 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.060 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.060 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.970479+0000", 2024-09-15T03:47:17.060 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9237e-05 2024-09-15T03:47:17.060 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.060 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.061 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.061 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.971133+0000", 2024-09-15T03:47:17.061 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00065386100000000005 2024-09-15T03:47:17.061 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.061 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.061 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.061 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.061 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.061 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s7 69/68 ECSubWrite(tid=169, reqid=client.4521.0:527, at_version=69'50, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:17.061 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.974164+0000", 2024-09-15T03:47:17.061 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.030952514, 2024-09-15T03:47:17.061 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0048914409999999998, 2024-09-15T03:47:17.061 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.061 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.062 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.062 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.062 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.974164+0000", 2024-09-15T03:47:17.062 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.062 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.062 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.062 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.974164+0000", 2024-09-15T03:47:17.062 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.063 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.063 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.063 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.063 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.974165+0000", 2024-09-15T03:47:17.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1699999999999999e-07 2024-09-15T03:47:17.063 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.063 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.063 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.063 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.977443+0000", 2024-09-15T03:47:17.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032781960000000001 2024-09-15T03:47:17.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.064 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.064 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.064 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.977456+0000", 2024-09-15T03:47:17.064 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.341e-05 2024-09-15T03:47:17.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.064 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.064 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.064 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.977462+0000", 2024-09-15T03:47:17.064 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9739999999999999e-06 2024-09-15T03:47:17.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.064 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.065 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.065 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.977510+0000", 2024-09-15T03:47:17.065 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7635e-05 2024-09-15T03:47:17.065 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.065 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.065 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:17.065 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.977524+0000", 2024-09-15T03:47:17.065 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3538e-05 2024-09-15T03:47:17.065 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.065 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.065 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:17.065 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.978067+0000", 2024-09-15T03:47:17.065 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00054367700000000001 2024-09-15T03:47:17.066 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.066 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.066 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:17.066 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.978076+0000", 2024-09-15T03:47:17.066 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.524e-06 2024-09-15T03:47:17.066 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.066 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.066 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:17.066 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.978598+0000", 2024-09-15T03:47:17.066 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00052229700000000002 2024-09-15T03:47:17.066 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.066 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.066 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:17.067 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.978648+0000", 2024-09-15T03:47:17.067 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0358999999999997e-05 2024-09-15T03:47:17.067 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.067 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.067 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.067 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.979056+0000", 2024-09-15T03:47:17.067 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00040741399999999999 2024-09-15T03:47:17.067 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.067 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.067 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.067 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.067 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.067 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(3.bs5 69/68 [PushOp(3:d99ddedc:::smithi00132172-44:7, version: 47'53, data_included: [0~184320], data_size: 184320, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:d99ddedc:::smithi00132172-44:7@47'53, size: 737280, copy_subset: [], clone_subset: {}, snapset: 21=[]:{7=[3,2],d=[c],21=[21,20,18,17,16,12,11]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:737280, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T03:47:17.068 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.975728+0000", 2024-09-15T03:47:17.068 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.029388749, 2024-09-15T03:47:17.068 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012971700000000001, 2024-09-15T03:47:17.068 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.068 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.068 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.068 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.068 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.068 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.975728+0000", 2024-09-15T03:47:17.068 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.068 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.068 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.068 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.069 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.975728+0000", 2024-09-15T03:47:17.069 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.069 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.069 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.069 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.069 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.975729+0000", 2024-09-15T03:47:17.069 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3099999999999997e-07 2024-09-15T03:47:17.069 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.069 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.069 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.069 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.976538+0000", 2024-09-15T03:47:17.069 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00080902000000000003 2024-09-15T03:47:17.070 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.070 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.070 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.070 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.976564+0000", 2024-09-15T03:47:17.070 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5939e-05 2024-09-15T03:47:17.070 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.070 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.070 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.070 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.976574+0000", 2024-09-15T03:47:17.070 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0616e-05 2024-09-15T03:47:17.070 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.070 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.070 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.070 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.976665+0000", 2024-09-15T03:47:17.071 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1070999999999996e-05 2024-09-15T03:47:17.071 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.071 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.071 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.071 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.977025+0000", 2024-09-15T03:47:17.071 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00035979300000000002 2024-09-15T03:47:17.071 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.071 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.071 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.071 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.071 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.071 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(3.2s2 69/68 [PushOp(3:48ceb02f:::smithi00132172-5:19, version: 65'26, data_included: [0~176128], data_size: 176128, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:48ceb02f:::smithi00132172-5:19@65'26, size: 704512, copy_subset: [], clone_subset: {}, snapset: 19=[]:{14=[12,11,c,3,2],19=[18,17,16]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:704512, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T03:47:17.071 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.977473+0000", 2024-09-15T03:47:17.072 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.027643584, 2024-09-15T03:47:17.072 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001054419, 2024-09-15T03:47:17.072 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.072 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.072 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.072 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.072 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.072 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.977473+0000", 2024-09-15T03:47:17.072 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.072 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.072 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.072 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.072 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.977473+0000", 2024-09-15T03:47:17.073 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.073 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.073 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.073 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.073 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.977474+0000", 2024-09-15T03:47:17.073 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9800000000000004e-07 2024-09-15T03:47:17.073 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.073 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.073 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.073 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.977849+0000", 2024-09-15T03:47:17.073 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00037519799999999998 2024-09-15T03:47:17.073 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.073 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.073 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.074 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.977868+0000", 2024-09-15T03:47:17.074 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.855e-05 2024-09-15T03:47:17.074 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.074 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.074 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.074 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.977878+0000", 2024-09-15T03:47:17.074 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0291e-05 2024-09-15T03:47:17.074 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.074 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.074 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.074 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.978222+0000", 2024-09-15T03:47:17.074 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034374399999999999 2024-09-15T03:47:17.074 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.075 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.075 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.075 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.978528+0000", 2024-09-15T03:47:17.075 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030613800000000002 2024-09-15T03:47:17.075 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.075 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.075 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.075 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.075 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.075 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(3.2s2 69/68 [PushOp(3:48ceb02f:::smithi00132172-5:head, version: 65'27, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:48ceb02f:::smithi00132172-5:head@65'27, size: 2097152, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T03:47:17.075 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.984730+0000", 2024-09-15T03:47:17.075 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.0203864010000001, 2024-09-15T03:47:17.076 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00086024199999999999, 2024-09-15T03:47:17.076 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.076 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.076 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.076 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.076 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.076 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.984730+0000", 2024-09-15T03:47:17.076 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.076 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.076 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.076 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.076 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.984730+0000", 2024-09-15T03:47:17.076 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.077 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.077 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.077 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.077 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.984731+0000", 2024-09-15T03:47:17.077 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7999999999999995e-07 2024-09-15T03:47:17.077 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.077 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.077 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.077 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.985412+0000", 2024-09-15T03:47:17.077 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00068056499999999995 2024-09-15T03:47:17.077 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.077 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.077 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.078 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.985424+0000", 2024-09-15T03:47:17.078 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2367e-05 2024-09-15T03:47:17.078 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.078 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.078 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.078 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.985432+0000", 2024-09-15T03:47:17.078 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.515e-06 2024-09-15T03:47:17.078 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.078 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.078 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.078 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.985498+0000", 2024-09-15T03:47:17.078 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6686000000000005e-05 2024-09-15T03:47:17.078 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.078 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.079 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.079 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.985591+0000", 2024-09-15T03:47:17.079 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2429000000000006e-05 2024-09-15T03:47:17.079 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.079 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.079 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.079 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.079 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.079 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(3.2s2 69/68 [PushOp(3:48ceb02f:::smithi00132172-5:head, version: 65'27, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:48ceb02f:::smithi00132172-5:head@65'27, size: 2097152, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T03:47:17.079 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.988201+0000", 2024-09-15T03:47:17.079 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.016916009, 2024-09-15T03:47:17.079 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013257729999999999, 2024-09-15T03:47:17.079 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.080 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.080 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.080 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.080 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.080 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.988201+0000", 2024-09-15T03:47:17.080 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.080 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.080 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.080 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.081 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.988201+0000", 2024-09-15T03:47:17.081 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.081 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.081 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.081 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.081 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.988202+0000", 2024-09-15T03:47:17.081 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.75e-07 2024-09-15T03:47:17.081 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.081 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.081 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.081 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.989211+0000", 2024-09-15T03:47:17.081 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001009289 2024-09-15T03:47:17.081 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.082 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.082 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.082 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.989229+0000", 2024-09-15T03:47:17.082 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8306000000000001e-05 2024-09-15T03:47:17.082 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.082 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.082 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.082 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.989240+0000", 2024-09-15T03:47:17.082 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0689e-05 2024-09-15T03:47:17.082 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.082 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.082 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.082 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.989296+0000", 2024-09-15T03:47:17.083 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6591000000000001e-05 2024-09-15T03:47:17.083 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.083 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.083 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.083 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.989527+0000", 2024-09-15T03:47:17.083 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023022299999999999 2024-09-15T03:47:17.083 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.083 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.083 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.083 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.083 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.083 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(3.7s7 69/68 [PushOp(3:e40d1694:::smithi00132172-50:1a, version: 62'44, data_included: [262144~249856], data_size: 249856, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:e40d1694:::smithi00132172-50:1a@62'44, size: 2048000, copy_subset: [], clone_subset: {}, snapset: 1a=[]:{12=[12,11,c,3,2],1a=[18,17,16]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2048000, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T03:47:17.083 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.995149+0000", 2024-09-15T03:47:17.084 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.009968247, 2024-09-15T03:47:17.084 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010218779999999999, 2024-09-15T03:47:17.084 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.084 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.084 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.084 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.084 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.995149+0000", 2024-09-15T03:47:17.084 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.084 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.084 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.084 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.995149+0000", 2024-09-15T03:47:17.085 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.085 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.085 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.085 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.085 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.995149+0000", 2024-09-15T03:47:17.085 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5499999999999997e-07 2024-09-15T03:47:17.085 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.085 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.085 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.085 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.995584+0000", 2024-09-15T03:47:17.085 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043478400000000002 2024-09-15T03:47:17.085 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.085 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.086 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.086 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.995620+0000", 2024-09-15T03:47:17.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5408999999999999e-05 2024-09-15T03:47:17.086 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.086 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.086 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.086 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.995635+0000", 2024-09-15T03:47:17.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5396999999999998e-05 2024-09-15T03:47:17.086 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.086 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.086 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.086 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.995717+0000", 2024-09-15T03:47:17.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2262000000000006e-05 2024-09-15T03:47:17.086 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.087 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.087 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.087 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.996170+0000", 2024-09-15T03:47:17.087 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045327099999999999 2024-09-15T03:47:17.087 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.087 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.087 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.087 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.087 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.087 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(3.bs5 69/68 [PushOp(3:d52faf55:::smithi00132172-40:21, version: 65'71, data_included: [262144~188416], data_size: 188416, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:d52faf55:::smithi00132172-40:21@65'71, size: 1802240, copy_subset: [], clone_subset: {}, snapset: 21=[]:{7=[3,2],21=[21,20,18,17,16,12,11,c]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1802240, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T03:47:17.087 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:16.014167+0000", 2024-09-15T03:47:17.087 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.990949635, 2024-09-15T03:47:17.087 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001026753, 2024-09-15T03:47:17.088 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.088 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.088 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.088 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.088 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.088 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.014167+0000", 2024-09-15T03:47:17.088 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.088 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.088 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.088 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.088 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.014167+0000", 2024-09-15T03:47:17.088 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.088 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.089 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.089 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.089 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.014168+0000", 2024-09-15T03:47:17.089 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2800000000000002e-07 2024-09-15T03:47:17.089 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.089 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.089 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.089 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.014694+0000", 2024-09-15T03:47:17.089 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00052610600000000003 2024-09-15T03:47:17.089 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.089 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.089 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.089 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.014708+0000", 2024-09-15T03:47:17.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4620999999999999e-05 2024-09-15T03:47:17.090 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.090 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.090 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.014720+0000", 2024-09-15T03:47:17.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.182e-05 2024-09-15T03:47:17.090 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.090 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.090 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.014798+0000", 2024-09-15T03:47:17.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7797000000000001e-05 2024-09-15T03:47:17.090 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.091 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.091 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.015194+0000", 2024-09-15T03:47:17.091 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000395981 2024-09-15T03:47:17.091 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.091 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.091 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.091 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.091 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.091 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(3.8s7 69/68 [PushOp(3:1af4eaf3:::smithi00132172-20:1a, version: 61'16, data_included: [0~81920], data_size: 81920, omap_header_size: 0, omap_entries_size: 0, attrset_size: 23, recovery_info: ObjectRecoveryInfo(3:1af4eaf3:::smithi00132172-20:1a@61'16, size: 327680, copy_subset: [], clone_subset: {}, snapset: 1a=[]:{17=[17,16,12,11,c,3,2],1a=[18]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T03:47:17.091 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:16.022538+0000", 2024-09-15T03:47:17.091 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.98257910299999995, 2024-09-15T03:47:17.091 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0011816159999999999, 2024-09-15T03:47:17.092 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.092 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.092 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.092 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.092 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.022538+0000", 2024-09-15T03:47:17.092 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.092 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.092 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.092 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.022538+0000", 2024-09-15T03:47:17.092 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.022538+0000", 2024-09-15T03:47:17.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4000000000000001e-07 2024-09-15T03:47:17.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.022786+0000", 2024-09-15T03:47:17.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024760599999999999 2024-09-15T03:47:17.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.022814+0000", 2024-09-15T03:47:17.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8178999999999999e-05 2024-09-15T03:47:17.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.094 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.022822+0000", 2024-09-15T03:47:17.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7770000000000002e-06 2024-09-15T03:47:17.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.094 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.023062+0000", 2024-09-15T03:47:17.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000239836 2024-09-15T03:47:17.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.095 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.095 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.023719+0000", 2024-09-15T03:47:17.095 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00065757800000000003 2024-09-15T03:47:17.095 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.095 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.095 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.095 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.095 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.095 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(3.8s7 69/68 [PushOp(3:1af4eaf3:::smithi00132172-20:head, version: 62'19, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 24, recovery_info: ObjectRecoveryInfo(3:1af4eaf3:::smithi00132172-20:head@62'19, size: 1818624, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T03:47:17.095 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:16.032881+0000", 2024-09-15T03:47:17.095 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.97223559199999998, 2024-09-15T03:47:17.095 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0011224449999999999, 2024-09-15T03:47:17.095 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.095 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.096 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.096 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.096 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.032881+0000", 2024-09-15T03:47:17.096 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.096 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.096 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.096 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.032881+0000", 2024-09-15T03:47:17.096 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.096 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.096 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.097 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.032882+0000", 2024-09-15T03:47:17.097 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5000000000000002e-07 2024-09-15T03:47:17.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.097 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.097 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.033695+0000", 2024-09-15T03:47:17.097 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00081264399999999995 2024-09-15T03:47:17.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.097 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.097 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.033708+0000", 2024-09-15T03:47:17.097 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3288999999999999e-05 2024-09-15T03:47:17.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.098 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.098 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.033715+0000", 2024-09-15T03:47:17.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7780000000000002e-06 2024-09-15T03:47:17.098 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.098 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.098 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.098 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.033775+0000", 2024-09-15T03:47:17.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0004999999999999e-05 2024-09-15T03:47:17.098 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.098 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.098 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.098 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:16.034004+0000", 2024-09-15T03:47:17.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022907900000000001 2024-09-15T03:47:17.098 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.099 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.099 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.099 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.099 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.099 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:17.100 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:47:17.101 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:17.099+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:17.198 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:17.198 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:17.198 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:17.198 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:17.201 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:17.199+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:17.206 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:47:17.302 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:17.302+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:17.342 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:17.342 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:17.342 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:17.342 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:17.342 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:17.350 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:47:17.403 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:17.403+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:17.450 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:17.450 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:17.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:17.450 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:47:17.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.450 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:185 3.as0 3:5515acd6:::smithi00132172-13:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:47:17.450 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:47:17.450 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 37.343125133000001, 2024-09-15T03:47:17.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01268643, 2024-09-15T03:47:17.450 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.450 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:17.451 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:17.451 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:17.451 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:17.451 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 185 2024-09-15T03:47:17.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.451 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:47:17.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104709+0000", 2024-09-15T03:47:17.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104710+0000", 2024-09-15T03:47:17.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0200000000000003e-07 2024-09-15T03:47:17.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104713+0000", 2024-09-15T03:47:17.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2339999999999999e-06 2024-09-15T03:47:17.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104718+0000", 2024-09-15T03:47:17.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3449999999999996e-06 2024-09-15T03:47:17.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104722+0000", 2024-09-15T03:47:17.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8399999999999997e-06 2024-09-15T03:47:17.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104849+0000", 2024-09-15T03:47:17.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012709599999999999 2024-09-15T03:47:17.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:17.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104856+0000", 2024-09-15T03:47:17.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.652e-06 2024-09-15T03:47:17.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106851+0000", 2024-09-15T03:47:17.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019954840000000001 2024-09-15T03:47:17.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:17.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106935+0000", 2024-09-15T03:47:17.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4095e-05 2024-09-15T03:47:17.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:17.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107453+0000", 2024-09-15T03:47:17.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00051729499999999997 2024-09-15T03:47:17.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:17.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107554+0000", 2024-09-15T03:47:17.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000101251 2024-09-15T03:47:17.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:17.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108104+0000", 2024-09-15T03:47:17.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00054995100000000002 2024-09-15T03:47:17.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:17.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108503+0000", 2024-09-15T03:47:17.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039953500000000002 2024-09-15T03:47:17.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:17.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109405+0000", 2024-09-15T03:47:17.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00090170099999999998 2024-09-15T03:47:17.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:17.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114365+0000", 2024-09-15T03:47:17.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049602120000000003 2024-09-15T03:47:17.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117396+0000", 2024-09-15T03:47:17.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030303370000000001 2024-09-15T03:47:17.457 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.457 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.458 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.458 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:186 3.as0 3:5515acd6:::smithi00132172-13:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:47:17.458 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:47:17.458 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 37.343085342999998, 2024-09-15T03:47:17.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012817485999999999, 2024-09-15T03:47:17.458 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.458 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:17.458 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:17.458 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:17.458 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:17.458 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 186 2024-09-15T03:47:17.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.459 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.459 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:47:17.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.459 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104749+0000", 2024-09-15T03:47:17.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104750+0000", 2024-09-15T03:47:17.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1500000000000004e-07 2024-09-15T03:47:17.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.460 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104754+0000", 2024-09-15T03:47:17.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.946e-06 2024-09-15T03:47:17.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.460 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104769+0000", 2024-09-15T03:47:17.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.507e-05 2024-09-15T03:47:17.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.461 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.461 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104776+0000", 2024-09-15T03:47:17.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7849999999999992e-06 2024-09-15T03:47:17.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.462 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.462 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104888+0000", 2024-09-15T03:47:17.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011164500000000001 2024-09-15T03:47:17.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.462 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.462 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:17.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104894+0000", 2024-09-15T03:47:17.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.942e-06 2024-09-15T03:47:17.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108172+0000", 2024-09-15T03:47:17.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032781939999999999 2024-09-15T03:47:17.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:17.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108257+0000", 2024-09-15T03:47:17.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4488999999999997e-05 2024-09-15T03:47:17.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:17.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108732+0000", 2024-09-15T03:47:17.464 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047547399999999998 2024-09-15T03:47:17.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.464 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.464 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:17.464 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108823+0000", 2024-09-15T03:47:17.464 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1106000000000005e-05 2024-09-15T03:47:17.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.464 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.464 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:17.464 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108961+0000", 2024-09-15T03:47:17.464 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013735099999999999 2024-09-15T03:47:17.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.465 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.465 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:17.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109065+0000", 2024-09-15T03:47:17.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104715 2024-09-15T03:47:17.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.465 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.465 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:17.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.109406+0000", 2024-09-15T03:47:17.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034062300000000002 2024-09-15T03:47:17.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.465 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.465 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:17.466 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114529+0000", 2024-09-15T03:47:17.466 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.005123219 2024-09-15T03:47:17.466 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.466 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.466 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.466 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117566+0000", 2024-09-15T03:47:17.466 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030372120000000001 2024-09-15T03:47:17.466 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.466 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.466 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.466 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.466 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.467 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:188 3.fs0 3:fe9c5513:::smithi00132172-47:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:47:17.467 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:47:17.467 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 37.343002646000002, 2024-09-15T03:47:17.467 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012637061, 2024-09-15T03:47:17.467 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.467 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:17.467 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:17.467 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:17.467 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:17.467 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 188 2024-09-15T03:47:17.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.467 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.468 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:47:17.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.468 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:47:17.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.468 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104832+0000", 2024-09-15T03:47:17.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2799999999999996e-07 2024-09-15T03:47:17.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104835+0000", 2024-09-15T03:47:17.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7190000000000001e-06 2024-09-15T03:47:17.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104842+0000", 2024-09-15T03:47:17.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4580000000000004e-06 2024-09-15T03:47:17.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.470 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.470 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104846+0000", 2024-09-15T03:47:17.470 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1250000000000003e-06 2024-09-15T03:47:17.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.470 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.470 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104884+0000", 2024-09-15T03:47:17.470 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7075e-05 2024-09-15T03:47:17.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.470 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.470 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:17.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104888+0000", 2024-09-15T03:47:17.470 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8520000000000003e-06 2024-09-15T03:47:17.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.471 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.471 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.471 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106860+0000", 2024-09-15T03:47:17.471 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019719440000000002 2024-09-15T03:47:17.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.471 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.471 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:17.471 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.106938+0000", 2024-09-15T03:47:17.471 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7421000000000003e-05 2024-09-15T03:47:17.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.471 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.471 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:17.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107368+0000", 2024-09-15T03:47:17.472 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043042999999999998 2024-09-15T03:47:17.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.472 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:17.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107460+0000", 2024-09-15T03:47:17.472 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1392999999999997e-05 2024-09-15T03:47:17.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.472 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:17.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107473+0000", 2024-09-15T03:47:17.472 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3702e-05 2024-09-15T03:47:17.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.473 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:17.473 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107504+0000", 2024-09-15T03:47:17.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1000000000000001e-05 2024-09-15T03:47:17.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.473 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:17.473 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108376+0000", 2024-09-15T03:47:17.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00087162000000000003 2024-09-15T03:47:17.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.473 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:17.473 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.111657+0000", 2024-09-15T03:47:17.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032815589999999999 2024-09-15T03:47:17.474 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.474 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.474 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.474 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117469+0000", 2024-09-15T03:47:17.474 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058112349999999997 2024-09-15T03:47:17.474 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.474 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.474 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.474 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.474 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.474 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:189 3.fs0 3:fe9c5513:::smithi00132172-47:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-15T03:47:17.474 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:47:17.474 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 37.342983459999999, 2024-09-15T03:47:17.474 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012731009999999999, 2024-09-15T03:47:17.475 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.475 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:17.475 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:17.475 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:17.475 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:17.475 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 189 2024-09-15T03:47:17.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.475 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.475 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.475 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.475 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:47:17.475 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.475 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.475 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:47:17.476 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104851+0000", 2024-09-15T03:47:17.476 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7899999999999999e-07 2024-09-15T03:47:17.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104853+0000", 2024-09-15T03:47:17.476 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.933e-06 2024-09-15T03:47:17.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.477 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.477 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.477 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104859+0000", 2024-09-15T03:47:17.477 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.592e-06 2024-09-15T03:47:17.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.477 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.477 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.477 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104863+0000", 2024-09-15T03:47:17.477 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4000000000000002e-06 2024-09-15T03:47:17.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.477 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.477 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.477 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104910+0000", 2024-09-15T03:47:17.477 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6678000000000003e-05 2024-09-15T03:47:17.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.478 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.478 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:17.478 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.104916+0000", 2024-09-15T03:47:17.478 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1079999999999998e-06 2024-09-15T03:47:17.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.478 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.478 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.478 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107550+0000", 2024-09-15T03:47:17.478 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026344110000000001 2024-09-15T03:47:17.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.478 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.479 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:17.479 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107624+0000", 2024-09-15T03:47:17.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4134000000000005e-05 2024-09-15T03:47:17.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.479 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.479 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:17.479 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.107988+0000", 2024-09-15T03:47:17.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036353600000000002 2024-09-15T03:47:17.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.479 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.479 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:17.479 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108066+0000", 2024-09-15T03:47:17.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8047999999999995e-05 2024-09-15T03:47:17.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.480 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:17.480 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108105+0000", 2024-09-15T03:47:17.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8868000000000001e-05 2024-09-15T03:47:17.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.480 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:17.480 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108504+0000", 2024-09-15T03:47:17.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039916600000000001 2024-09-15T03:47:17.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.480 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:17.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.108955+0000", 2024-09-15T03:47:17.481 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045126200000000002 2024-09-15T03:47:17.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.481 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.481 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:17.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.114394+0000", 2024-09-15T03:47:17.481 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0054382750000000002 2024-09-15T03:47:17.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.481 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.481 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:40.117582+0000", 2024-09-15T03:47:17.481 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031882199999999999 2024-09-15T03:47:17.482 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.482 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.482 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.482 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 38/31 ECSubWrite(tid=36, reqid=client.4521.0:220, at_version=38'23, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:47:17.482 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.148035+0000", 2024-09-15T03:47:17.482 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 32.299799354000001, 2024-09-15T03:47:17.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010074534, 2024-09-15T03:47:17.482 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.482 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.482 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148035+0000", 2024-09-15T03:47:17.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148035+0000", 2024-09-15T03:47:17.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148036+0000", 2024-09-15T03:47:17.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3939999999999999e-06 2024-09-15T03:47:17.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.484 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.484 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148681+0000", 2024-09-15T03:47:17.484 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00064493899999999997 2024-09-15T03:47:17.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.484 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.484 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148695+0000", 2024-09-15T03:47:17.484 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4078e-05 2024-09-15T03:47:17.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.484 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148701+0000", 2024-09-15T03:47:17.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7230000000000001e-06 2024-09-15T03:47:17.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148742+0000", 2024-09-15T03:47:17.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0781000000000001e-05 2024-09-15T03:47:17.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:17.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148749+0000", 2024-09-15T03:47:17.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6109999999999996e-06 2024-09-15T03:47:17.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:17.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149077+0000", 2024-09-15T03:47:17.486 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000327597 2024-09-15T03:47:17.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.486 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:17.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149087+0000", 2024-09-15T03:47:17.486 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4739999999999994e-06 2024-09-15T03:47:17.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.486 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:17.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149230+0000", 2024-09-15T03:47:17.486 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014398300000000001 2024-09-15T03:47:17.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.487 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.487 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:17.487 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.149247+0000", 2024-09-15T03:47:17.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6201e-05 2024-09-15T03:47:17.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.487 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.487 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.487 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.158109+0000", 2024-09-15T03:47:17.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0088627529999999993 2024-09-15T03:47:17.487 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.487 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.487 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.488 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.488 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:231 3.fs0 3:f40d4749:::smithi00132172-35:head [copy-from ver 6] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:17.488 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.155669+0000", 2024-09-15T03:47:17.488 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 32.292165654000001, 2024-09-15T03:47:17.488 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010370569, 2024-09-15T03:47:17.488 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.488 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:17.488 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:17.488 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:17.488 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:17.488 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 231 2024-09-15T03:47:17.488 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.488 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.489 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.489 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155669+0000", 2024-09-15T03:47:17.489 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.489 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.489 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155669+0000", 2024-09-15T03:47:17.489 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.489 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.489 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155671+0000", 2024-09-15T03:47:17.490 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0829999999999998e-06 2024-09-15T03:47:17.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.490 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.490 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155676+0000", 2024-09-15T03:47:17.490 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6210000000000001e-06 2024-09-15T03:47:17.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.490 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.490 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155690+0000", 2024-09-15T03:47:17.490 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4166e-05 2024-09-15T03:47:17.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.490 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155698+0000", 2024-09-15T03:47:17.491 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5560000000000002e-06 2024-09-15T03:47:17.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.491 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.491 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155748+0000", 2024-09-15T03:47:17.491 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9851000000000001e-05 2024-09-15T03:47:17.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.491 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.491 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:17.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155872+0000", 2024-09-15T03:47:17.491 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000124091 2024-09-15T03:47:17.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.491 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.492 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:17.492 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163470+0000", 2024-09-15T03:47:17.492 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075976730000000001 2024-09-15T03:47:17.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.492 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.492 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:17.492 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163801+0000", 2024-09-15T03:47:17.492 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033157999999999999 2024-09-15T03:47:17.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.492 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.492 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:17.492 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163814+0000", 2024-09-15T03:47:17.492 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2659e-05 2024-09-15T03:47:17.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.493 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:17.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163935+0000", 2024-09-15T03:47:17.493 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000121155 2024-09-15T03:47:17.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.493 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:17.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.164131+0000", 2024-09-15T03:47:17.493 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019600499999999999 2024-09-15T03:47:17.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.494 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.494 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:17.494 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.165998+0000", 2024-09-15T03:47:17.494 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001866882 2024-09-15T03:47:17.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.494 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.494 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.494 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.166039+0000", 2024-09-15T03:47:17.494 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1247e-05 2024-09-15T03:47:17.494 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.494 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.494 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.495 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.495 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:232 3.fs0 3:f40d4749:::smithi00132172-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:17.495 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:47:17.495 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 32.292128365000003, 2024-09-15T03:47:17.495 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084205730000000006, 2024-09-15T03:47:17.495 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.495 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:17.495 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:17.495 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:17.495 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:17.495 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 232 2024-09-15T03:47:17.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.495 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.496 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.496 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.496 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:47:17.496 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.496 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.496 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.496 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:47:17.496 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.496 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.496 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.496 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155706+0000", 2024-09-15T03:47:17.497 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7399999999999999e-07 2024-09-15T03:47:17.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.497 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.497 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.497 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155708+0000", 2024-09-15T03:47:17.497 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6500000000000001e-06 2024-09-15T03:47:17.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.497 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.497 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.497 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155713+0000", 2024-09-15T03:47:17.497 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2519999999999999e-06 2024-09-15T03:47:17.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.497 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.498 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.498 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.155716+0000", 2024-09-15T03:47:17.498 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4430000000000002e-06 2024-09-15T03:47:17.498 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.498 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.498 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.499 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156060+0000", 2024-09-15T03:47:17.500 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034440000000000002 2024-09-15T03:47:17.500 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.500 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.500 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:17.500 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.156112+0000", 2024-09-15T03:47:17.500 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1928000000000002e-05 2024-09-15T03:47:17.500 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.500 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.500 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.500 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.163929+0000", 2024-09-15T03:47:17.500 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0078168999999999999 2024-09-15T03:47:17.500 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.501 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.501 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:17.501 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.164027+0000", 2024-09-15T03:47:17.501 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8109999999999994e-05 2024-09-15T03:47:17.501 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.501 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.501 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.501 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.164126+0000", 2024-09-15T03:47:17.501 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9516000000000001e-05 2024-09-15T03:47:17.501 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.501 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.501 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.501 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.502 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.502 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 39/31 ECSubWrite(tid=43, reqid=client.4521.0:252, at_version=39'27, trim_to=0'0, roll_forward_to=39'25))", 2024-09-15T03:47:17.502 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:47:17.502 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.269488672000001, 2024-09-15T03:47:17.502 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082900309999999998, 2024-09-15T03:47:17.502 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.502 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.502 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.502 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.502 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.502 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:47:17.502 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.502 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.503 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:47:17.503 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.503 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.503 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178346+0000", 2024-09-15T03:47:17.503 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6199999999999998e-07 2024-09-15T03:47:17.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.503 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.503 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178642+0000", 2024-09-15T03:47:17.503 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029559399999999999 2024-09-15T03:47:17.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.504 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.504 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.504 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178653+0000", 2024-09-15T03:47:17.504 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0865e-05 2024-09-15T03:47:17.504 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.504 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.504 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.504 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178658+0000", 2024-09-15T03:47:17.504 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3720000000000004e-06 2024-09-15T03:47:17.504 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.504 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.504 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.504 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178695+0000", 2024-09-15T03:47:17.504 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6708999999999997e-05 2024-09-15T03:47:17.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.505 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.505 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:17.505 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178701+0000", 2024-09-15T03:47:17.505 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7989999999999996e-06 2024-09-15T03:47:17.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.505 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.505 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:17.505 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178926+0000", 2024-09-15T03:47:17.505 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022405299999999999 2024-09-15T03:47:17.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.505 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.505 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:17.506 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178935+0000", 2024-09-15T03:47:17.506 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9649999999999997e-06 2024-09-15T03:47:17.506 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.506 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.506 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:17.506 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179230+0000", 2024-09-15T03:47:17.506 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029551000000000002 2024-09-15T03:47:17.506 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.506 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.506 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:17.506 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179247+0000", 2024-09-15T03:47:17.506 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6685999999999999e-05 2024-09-15T03:47:17.506 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.506 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.506 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.507 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.186636+0000", 2024-09-15T03:47:17.507 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0073889159999999997 2024-09-15T03:47:17.507 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.507 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.507 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.507 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.507 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.507 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 49/31 ECSubWrite(tid=87, reqid=client.4521.0:335, at_version=49'21, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:17.507 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.858534+0000", 2024-09-15T03:47:17.507 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.589300536, 2024-09-15T03:47:17.507 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094628430000000003, 2024-09-15T03:47:17.507 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.507 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.507 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.507 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.508 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.508 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858534+0000", 2024-09-15T03:47:17.508 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.508 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.508 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.508 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858534+0000", 2024-09-15T03:47:17.508 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.508 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.508 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.508 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858534+0000", 2024-09-15T03:47:17.508 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2300000000000001e-07 2024-09-15T03:47:17.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.509 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.509 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858553+0000", 2024-09-15T03:47:17.509 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8221000000000001e-05 2024-09-15T03:47:17.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.509 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.509 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858564+0000", 2024-09-15T03:47:17.509 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1471000000000001e-05 2024-09-15T03:47:17.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.509 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.509 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858570+0000", 2024-09-15T03:47:17.509 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7640000000000002e-06 2024-09-15T03:47:17.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.510 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858621+0000", 2024-09-15T03:47:17.510 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0943999999999999e-05 2024-09-15T03:47:17.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.510 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:17.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858626+0000", 2024-09-15T03:47:17.510 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4949999999999999e-06 2024-09-15T03:47:17.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.510 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:17.511 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858715+0000", 2024-09-15T03:47:17.511 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9061000000000007e-05 2024-09-15T03:47:17.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.511 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.511 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:17.511 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858722+0000", 2024-09-15T03:47:17.511 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9689999999999997e-06 2024-09-15T03:47:17.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.511 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.511 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:17.511 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858748+0000", 2024-09-15T03:47:17.511 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6179000000000001e-05 2024-09-15T03:47:17.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.512 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.512 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:17.512 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.858761+0000", 2024-09-15T03:47:17.512 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2159e-05 2024-09-15T03:47:17.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.512 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.512 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.512 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.867997+0000", 2024-09-15T03:47:17.512 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092359569999999995 2024-09-15T03:47:17.512 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.512 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.512 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.513 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.513 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.513 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 49/31 ECSubWrite(tid=88, reqid=client.4521.0:336, at_version=49'22, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:17.513 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:47:17.513 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.588572893999999, 2024-09-15T03:47:17.513 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0088255839999999992, 2024-09-15T03:47:17.513 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.513 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.513 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.513 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.513 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.513 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:47:17.513 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.514 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.514 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.514 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.514 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:47:17.514 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.514 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.514 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.514 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.514 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859262+0000", 2024-09-15T03:47:17.514 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2399999999999999e-07 2024-09-15T03:47:17.514 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.514 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.515 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.515 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859275+0000", 2024-09-15T03:47:17.515 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3395e-05 2024-09-15T03:47:17.515 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.515 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.515 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.515 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859285+0000", 2024-09-15T03:47:17.515 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8959999999999998e-06 2024-09-15T03:47:17.515 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.515 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.515 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.515 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859289+0000", 2024-09-15T03:47:17.515 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2549999999999999e-06 2024-09-15T03:47:17.515 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.516 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.516 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859327+0000", 2024-09-15T03:47:17.516 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7753000000000003e-05 2024-09-15T03:47:17.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.516 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:17.516 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859335+0000", 2024-09-15T03:47:17.516 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3000000000000002e-06 2024-09-15T03:47:17.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.516 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:17.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859391+0000", 2024-09-15T03:47:17.517 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6085000000000003e-05 2024-09-15T03:47:17.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.517 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.517 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:17.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859415+0000", 2024-09-15T03:47:17.517 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3295999999999999e-05 2024-09-15T03:47:17.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.517 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.517 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:17.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859432+0000", 2024-09-15T03:47:17.517 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7087000000000001e-05 2024-09-15T03:47:17.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.517 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.518 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:17.518 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.859442+0000", 2024-09-15T03:47:17.518 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0339e-05 2024-09-15T03:47:17.518 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.518 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.518 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.518 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:55.868087+0000", 2024-09-15T03:47:17.518 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0086447539999999993 2024-09-15T03:47:17.518 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.518 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.518 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.519 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.519 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.519 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s6 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:17.519 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.846035+0000", 2024-09-15T03:47:17.519 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 19.601799667000002, 2024-09-15T03:47:17.519 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010491004, 2024-09-15T03:47:17.519 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.519 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.519 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.519 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.519 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.519 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846035+0000", 2024-09-15T03:47:17.519 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.519 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.519 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.520 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.520 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846035+0000", 2024-09-15T03:47:17.520 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.520 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.520 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846035+0000", 2024-09-15T03:47:17.520 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8599999999999997e-07 2024-09-15T03:47:17.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.520 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.520 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846068+0000", 2024-09-15T03:47:17.520 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2138000000000003e-05 2024-09-15T03:47:17.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.521 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.521 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846081+0000", 2024-09-15T03:47:17.521 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3127e-05 2024-09-15T03:47:17.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.521 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.521 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846092+0000", 2024-09-15T03:47:17.521 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1634000000000001e-05 2024-09-15T03:47:17.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.521 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.521 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846136+0000", 2024-09-15T03:47:17.522 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3999999999999999e-05 2024-09-15T03:47:17.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:17.522 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846148+0000", 2024-09-15T03:47:17.522 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1939e-05 2024-09-15T03:47:17.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:17.522 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846220+0000", 2024-09-15T03:47:17.522 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1248999999999995e-05 2024-09-15T03:47:17.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:17.522 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846231+0000", 2024-09-15T03:47:17.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1379999999999999e-05 2024-09-15T03:47:17.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.523 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:17.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847412+0000", 2024-09-15T03:47:17.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001181407 2024-09-15T03:47:17.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.523 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:17.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847435+0000", 2024-09-15T03:47:17.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3084000000000001e-05 2024-09-15T03:47:17.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.524 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.524 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856526+0000", 2024-09-15T03:47:17.524 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0090901599999999999 2024-09-15T03:47:17.524 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.524 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.524 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.524 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.524 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.524 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s6 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:17.524 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.846101+0000", 2024-09-15T03:47:17.524 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 19.601732858999998, 2024-09-15T03:47:17.524 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010577013999999999, 2024-09-15T03:47:17.524 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.524 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.525 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.525 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.525 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.525 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846101+0000", 2024-09-15T03:47:17.525 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.525 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.525 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.525 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.525 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846101+0000", 2024-09-15T03:47:17.525 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.525 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.525 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.525 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.526 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846102+0000", 2024-09-15T03:47:17.526 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5100000000000001e-07 2024-09-15T03:47:17.526 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.526 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.526 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.526 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846110+0000", 2024-09-15T03:47:17.526 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7069999999999998e-06 2024-09-15T03:47:17.526 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.526 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.526 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.526 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846115+0000", 2024-09-15T03:47:17.526 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2799999999999997e-06 2024-09-15T03:47:17.526 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.526 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.527 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.527 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846117+0000", 2024-09-15T03:47:17.527 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5909999999999998e-06 2024-09-15T03:47:17.527 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.527 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.527 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.527 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846259+0000", 2024-09-15T03:47:17.527 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014169900000000001 2024-09-15T03:47:17.527 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.527 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.527 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:17.527 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846268+0000", 2024-09-15T03:47:17.527 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5159999999999994e-06 2024-09-15T03:47:17.527 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.528 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.528 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:17.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.846335+0000", 2024-09-15T03:47:17.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7936999999999997e-05 2024-09-15T03:47:17.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.528 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.528 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:17.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847415+0000", 2024-09-15T03:47:17.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001079874 2024-09-15T03:47:17.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.528 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.528 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:17.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847453+0000", 2024-09-15T03:47:17.529 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7808999999999997e-05 2024-09-15T03:47:17.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.529 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.529 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:17.529 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847477+0000", 2024-09-15T03:47:17.529 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3609e-05 2024-09-15T03:47:17.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.529 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.529 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.529 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856678+0000", 2024-09-15T03:47:17.529 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.009201641 2024-09-15T03:47:17.529 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.529 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.530 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.530 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.530 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.530 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 51/31 ECSubWrite(tid=93, reqid=client.4521.0:352, at_version=51'24, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:17.530 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.847310+0000", 2024-09-15T03:47:17.530 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 19.600524553, 2024-09-15T03:47:17.530 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012670010000000001, 2024-09-15T03:47:17.530 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.530 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.530 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.530 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.530 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.530 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847310+0000", 2024-09-15T03:47:17.530 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.531 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.531 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.531 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847310+0000", 2024-09-15T03:47:17.531 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.531 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.531 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.531 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847310+0000", 2024-09-15T03:47:17.531 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8899999999999999e-07 2024-09-15T03:47:17.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.531 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.531 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.531 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847750+0000", 2024-09-15T03:47:17.531 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000439238 2024-09-15T03:47:17.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.532 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847767+0000", 2024-09-15T03:47:17.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7362000000000001e-05 2024-09-15T03:47:17.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.532 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847774+0000", 2024-09-15T03:47:17.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0169999999999999e-06 2024-09-15T03:47:17.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.532 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847831+0000", 2024-09-15T03:47:17.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7207999999999998e-05 2024-09-15T03:47:17.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:17.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847841+0000", 2024-09-15T03:47:17.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7829999999999993e-06 2024-09-15T03:47:17.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:17.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.848145+0000", 2024-09-15T03:47:17.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030431300000000001 2024-09-15T03:47:17.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:17.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.848179+0000", 2024-09-15T03:47:17.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3423999999999997e-05 2024-09-15T03:47:17.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.534 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:17.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850141+0000", 2024-09-15T03:47:17.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019623380000000001 2024-09-15T03:47:17.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.534 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:17.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850164+0000", 2024-09-15T03:47:17.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2969e-05 2024-09-15T03:47:17.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.535 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.535 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.535 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.859980+0000", 2024-09-15T03:47:17.535 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098156690000000008 2024-09-15T03:47:17.535 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.535 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.535 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.535 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.535 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 51/31 ECSubWrite(tid=94, reqid=client.4521.0:353, at_version=51'25, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:17.535 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.850957+0000", 2024-09-15T03:47:17.535 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 19.596876764000001, 2024-09-15T03:47:17.535 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092043439999999997, 2024-09-15T03:47:17.536 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.536 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.536 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.536 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850957+0000", 2024-09-15T03:47:17.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.536 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.536 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850957+0000", 2024-09-15T03:47:17.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.537 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.537 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850958+0000", 2024-09-15T03:47:17.537 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9500000000000002e-07 2024-09-15T03:47:17.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.537 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.537 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851297+0000", 2024-09-15T03:47:17.537 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033833300000000001 2024-09-15T03:47:17.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.537 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851311+0000", 2024-09-15T03:47:17.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4144999999999999e-05 2024-09-15T03:47:17.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851318+0000", 2024-09-15T03:47:17.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8700000000000003e-06 2024-09-15T03:47:17.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851390+0000", 2024-09-15T03:47:17.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.203e-05 2024-09-15T03:47:17.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.539 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:17.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851402+0000", 2024-09-15T03:47:17.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2146e-05 2024-09-15T03:47:17.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.539 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:17.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851556+0000", 2024-09-15T03:47:17.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015461199999999999 2024-09-15T03:47:17.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:17.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851570+0000", 2024-09-15T03:47:17.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4106e-05 2024-09-15T03:47:17.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:17.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853231+0000", 2024-09-15T03:47:17.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001660559 2024-09-15T03:47:17.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:17.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853246+0000", 2024-09-15T03:47:17.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5457000000000001e-05 2024-09-15T03:47:17.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.541 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.860162+0000", 2024-09-15T03:47:17.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069153909999999999 2024-09-15T03:47:17.541 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.541 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.541 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.542 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 51/31 ECSubWrite(tid=95, reqid=client.4521.0:354, at_version=51'26, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:17.542 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.851558+0000", 2024-09-15T03:47:17.542 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 19.596276626000002, 2024-09-15T03:47:17.542 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0087097059999999994, 2024-09-15T03:47:17.542 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.542 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.542 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.542 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851558+0000", 2024-09-15T03:47:17.542 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.543 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.543 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.543 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851558+0000", 2024-09-15T03:47:17.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.543 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.543 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.543 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.543 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851559+0000", 2024-09-15T03:47:17.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7599999999999996e-07 2024-09-15T03:47:17.543 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.543 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.543 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851599+0000", 2024-09-15T03:47:17.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0472000000000001e-05 2024-09-15T03:47:17.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851614+0000", 2024-09-15T03:47:17.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4640999999999999e-05 2024-09-15T03:47:17.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851620+0000", 2024-09-15T03:47:17.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3430000000000001e-06 2024-09-15T03:47:17.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.545 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.545 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851680+0000", 2024-09-15T03:47:17.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9648000000000002e-05 2024-09-15T03:47:17.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.545 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.545 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:17.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851691+0000", 2024-09-15T03:47:17.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1012999999999999e-05 2024-09-15T03:47:17.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.545 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.545 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:17.546 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.851803+0000", 2024-09-15T03:47:17.546 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112089 2024-09-15T03:47:17.546 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.546 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:17.546 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853234+0000", 2024-09-15T03:47:17.546 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0014310810000000001 2024-09-15T03:47:17.546 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.546 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:17.546 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853304+0000", 2024-09-15T03:47:17.546 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0656999999999998e-05 2024-09-15T03:47:17.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.547 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.547 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:17.547 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.853317+0000", 2024-09-15T03:47:17.547 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2186e-05 2024-09-15T03:47:17.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.547 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.547 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.547 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.860267+0000", 2024-09-15T03:47:17.547 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069506999999999998 2024-09-15T03:47:17.547 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.547 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.547 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.548 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:424 3.as0 3:5515acd6:::smithi00132172-13:head [rollback 17] snapc 1a=[1a,18,17,16,15,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:47:17.548 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.898858+0000", 2024-09-15T03:47:17.548 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.548976424999999, 2024-09-15T03:47:17.548 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013029552999999999, 2024-09-15T03:47:17.548 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.548 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:17.548 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:17.548 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:17.548 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:17.548 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 424 2024-09-15T03:47:17.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.549 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.549 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.549 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.549 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898858+0000", 2024-09-15T03:47:17.549 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.549 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.549 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.549 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898858+0000", 2024-09-15T03:47:17.549 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.549 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.549 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.549 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898858+0000", 2024-09-15T03:47:17.550 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2300000000000002e-07 2024-09-15T03:47:17.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.550 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.550 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898860+0000", 2024-09-15T03:47:17.550 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.623e-06 2024-09-15T03:47:17.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.550 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.550 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898863+0000", 2024-09-15T03:47:17.550 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5870000000000001e-06 2024-09-15T03:47:17.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898865+0000", 2024-09-15T03:47:17.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.075e-06 2024-09-15T03:47:17.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.899786+0000", 2024-09-15T03:47:17.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00092033499999999999 2024-09-15T03:47:17.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:17.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.899886+0000", 2024-09-15T03:47:17.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000100516 2024-09-15T03:47:17.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:17.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900376+0000", 2024-09-15T03:47:17.552 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00048932700000000001 2024-09-15T03:47:17.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:17.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900486+0000", 2024-09-15T03:47:17.552 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011024 2024-09-15T03:47:17.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:17.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900492+0000", 2024-09-15T03:47:17.553 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.908e-06 2024-09-15T03:47:17.553 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.553 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.553 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:17.553 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900724+0000", 2024-09-15T03:47:17.553 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000232382 2024-09-15T03:47:17.553 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.553 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.553 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:17.553 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901279+0000", 2024-09-15T03:47:17.553 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00055503599999999996 2024-09-15T03:47:17.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.554 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.554 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:17.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.902348+0000", 2024-09-15T03:47:17.554 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010692130000000001 2024-09-15T03:47:17.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.554 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.554 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.911887+0000", 2024-09-15T03:47:17.554 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.009538988 2024-09-15T03:47:17.554 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.554 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.555 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.555 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.555 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.555 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:425 3.as0 3:5515acd6:::smithi00132172-13:head [append 0~0] snapc 1a=[1a,18,17,16,15,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:47:17.555 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.898884+0000", 2024-09-15T03:47:17.555 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.548950035000001, 2024-09-15T03:47:17.555 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013107466999999999, 2024-09-15T03:47:17.555 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.555 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:17.555 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:17.555 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:17.555 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:17.555 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 425 2024-09-15T03:47:17.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.556 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.556 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.556 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898884+0000", 2024-09-15T03:47:17.556 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.556 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.556 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898884+0000", 2024-09-15T03:47:17.556 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898884+0000", 2024-09-15T03:47:17.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4499999999999998e-07 2024-09-15T03:47:17.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898886+0000", 2024-09-15T03:47:17.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0140000000000001e-06 2024-09-15T03:47:17.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898890+0000", 2024-09-15T03:47:17.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9179999999999996e-06 2024-09-15T03:47:17.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.898893+0000", 2024-09-15T03:47:17.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2900000000000001e-06 2024-09-15T03:47:17.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900560+0000", 2024-09-15T03:47:17.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00166749 2024-09-15T03:47:17.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:17.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900717+0000", 2024-09-15T03:47:17.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000157174 2024-09-15T03:47:17.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:17.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901127+0000", 2024-09-15T03:47:17.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00040965100000000003 2024-09-15T03:47:17.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:17.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901239+0000", 2024-09-15T03:47:17.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112324 2024-09-15T03:47:17.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.560 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.560 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:17.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901246+0000", 2024-09-15T03:47:17.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8730000000000001e-06 2024-09-15T03:47:17.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.560 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.560 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:17.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903551+0000", 2024-09-15T03:47:17.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023044459999999999 2024-09-15T03:47:17.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.560 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.560 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:17.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903565+0000", 2024-09-15T03:47:17.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4066999999999999e-05 2024-09-15T03:47:17.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.561 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:17.561 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903748+0000", 2024-09-15T03:47:17.561 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018303899999999999 2024-09-15T03:47:17.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.561 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.561 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.911992+0000", 2024-09-15T03:47:17.561 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082439360000000003 2024-09-15T03:47:17.561 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.561 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.561 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.562 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 60/31 ECSubWrite(tid=129, reqid=client.4521.0:422, at_version=60'41, trim_to=0'0, roll_forward_to=60'39))", 2024-09-15T03:47:17.562 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.901153+0000", 2024-09-15T03:47:17.562 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.546680809, 2024-09-15T03:47:17.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017516662999999998, 2024-09-15T03:47:17.562 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.562 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.562 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.562 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.562 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901153+0000", 2024-09-15T03:47:17.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901153+0000", 2024-09-15T03:47:17.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901154+0000", 2024-09-15T03:47:17.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3000000000000001e-07 2024-09-15T03:47:17.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901593+0000", 2024-09-15T03:47:17.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043868200000000003 2024-09-15T03:47:17.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901605+0000", 2024-09-15T03:47:17.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1987000000000001e-05 2024-09-15T03:47:17.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901611+0000", 2024-09-15T03:47:17.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1720000000000004e-06 2024-09-15T03:47:17.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901648+0000", 2024-09-15T03:47:17.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7734999999999997e-05 2024-09-15T03:47:17.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:17.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901657+0000", 2024-09-15T03:47:17.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.242e-06 2024-09-15T03:47:17.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:17.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903437+0000", 2024-09-15T03:47:17.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001780404 2024-09-15T03:47:17.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:17.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903553+0000", 2024-09-15T03:47:17.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000115927 2024-09-15T03:47:17.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:17.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906054+0000", 2024-09-15T03:47:17.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002500604 2024-09-15T03:47:17.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:17.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906067+0000", 2024-09-15T03:47:17.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3699e-05 2024-09-15T03:47:17.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.567 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.567 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.567 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.918670+0000", 2024-09-15T03:47:17.567 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012602781 2024-09-15T03:47:17.567 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.567 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.567 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.567 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.567 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s6 60/31 ECSubWrite(tid=132, reqid=client.4521.0:422, at_version=0'0, trim_to=0'0, roll_forward_to=60'41))", 2024-09-15T03:47:17.567 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.906920+0000", 2024-09-15T03:47:17.567 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.540913983999999, 2024-09-15T03:47:17.567 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011799888, 2024-09-15T03:47:17.568 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.568 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.568 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.568 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906920+0000", 2024-09-15T03:47:17.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.568 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906920+0000", 2024-09-15T03:47:17.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.569 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.569 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906921+0000", 2024-09-15T03:47:17.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4600000000000005e-07 2024-09-15T03:47:17.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.569 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.569 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906929+0000", 2024-09-15T03:47:17.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7840000000000004e-06 2024-09-15T03:47:17.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.569 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906938+0000", 2024-09-15T03:47:17.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4299999999999995e-06 2024-09-15T03:47:17.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906943+0000", 2024-09-15T03:47:17.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1209999999999998e-06 2024-09-15T03:47:17.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906988+0000", 2024-09-15T03:47:17.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5086999999999997e-05 2024-09-15T03:47:17.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:17.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906998+0000", 2024-09-15T03:47:17.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5610000000000004e-06 2024-09-15T03:47:17.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:17.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.907078+0000", 2024-09-15T03:47:17.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9989999999999998e-05 2024-09-15T03:47:17.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:17.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.907087+0000", 2024-09-15T03:47:17.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4110000000000002e-06 2024-09-15T03:47:17.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:17.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.908905+0000", 2024-09-15T03:47:17.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0018175229999999999 2024-09-15T03:47:17.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:17.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.908914+0000", 2024-09-15T03:47:17.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5829999999999996e-06 2024-09-15T03:47:17.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.573 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.918720+0000", 2024-09-15T03:47:17.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098058520000000003 2024-09-15T03:47:17.573 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.573 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.573 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.573 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.573 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs6 65/64 ECSubWrite(tid=110, reqid=client.4521.0:474, at_version=65'72, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:17.573 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:11.849490+0000", 2024-09-15T03:47:17.573 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.598344333, 2024-09-15T03:47:17.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0099941470000000001, 2024-09-15T03:47:17.573 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:17.573 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:17.574 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:17.574 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.574 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:17.574 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.849490+0000", 2024-09-15T03:47:17.574 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.574 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.574 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:17.574 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.849490+0000", 2024-09-15T03:47:17.574 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:17.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.574 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.574 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:17.574 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.849490+0000", 2024-09-15T03:47:17.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8099999999999998e-07 2024-09-15T03:47:17.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.575 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:17.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.849594+0000", 2024-09-15T03:47:17.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000103806 2024-09-15T03:47:17.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.575 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:17.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.849602+0000", 2024-09-15T03:47:17.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6159999999999996e-06 2024-09-15T03:47:17.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:17.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.849607+0000", 2024-09-15T03:47:17.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.463e-06 2024-09-15T03:47:17.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:17.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.849732+0000", 2024-09-15T03:47:17.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012485900000000001 2024-09-15T03:47:17.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:17.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.849740+0000", 2024-09-15T03:47:17.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3180000000000007e-06 2024-09-15T03:47:17.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.577 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:17.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.850040+0000", 2024-09-15T03:47:17.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000299141 2024-09-15T03:47:17.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.577 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:17.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.850085+0000", 2024-09-15T03:47:17.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.566e-05 2024-09-15T03:47:17.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.577 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:17.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.850683+0000", 2024-09-15T03:47:17.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00059757400000000002 2024-09-15T03:47:17.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.578 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:17.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.850729+0000", 2024-09-15T03:47:17.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6548999999999998e-05 2024-09-15T03:47:17.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:17.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:17.578 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:17.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.859484+0000", 2024-09-15T03:47:17.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0087547800000000002 2024-09-15T03:47:17.578 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.578 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.578 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.579 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:17.579 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:17.579 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:17.580 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:47:17.581 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:17.579+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:17.681 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:17.680+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:17.686 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:17.686 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:17.686 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-15T03:47:17.686 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:17.694 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:47:17.725 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:47:17.781 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:17.779+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:17.837 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:17.837 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:17.837 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-15T03:47:17.838 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-15T03:47:17.838 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:17.847 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:47:17.854 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:17.854 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:17.870 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:17.871 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:17.882 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:17.881+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:17.885 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:17.885 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:17.900 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:17.900 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:17.916 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:17.916 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:17.921 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-15T03:47:17.922 INFO:tasks.rados.rados.0.smithi001.stdout:208: done (1 left) 2024-09-15T03:47:17.922 INFO:tasks.rados.rados.0.smithi001.stdout:209: snap_create 2024-09-15T03:47:17.942 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:17.943 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:17.950 INFO:tasks.rados.rados.0.smithi001.stdout:207: finishing write tid 1 to smithi00132172-20 2024-09-15T03:47:17.953 INFO:tasks.rados.rados.0.smithi001.stdout:207: finishing write tid 2 to smithi00132172-20 2024-09-15T03:47:17.955 INFO:tasks.rados.rados.0.smithi001.stdout:207: finishing write tid 3 to smithi00132172-20 2024-09-15T03:47:17.956 INFO:tasks.rados.rados.0.smithi001.stdout:207: finishing write tid 4 to smithi00132172-20 2024-09-15T03:47:17.961 INFO:tasks.rados.rados.0.smithi001.stdout:207: finishing write tid 5 to smithi00132172-20 2024-09-15T03:47:17.961 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 20 v 24 (ObjNum 99 snap 24 seq_num 99) dirty exists 2024-09-15T03:47:17.961 INFO:tasks.rados.rados.0.smithi001.stdout:207: left oid 20 (ObjNum 99 snap 24 seq_num 99) 2024-09-15T03:47:17.965 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:17.965 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:17.970 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:17.970 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:17.979 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:47:17.981 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:17.982 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:17.983 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:17.982+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:17.996 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:17.996 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:18.011 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:18.011 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:18.025 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:47:18.025 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:47:18.040 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:18.040 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:18.084 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:18.084 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:18.084 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:18.083+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:18.092 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:47:18.185 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:18.183+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:18.232 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:18.233 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:18.241 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:47:18.286 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:18.285+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:18.341 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:18.341 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:18.349 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:47:18.394 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:18.393+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:18.450 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:18.450 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:18.458 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:47:18.500 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:18.499+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:18.589 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:18.588+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:18.600 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:18.600 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:18.608 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:47:18.690 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:18.689+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:18.708 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:18.709 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:18.717 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:47:18.791 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:18.790+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:18.818 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:18.818 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:18.827 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:47:18.891 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:18.891+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:18.924 INFO:tasks.rados.rados.0.smithi001.stdout:207: done (1 left) 2024-09-15T03:47:18.925 INFO:tasks.rados.rados.0.smithi001.stdout:209: done (0 left) 2024-09-15T03:47:18.925 INFO:tasks.rados.rados.0.smithi001.stdout:210: snap_create 2024-09-15T03:47:18.929 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-15T03:47:18.929 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T03:47:18.929 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-15T03:47:18.929 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-15T03:47:18.930 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:18.930 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:18.938 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:47:18.992 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:18.992+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:19.084 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:19.084 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:19.093 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:47:19.094 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:19.092+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:19.193 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:19.193+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:19.208 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:19.208 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:19.210 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:47:19.210 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:19.210 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 72, 2024-09-15T03:47:19.210 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "f27b540e-e5e3-4e95-afe8-c95ee66b0a62", 2024-09-15T03:47:19.210 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-09-15T03:46:09.843094+0000", 2024-09-15T03:47:19.210 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-09-15T03:47:18.920644+0000", 2024-09-15T03:47:19.211 INFO:teuthology.orchestra.run.smithi001.stdout: "last_up_change": "2024-09-15T03:46:15.871985+0000", 2024-09-15T03:47:19.211 INFO:teuthology.orchestra.run.smithi001.stdout: "last_in_change": "2024-09-15T03:47:13.897657+0000", 2024-09-15T03:47:19.211 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-15T03:47:19.211 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_num": 5799936, 2024-09-15T03:47:19.211 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_set": [ 2024-09-15T03:47:19.211 INFO:teuthology.orchestra.run.smithi001.stdout: "pglog_hardlimit", 2024-09-15T03:47:19.211 INFO:teuthology.orchestra.run.smithi001.stdout: "purged_snapdirs", 2024-09-15T03:47:19.211 INFO:teuthology.orchestra.run.smithi001.stdout: "recovery_deletes", 2024-09-15T03:47:19.211 INFO:teuthology.orchestra.run.smithi001.stdout: "sortbitwise" 2024-09-15T03:47:19.211 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:47:19.211 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_version": 5, 2024-09-15T03:47:19.211 INFO:teuthology.orchestra.run.smithi001.stdout: "full_ratio": 0.94999998807907104, 2024-09-15T03:47:19.211 INFO:teuthology.orchestra.run.smithi001.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-15T03:47:19.212 INFO:teuthology.orchestra.run.smithi001.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-15T03:47:19.212 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_snapshot": "", 2024-09-15T03:47:19.212 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_max": 3, 2024-09-15T03:47:19.212 INFO:teuthology.orchestra.run.smithi001.stdout: "max_osd": 12, 2024-09-15T03:47:19.212 INFO:teuthology.orchestra.run.smithi001.stdout: "require_min_compat_client": "luminous", 2024-09-15T03:47:19.212 INFO:teuthology.orchestra.run.smithi001.stdout: "min_compat_client": "jewel", 2024-09-15T03:47:19.212 INFO:teuthology.orchestra.run.smithi001.stdout: "require_osd_release": "quincy", 2024-09-15T03:47:19.212 INFO:teuthology.orchestra.run.smithi001.stdout: "pools": [ 2024-09-15T03:47:19.212 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.212 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 1, 2024-09-15T03:47:19.212 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": ".mgr", 2024-09-15T03:47:19.212 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-15T03:46:16.185530+0000", 2024-09-15T03:47:19.213 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-09-15T03:47:19.213 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool", 2024-09-15T03:47:19.213 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-15T03:47:19.213 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-15T03:47:19.213 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 1, 2024-09-15T03:47:19.213 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-15T03:47:19.213 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-15T03:47:19.213 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-15T03:47:19.213 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T03:47:19.213 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T03:47:19.213 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-15T03:47:19.213 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-15T03:47:19.213 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 1, 2024-09-15T03:47:19.214 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 1, 2024-09-15T03:47:19.214 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 1, 2024-09-15T03:47:19.214 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 1, 2024-09-15T03:47:19.214 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 1, 2024-09-15T03:47:19.214 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-15T03:47:19.214 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-15T03:47:19.214 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-15T03:47:19.214 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-15T03:47:19.214 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-15T03:47:19.214 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-15T03:47:19.214 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-15T03:47:19.214 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.215 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "20", 2024-09-15T03:47:19.215 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-15T03:47:19.215 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T03:47:19.215 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T03:47:19.215 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-15T03:47:19.215 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-15T03:47:19.215 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 0, 2024-09-15T03:47:19.215 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 0, 2024-09-15T03:47:19.215 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-15T03:47:19.215 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-15T03:47:19.215 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-15T03:47:19.215 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-15T03:47:19.215 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-15T03:47:19.216 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-15T03:47:19.216 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-15T03:47:19.216 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-15T03:47:19.216 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-15T03:47:19.216 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-15T03:47:19.216 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-15T03:47:19.216 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T03:47:19.216 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T03:47:19.216 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T03:47:19.216 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-15T03:47:19.216 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-15T03:47:19.216 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-15T03:47:19.217 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-15T03:47:19.217 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-15T03:47:19.217 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.217 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-15T03:47:19.217 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-15T03:47:19.217 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-15T03:47:19.217 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-15T03:47:19.217 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-15T03:47:19.217 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T03:47:19.217 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-15T03:47:19.217 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-15T03:47:19.217 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-15T03:47:19.217 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-15T03:47:19.218 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-15T03:47:19.218 INFO:teuthology.orchestra.run.smithi001.stdout: "options": { 2024-09-15T03:47:19.218 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_max": 32, 2024-09-15T03:47:19.218 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_min": 1 2024-09-15T03:47:19.218 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.218 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-15T03:47:19.218 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": {} 2024-09-15T03:47:19.218 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.218 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.218 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.218 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 2, 2024-09-15T03:47:19.219 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": "rbd", 2024-09-15T03:47:19.219 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-15T03:46:22.284406+0000", 2024-09-15T03:47:19.219 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 8193, 2024-09-15T03:47:19.219 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-15T03:47:19.219 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-15T03:47:19.219 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-15T03:47:19.219 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 1, 2024-09-15T03:47:19.219 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-15T03:47:19.219 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-15T03:47:19.219 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-15T03:47:19.219 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T03:47:19.219 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T03:47:19.220 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-15T03:47:19.220 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-15T03:47:19.220 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 8, 2024-09-15T03:47:19.220 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 8, 2024-09-15T03:47:19.220 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 8, 2024-09-15T03:47:19.220 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 8, 2024-09-15T03:47:19.220 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 8, 2024-09-15T03:47:19.220 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-15T03:47:19.220 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-15T03:47:19.220 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-15T03:47:19.220 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-15T03:47:19.220 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-15T03:47:19.220 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-15T03:47:19.221 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-15T03:47:19.221 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.221 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "24", 2024-09-15T03:47:19.221 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-15T03:47:19.221 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T03:47:19.221 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T03:47:19.221 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-15T03:47:19.221 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-15T03:47:19.221 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 3, 2024-09-15T03:47:19.221 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 24, 2024-09-15T03:47:19.221 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-15T03:47:19.221 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-15T03:47:19.221 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-15T03:47:19.222 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-15T03:47:19.222 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-15T03:47:19.222 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-15T03:47:19.222 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-15T03:47:19.222 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-15T03:47:19.222 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-15T03:47:19.222 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-15T03:47:19.222 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-15T03:47:19.222 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T03:47:19.222 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T03:47:19.222 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T03:47:19.222 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-15T03:47:19.222 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-15T03:47:19.223 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-15T03:47:19.223 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-15T03:47:19.223 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-15T03:47:19.223 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.223 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-15T03:47:19.223 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-15T03:47:19.223 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-15T03:47:19.223 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-15T03:47:19.223 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-15T03:47:19.223 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T03:47:19.223 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-15T03:47:19.223 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-15T03:47:19.223 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-15T03:47:19.224 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-15T03:47:19.224 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-15T03:47:19.224 INFO:teuthology.orchestra.run.smithi001.stdout: "options": {}, 2024-09-15T03:47:19.224 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-15T03:47:19.224 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd": {} 2024-09-15T03:47:19.224 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.224 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.224 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.224 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 3, 2024-09-15T03:47:19.224 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": "unique_pool_0", 2024-09-15T03:47:19.224 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-15T03:46:34.105566+0000", 2024-09-15T03:47:19.224 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 8193, 2024-09-15T03:47:19.225 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-15T03:47:19.225 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 3, 2024-09-15T03:47:19.225 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 8, 2024-09-15T03:47:19.225 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 5, 2024-09-15T03:47:19.225 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 1, 2024-09-15T03:47:19.225 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-15T03:47:19.225 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-15T03:47:19.225 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T03:47:19.225 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T03:47:19.225 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-15T03:47:19.225 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-15T03:47:19.225 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 16, 2024-09-15T03:47:19.225 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 16, 2024-09-15T03:47:19.226 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 16, 2024-09-15T03:47:19.226 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 16, 2024-09-15T03:47:19.226 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 16, 2024-09-15T03:47:19.226 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-15T03:47:19.226 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-15T03:47:19.226 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-15T03:47:19.226 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-15T03:47:19.226 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-15T03:47:19.226 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-15T03:47:19.226 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-15T03:47:19.226 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.226 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "72", 2024-09-15T03:47:19.226 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-15T03:47:19.227 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T03:47:19.227 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T03:47:19.227 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-15T03:47:19.227 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-15T03:47:19.227 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 38, 2024-09-15T03:47:19.227 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 72, 2024-09-15T03:47:19.227 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-15T03:47:19.227 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-15T03:47:19.227 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-15T03:47:19.227 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-15T03:47:19.227 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-15T03:47:19.227 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-15T03:47:19.228 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-15T03:47:19.228 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-15T03:47:19.228 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-15T03:47:19.228 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-15T03:47:19.228 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-15T03:47:19.228 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T03:47:19.228 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T03:47:19.228 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T03:47:19.228 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-15T03:47:19.228 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-15T03:47:19.228 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "lrcprofile", 2024-09-15T03:47:19.228 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-15T03:47:19.228 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-15T03:47:19.229 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.229 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-15T03:47:19.229 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-15T03:47:19.229 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-15T03:47:19.229 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-15T03:47:19.229 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-15T03:47:19.229 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T03:47:19.229 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-15T03:47:19.229 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-15T03:47:19.229 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 16384, 2024-09-15T03:47:19.229 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-15T03:47:19.229 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-15T03:47:19.230 INFO:teuthology.orchestra.run.smithi001.stdout: "options": {}, 2024-09-15T03:47:19.230 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-15T03:47:19.230 INFO:teuthology.orchestra.run.smithi001.stdout: "rados": {} 2024-09-15T03:47:19.230 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.230 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.230 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:47:19.230 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-15T03:47:19.230 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.230 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 0, 2024-09-15T03:47:19.230 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "cd433a6a-c875-4e6e-b89c-938caecbae3a", 2024-09-15T03:47:19.230 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:19.230 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:19.230 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:19.230 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:19.231 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:19.231 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:19.231 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:19.231 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 70, 2024-09-15T03:47:19.231 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:19.231 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:19.231 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:19.231 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.231 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.231 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.231 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6800", 2024-09-15T03:47:19.231 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:19.232 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.232 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.232 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.232 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6801", 2024-09-15T03:47:19.232 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:19.232 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.232 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.232 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.232 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:19.232 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.232 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.232 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.233 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6802", 2024-09-15T03:47:19.233 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:19.233 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.233 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.233 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.233 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6803", 2024-09-15T03:47:19.233 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:19.233 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.233 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.233 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.233 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:19.233 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.233 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.234 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.234 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6806", 2024-09-15T03:47:19.234 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:19.234 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.234 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.234 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.234 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6807", 2024-09-15T03:47:19.234 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:19.234 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.234 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.234 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.234 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:19.234 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.235 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.235 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.235 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6804", 2024-09-15T03:47:19.235 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:19.235 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.235 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.235 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.235 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6805", 2024-09-15T03:47:19.235 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:19.235 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.235 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.235 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.236 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6801/28908", 2024-09-15T03:47:19.236 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6803/28908", 2024-09-15T03:47:19.236 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6807/28908", 2024-09-15T03:47:19.236 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6805/28908", 2024-09-15T03:47:19.236 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:19.236 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:19.236 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:19.236 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.236 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.236 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.236 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 1, 2024-09-15T03:47:19.236 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "789930f9-42bf-476b-9f09-a21bffe33a79", 2024-09-15T03:47:19.237 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:19.237 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:19.237 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:19.237 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:19.237 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:19.237 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:19.237 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:19.237 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 70, 2024-09-15T03:47:19.237 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:19.237 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:19.237 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:19.237 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.237 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.238 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.238 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6808", 2024-09-15T03:47:19.238 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:19.238 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.238 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.238 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.238 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6809", 2024-09-15T03:47:19.238 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:19.238 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.238 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.238 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.238 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:19.239 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.239 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.239 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.239 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6810", 2024-09-15T03:47:19.239 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:19.239 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.239 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.239 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.239 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6811", 2024-09-15T03:47:19.239 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:19.239 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.239 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.239 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.240 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:19.240 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.240 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.240 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.240 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6814", 2024-09-15T03:47:19.240 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:19.240 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.240 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.240 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.240 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6815", 2024-09-15T03:47:19.240 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:19.240 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.241 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.241 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.241 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:19.241 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.241 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.241 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.241 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6812", 2024-09-15T03:47:19.241 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:19.241 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.241 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.241 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.241 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6813", 2024-09-15T03:47:19.241 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:19.242 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.242 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.242 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.242 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6809/28913", 2024-09-15T03:47:19.242 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6811/28913", 2024-09-15T03:47:19.242 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6815/28913", 2024-09-15T03:47:19.242 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6813/28913", 2024-09-15T03:47:19.242 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:19.242 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:19.242 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:19.242 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.242 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.242 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.243 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 2, 2024-09-15T03:47:19.243 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "9a83fec5-0d63-461c-b516-289443b3882b", 2024-09-15T03:47:19.243 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:19.243 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:19.243 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:19.243 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:19.243 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:19.243 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:19.243 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:19.243 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 63, 2024-09-15T03:47:19.243 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:19.243 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:19.243 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:19.244 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.244 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.244 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.244 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6821", 2024-09-15T03:47:19.244 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:19.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.244 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.244 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.244 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6823", 2024-09-15T03:47:19.244 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:19.244 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.244 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.245 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:19.245 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.245 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.245 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.245 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6825", 2024-09-15T03:47:19.245 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:19.245 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.245 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.245 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.245 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6827", 2024-09-15T03:47:19.245 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:19.245 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.245 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.245 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.246 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:19.246 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.246 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.246 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.246 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6830", 2024-09-15T03:47:19.246 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:19.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.246 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.246 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.246 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6831", 2024-09-15T03:47:19.246 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:19.246 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.246 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.247 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:19.247 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.247 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.247 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6828", 2024-09-15T03:47:19.247 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:19.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.247 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.247 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.247 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6829", 2024-09-15T03:47:19.247 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:19.247 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.248 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.248 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6823/28911", 2024-09-15T03:47:19.248 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6827/28911", 2024-09-15T03:47:19.248 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6831/28911", 2024-09-15T03:47:19.248 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6829/28911", 2024-09-15T03:47:19.248 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:19.248 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:19.248 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:19.248 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.248 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.248 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.248 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 3, 2024-09-15T03:47:19.249 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "d944bf89-e183-496a-9ca3-7949741b4be4", 2024-09-15T03:47:19.249 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:19.249 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:19.249 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:19.249 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:19.249 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:19.249 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:19.249 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:19.249 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 70, 2024-09-15T03:47:19.249 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:19.249 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:19.249 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:19.250 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.250 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.250 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6816", 2024-09-15T03:47:19.250 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:19.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.250 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.250 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.250 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6817", 2024-09-15T03:47:19.250 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:19.250 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.250 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.251 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:19.251 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.251 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.251 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.251 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6818", 2024-09-15T03:47:19.251 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:19.251 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.251 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.251 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.251 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6819", 2024-09-15T03:47:19.251 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:19.251 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.252 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.252 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.252 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:19.252 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.252 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.252 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6824", 2024-09-15T03:47:19.252 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:19.252 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.252 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.252 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.253 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6826", 2024-09-15T03:47:19.253 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:19.253 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.253 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.253 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:19.253 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.253 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.253 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6820", 2024-09-15T03:47:19.253 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:19.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.253 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.254 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.254 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6822", 2024-09-15T03:47:19.254 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:19.254 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.254 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.254 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6817/28914", 2024-09-15T03:47:19.254 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6819/28914", 2024-09-15T03:47:19.254 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6826/28914", 2024-09-15T03:47:19.254 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6822/28914", 2024-09-15T03:47:19.254 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:19.254 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:19.255 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:19.255 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.255 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.255 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.255 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 4, 2024-09-15T03:47:19.255 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "15ec2dbd-d099-4aff-94b2-1bf9dbb00b56", 2024-09-15T03:47:19.255 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:19.255 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:19.255 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:19.255 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:19.255 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:19.255 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:19.255 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:19.256 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 70, 2024-09-15T03:47:19.256 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:19.256 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:19.256 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:19.256 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.256 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.256 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.256 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6803", 2024-09-15T03:47:19.256 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:19.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.256 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.256 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.256 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6805", 2024-09-15T03:47:19.257 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:19.257 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.257 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.257 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:19.257 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.257 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.257 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6807", 2024-09-15T03:47:19.257 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:19.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.257 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.258 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.258 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6809", 2024-09-15T03:47:19.258 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:19.258 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.258 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.258 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:19.258 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.258 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.258 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.258 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6814", 2024-09-15T03:47:19.258 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:19.258 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.259 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.259 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.259 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6815", 2024-09-15T03:47:19.259 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:19.259 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.259 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.259 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.259 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:19.259 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.259 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.259 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.259 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6811", 2024-09-15T03:47:19.259 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:19.260 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.260 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.260 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.260 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6813", 2024-09-15T03:47:19.260 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:19.260 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.260 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.260 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.260 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.67:6805/28468", 2024-09-15T03:47:19.260 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.67:6809/28468", 2024-09-15T03:47:19.261 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.67:6815/28468", 2024-09-15T03:47:19.261 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.67:6813/28468", 2024-09-15T03:47:19.261 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:19.261 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:19.261 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:19.261 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.261 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.261 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.261 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 5, 2024-09-15T03:47:19.261 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "1e45de07-746b-4612-9a52-d05509cae9a3", 2024-09-15T03:47:19.261 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:19.261 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:19.261 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:19.262 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:19.262 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:19.262 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:19.262 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:19.262 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 61, 2024-09-15T03:47:19.262 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:19.262 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:19.262 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:19.262 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.262 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.262 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.262 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6824", 2024-09-15T03:47:19.262 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:19.263 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.263 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.263 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.263 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6825", 2024-09-15T03:47:19.263 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:19.263 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.263 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.263 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.263 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:19.263 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.263 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.263 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.264 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6826", 2024-09-15T03:47:19.264 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:19.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.264 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.264 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.264 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6827", 2024-09-15T03:47:19.264 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:19.264 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.264 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.264 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:19.264 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.264 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.265 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.265 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6830", 2024-09-15T03:47:19.265 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:19.265 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.265 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.265 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.265 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6831", 2024-09-15T03:47:19.265 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:19.265 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.265 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.265 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.265 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:19.265 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.266 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.266 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.266 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6828", 2024-09-15T03:47:19.266 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:19.266 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.266 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.266 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.266 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6829", 2024-09-15T03:47:19.266 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:19.266 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.266 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.266 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.266 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.67:6825/28495", 2024-09-15T03:47:19.267 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.67:6827/28495", 2024-09-15T03:47:19.267 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.67:6831/28495", 2024-09-15T03:47:19.267 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.67:6829/28495", 2024-09-15T03:47:19.267 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:19.267 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:19.267 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:19.267 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.267 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.267 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.268 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 6, 2024-09-15T03:47:19.268 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "ccf785f8-751d-473e-b17c-3df9346be9f8", 2024-09-15T03:47:19.268 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:19.268 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 0, 2024-09-15T03:47:19.268 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-09-15T03:47:19.268 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:19.268 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:19.268 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:19.268 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:19.268 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 31, 2024-09-15T03:47:19.268 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:19.268 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:19.269 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:19.269 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.269 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.269 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6800", 2024-09-15T03:47:19.269 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:19.269 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.269 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.269 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6801", 2024-09-15T03:47:19.269 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:19.269 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.269 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.269 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.270 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:19.270 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.270 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.270 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6802", 2024-09-15T03:47:19.270 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:19.270 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.270 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.270 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6804", 2024-09-15T03:47:19.270 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:19.270 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.270 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.271 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.271 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:19.271 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.271 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.271 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.271 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6810", 2024-09-15T03:47:19.271 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:19.271 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.271 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.271 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.271 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6812", 2024-09-15T03:47:19.271 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:19.271 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.272 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.272 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:19.272 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.272 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.272 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.272 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6806", 2024-09-15T03:47:19.272 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:19.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.272 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.272 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.272 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6808", 2024-09-15T03:47:19.272 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:19.273 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.273 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.273 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.67:6801/28469", 2024-09-15T03:47:19.273 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.67:6804/28469", 2024-09-15T03:47:19.273 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.67:6812/28469", 2024-09-15T03:47:19.273 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.67:6808/28469", 2024-09-15T03:47:19.273 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:19.273 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:19.273 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:19.273 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.274 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 7, 2024-09-15T03:47:19.274 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "f9f59987-869d-42eb-9eb6-08c39e3afe6d", 2024-09-15T03:47:19.274 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:19.274 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:19.274 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:19.274 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:19.274 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:19.274 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:19.274 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:19.274 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 70, 2024-09-15T03:47:19.274 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:19.274 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:19.274 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:19.275 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.275 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.275 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.275 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6816", 2024-09-15T03:47:19.275 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:19.275 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.275 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.275 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.275 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6817", 2024-09-15T03:47:19.275 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:19.275 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.275 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.275 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.276 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:19.276 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.276 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.276 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.276 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6818", 2024-09-15T03:47:19.276 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:19.276 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.276 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.276 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.276 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6819", 2024-09-15T03:47:19.276 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:19.276 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.276 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.277 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.277 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:19.277 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.277 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.277 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.277 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6822", 2024-09-15T03:47:19.277 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:19.277 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.277 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.277 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.277 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6823", 2024-09-15T03:47:19.277 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:19.277 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.278 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.278 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.278 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:19.278 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.278 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.278 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.278 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6820", 2024-09-15T03:47:19.278 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:19.278 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.278 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.278 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.278 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6821", 2024-09-15T03:47:19.278 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:19.279 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.279 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.279 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.67:6817/28470", 2024-09-15T03:47:19.279 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.67:6819/28470", 2024-09-15T03:47:19.279 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.67:6823/28470", 2024-09-15T03:47:19.279 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.67:6821/28470", 2024-09-15T03:47:19.279 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:19.279 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:19.279 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:19.279 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.279 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.280 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 8, 2024-09-15T03:47:19.280 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "69ddfe52-11df-4608-9a10-e8907c4cd87f", 2024-09-15T03:47:19.280 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:19.280 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:19.280 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:19.280 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:19.280 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:19.280 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:19.280 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:19.280 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 70, 2024-09-15T03:47:19.280 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:19.280 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:19.280 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:19.281 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.281 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.281 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.281 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6808", 2024-09-15T03:47:19.281 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:19.281 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.281 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.281 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.281 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6809", 2024-09-15T03:47:19.281 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:19.281 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.281 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.281 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.282 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:19.282 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.282 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.282 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.282 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6810", 2024-09-15T03:47:19.282 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:19.282 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.282 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.282 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.282 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6811", 2024-09-15T03:47:19.282 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:19.282 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.282 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.283 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.283 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:19.283 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.283 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.283 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.283 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6814", 2024-09-15T03:47:19.283 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:19.283 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.283 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.283 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.283 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6815", 2024-09-15T03:47:19.283 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:19.283 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.283 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.284 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:19.284 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.284 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.284 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.284 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6812", 2024-09-15T03:47:19.284 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:19.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.284 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.284 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.284 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6813", 2024-09-15T03:47:19.284 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:19.284 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.285 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.285 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.285 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.190:6809/29472", 2024-09-15T03:47:19.285 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.190:6811/29472", 2024-09-15T03:47:19.285 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.190:6815/29472", 2024-09-15T03:47:19.285 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.190:6813/29472", 2024-09-15T03:47:19.285 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:19.285 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:19.285 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:19.285 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.285 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.285 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.285 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 9, 2024-09-15T03:47:19.286 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "f92e1609-2a1a-4814-9ac0-44a73373d229", 2024-09-15T03:47:19.286 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:19.286 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:19.286 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:19.286 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:19.286 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:19.286 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:19.286 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:19.286 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 70, 2024-09-15T03:47:19.286 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:19.286 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:19.286 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:19.286 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.287 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.287 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.287 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6800", 2024-09-15T03:47:19.287 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:19.287 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.287 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.287 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.287 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6801", 2024-09-15T03:47:19.287 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:19.287 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.287 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.287 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.287 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:19.288 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.288 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.288 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.288 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6802", 2024-09-15T03:47:19.288 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:19.288 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.288 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.288 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.288 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6803", 2024-09-15T03:47:19.288 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:19.289 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.289 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.289 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.289 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:19.289 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.289 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.289 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.289 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6806", 2024-09-15T03:47:19.289 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:19.289 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.289 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.289 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.289 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6807", 2024-09-15T03:47:19.290 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:19.290 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.290 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.290 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.290 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:19.290 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.290 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.290 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.291 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6804", 2024-09-15T03:47:19.291 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:19.291 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.291 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.291 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.291 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6805", 2024-09-15T03:47:19.291 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:19.291 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.291 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.292 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.292 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.190:6801/29473", 2024-09-15T03:47:19.292 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.190:6803/29473", 2024-09-15T03:47:19.292 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.190:6807/29473", 2024-09-15T03:47:19.292 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.190:6805/29473", 2024-09-15T03:47:19.292 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:19.292 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:19.292 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:19.292 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.293 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.293 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.293 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 10, 2024-09-15T03:47:19.293 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "5f2e8ca3-8c65-4595-8e82-27dc3768ab99", 2024-09-15T03:47:19.293 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:19.293 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:19.293 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:19.293 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:19.293 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:19.293 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:19.294 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:19.294 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 70, 2024-09-15T03:47:19.294 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:19.294 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:19.294 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:19.294 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.294 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.294 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.294 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6816", 2024-09-15T03:47:19.294 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:19.294 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.294 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.294 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.295 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6817", 2024-09-15T03:47:19.295 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:19.295 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.295 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.295 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.295 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:19.295 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.295 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.295 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.295 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6818", 2024-09-15T03:47:19.295 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:19.295 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.296 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.296 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.296 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6819", 2024-09-15T03:47:19.296 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:19.296 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.296 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.296 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:19.296 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.296 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.296 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.296 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6822", 2024-09-15T03:47:19.296 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:19.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.297 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.297 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.297 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6823", 2024-09-15T03:47:19.297 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:19.297 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.297 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.297 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:19.297 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.297 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.297 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.297 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6820", 2024-09-15T03:47:19.298 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:19.298 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.298 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.298 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6821", 2024-09-15T03:47:19.298 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:19.298 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.298 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.298 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.298 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.190:6817/29474", 2024-09-15T03:47:19.298 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.190:6819/29474", 2024-09-15T03:47:19.298 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.190:6823/29474", 2024-09-15T03:47:19.298 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.190:6821/29474", 2024-09-15T03:47:19.299 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:19.299 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:19.299 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:19.299 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.299 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 11, 2024-09-15T03:47:19.299 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "cedf7df4-f875-42f5-a8cf-13cb82f05cc2", 2024-09-15T03:47:19.299 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:19.299 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:19.299 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:19.299 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:19.299 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:19.300 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:19.300 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:19.300 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 70, 2024-09-15T03:47:19.300 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:19.300 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:19.300 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:19.300 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.300 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.300 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6824", 2024-09-15T03:47:19.300 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:19.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.301 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6825", 2024-09-15T03:47:19.301 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:19.301 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.301 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.301 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:19.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.301 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6826", 2024-09-15T03:47:19.301 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:19.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.302 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.302 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6827", 2024-09-15T03:47:19.302 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:19.302 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.302 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.302 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:19.302 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.302 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.302 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6830", 2024-09-15T03:47:19.303 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:19.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.303 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.303 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6831", 2024-09-15T03:47:19.303 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:19.303 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.303 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.303 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:19.303 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:19.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.303 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:19.304 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6828", 2024-09-15T03:47:19.304 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:19.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.304 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:19.304 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6829", 2024-09-15T03:47:19.304 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:19.304 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.304 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.304 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.190:6825/29475", 2024-09-15T03:47:19.304 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.190:6827/29475", 2024-09-15T03:47:19.304 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.190:6831/29475", 2024-09-15T03:47:19.305 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.190:6829/29475", 2024-09-15T03:47:19.305 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:19.305 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:19.305 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:19.305 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.305 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.305 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:47:19.305 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_xinfo": [ 2024-09-15T03:47:19.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.305 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 0, 2024-09-15T03:47:19.305 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:19.305 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:19.305 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:19.306 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:19.306 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:19.306 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.943165+0000", 2024-09-15T03:47:19.306 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:19.306 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.306 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 1, 2024-09-15T03:47:19.306 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:19.306 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:19.306 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:19.306 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:19.306 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:19.306 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.946049+0000", 2024-09-15T03:47:19.307 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:19.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.307 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.307 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 2, 2024-09-15T03:47:19.307 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:19.307 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:19.307 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:19.307 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:19.307 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:19.307 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.981045+0000", 2024-09-15T03:47:19.307 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:19.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.307 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.308 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 3, 2024-09-15T03:47:19.308 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:19.308 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:19.308 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:19.308 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:19.308 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:19.308 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:14.019932+0000", 2024-09-15T03:47:19.308 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:19.308 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.308 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.308 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 4, 2024-09-15T03:47:19.308 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:19.308 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:19.309 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:19.309 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:19.309 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:19.309 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.936197+0000", 2024-09-15T03:47:19.309 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:19.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.309 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.309 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 5, 2024-09-15T03:47:19.309 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:19.309 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:19.309 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:19.309 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:19.309 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:19.310 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.982429+0000", 2024-09-15T03:47:19.310 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:19.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.310 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 6, 2024-09-15T03:47:19.310 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:19.310 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:19.310 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:19.310 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:19.310 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 65536, 2024-09-15T03:47:19.310 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.953833+0000", 2024-09-15T03:47:19.310 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:19.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.311 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 7, 2024-09-15T03:47:19.311 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:19.311 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:19.311 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:19.311 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:19.311 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:19.311 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:47:18.853134+0000", 2024-09-15T03:47:19.311 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:19.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.311 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 8, 2024-09-15T03:47:19.311 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:19.311 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:19.312 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:19.312 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:19.312 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:19.312 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.996438+0000", 2024-09-15T03:47:19.312 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:19.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.312 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.312 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 9, 2024-09-15T03:47:19.312 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:19.312 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:19.312 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:19.312 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:19.312 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:19.313 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.926618+0000", 2024-09-15T03:47:19.313 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:19.313 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.313 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.313 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 10, 2024-09-15T03:47:19.313 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:19.313 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:19.313 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:19.313 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:19.313 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:19.313 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.993934+0000", 2024-09-15T03:47:19.313 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:19.313 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.314 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 11, 2024-09-15T03:47:19.314 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:19.314 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:19.314 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:19.314 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:19.314 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:19.314 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:14.022484+0000", 2024-09-15T03:47:19.314 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:19.314 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.314 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:47:19.314 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap": [], 2024-09-15T03:47:19.314 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap_items": [], 2024-09-15T03:47:19.315 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_temp": [], 2024-09-15T03:47:19.315 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_temp": [], 2024-09-15T03:47:19.315 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklist": {}, 2024-09-15T03:47:19.315 INFO:teuthology.orchestra.run.smithi001.stdout: "range_blocklist": {}, 2024-09-15T03:47:19.315 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profiles": { 2024-09-15T03:47:19.315 INFO:teuthology.orchestra.run.smithi001.stdout: "default": { 2024-09-15T03:47:19.315 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-failure-domain": "osd", 2024-09-15T03:47:19.315 INFO:teuthology.orchestra.run.smithi001.stdout: "k": "2", 2024-09-15T03:47:19.315 INFO:teuthology.orchestra.run.smithi001.stdout: "m": "1", 2024-09-15T03:47:19.315 INFO:teuthology.orchestra.run.smithi001.stdout: "plugin": "jerasure", 2024-09-15T03:47:19.315 INFO:teuthology.orchestra.run.smithi001.stdout: "technique": "reed_sol_van" 2024-09-15T03:47:19.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.315 INFO:teuthology.orchestra.run.smithi001.stdout: "lrcprofile": { 2024-09-15T03:47:19.315 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-device-class": "", 2024-09-15T03:47:19.316 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-failure-domain": "osd", 2024-09-15T03:47:19.316 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-root": "default", 2024-09-15T03:47:19.316 INFO:teuthology.orchestra.run.smithi001.stdout: "k": "4", 2024-09-15T03:47:19.316 INFO:teuthology.orchestra.run.smithi001.stdout: "l": "3", 2024-09-15T03:47:19.316 INFO:teuthology.orchestra.run.smithi001.stdout: "m": "2", 2024-09-15T03:47:19.316 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "lrcprofile", 2024-09-15T03:47:19.316 INFO:teuthology.orchestra.run.smithi001.stdout: "plugin": "lrc" 2024-09-15T03:47:19.316 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.316 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps_queue": [ 2024-09-15T03:47:19.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.316 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 3, 2024-09-15T03:47:19.316 INFO:teuthology.orchestra.run.smithi001.stdout: "snaps": [ 2024-09-15T03:47:19.317 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.317 INFO:teuthology.orchestra.run.smithi001.stdout: "begin": 21, 2024-09-15T03:47:19.317 INFO:teuthology.orchestra.run.smithi001.stdout: "length": 1 2024-09-15T03:47:19.317 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:19.317 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:19.317 INFO:teuthology.orchestra.run.smithi001.stdout: "begin": 26, 2024-09-15T03:47:19.317 INFO:teuthology.orchestra.run.smithi001.stdout: "length": 1 2024-09-15T03:47:19.317 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.317 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:19.317 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.317 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:47:19.317 INFO:teuthology.orchestra.run.smithi001.stdout: "new_removed_snaps": [], 2024-09-15T03:47:19.317 INFO:teuthology.orchestra.run.smithi001.stdout: "new_purged_snaps": [], 2024-09-15T03:47:19.318 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_node_flags": {}, 2024-09-15T03:47:19.318 INFO:teuthology.orchestra.run.smithi001.stdout: "device_class_flags": {}, 2024-09-15T03:47:19.318 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": { 2024-09-15T03:47:19.318 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode_enabled": false, 2024-09-15T03:47:19.318 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_bucket_count": 0, 2024-09-15T03:47:19.318 INFO:teuthology.orchestra.run.smithi001.stdout: "degraded_stretch_mode": 0, 2024-09-15T03:47:19.318 INFO:teuthology.orchestra.run.smithi001.stdout: "recovering_stretch_mode": 0, 2024-09-15T03:47:19.318 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode_bucket": 0 2024-09-15T03:47:19.318 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:19.318 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:19.320 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:47:19.321 INFO:tasks.thrashosds.thrasher:j is {'epoch': 72, 'fsid': 'f27b540e-e5e3-4e95-afe8-c95ee66b0a62', 'created': '2024-09-15T03:46:09.843094+0000', 'modified': '2024-09-15T03:47:18.920644+0000', 'last_up_change': '2024-09-15T03:46:15.871985+0000', 'last_in_change': '2024-09-15T03:47:13.897657+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-15T03:46:16.185530+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-15T03:46:22.284406+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 24, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-15T03:46:34.105566+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 8, 'min_size': 5, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 38, 'snap_epoch': 72, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'lrcprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 16384, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'cd433a6a-c875-4e6e-b89c-938caecbae3a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6800', 'nonce': 28908}, {'type': 'v1', 'addr': '172.21.15.1:6801', 'nonce': 28908}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6802', 'nonce': 28908}, {'type': 'v1', 'addr': '172.21.15.1:6803', 'nonce': 28908}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6806', 'nonce': 28908}, {'type': 'v1', 'addr': '172.21.15.1:6807', 'nonce': 28908}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6804', 'nonce': 28908}, {'type': 'v1', 'addr': '172.21.15.1:6805', 'nonce': 28908}]}, 'public_addr': '172.21.15.1:6801/28908', 'cluster_addr': '172.21.15.1:6803/28908', 'heartbeat_back_addr': '172.21.15.1:6807/28908', 'heartbeat_front_addr': '172.21.15.1:6805/28908', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '789930f9-42bf-476b-9f09-a21bffe33a79', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6808', 'nonce': 28913}, {'type': 'v1', 'addr': '172.21.15.1:6809', 'nonce': 28913}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6810', 'nonce': 28913}, {'type': 'v1', 'addr': '172.21.15.1:6811', 'nonce': 28913}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6814', 'nonce': 28913}, {'type': 'v1', 'addr': '172.21.15.1:6815', 'nonce': 28913}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6812', 'nonce': 28913}, {'type': 'v1', 'addr': '172.21.15.1:6813', 'nonce': 28913}]}, 'public_addr': '172.21.15.1:6809/28913', 'cluster_addr': '172.21.15.1:6811/28913', 'heartbeat_back_addr': '172.21.15.1:6815/28913', 'heartbeat_front_addr': '172.21.15.1:6813/28913', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '9a83fec5-0d63-461c-b516-289443b3882b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 63, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6821', 'nonce': 28911}, {'type': 'v1', 'addr': '172.21.15.1:6823', 'nonce': 28911}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6825', 'nonce': 28911}, {'type': 'v1', 'addr': '172.21.15.1:6827', 'nonce': 28911}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6830', 'nonce': 28911}, {'type': 'v1', 'addr': '172.21.15.1:6831', 'nonce': 28911}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6828', 'nonce': 28911}, {'type': 'v1', 'addr': '172.21.15.1:6829', 'nonce': 28911}]}, 'public_addr': '172.21.15.1:6823/28911', 'cluster_addr': '172.21.15.1:6827/28911', 'heartbeat_back_addr': '172.21.15.1:6831/28911', 'heartbeat_front_addr': '172.21.15.1:6829/28911', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd944bf89-e183-496a-9ca3-7949741b4be4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6816', 'nonce': 28914}, {'type': 'v1', 'addr': '172.21.15.1:6817', 'nonce': 28914}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6818', 'nonce': 28914}, {'type': 'v1', 'addr': '172.21.15.1:6819', 'nonce': 28914}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6824', 'nonce': 28914}, {'type': 'v1', 'addr': '172.21.15.1:6826', 'nonce': 28914}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6820', 'nonce': 28914}, {'type': 'v1', 'addr': '172.21.15.1:6822', 'nonce': 28914}]}, 'public_addr': '172.21.15.1:6817/28914', 'cluster_addr': '172.21.15.1:6819/28914', 'heartbeat_back_addr': '172.21.15.1:6826/28914', 'heartbeat_front_addr': '172.21.15.1:6822/28914', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '15ec2dbd-d099-4aff-94b2-1bf9dbb00b56', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6803', 'nonce': 28468}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 28468}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6807', 'nonce': 28468}, {'type': 'v1', 'addr': '172.21.15.67:6809', 'nonce': 28468}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 28468}, {'type': 'v1', 'addr': '172.21.15.67:6815', 'nonce': 28468}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6811', 'nonce': 28468}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 28468}]}, 'public_addr': '172.21.15.67:6805/28468', 'cluster_addr': '172.21.15.67:6809/28468', 'heartbeat_back_addr': '172.21.15.67:6815/28468', 'heartbeat_front_addr': '172.21.15.67:6813/28468', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1e45de07-746b-4612-9a52-d05509cae9a3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 61, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6824', 'nonce': 28495}, {'type': 'v1', 'addr': '172.21.15.67:6825', 'nonce': 28495}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6826', 'nonce': 28495}, {'type': 'v1', 'addr': '172.21.15.67:6827', 'nonce': 28495}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6830', 'nonce': 28495}, {'type': 'v1', 'addr': '172.21.15.67:6831', 'nonce': 28495}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6828', 'nonce': 28495}, {'type': 'v1', 'addr': '172.21.15.67:6829', 'nonce': 28495}]}, 'public_addr': '172.21.15.67:6825/28495', 'cluster_addr': '172.21.15.67:6827/28495', 'heartbeat_back_addr': '172.21.15.67:6831/28495', 'heartbeat_front_addr': '172.21.15.67:6829/28495', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ccf785f8-751d-473e-b17c-3df9346be9f8', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 28469}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 28469}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 28469}, {'type': 'v1', 'addr': '172.21.15.67:6804', 'nonce': 28469}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 28469}, {'type': 'v1', 'addr': '172.21.15.67:6812', 'nonce': 28469}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 28469}, {'type': 'v1', 'addr': '172.21.15.67:6808', 'nonce': 28469}]}, 'public_addr': '172.21.15.67:6801/28469', 'cluster_addr': '172.21.15.67:6804/28469', 'heartbeat_back_addr': '172.21.15.67:6812/28469', 'heartbeat_front_addr': '172.21.15.67:6808/28469', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f9f59987-869d-42eb-9eb6-08c39e3afe6d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 28470}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 28470}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6818', 'nonce': 28470}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 28470}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6822', 'nonce': 28470}, {'type': 'v1', 'addr': '172.21.15.67:6823', 'nonce': 28470}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 28470}, {'type': 'v1', 'addr': '172.21.15.67:6821', 'nonce': 28470}]}, 'public_addr': '172.21.15.67:6817/28470', 'cluster_addr': '172.21.15.67:6819/28470', 'heartbeat_back_addr': '172.21.15.67:6823/28470', 'heartbeat_front_addr': '172.21.15.67:6821/28470', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '69ddfe52-11df-4608-9a10-e8907c4cd87f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6808', 'nonce': 29472}, {'type': 'v1', 'addr': '172.21.15.190:6809', 'nonce': 29472}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6810', 'nonce': 29472}, {'type': 'v1', 'addr': '172.21.15.190:6811', 'nonce': 29472}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6814', 'nonce': 29472}, {'type': 'v1', 'addr': '172.21.15.190:6815', 'nonce': 29472}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6812', 'nonce': 29472}, {'type': 'v1', 'addr': '172.21.15.190:6813', 'nonce': 29472}]}, 'public_addr': '172.21.15.190:6809/29472', 'cluster_addr': '172.21.15.190:6811/29472', 'heartbeat_back_addr': '172.21.15.190:6815/29472', 'heartbeat_front_addr': '172.21.15.190:6813/29472', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'f92e1609-2a1a-4814-9ac0-44a73373d229', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6800', 'nonce': 29473}, {'type': 'v1', 'addr': '172.21.15.190:6801', 'nonce': 29473}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6802', 'nonce': 29473}, {'type': 'v1', 'addr': '172.21.15.190:6803', 'nonce': 29473}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6806', 'nonce': 29473}, {'type': 'v1', 'addr': '172.21.15.190:6807', 'nonce': 29473}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6804', 'nonce': 29473}, {'type': 'v1', 'addr': '172.21.15.190:6805', 'nonce': 29473}]}, 'public_addr': '172.21.15.190:6801/29473', 'cluster_addr': '172.21.15.190:6803/29473', 'heartbeat_back_addr': '172.21.15.190:6807/29473', 'heartbeat_front_addr': '172.21.15.190:6805/29473', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '5f2e8ca3-8c65-4595-8e82-27dc3768ab99', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6816', 'nonce': 29474}, {'type': 'v1', 'addr': '172.21.15.190:6817', 'nonce': 29474}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6818', 'nonce': 29474}, {'type': 'v1', 'addr': '172.21.15.190:6819', 'nonce': 29474}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6822', 'nonce': 29474}, {'type': 'v1', 'addr': '172.21.15.190:6823', 'nonce': 29474}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6820', 'nonce': 29474}, {'type': 'v1', 'addr': '172.21.15.190:6821', 'nonce': 29474}]}, 'public_addr': '172.21.15.190:6817/29474', 'cluster_addr': '172.21.15.190:6819/29474', 'heartbeat_back_addr': '172.21.15.190:6823/29474', 'heartbeat_front_addr': '172.21.15.190:6821/29474', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'cedf7df4-f875-42f5-a8cf-13cb82f05cc2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6824', 'nonce': 29475}, {'type': 'v1', 'addr': '172.21.15.190:6825', 'nonce': 29475}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6826', 'nonce': 29475}, {'type': 'v1', 'addr': '172.21.15.190:6827', 'nonce': 29475}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6830', 'nonce': 29475}, {'type': 'v1', 'addr': '172.21.15.190:6831', 'nonce': 29475}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6828', 'nonce': 29475}, {'type': 'v1', 'addr': '172.21.15.190:6829', 'nonce': 29475}]}, 'public_addr': '172.21.15.190:6825/29475', 'cluster_addr': '172.21.15.190:6827/29475', 'heartbeat_back_addr': '172.21.15.190:6831/29475', 'heartbeat_front_addr': '172.21.15.190:6829/29475', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.943165+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.946049+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.981045+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:14.019932+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.936197+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.982429+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.953833+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:47:18.853134+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.996438+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.926618+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.993934+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:14.022484+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'lrcprofile': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'k': '4', 'l': '3', 'm': '2', 'name': 'lrcprofile', 'plugin': 'lrc'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 21, 'length': 1}, {'begin': 26, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-15T03:47:19.321 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:47:19.322 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:19.318+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:19.420 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:19.419+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:19.425 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:19.425 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:19.433 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:47:19.521 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:19.520+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:19.574 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:19.575 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:19.583 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:47:19.622 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:19.620+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:19.634 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:47:19.635 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-15T03:47:19.645 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2024-09-15T03:47:18.925341+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":75940403,"num_objects":118,"num_object_clones":64,"num_object_copies":920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118,"num_whiteouts":9,"num_read":327,"num_read_kb":61449,"num_write":548,"num_write_kb":78120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":140558355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":677,"ondisk_log_size":677,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":150,"num_osds":11,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":1030475776,"kb_used":9105200,"kb_used_data":1021399368,"kb_used_omap":21098,"kb_used_meta":0,"kb_avail":1021370576,"statfs":{"total":1055207194624,"available":1045883469824,"internally_reserved":29483008,"allocated":1045912952832,"data_stored":1045912952832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21604959,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":25,"commit_latency_ns":0,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14368768,"num_objects":10,"num_object_clones":9,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":-1,"num_read":18,"num_read_kb":11863,"num_write":53,"num_write_kb":14384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":89292800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.092892"},"pg_stats":[{"pgid":"3.f","version":"69'34","reported_seq":226,"reported_epoch":69,"state":"active+undersized+remapped","last_fresh":"2024-09-15T03:47:16.058733+0000","last_change":"2024-09-15T03:47:16.059096+0000","last_active":"2024-09-15T03:47:16.058733+0000","last_peered":"2024-09-15T03:47:16.058733+0000","last_clean":"2024-09-15T03:47:12.918694+0000","last_became_active":"2024-09-15T03:47:15.892094+0000","last_became_peered":"2024-09-15T03:47:15.892094+0000","last_unstale":"2024-09-15T03:47:16.058733+0000","last_undegraded":"2024-09-15T03:47:16.058733+0000","last_fullsized":"2024-09-15T03:47:15.835895+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:46.805002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032919603999999998,"stat_sum":{"num_bytes":4227072,"num_objects":8,"num_object_clones":5,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":9994240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0,7,2,8,4,10],"acting":[11,1,0,7,2,8,4,2147483647],"avail_no_missing":["11(0)","0(2)","1(1)","2(4)","4(6)","7(3)","8(5)","9(7)","10(7)"],"object_location_counts":[{"shards":"0(2),1(1),2(4),4(6),7(3),8(5),11(0)","objects":8}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.e","version":"65'74","reported_seq":196,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-15T03:47:14.847468+0000","last_change":"2024-09-15T03:47:08.906305+0000","last_active":"2024-09-15T03:47:14.847468+0000","last_peered":"2024-09-15T03:47:14.847468+0000","last_clean":"2024-09-15T03:47:14.847468+0000","last_became_active":"2024-09-15T03:46:36.170536+0000","last_became_peered":"2024-09-15T03:46:36.170536+0000","last_unstale":"2024-09-15T03:47:14.847468+0000","last_undegraded":"2024-09-15T03:47:14.847468+0000","last_fullsized":"2024-09-15T03:47:14.847468+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:40.141809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016129392999999999,"stat_sum":{"num_bytes":10813440,"num_objects":16,"num_object_clones":9,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":2,"num_read":42,"num_read_kb":11921,"num_write":37,"num_write_kb":10256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,11,7,8,5,9],"acting":[2,0,1,11,7,8,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.d","version":"66'26","reported_seq":178,"reported_epoch":69,"state":"active+undersized+remapped","last_fresh":"2024-09-15T03:47:15.993206+0000","last_change":"2024-09-15T03:47:15.993586+0000","last_active":"2024-09-15T03:47:15.993206+0000","last_peered":"2024-09-15T03:47:15.993206+0000","last_clean":"2024-09-15T03:47:12.926800+0000","last_became_active":"2024-09-15T03:47:15.892093+0000","last_became_peered":"2024-09-15T03:47:15.892093+0000","last_unstale":"2024-09-15T03:47:15.993206+0000","last_undegraded":"2024-09-15T03:47:15.993206+0000","last_fullsized":"2024-09-15T03:47:15.840845+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:52.572176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0098486059999999993,"stat_sum":{"num_bytes":5931008,"num_objects":5,"num_object_clones":3,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":18,"num_read_kb":8406,"num_write":15,"num_write_kb":5792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8044544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,1,3,10,7,9,4],"acting":[8,11,1,3,2147483647,7,9,4],"avail_no_missing":["8(0)","0(4)","1(2)","3(3)","4(7)","7(5)","9(6)","10(4)","11(1)"],"object_location_counts":[{"shards":"1(2),3(3),4(7),7(5),8(0),9(6),11(1)","objects":5}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.c","version":"58'14","reported_seq":179,"reported_epoch":69,"state":"active+undersized+remapped","last_fresh":"2024-09-15T03:47:16.029433+0000","last_change":"2024-09-15T03:47:16.029745+0000","last_active":"2024-09-15T03:47:16.029433+0000","last_peered":"2024-09-15T03:47:16.029433+0000","last_clean":"2024-09-15T03:47:12.917272+0000","last_became_active":"2024-09-15T03:47:15.892020+0000","last_became_peered":"2024-09-15T03:47:15.892020+0000","last_unstale":"2024-09-15T03:47:16.029433+0000","last_undegraded":"2024-09-15T03:47:16.029433+0000","last_fullsized":"2024-09-15T03:47:15.834036+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:44.993488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030910460000000001,"stat_sum":{"num_bytes":1720320,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":9,"num_read_kb":3460,"num_write":9,"num_write_kb":2224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3440640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,11,1,0,10,5,2],"acting":[9,7,11,1,0,2147483647,5,2],"avail_no_missing":["9(0)","0(4)","1(3)","2(7)","4(5)","5(6)","7(1)","10(5)","11(2)"],"object_location_counts":[{"shards":"0(4),1(3),2(7),5(6),7(1),9(0),11(2)","objects":3}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.b","version":"65'81","reported_seq":343,"reported_epoch":69,"state":"active+undersized+remapped","last_fresh":"2024-09-15T03:47:16.053421+0000","last_change":"2024-09-15T03:47:16.053739+0000","last_active":"2024-09-15T03:47:16.053421+0000","last_peered":"2024-09-15T03:47:16.053421+0000","last_clean":"2024-09-15T03:47:12.903860+0000","last_became_active":"2024-09-15T03:47:15.891993+0000","last_became_peered":"2024-09-15T03:47:15.891993+0000","last_unstale":"2024-09-15T03:47:16.053421+0000","last_undegraded":"2024-09-15T03:47:16.053421+0000","last_fullsized":"2024-09-15T03:47:15.834600+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:00.499964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012289705,"stat_sum":{"num_bytes":11681792,"num_objects":18,"num_object_clones":11,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":38,"num_read_kb":10143,"num_write":76,"num_write_kb":10688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":26148864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8,4,1,10,11,9],"acting":[7,3,8,4,1,2147483647,11,9],"avail_no_missing":["7(0)","1(4)","2(5)","3(1)","4(3)","8(2)","9(7)","10(5)","11(6)"],"object_location_counts":[{"shards":"1(4),3(1),4(3),7(0),8(2),9(7),11(6)","objects":18}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.a","version":"65'51","reported_seq":195,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-15T03:47:14.840677+0000","last_change":"2024-09-15T03:47:11.879632+0000","last_active":"2024-09-15T03:47:14.840677+0000","last_peered":"2024-09-15T03:47:14.840677+0000","last_clean":"2024-09-15T03:47:14.840677+0000","last_became_active":"2024-09-15T03:47:11.845457+0000","last_became_peered":"2024-09-15T03:47:11.845457+0000","last_unstale":"2024-09-15T03:47:14.840677+0000","last_undegraded":"2024-09-15T03:47:14.840677+0000","last_fullsized":"2024-09-15T03:47:14.840677+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:39.617129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033286889,"stat_sum":{"num_bytes":4620288,"num_objects":7,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":23,"num_write_kb":4512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,9,3,1,2,5],"acting":[11,8,7,9,3,1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.9","version":"69'27","reported_seq":217,"reported_epoch":69,"state":"active+undersized+remapped","last_fresh":"2024-09-15T03:47:16.055663+0000","last_change":"2024-09-15T03:47:16.056132+0000","last_active":"2024-09-15T03:47:16.055663+0000","last_peered":"2024-09-15T03:47:16.055663+0000","last_clean":"2024-09-15T03:47:12.903714+0000","last_became_active":"2024-09-15T03:47:15.895510+0000","last_became_peered":"2024-09-15T03:47:15.895510+0000","last_unstale":"2024-09-15T03:47:16.055663+0000","last_undegraded":"2024-09-15T03:47:15.836238+0000","last_fullsized":"2024-09-15T03:47:15.835430+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:35.874629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014071818,"stat_sum":{"num_bytes":3817472,"num_objects":7,"num_object_clones":4,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":9,"num_read_kb":5,"num_write":36,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":8224768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,10,1,9,4,8,11],"acting":[3,2,2147483647,1,9,4,8,11],"avail_no_missing":["3(0)","0(2)","1(3)","2(1)","4(5)","8(6)","9(4)","10(2)","11(7)"],"object_location_counts":[{"shards":"1(3),2(1),3(0),4(5),8(6),9(4),11(7)","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.8","version":"62'19","reported_seq":169,"reported_epoch":69,"state":"active+undersized+remapped","last_fresh":"2024-09-15T03:47:16.041537+0000","last_change":"2024-09-15T03:47:16.041859+0000","last_active":"2024-09-15T03:47:16.041537+0000","last_peered":"2024-09-15T03:47:16.041537+0000","last_clean":"2024-09-15T03:47:12.903476+0000","last_became_active":"2024-09-15T03:47:15.893351+0000","last_became_peered":"2024-09-15T03:47:15.893351+0000","last_unstale":"2024-09-15T03:47:16.041537+0000","last_undegraded":"2024-09-15T03:47:16.041537+0000","last_fullsized":"2024-09-15T03:47:15.834965+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:52.082203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016346408,"stat_sum":{"num_bytes":2408448,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":3654,"num_write":28,"num_write_kb":2352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3063808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,11,4,1,0,10],"acting":[3,7,2,11,4,1,0,2147483647],"avail_no_missing":["3(0)","0(6)","1(5)","2(2)","4(4)","5(7)","7(1)","10(7)","11(3)"],"object_location_counts":[{"shards":"0(6),1(5),2(2),3(0),4(4),7(1),11(3)","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"1.0","version":"20'92","reported_seq":169,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-15T03:47:14.842620+0000","last_change":"2024-09-15T03:46:18.011841+0000","last_active":"2024-09-15T03:47:14.842620+0000","last_peered":"2024-09-15T03:47:14.842620+0000","last_clean":"2024-09-15T03:47:14.842620+0000","last_became_active":"2024-09-15T03:46:18.011130+0000","last_became_peered":"2024-09-15T03:46:18.011130+0000","last_unstale":"2024-09-15T03:47:14.842620+0000","last_undegraded":"2024-09-15T03:47:14.842620+0000","last_fullsized":"2024-09-15T03:47:14.842620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_clean_scrub_stamp":"2024-09-15T03:46:16.886982+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:32.556840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"22'1","reported_seq":61,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-15T03:47:14.838155+0000","last_change":"2024-09-15T03:46:26.776623+0000","last_active":"2024-09-15T03:47:14.838155+0000","last_peered":"2024-09-15T03:47:14.838155+0000","last_clean":"2024-09-15T03:47:14.838155+0000","last_became_active":"2024-09-15T03:46:24.038633+0000","last_became_peered":"2024-09-15T03:46:24.038633+0000","last_unstale":"2024-09-15T03:47:14.838155+0000","last_undegraded":"2024-09-15T03:47:14.838155+0000","last_fullsized":"2024-09-15T03:47:14.838155+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:45.125997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030178200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"65'27","reported_seq":237,"reported_epoch":69,"state":"active+undersized+remapped","last_fresh":"2024-09-15T03:47:15.997728+0000","last_change":"2024-09-15T03:47:15.998023+0000","last_active":"2024-09-15T03:47:15.997728+0000","last_peered":"2024-09-15T03:47:15.997728+0000","last_clean":"2024-09-15T03:47:12.917304+0000","last_became_active":"2024-09-15T03:47:15.889368+0000","last_became_peered":"2024-09-15T03:47:15.889368+0000","last_unstale":"2024-09-15T03:47:15.997728+0000","last_undegraded":"2024-09-15T03:47:15.997728+0000","last_fullsized":"2024-09-15T03:47:15.835491+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:39.085298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097624440000000003,"stat_sum":{"num_bytes":2506752,"num_objects":6,"num_object_clones":3,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":16,"num_read_kb":4134,"num_write":14,"num_write_kb":2528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":7831552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,7,8,9,11,1],"acting":[4,2,2147483647,7,8,2147483647,11,1],"avail_no_missing":["4(0)","0(5)","1(2)","1(7)","2(1)","7(3)","8(4)","9(5)","9(7)","10(2)","11(6)"],"object_location_counts":[{"shards":"1(7),2(1),4(0),7(3),8(4),11(6)","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":58,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-15T03:47:14.837689+0000","last_change":"2024-09-15T03:46:26.044503+0000","last_active":"2024-09-15T03:47:14.837689+0000","last_peered":"2024-09-15T03:47:14.837689+0000","last_clean":"2024-09-15T03:47:14.837689+0000","last_became_active":"2024-09-15T03:46:24.063106+0000","last_became_peered":"2024-09-15T03:46:24.063106+0000","last_unstale":"2024-09-15T03:47:14.837689+0000","last_undegraded":"2024-09-15T03:47:14.837689+0000","last_fullsized":"2024-09-15T03:47:14.837689+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:24.101828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000393075,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"32'2","reported_seq":90,"reported_epoch":69,"state":"active+undersized+remapped","last_fresh":"2024-09-15T03:47:16.015730+0000","last_change":"2024-09-15T03:47:16.016238+0000","last_active":"2024-09-15T03:47:16.015730+0000","last_peered":"2024-09-15T03:47:16.015730+0000","last_clean":"2024-09-15T03:47:12.917256+0000","last_became_active":"2024-09-15T03:47:15.892074+0000","last_became_peered":"2024-09-15T03:47:15.892074+0000","last_unstale":"2024-09-15T03:47:16.015730+0000","last_undegraded":"2024-09-15T03:47:16.015730+0000","last_fullsized":"2024-09-15T03:47:15.836692+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:36.160697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001566914,"stat_sum":{"num_bytes":524288,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1537,"num_write":2,"num_write_kb":512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":524288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,3,10,5,2,9,7],"acting":[0,11,3,2147483647,5,2,9,7],"avail_no_missing":["0(0)","2(5)","3(2)","5(4)","7(7)","8(3)","9(6)","10(3)","11(1)"],"object_location_counts":[{"shards":"0(0),2(5),3(2),5(4),7(7),9(6),11(1)","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":60,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-15T03:47:14.840803+0000","last_change":"2024-09-15T03:46:26.042740+0000","last_active":"2024-09-15T03:47:14.840803+0000","last_peered":"2024-09-15T03:47:14.840803+0000","last_clean":"2024-09-15T03:47:14.840803+0000","last_became_active":"2024-09-15T03:46:24.037471+0000","last_became_peered":"2024-09-15T03:46:24.037471+0000","last_unstale":"2024-09-15T03:47:14.840803+0000","last_undegraded":"2024-09-15T03:47:14.840803+0000","last_fullsized":"2024-09-15T03:47:14.840803+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:29.654520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026490299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"47'26","reported_seq":231,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-15T03:47:17.942688+0000","last_change":"2024-09-15T03:47:17.942688+0000","last_active":"2024-09-15T03:47:17.942688+0000","last_peered":"2024-09-15T03:47:17.942688+0000","last_clean":"2024-09-15T03:47:17.942688+0000","last_became_active":"2024-09-15T03:47:17.937055+0000","last_became_peered":"2024-09-15T03:47:17.937055+0000","last_unstale":"2024-09-15T03:47:17.942688+0000","last_undegraded":"2024-09-15T03:47:17.942688+0000","last_fullsized":"2024-09-15T03:47:17.942688+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:55.089176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001461019,"stat_sum":{"num_bytes":2752512,"num_objects":5,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":13,"num_read_kb":2070,"num_write":15,"num_write_kb":2688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9207808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,9,4,3,8,0,10],"acting":[1,2,9,4,3,8,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.2","version":"24'2","reported_seq":82,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-15T03:47:14.837594+0000","last_change":"2024-09-15T03:47:11.841507+0000","last_active":"2024-09-15T03:47:14.837594+0000","last_peered":"2024-09-15T03:47:14.837594+0000","last_clean":"2024-09-15T03:47:14.837594+0000","last_became_active":"2024-09-15T03:47:11.841215+0000","last_became_peered":"2024-09-15T03:47:11.841215+0000","last_unstale":"2024-09-15T03:47:14.837594+0000","last_undegraded":"2024-09-15T03:47:14.837594+0000","last_fullsized":"2024-09-15T03:47:14.837594+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:51.577501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000944677,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"65'74","reported_seq":231,"reported_epoch":69,"state":"active+undersized+remapped","last_fresh":"2024-09-15T03:47:16.044943+0000","last_change":"2024-09-15T03:47:16.045411+0000","last_active":"2024-09-15T03:47:16.044943+0000","last_peered":"2024-09-15T03:47:16.044943+0000","last_clean":"2024-09-15T03:47:11.914398+0000","last_became_active":"2024-09-15T03:47:15.892194+0000","last_became_peered":"2024-09-15T03:47:15.892194+0000","last_unstale":"2024-09-15T03:47:16.044943+0000","last_undegraded":"2024-09-15T03:47:16.044943+0000","last_fullsized":"2024-09-15T03:47:15.840836+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:42.648657+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.042282048000000003,"stat_sum":{"num_bytes":8404992,"num_objects":13,"num_object_clones":7,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":20,"num_read_kb":9226,"num_write":75,"num_write_kb":9584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":12795904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,4,7,1,2,9,0],"acting":[2147483647,8,4,7,1,2,9,0],"avail_no_missing":["8(1)","0(7)","1(4)","2(5)","4(2)","7(3)","9(6)","10(0)","11(0)"],"object_location_counts":[{"shards":"0(7),1(4),2(5),4(2),7(3),8(1),9(6)","objects":13}],"blocked_by":[],"up_primary":10,"acting_primary":8,"purged_snaps":[{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.5","version":"65'35","reported_seq":197,"reported_epoch":69,"state":"active+undersized+remapped","last_fresh":"2024-09-15T03:47:16.013706+0000","last_change":"2024-09-15T03:47:16.014130+0000","last_active":"2024-09-15T03:47:16.013706+0000","last_peered":"2024-09-15T03:47:16.013706+0000","last_clean":"2024-09-15T03:47:11.874747+0000","last_became_active":"2024-09-15T03:47:15.891795+0000","last_became_peered":"2024-09-15T03:47:15.891795+0000","last_unstale":"2024-09-15T03:47:16.013706+0000","last_undegraded":"2024-09-15T03:47:16.013706+0000","last_fullsized":"2024-09-15T03:47:15.837153+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:53.214928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059856140000000002,"stat_sum":{"num_bytes":5210112,"num_objects":4,"num_object_clones":3,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":15,"num_write_kb":5088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":23789568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4,9,8,7,3,5],"acting":[2147483647,0,4,9,8,7,3,5],"avail_no_missing":["0(1)","1(0)","3(6)","4(2)","5(7)","7(5)","8(4)","9(3)","10(0)"],"object_location_counts":[{"shards":"0(1),3(6),4(2),5(7),7(5),8(4),9(3)","objects":4}],"blocked_by":[],"up_primary":10,"acting_primary":0,"purged_snaps":[{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":69,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-15T03:47:14.833641+0000","last_change":"2024-09-15T03:47:08.903628+0000","last_active":"2024-09-15T03:47:14.833641+0000","last_peered":"2024-09-15T03:47:14.833641+0000","last_clean":"2024-09-15T03:47:14.833641+0000","last_became_active":"2024-09-15T03:47:08.903372+0000","last_became_peered":"2024-09-15T03:47:08.903372+0000","last_unstale":"2024-09-15T03:47:14.833641+0000","last_undegraded":"2024-09-15T03:47:14.833641+0000","last_fullsized":"2024-09-15T03:47:14.833641+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:34.437807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034123999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"65'26","reported_seq":184,"reported_epoch":69,"state":"active+undersized+remapped","last_fresh":"2024-09-15T03:47:16.011066+0000","last_change":"2024-09-15T03:47:16.011505+0000","last_active":"2024-09-15T03:47:16.011066+0000","last_peered":"2024-09-15T03:47:16.011066+0000","last_clean":"2024-09-15T03:47:12.903613+0000","last_became_active":"2024-09-15T03:47:15.892780+0000","last_became_peered":"2024-09-15T03:47:15.892780+0000","last_unstale":"2024-09-15T03:47:16.011066+0000","last_undegraded":"2024-09-15T03:47:16.011066+0000","last_fullsized":"2024-09-15T03:47:15.834753+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:36.080195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014140376,"stat_sum":{"num_bytes":1703936,"num_objects":3,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":1059,"num_write":9,"num_write_kb":2400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2424832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,8,0,4,7,10],"acting":[3,2,5,8,0,4,7,2147483647],"avail_no_missing":["3(0)","0(4)","1(7)","2(1)","4(5)","5(2)","7(6)","8(3)","10(7)"],"object_location_counts":[{"shards":"0(4),2(1),3(0),4(5),5(2),7(6),8(3)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":59,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-15T03:47:14.837581+0000","last_change":"2024-09-15T03:46:26.042111+0000","last_active":"2024-09-15T03:47:14.837581+0000","last_peered":"2024-09-15T03:47:14.837581+0000","last_clean":"2024-09-15T03:47:14.837581+0000","last_became_active":"2024-09-15T03:46:24.038945+0000","last_became_peered":"2024-09-15T03:46:24.038945+0000","last_unstale":"2024-09-15T03:47:14.837581+0000","last_undegraded":"2024-09-15T03:47:14.837581+0000","last_fullsized":"2024-09-15T03:47:14.837581+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:27.950416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032787499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"69'50","reported_seq":215,"reported_epoch":69,"state":"active+undersized+remapped","last_fresh":"2024-09-15T03:47:16.015511+0000","last_change":"2024-09-15T03:47:16.015831+0000","last_active":"2024-09-15T03:47:16.015511+0000","last_peered":"2024-09-15T03:47:16.015511+0000","last_clean":"2024-09-15T03:47:12.917423+0000","last_became_active":"2024-09-15T03:47:15.892204+0000","last_became_peered":"2024-09-15T03:47:15.892204+0000","last_unstale":"2024-09-15T03:47:16.015511+0000","last_undegraded":"2024-09-15T03:47:16.015511+0000","last_fullsized":"2024-09-15T03:47:15.835654+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:44.711735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097263309999999995,"stat_sum":{"num_bytes":7700480,"num_objects":10,"num_object_clones":5,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":26,"num_read_kb":5242,"num_write":46,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9666560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,7,2,1,11,10],"acting":[4,3,9,7,2,1,11,2147483647],"avail_no_missing":["4(0)","0(7)","1(5)","2(4)","3(1)","7(3)","9(2)","10(7)","11(6)"],"object_location_counts":[{"shards":"1(5),2(4),3(1),4(0),7(3),9(2),11(6)","objects":10}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":60,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-15T03:47:14.836907+0000","last_change":"2024-09-15T03:46:26.776405+0000","last_active":"2024-09-15T03:47:14.836907+0000","last_peered":"2024-09-15T03:47:14.836907+0000","last_clean":"2024-09-15T03:47:14.836907+0000","last_became_active":"2024-09-15T03:46:24.063380+0000","last_became_peered":"2024-09-15T03:46:24.063380+0000","last_unstale":"2024-09-15T03:47:14.836907+0000","last_undegraded":"2024-09-15T03:47:14.836907+0000","last_fullsized":"2024-09-15T03:47:14.836907+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:24.792912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034490600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"47'16","reported_seq":288,"reported_epoch":69,"state":"active+undersized+remapped","last_fresh":"2024-09-15T03:47:15.969519+0000","last_change":"2024-09-15T03:47:15.969900+0000","last_active":"2024-09-15T03:47:15.969519+0000","last_peered":"2024-09-15T03:47:15.969519+0000","last_clean":"2024-09-15T03:47:12.917367+0000","last_became_active":"2024-09-15T03:47:15.892309+0000","last_became_peered":"2024-09-15T03:47:15.892309+0000","last_unstale":"2024-09-15T03:47:15.969519+0000","last_undegraded":"2024-09-15T03:47:15.969519+0000","last_fullsized":"2024-09-15T03:47:15.837449+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:01.519540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023525909999999998,"stat_sum":{"num_bytes":1327104,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":515,"num_write":7,"num_write_kb":1296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":7012352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,9,8,3,11,1,10],"acting":[4,7,9,8,3,2147483647,1,10],"avail_no_missing":["4(0)","1(6)","2(5)","3(4)","7(1)","8(3)","9(2)","10(7)","11(5)","11(7)"],"object_location_counts":[{"shards":"1(6),3(4),4(0),7(1),8(3),9(2),10(7)","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":58,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-15T03:47:14.841949+0000","last_change":"2024-09-15T03:46:26.044851+0000","last_active":"2024-09-15T03:47:14.841949+0000","last_peered":"2024-09-15T03:47:14.841949+0000","last_clean":"2024-09-15T03:47:14.841949+0000","last_became_active":"2024-09-15T03:46:24.063335+0000","last_became_peered":"2024-09-15T03:46:24.063335+0000","last_unstale":"2024-09-15T03:47:14.841949+0000","last_undegraded":"2024-09-15T03:47:14.841949+0000","last_fullsized":"2024-09-15T03:47:14.841949+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:34.631264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065928000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":75350016,"num_objects":114,"num_object_clones":64,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":9,"num_read":251,"num_read_kb":61385,"num_write":423,"num_write_kb":75968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":140558336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":582,"ondisk_log_size":582,"up":128,"acting":128,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":11,"up_from":17,"seq":73014444046,"num_pgs":15,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":825916,"kb_used_data":92853828,"kb_used_omap":1950,"kb_used_meta":0,"kb_avail":92853700,"statfs":{"total":95927926784,"available":95082188800,"internally_reserved":131072,"allocated":95082319872,"data_stored":95082319872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1997269,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]}]},{"osd":10,"up_from":17,"seq":73014444046,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":845160,"kb_used_data":92858984,"kb_used_omap":1135,"kb_used_meta":0,"kb_avail":92834456,"statfs":{"total":95927926784,"available":95062482944,"internally_reserved":25116672,"allocated":95087599616,"data_stored":95087599616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1162261,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":12,"commit_latency_ns":0,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]}]},{"osd":9,"up_from":17,"seq":73014444046,"num_pgs":15,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":830736,"kb_used_data":92849020,"kb_used_omap":2113,"kb_used_meta":0,"kb_avail":92848880,"statfs":{"total":95927926784,"available":95077253120,"internally_reserved":143360,"allocated":95077396480,"data_stored":95077396480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2164054,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]}]},{"osd":8,"up_from":17,"seq":73014444046,"num_pgs":17,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":827224,"kb_used_data":92852536,"kb_used_omap":2178,"kb_used_meta":0,"kb_avail":92852392,"statfs":{"total":95927926784,"available":95080849408,"internally_reserved":147456,"allocated":95080996864,"data_stored":95080996864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2231180,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]}]},{"osd":7,"up_from":17,"seq":73014444046,"num_pgs":15,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":829228,"kb_used_data":92850640,"kb_used_omap":2355,"kb_used_meta":0,"kb_avail":92850388,"statfs":{"total":95927926784,"available":95078797312,"internally_reserved":258048,"allocated":95079055360,"data_stored":95079055360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2411617,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.58099999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.63300000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.59699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.51700000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.61599999999999999}]},{"osd":9,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.70099999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.67500000000000004}]},{"osd":11,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.68600000000000005}]}]},{"osd":6,"up_from":0,"seq":73014444046,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":17,"seq":73014444046,"num_pgs":10,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":811296,"kb_used_data":92868464,"kb_used_omap":927,"kb_used_meta":0,"kb_avail":92868320,"statfs":{"total":95927926784,"available":95097159680,"internally_reserved":147456,"allocated":95097307136,"data_stored":95097307136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":950157,"internal_metadata":0},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]}]},{"osd":4,"up_from":17,"seq":73014444046,"num_pgs":15,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":826028,"kb_used_data":92853732,"kb_used_omap":2096,"kb_used_meta":0,"kb_avail":92853588,"statfs":{"total":95927926784,"available":95082074112,"internally_reserved":147456,"allocated":95082221568,"data_stored":95082221568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2146860,"internal_metadata":0},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]}]},{"osd":3,"up_from":17,"seq":73014444046,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":823436,"kb_used_data":92857108,"kb_used_omap":1615,"kb_used_meta":0,"kb_avail":92856180,"statfs":{"total":95927926784,"available":95084728320,"internally_reserved":950272,"allocated":95085678592,"data_stored":95085678592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1654613,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]}]},{"osd":2,"up_from":17,"seq":73014444046,"num_pgs":15,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":824852,"kb_used_data":92854908,"kb_used_omap":2167,"kb_used_meta":0,"kb_avail":92854764,"statfs":{"total":95927926784,"available":95083278336,"internally_reserved":147456,"allocated":95083425792,"data_stored":95083425792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2219703,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.247}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]}]},{"osd":1,"up_from":17,"seq":73014444047,"num_pgs":17,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":836808,"kb_used_data":92844908,"kb_used_omap":2539,"kb_used_meta":0,"kb_avail":92842808,"statfs":{"total":95927926784,"available":95071035392,"internally_reserved":2150400,"allocated":95073185792,"data_stored":95073185792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2600586,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.379},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.35899999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.40000000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.44600000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":2.6920000000000002,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":0.54600000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.7440000000000002,"5min":2.7440000000000002,"15min":2.7440000000000002},"last":0.60299999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":3.278,"5min":3.278,"15min":3.278},"last":0.65300000000000002}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.72099999999999997}]},{"osd":8,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":119.93899999999999,"5min":119.93899999999999,"15min":119.93899999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2861.7190000000001,"5min":2861.7190000000001,"15min":2861.7190000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.946,"5min":1.946,"15min":1.946},"last":0.68799999999999994}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":119.937,"5min":119.937,"15min":119.937},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":2861.6010000000001,"5min":2861.6010000000001,"15min":2861.6010000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.70099999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":119.96599999999999,"5min":119.96599999999999,"15min":119.96599999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2861.5639999999999,"5min":2861.5639999999999,"15min":2861.5639999999999},"last":0.73199999999999998}]}]},{"osd":0,"up_from":17,"seq":73014444046,"num_pgs":17,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":824516,"kb_used_data":92855240,"kb_used_omap":2018,"kb_used_meta":0,"kb_avail":92855100,"statfs":{"total":95927926784,"available":95083622400,"internally_reserved":143360,"allocated":95083765760,"data_stored":95083765760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2066659,"internal_metadata":0},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]}]}],"pool_statfs":[]}} 2024-09-15T03:47:19.650 INFO:tasks.thrashosds.thrasher:Setting 3.a to [4, 10, 8, 9, 5, 6, 2, 0] 2024-09-15T03:47:19.650 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.a', '4', '10', '8', '9', '5', '6', '2', '0'] 2024-09-15T03:47:19.651 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.a 4 10 8 9 5 6 2 0 2024-09-15T03:47:19.683 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:19.683 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:19.691 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:47:19.722 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:19.721+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:19.792 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:19.792 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:19.801 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:47:19.823 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:19.822+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:19.833 INFO:tasks.rados.rados.0.smithi001.stdout:210: done (0 left) 2024-09-15T03:47:19.834 INFO:tasks.rados.rados.0.smithi001.stdout:211: read oid 23 snap 1 2024-09-15T03:47:19.834 INFO:tasks.rados.rados.0.smithi001.stdout:211: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-15T03:47:19.834 INFO:tasks.rados.rados.0.smithi001.stdout:212: read oid 47 snap -1 2024-09-15T03:47:19.834 INFO:tasks.rados.rados.0.smithi001.stdout:212: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-15T03:47:19.834 INFO:tasks.rados.rados.0.smithi001.stdout:213: copy_from oid 32 from oid 48 current snap is 27 2024-09-15T03:47:19.834 INFO:tasks.rados.rados.0.smithi001.stdout:214: read oid 41 snap 0 2024-09-15T03:47:19.834 INFO:tasks.rados.rados.0.smithi001.stdout:214: expect deleted 2024-09-15T03:47:19.834 INFO:tasks.rados.rados.0.smithi001.stdout:215: rollback oid 49 current snap is 27 2024-09-15T03:47:19.834 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 49 to 22 2024-09-15T03:47:19.834 INFO:tasks.rados.rados.0.smithi001.stdout:216: setattr oid 8 current snap is 27 2024-09-15T03:47:19.835 INFO:tasks.rados.rados.0.smithi001.stdout:217: setattr oid 15 current snap is 27 2024-09-15T03:47:19.836 INFO:tasks.rados.rados.0.smithi001.stdout:218: setattr oid 40 current snap is 27 2024-09-15T03:47:19.837 INFO:tasks.rados.rados.0.smithi001.stdout:214: done (7 left) 2024-09-15T03:47:19.837 INFO:tasks.rados.rados.0.smithi001.stdout:219: delete oid 5 current snap is 27 2024-09-15T03:47:19.843 INFO:tasks.rados.rados.0.smithi001.stdout:211: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-15T03:47:19.858 INFO:tasks.rados.rados.0.smithi001.stdout:215: finishing rollback tid 0 to smithi00132172-49 2024-09-15T03:47:19.858 INFO:tasks.rados.rados.0.smithi001.stdout:215: finishing rollback tid 1 to smithi00132172-49 2024-09-15T03:47:19.858 INFO:tasks.rados.rados.0.smithi001.stdout:211: done (7 left) 2024-09-15T03:47:19.858 INFO:tasks.rados.rados.0.smithi001.stdout:219: done (6 left) 2024-09-15T03:47:19.858 INFO:tasks.rados.rados.0.smithi001.stdout:220: read oid 50 snap -1 2024-09-15T03:47:19.858 INFO:tasks.rados.rados.0.smithi001.stdout:220: expect (ObjNum 72 snap 12 seq_num 72) 2024-09-15T03:47:19.858 INFO:tasks.rados.rados.0.smithi001.stdout:221: delete oid 31 current snap is 27 2024-09-15T03:47:19.858 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 8 v 18 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-15T03:47:19.858 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 15 v 83 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-15T03:47:19.858 INFO:tasks.rados.rados.0.smithi001.stdout:213: finishing copy_from racing read to smithi00132172-32 2024-09-15T03:47:19.858 INFO:tasks.rados.rados.0.smithi001.stdout:215: finishing rollback tid 2 to smithi00132172-49 2024-09-15T03:47:19.858 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 49 v 30 (ObjNum 79 snap 16 seq_num 79) dirty exists 2024-09-15T03:47:19.859 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 40 v 85 (ObjNum 92 snap 21 seq_num 92) dirty exists 2024-09-15T03:47:19.859 INFO:tasks.rados.rados.0.smithi001.stdout:212: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-15T03:47:19.881 INFO:tasks.rados.rados.0.smithi001.stdout:213: finishing copy_from to smithi00132172-32 2024-09-15T03:47:19.881 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 32 v 49 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-15T03:47:19.881 INFO:tasks.rados.rados.0.smithi001.stdout:220: expect (ObjNum 72 snap 12 seq_num 72) 2024-09-15T03:47:19.913 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:19.914 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:19.922 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:47:19.924 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:19.923+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:19.952 INFO:tasks.rados.rados.0.smithi001.stdout:212: done (7 left) 2024-09-15T03:47:19.953 INFO:tasks.rados.rados.0.smithi001.stdout:213: done (6 left) 2024-09-15T03:47:19.953 INFO:tasks.rados.rados.0.smithi001.stdout:215: done (5 left) 2024-09-15T03:47:19.953 INFO:tasks.rados.rados.0.smithi001.stdout:216: done (4 left) 2024-09-15T03:47:19.953 INFO:tasks.rados.rados.0.smithi001.stdout:217: done (3 left) 2024-09-15T03:47:19.953 INFO:tasks.rados.rados.0.smithi001.stdout:218: done (2 left) 2024-09-15T03:47:19.954 INFO:tasks.rados.rados.0.smithi001.stdout:220: done (1 left) 2024-09-15T03:47:19.954 INFO:tasks.rados.rados.0.smithi001.stdout:221: done (0 left) 2024-09-15T03:47:19.954 INFO:tasks.rados.rados.0.smithi001.stdout:222: snap_create 2024-09-15T03:47:19.954 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T03:47:20.022 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:20.023 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:20.025 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:20.024+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:20.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:47:20.051 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:47:20.126 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:20.124+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:20.169 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:20.170 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:20.171 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:20.171 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:20.180 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:47:20.183 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:20.184 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:20.199 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:20.200 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:20.213 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:20.214 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:20.227 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:20.226+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:20.228 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:20.228 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:20.242 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:20.243 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:20.257 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:20.257 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:20.272 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:20.272 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:20.285 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:20.285 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:20.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:20.285 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:20.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.285 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=21, reqid=client.4521.0:184, at_version=33'6, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:20.285 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.176130626000003, 2024-09-15T03:47:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010901256999999999, 2024-09-15T03:47:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0200000000000001e-07 2024-09-15T03:47:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106529+0000", 2024-09-15T03:47:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9539e-05 2024-09-15T03:47:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106542+0000", 2024-09-15T03:47:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2731999999999999e-05 2024-09-15T03:47:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106549+0000", 2024-09-15T03:47:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3339999999999996e-06 2024-09-15T03:47:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106613+0000", 2024-09-15T03:47:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4527000000000001e-05 2024-09-15T03:47:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106622+0000", 2024-09-15T03:47:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.755e-06 2024-09-15T03:47:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106841+0000", 2024-09-15T03:47:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021906200000000001 2024-09-15T03:47:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106869+0000", 2024-09-15T03:47:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.739e-05 2024-09-15T03:47:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106948+0000", 2024-09-15T03:47:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9422e-05 2024-09-15T03:47:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106965+0000", 2024-09-15T03:47:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7010000000000001e-05 2024-09-15T03:47:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.117410+0000", 2024-09-15T03:47:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010445284000000001 2024-09-15T03:47:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.174217139, 2024-09-15T03:47:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015296944, 2024-09-15T03:47:20.292 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.292 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:20.292 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5499999999999998e-07 2024-09-15T03:47:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108447+0000", 2024-09-15T03:47:20.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3314000000000001e-05 2024-09-15T03:47:20.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108460+0000", 2024-09-15T03:47:20.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3444e-05 2024-09-15T03:47:20.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108464+0000", 2024-09-15T03:47:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3610000000000003e-06 2024-09-15T03:47:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108494+0000", 2024-09-15T03:47:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9688000000000001e-05 2024-09-15T03:47:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108502+0000", 2024-09-15T03:47:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.678e-06 2024-09-15T03:47:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108601+0000", 2024-09-15T03:47:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9748999999999997e-05 2024-09-15T03:47:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108610+0000", 2024-09-15T03:47:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2490000000000001e-06 2024-09-15T03:47:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109977+0000", 2024-09-15T03:47:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013669450000000001 2024-09-15T03:47:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109993+0000", 2024-09-15T03:47:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6688e-05 2024-09-15T03:47:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123720+0000", 2024-09-15T03:47:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013726172999999999 2024-09-15T03:47:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.174167478000001, 2024-09-15T03:47:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015105210000000001, 2024-09-15T03:47:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108473+0000", 2024-09-15T03:47:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.84e-07 2024-09-15T03:47:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108486+0000", 2024-09-15T03:47:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3845999999999999e-05 2024-09-15T03:47:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108491+0000", 2024-09-15T03:47:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4719999999999997e-06 2024-09-15T03:47:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108495+0000", 2024-09-15T03:47:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.822e-06 2024-09-15T03:47:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108546+0000", 2024-09-15T03:47:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0853000000000003e-05 2024-09-15T03:47:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108555+0000", 2024-09-15T03:47:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3319999999999998e-06 2024-09-15T03:47:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108638+0000", 2024-09-15T03:47:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3580999999999998e-05 2024-09-15T03:47:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109983+0000", 2024-09-15T03:47:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001344392 2024-09-15T03:47:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110023+0000", 2024-09-15T03:47:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9895999999999999e-05 2024-09-15T03:47:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110040+0000", 2024-09-15T03:47:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.747e-05 2024-09-15T03:47:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123577+0000", 2024-09-15T03:47:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013537262 2024-09-15T03:47:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.174133599000001, 2024-09-15T03:47:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015311313999999999, 2024-09-15T03:47:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8500000000000002e-07 2024-09-15T03:47:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108517+0000", 2024-09-15T03:47:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0543000000000001e-05 2024-09-15T03:47:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108524+0000", 2024-09-15T03:47:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7170000000000002e-06 2024-09-15T03:47:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108528+0000", 2024-09-15T03:47:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3970000000000004e-06 2024-09-15T03:47:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108632+0000", 2024-09-15T03:47:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103687 2024-09-15T03:47:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108637+0000", 2024-09-15T03:47:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4389999999999999e-06 2024-09-15T03:47:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108713+0000", 2024-09-15T03:47:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6223000000000005e-05 2024-09-15T03:47:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109984+0000", 2024-09-15T03:47:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012700719999999999 2024-09-15T03:47:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110023+0000", 2024-09-15T03:47:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9864999999999999e-05 2024-09-15T03:47:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110041+0000", 2024-09-15T03:47:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8040999999999999e-05 2024-09-15T03:47:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123817+0000", 2024-09-15T03:47:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013776045000000001 2024-09-15T03:47:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.173639991000002, 2024-09-15T03:47:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014864502, 2024-09-15T03:47:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:47:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109016+0000", 2024-09-15T03:47:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5889999999999999e-05 2024-09-15T03:47:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109029+0000", 2024-09-15T03:47:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2812e-05 2024-09-15T03:47:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109035+0000", 2024-09-15T03:47:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0109999999999997e-06 2024-09-15T03:47:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109073+0000", 2024-09-15T03:47:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8260000000000003e-05 2024-09-15T03:47:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109080+0000", 2024-09-15T03:47:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0190000000000001e-06 2024-09-15T03:47:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109165+0000", 2024-09-15T03:47:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4957000000000007e-05 2024-09-15T03:47:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109984+0000", 2024-09-15T03:47:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081866399999999998 2024-09-15T03:47:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110024+0000", 2024-09-15T03:47:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9644999999999997e-05 2024-09-15T03:47:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110072+0000", 2024-09-15T03:47:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8418999999999998e-05 2024-09-15T03:47:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123864+0000", 2024-09-15T03:47:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013792234 2024-09-15T03:47:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=27, reqid=client.4521.0:189, at_version=0'0, trim_to=0'0, roll_forward_to=33'10))", 2024-09-15T03:47:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.114624+0000", 2024-09-15T03:47:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.168016002000002, 2024-09-15T03:47:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092946850000000004, 2024-09-15T03:47:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114624+0000", 2024-09-15T03:47:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114624+0000", 2024-09-15T03:47:20.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114624+0000", 2024-09-15T03:47:20.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3600000000000004e-07 2024-09-15T03:47:20.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114633+0000", 2024-09-15T03:47:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1579999999999994e-06 2024-09-15T03:47:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114644+0000", 2024-09-15T03:47:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.04e-05 2024-09-15T03:47:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114649+0000", 2024-09-15T03:47:20.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8760000000000004e-06 2024-09-15T03:47:20.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114687+0000", 2024-09-15T03:47:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8438e-05 2024-09-15T03:47:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114697+0000", 2024-09-15T03:47:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6479999999999998e-06 2024-09-15T03:47:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114818+0000", 2024-09-15T03:47:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121116 2024-09-15T03:47:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114830+0000", 2024-09-15T03:47:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1844e-05 2024-09-15T03:47:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114921+0000", 2024-09-15T03:47:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1461999999999999e-05 2024-09-15T03:47:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114934+0000", 2024-09-15T03:47:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2857000000000001e-05 2024-09-15T03:47:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123918+0000", 2024-09-15T03:47:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089843500000000003 2024-09-15T03:47:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=34, reqid=client.4521.0:197, at_version=34'21, trim_to=0'0, roll_forward_to=34'18))", 2024-09-15T03:47:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.158628110999999, 2024-09-15T03:47:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012992399, 2024-09-15T03:47:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124013+0000", 2024-09-15T03:47:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6299999999999993e-07 2024-09-15T03:47:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124044+0000", 2024-09-15T03:47:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0995999999999999e-05 2024-09-15T03:47:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124062+0000", 2024-09-15T03:47:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8413000000000001e-05 2024-09-15T03:47:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124069+0000", 2024-09-15T03:47:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3930000000000002e-06 2024-09-15T03:47:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124150+0000", 2024-09-15T03:47:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1097e-05 2024-09-15T03:47:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124161+0000", 2024-09-15T03:47:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0720000000000001e-05 2024-09-15T03:47:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124402+0000", 2024-09-15T03:47:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000240442 2024-09-15T03:47:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125141+0000", 2024-09-15T03:47:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00073915500000000002 2024-09-15T03:47:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125191+0000", 2024-09-15T03:47:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9951999999999999e-05 2024-09-15T03:47:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125251+0000", 2024-09-15T03:47:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9938000000000001e-05 2024-09-15T03:47:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.137004+0000", 2024-09-15T03:47:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011753329999999999 2024-09-15T03:47:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=35, reqid=client.4521.0:197, at_version=0'0, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:47:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.156688066999997, 2024-09-15T03:47:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011202456, 2024-09-15T03:47:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125953+0000", 2024-09-15T03:47:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9100000000000002e-07 2024-09-15T03:47:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125964+0000", 2024-09-15T03:47:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1399999999999999e-05 2024-09-15T03:47:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125979+0000", 2024-09-15T03:47:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4810000000000001e-05 2024-09-15T03:47:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126004+0000", 2024-09-15T03:47:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4913e-05 2024-09-15T03:47:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126059+0000", 2024-09-15T03:47:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5668999999999998e-05 2024-09-15T03:47:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126071+0000", 2024-09-15T03:47:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1219999999999999e-05 2024-09-15T03:47:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126151+0000", 2024-09-15T03:47:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0572e-05 2024-09-15T03:47:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126163+0000", 2024-09-15T03:47:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2101e-05 2024-09-15T03:47:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.127945+0000", 2024-09-15T03:47:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001781803 2024-09-15T03:47:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.127982+0000", 2024-09-15T03:47:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7129999999999999e-05 2024-09-15T03:47:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.137154+0000", 2024-09-15T03:47:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0091719469999999997 2024-09-15T03:47:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:243 3.3s0 3:c7c3ef16:::smithi00132172-25:head [create,setxattr iZnWfebab`ptjURR\\f`psW]tpS`qpUaeWkY[Oqep (1330) in=40b,setxattr f[mZ_qQmcZejQ`fbdeic_tkPUaTYcnm[WprjVmc] (12488) in=40b,setxattr tkZgQ]S_pdlhZnqrVaZRa`oQQr_USmUfhado^dXT (1121) in=40b,setxattr srWjTqXhcth`hfZmQY_fg]UqNjWofa^dfgOV`sYP (455) in=40b,setxattr ^_mdiRroj_TtQ^gcNj`XtkW[ooqY`PWXqYmcOk_n (138) in=40b,setxattr jQlajdtcalhkcfYaakZfNb[^_TjkfcS`Vrk]q_d[ (806) in=40b,setxattr ZfYgkVcrsXcnpchgfcbldNrUTasnQNmfZW]Nbh\\o (561) in=40b,setxattr _hkYOqjPNhmhUlSNktfeQpsqSWjO\\fbf[fpTlZiO (1227) in=40b,setxattr ]mdV^Xdc\\sViXdkXNU[js_Qhh[b[sbfnZVXkdids (11788) in=40b,setxattr tb^mbOrNhfrhtce^g (455) in=17b,setxattr koTkVgrfU\\m`StnZ`Rirjhbhg`PhVqm^a^Xgaaqn (35) in=40b,setxattr aUtZPUPXPnQmT]\\\\n\\lZpnXakdNkTldVQoQhPUf] (559) in=40b,setxattr mrhR]^U`Tteo[Topmql`SNma\\pqfk`if]OppnYt` (770) in=40b,setxattr Oetk]RrnWeXbVctVtVrhXf_RfTdelqX^lWp[sTkp (1226) in=40b,setxattr WtedXoOd`VqnbksOqQkatk`gdTnW^lXbUtrlYgkl (3576) in=40b,setxattr dddOWN\\j\\XXg^fr[rZcskeVX]jOQ^ZmoWepk[]\\a (17824) in=40b,setxattr TeRO^djPgNZiemQ_aRXgP[]jgUZa\\^\\iUgesaqaN (208) in=40b,setxattr teqm]mUobZbWjcdQ_jTa[f[k[Sd\\lh]n\\osigqkN (7436) in=40b,setxattr dphaol[XkeitbkomWYaQctWfataUhbSiNntkWo`P (1893) in=40b,setxattr oQsnhO]ZklSVYQ[WUpaelY[p[Qie^UepOt`Z[q (455) in=38b] snapc 7=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:47:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 34.113724179000002, 2024-09-15T03:47:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012365956000000001, 2024-09-15T03:47:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 243 2024-09-15T03:47:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168917+0000", 2024-09-15T03:47:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.725e-06 2024-09-15T03:47:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169030+0000", 2024-09-15T03:47:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113035 2024-09-15T03:47:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169047+0000", 2024-09-15T03:47:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6701000000000002e-05 2024-09-15T03:47:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169054+0000", 2024-09-15T03:47:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1369999999999996e-06 2024-09-15T03:47:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169140+0000", 2024-09-15T03:47:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6069999999999994e-05 2024-09-15T03:47:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169270+0000", 2024-09-15T03:47:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130071 2024-09-15T03:47:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.170661+0000", 2024-09-15T03:47:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001390666 2024-09-15T03:47:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171751+0000", 2024-09-15T03:47:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010899079999999999 2024-09-15T03:47:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171812+0000", 2024-09-15T03:47:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0797000000000001e-05 2024-09-15T03:47:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172279+0000", 2024-09-15T03:47:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046704500000000003 2024-09-15T03:47:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172299+0000", 2024-09-15T03:47:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0387999999999999e-05 2024-09-15T03:47:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.176110+0000", 2024-09-15T03:47:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038112340000000001 2024-09-15T03:47:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.181282+0000", 2024-09-15T03:47:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051711789999999997 2024-09-15T03:47:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:258 3.3s0 3:c9b663a2:::smithi00132172-7:head [delete] snapc 7=[6,5,3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e40)", 2024-09-15T03:47:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:47.146116+0000", 2024-09-15T03:47:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.136523361000002, 2024-09-15T03:47:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096428380000000008, 2024-09-15T03:47:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 258 2024-09-15T03:47:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146116+0000", 2024-09-15T03:47:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146116+0000", 2024-09-15T03:47:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146119+0000", 2024-09-15T03:47:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.227e-06 2024-09-15T03:47:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146127+0000", 2024-09-15T03:47:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1529999999999997e-06 2024-09-15T03:47:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146146+0000", 2024-09-15T03:47:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9318999999999999e-05 2024-09-15T03:47:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146154+0000", 2024-09-15T03:47:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7200000000000006e-06 2024-09-15T03:47:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146233+0000", 2024-09-15T03:47:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8725000000000003e-05 2024-09-15T03:47:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146244+0000", 2024-09-15T03:47:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1725e-05 2024-09-15T03:47:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.149207+0000", 2024-09-15T03:47:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029627999999999998 2024-09-15T03:47:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.149326+0000", 2024-09-15T03:47:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011869199999999999 2024-09-15T03:47:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.150266+0000", 2024-09-15T03:47:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00094048500000000002 2024-09-15T03:47:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.150626+0000", 2024-09-15T03:47:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035946699999999998 2024-09-15T03:47:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.150751+0000", 2024-09-15T03:47:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012544800000000001 2024-09-15T03:47:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.151656+0000", 2024-09-15T03:47:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00090447499999999996 2024-09-15T03:47:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.153489+0000", 2024-09-15T03:47:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001833115 2024-09-15T03:47:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.155171+0000", 2024-09-15T03:47:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001682455 2024-09-15T03:47:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.155759+0000", 2024-09-15T03:47:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00058803200000000005 2024-09-15T03:47:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/28 ECSubWrite(tid=15, reqid=client.4521.0:337, at_version=49'10, trim_to=0'0, roll_forward_to=43'8))", 2024-09-15T03:47:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 24.424734269999998, 2024-09-15T03:47:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010500893000000001, 2024-09-15T03:47:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857906+0000", 2024-09-15T03:47:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.63e-07 2024-09-15T03:47:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857928+0000", 2024-09-15T03:47:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2155999999999999e-05 2024-09-15T03:47:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857937+0000", 2024-09-15T03:47:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2359999999999993e-06 2024-09-15T03:47:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857941+0000", 2024-09-15T03:47:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0960000000000003e-06 2024-09-15T03:47:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857999+0000", 2024-09-15T03:47:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7407000000000001e-05 2024-09-15T03:47:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858011+0000", 2024-09-15T03:47:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1759999999999999e-05 2024-09-15T03:47:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858240+0000", 2024-09-15T03:47:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022949299999999999 2024-09-15T03:47:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858251+0000", 2024-09-15T03:47:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1168999999999999e-05 2024-09-15T03:47:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858332+0000", 2024-09-15T03:47:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1204000000000004e-05 2024-09-15T03:47:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858360+0000", 2024-09-15T03:47:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7767999999999999e-05 2024-09-15T03:47:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.868406+0000", 2024-09-15T03:47:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010046141 2024-09-15T03:47:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/28 ECSubWrite(tid=16, reqid=client.4521.0:337, at_version=0'0, trim_to=0'0, roll_forward_to=49'10))", 2024-09-15T03:47:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:47:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 24.423378241999998, 2024-09-15T03:47:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093024639999999999, 2024-09-15T03:47:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:47:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:47:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859263+0000", 2024-09-15T03:47:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1969999999999999e-06 2024-09-15T03:47:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859275+0000", 2024-09-15T03:47:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2153999999999999e-05 2024-09-15T03:47:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859294+0000", 2024-09-15T03:47:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8875000000000001e-05 2024-09-15T03:47:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859301+0000", 2024-09-15T03:47:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9360000000000002e-06 2024-09-15T03:47:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859397+0000", 2024-09-15T03:47:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6831999999999997e-05 2024-09-15T03:47:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859407+0000", 2024-09-15T03:47:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0939999999999997e-06 2024-09-15T03:47:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859500+0000", 2024-09-15T03:47:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3338000000000007e-05 2024-09-15T03:47:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859528+0000", 2024-09-15T03:47:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8039999999999999e-05 2024-09-15T03:47:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859568+0000", 2024-09-15T03:47:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0221e-05 2024-09-15T03:47:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859583+0000", 2024-09-15T03:47:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5217e-05 2024-09-15T03:47:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.868564+0000", 2024-09-15T03:47:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089805600000000003 2024-09-15T03:47:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:349 3.3s0 3:c7c3ef16:::smithi00132172-25:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.444158244, 2024-09-15T03:47:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005145406, 2024-09-15T03:47:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 349 2024-09-15T03:47:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838485+0000", 2024-09-15T03:47:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0620000000000001e-06 2024-09-15T03:47:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838494+0000", 2024-09-15T03:47:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3179999999999996e-06 2024-09-15T03:47:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3689999999999998e-05 2024-09-15T03:47:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838812+0000", 2024-09-15T03:47:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029437300000000001 2024-09-15T03:47:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838905+0000", 2024-09-15T03:47:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3196999999999996e-05 2024-09-15T03:47:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838914+0000", 2024-09-15T03:47:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0890000000000006e-06 2024-09-15T03:47:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840773+0000", 2024-09-15T03:47:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018584820000000001 2024-09-15T03:47:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840889+0000", 2024-09-15T03:47:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115986 2024-09-15T03:47:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842016+0000", 2024-09-15T03:47:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001126878 2024-09-15T03:47:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842320+0000", 2024-09-15T03:47:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030409799999999999 2024-09-15T03:47:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842360+0000", 2024-09-15T03:47:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0750000000000001e-05 2024-09-15T03:47:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842448+0000", 2024-09-15T03:47:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7842000000000004e-05 2024-09-15T03:47:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844580+0000", 2024-09-15T03:47:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021313719999999999 2024-09-15T03:47:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843574+0000", 2024-09-15T03:47:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99899468199999997 2024-09-15T03:47:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843627+0000", 2024-09-15T03:47:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2587000000000003e-05 2024-09-15T03:47:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:350 3.3s0 3:c7c3ef16:::smithi00132172-25:head [rollback 14] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.443806583000001, 2024-09-15T03:47:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0050383839999999, 2024-09-15T03:47:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 350 2024-09-15T03:47:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838834+0000", 2024-09-15T03:47:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0200000000000001e-07 2024-09-15T03:47:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838839+0000", 2024-09-15T03:47:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9790000000000001e-06 2024-09-15T03:47:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838852+0000", 2024-09-15T03:47:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3138e-05 2024-09-15T03:47:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838858+0000", 2024-09-15T03:47:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8320000000000002e-06 2024-09-15T03:47:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839003+0000", 2024-09-15T03:47:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014513500000000001 2024-09-15T03:47:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839011+0000", 2024-09-15T03:47:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7719999999999994e-06 2024-09-15T03:47:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842395+0000", 2024-09-15T03:47:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033840110000000001 2024-09-15T03:47:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842580+0000", 2024-09-15T03:47:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000185067 2024-09-15T03:47:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843394+0000", 2024-09-15T03:47:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081377299999999995 2024-09-15T03:47:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843510+0000", 2024-09-15T03:47:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0001165 2024-09-15T03:47:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843524+0000", 2024-09-15T03:47:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4022999999999999e-05 2024-09-15T03:47:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843579+0000", 2024-09-15T03:47:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4654999999999998e-05 2024-09-15T03:47:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844583+0000", 2024-09-15T03:47:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010042429999999999 2024-09-15T03:47:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843772+0000", 2024-09-15T03:47:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99918911200000005 2024-09-15T03:47:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843872+0000", 2024-09-15T03:47:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9341999999999999e-05 2024-09-15T03:47:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:351 3.3s0 3:c7c3ef16:::smithi00132172-25:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.44374895, 2024-09-15T03:47:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0051167679999999, 2024-09-15T03:47:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 351 2024-09-15T03:47:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0800000000000004e-07 2024-09-15T03:47:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838895+0000", 2024-09-15T03:47:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.241e-06 2024-09-15T03:47:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838904+0000", 2024-09-15T03:47:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9549999999999998e-06 2024-09-15T03:47:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838909+0000", 2024-09-15T03:47:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7869999999999998e-06 2024-09-15T03:47:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839045+0000", 2024-09-15T03:47:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013556399999999999 2024-09-15T03:47:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839052+0000", 2024-09-15T03:47:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8199999999999999e-06 2024-09-15T03:47:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843587+0000", 2024-09-15T03:47:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045349739999999998 2024-09-15T03:47:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843708+0000", 2024-09-15T03:47:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120776 2024-09-15T03:47:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844246+0000", 2024-09-15T03:47:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053838200000000001 2024-09-15T03:47:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844335+0000", 2024-09-15T03:47:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9240999999999995e-05 2024-09-15T03:47:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844347+0000", 2024-09-15T03:47:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2187e-05 2024-09-15T03:47:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844460+0000", 2024-09-15T03:47:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112571 2024-09-15T03:47:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844584+0000", 2024-09-15T03:47:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012403800000000001 2024-09-15T03:47:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843972+0000", 2024-09-15T03:47:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99938787299999998 2024-09-15T03:47:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844008+0000", 2024-09-15T03:47:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5750999999999997e-05 2024-09-15T03:47:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.436614999, 2024-09-15T03:47:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013010173, 2024-09-15T03:47:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846026+0000", 2024-09-15T03:47:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1230000000000001e-06 2024-09-15T03:47:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846047+0000", 2024-09-15T03:47:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0774000000000001e-05 2024-09-15T03:47:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846064+0000", 2024-09-15T03:47:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7560000000000001e-05 2024-09-15T03:47:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846072+0000", 2024-09-15T03:47:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5020000000000003e-06 2024-09-15T03:47:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846187+0000", 2024-09-15T03:47:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115717 2024-09-15T03:47:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846201+0000", 2024-09-15T03:47:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3329999999999999e-05 2024-09-15T03:47:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846303+0000", 2024-09-15T03:47:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101807 2024-09-15T03:47:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846312+0000", 2024-09-15T03:47:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5529999999999998e-06 2024-09-15T03:47:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846382+0000", 2024-09-15T03:47:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0146000000000002e-05 2024-09-15T03:47:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846520+0000", 2024-09-15T03:47:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013814099999999999 2024-09-15T03:47:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.859035+0000", 2024-09-15T03:47:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012514519999999999 2024-09-15T03:47:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.436518251999999, 2024-09-15T03:47:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013044694000000001, 2024-09-15T03:47:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846122+0000", 2024-09-15T03:47:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0399999999999996e-07 2024-09-15T03:47:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846174+0000", 2024-09-15T03:47:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1672999999999998e-05 2024-09-15T03:47:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846184+0000", 2024-09-15T03:47:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0514e-05 2024-09-15T03:47:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846190+0000", 2024-09-15T03:47:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2820000000000002e-06 2024-09-15T03:47:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846341+0000", 2024-09-15T03:47:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150803 2024-09-15T03:47:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846351+0000", 2024-09-15T03:47:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3899999999999999e-06 2024-09-15T03:47:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846450+0000", 2024-09-15T03:47:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9839000000000004e-05 2024-09-15T03:47:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846482+0000", 2024-09-15T03:47:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1089e-05 2024-09-15T03:47:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846615+0000", 2024-09-15T03:47:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013305899999999999 2024-09-15T03:47:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846634+0000", 2024-09-15T03:47:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8899999999999999e-05 2024-09-15T03:47:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.859166+0000", 2024-09-15T03:47:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012532641000000001 2024-09-15T03:47:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 60/31 ECSubWrite(tid=70, reqid=client.4521.0:421, at_version=60'22, trim_to=0'0, roll_forward_to=47'20))", 2024-09-15T03:47:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:06.837638+0000", 2024-09-15T03:47:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.445002177999999, 2024-09-15T03:47:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092785220000000009, 2024-09-15T03:47:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837638+0000", 2024-09-15T03:47:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837638+0000", 2024-09-15T03:47:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837638+0000", 2024-09-15T03:47:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6399999999999998e-07 2024-09-15T03:47:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837664+0000", 2024-09-15T03:47:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5788000000000001e-05 2024-09-15T03:47:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837675+0000", 2024-09-15T03:47:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1269e-05 2024-09-15T03:47:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837682+0000", 2024-09-15T03:47:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5120000000000002e-06 2024-09-15T03:47:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837771+0000", 2024-09-15T03:47:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8802999999999997e-05 2024-09-15T03:47:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837793+0000", 2024-09-15T03:47:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2141e-05 2024-09-15T03:47:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.838114+0000", 2024-09-15T03:47:20.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000320453 2024-09-15T03:47:20.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.838243+0000", 2024-09-15T03:47:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012918599999999999 2024-09-15T03:47:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.838298+0000", 2024-09-15T03:47:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5164999999999998e-05 2024-09-15T03:47:20.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.838665+0000", 2024-09-15T03:47:20.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036705199999999998 2024-09-15T03:47:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.846916+0000", 2024-09-15T03:47:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082508890000000008 2024-09-15T03:47:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 60/28 ECSubWrite(tid=73, reqid=client.4521.0:424, at_version=60'41, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.382209404999999, 2024-09-15T03:47:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011124821, 2024-09-15T03:47:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900431+0000", 2024-09-15T03:47:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2900000000000003e-07 2024-09-15T03:47:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900451+0000", 2024-09-15T03:47:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0066999999999999e-05 2024-09-15T03:47:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900466+0000", 2024-09-15T03:47:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5062000000000001e-05 2024-09-15T03:47:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900474+0000", 2024-09-15T03:47:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6310000000000003e-06 2024-09-15T03:47:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900524+0000", 2024-09-15T03:47:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0037000000000002e-05 2024-09-15T03:47:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900535+0000", 2024-09-15T03:47:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1209000000000001e-05 2024-09-15T03:47:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900659+0000", 2024-09-15T03:47:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012441000000000001 2024-09-15T03:47:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900971+0000", 2024-09-15T03:47:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031183799999999999 2024-09-15T03:47:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901144+0000", 2024-09-15T03:47:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172261 2024-09-15T03:47:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901734+0000", 2024-09-15T03:47:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059069700000000005 2024-09-15T03:47:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.911555+0000", 2024-09-15T03:47:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0098208800000000006 2024-09-15T03:47:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 60/28 ECSubWrite(tid=74, reqid=client.4521.0:425, at_version=60'42, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:06.901240+0000", 2024-09-15T03:47:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.381399825999999, 2024-09-15T03:47:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010429678, 2024-09-15T03:47:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901240+0000", 2024-09-15T03:47:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901240+0000", 2024-09-15T03:47:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901241+0000", 2024-09-15T03:47:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7999999999999993e-07 2024-09-15T03:47:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901273+0000", 2024-09-15T03:47:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2087e-05 2024-09-15T03:47:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901302+0000", 2024-09-15T03:47:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8705e-05 2024-09-15T03:47:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901311+0000", 2024-09-15T03:47:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9190000000000005e-06 2024-09-15T03:47:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901797+0000", 2024-09-15T03:47:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048606499999999999 2024-09-15T03:47:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901808+0000", 2024-09-15T03:47:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1075e-05 2024-09-15T03:47:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901934+0000", 2024-09-15T03:47:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012674299999999999 2024-09-15T03:47:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901964+0000", 2024-09-15T03:47:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9833999999999999e-05 2024-09-15T03:47:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.902006+0000", 2024-09-15T03:47:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1823999999999998e-05 2024-09-15T03:47:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.902020+0000", 2024-09-15T03:47:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.363e-05 2024-09-15T03:47:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.911670+0000", 2024-09-15T03:47:20.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096498160000000003 2024-09-15T03:47:20.444 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.444 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.444 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.444 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.444 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.444 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:20.446 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:20.446 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:20.444+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:20.446 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:20.447 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:20.447 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:20.448 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:47:20.449 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:47:20.449 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:47:20.449 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:20.451 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:20.546 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:20.544+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:20.551 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:20.552 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:20.552 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:20.552 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:20.560 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:47:20.647 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:20.646+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:20.660 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:20.660 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:20.660 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:20.660 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:20.660 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:20.668 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:47:20.748 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:20.747+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:20.770 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:20.770 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:20.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:20.770 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:20.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.770 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs3 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:20.770 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108456+0000", 2024-09-15T03:47:20.770 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.659774466000002, 2024-09-15T03:47:20.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015075257, 2024-09-15T03:47:20.771 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.771 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:20.771 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108456+0000", 2024-09-15T03:47:20.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108456+0000", 2024-09-15T03:47:20.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108457+0000", 2024-09-15T03:47:20.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9999999999999997e-07 2024-09-15T03:47:20.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108477+0000", 2024-09-15T03:47:20.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0344e-05 2024-09-15T03:47:20.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108497+0000", 2024-09-15T03:47:20.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0143999999999999e-05 2024-09-15T03:47:20.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108504+0000", 2024-09-15T03:47:20.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8499999999999996e-06 2024-09-15T03:47:20.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108554+0000", 2024-09-15T03:47:20.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9567999999999997e-05 2024-09-15T03:47:20.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108568+0000", 2024-09-15T03:47:20.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4326e-05 2024-09-15T03:47:20.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108654+0000", 2024-09-15T03:47:20.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6193999999999995e-05 2024-09-15T03:47:20.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108663+0000", 2024-09-15T03:47:20.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7479999999999999e-06 2024-09-15T03:47:20.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111234+0000", 2024-09-15T03:47:20.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025709079999999998 2024-09-15T03:47:20.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111259+0000", 2024-09-15T03:47:20.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4797999999999999e-05 2024-09-15T03:47:20.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123531+0000", 2024-09-15T03:47:20.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012272777 2024-09-15T03:47:20.776 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.776 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.776 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.776 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as2 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:20.776 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:47:20.776 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.659545811999998, 2024-09-15T03:47:20.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0149108, 2024-09-15T03:47:20.776 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.777 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:20.777 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:47:20.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:47:20.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:47:20.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1699999999999999e-07 2024-09-15T03:47:20.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108710+0000", 2024-09-15T03:47:20.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5032999999999999e-05 2024-09-15T03:47:20.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108725+0000", 2024-09-15T03:47:20.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4788e-05 2024-09-15T03:47:20.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108733+0000", 2024-09-15T03:47:20.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4610000000000008e-06 2024-09-15T03:47:20.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108778+0000", 2024-09-15T03:47:20.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4224999999999998e-05 2024-09-15T03:47:20.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108800+0000", 2024-09-15T03:47:20.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2189999999999999e-05 2024-09-15T03:47:20.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108884+0000", 2024-09-15T03:47:20.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4569999999999998e-05 2024-09-15T03:47:20.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111237+0000", 2024-09-15T03:47:20.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002352313 2024-09-15T03:47:20.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114139+0000", 2024-09-15T03:47:20.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002902057 2024-09-15T03:47:20.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114173+0000", 2024-09-15T03:47:20.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4461000000000001e-05 2024-09-15T03:47:20.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123596+0000", 2024-09-15T03:47:20.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094222850000000007 2024-09-15T03:47:20.786 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.786 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.786 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.786 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs3 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:20.786 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:47:20.786 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.659486534000003, 2024-09-15T03:47:20.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014886043, 2024-09-15T03:47:20.786 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.786 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:20.787 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:47:20.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:47:20.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:47:20.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5100000000000001e-07 2024-09-15T03:47:20.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108759+0000", 2024-09-15T03:47:20.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4945e-05 2024-09-15T03:47:20.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108768+0000", 2024-09-15T03:47:20.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2309999999999996e-06 2024-09-15T03:47:20.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108795+0000", 2024-09-15T03:47:20.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7829000000000001e-05 2024-09-15T03:47:20.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108841+0000", 2024-09-15T03:47:20.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5330999999999997e-05 2024-09-15T03:47:20.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108850+0000", 2024-09-15T03:47:20.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9959999999999999e-06 2024-09-15T03:47:20.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108940+0000", 2024-09-15T03:47:20.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9823999999999999e-05 2024-09-15T03:47:20.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111237+0000", 2024-09-15T03:47:20.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002297927 2024-09-15T03:47:20.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114139+0000", 2024-09-15T03:47:20.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029020029999999998 2024-09-15T03:47:20.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114212+0000", 2024-09-15T03:47:20.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2177999999999999e-05 2024-09-15T03:47:20.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123630+0000", 2024-09-15T03:47:20.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094184279999999995 2024-09-15T03:47:20.796 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.796 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.796 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.797 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as2 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:20.797 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.109397+0000", 2024-09-15T03:47:20.797 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.658833475999998, 2024-09-15T03:47:20.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014322056, 2024-09-15T03:47:20.797 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.797 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:20.797 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109397+0000", 2024-09-15T03:47:20.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109397+0000", 2024-09-15T03:47:20.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109398+0000", 2024-09-15T03:47:20.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1600000000000001e-07 2024-09-15T03:47:20.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109410+0000", 2024-09-15T03:47:20.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2294e-05 2024-09-15T03:47:20.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109420+0000", 2024-09-15T03:47:20.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.764e-06 2024-09-15T03:47:20.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109426+0000", 2024-09-15T03:47:20.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9309999999999996e-06 2024-09-15T03:47:20.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109457+0000", 2024-09-15T03:47:20.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1047000000000003e-05 2024-09-15T03:47:20.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109462+0000", 2024-09-15T03:47:20.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1499999999999998e-06 2024-09-15T03:47:20.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109513+0000", 2024-09-15T03:47:20.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0722e-05 2024-09-15T03:47:20.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111238+0000", 2024-09-15T03:47:20.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017253380000000001 2024-09-15T03:47:20.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114140+0000", 2024-09-15T03:47:20.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029017370000000002 2024-09-15T03:47:20.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114201+0000", 2024-09-15T03:47:20.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1047e-05 2024-09-15T03:47:20.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123719+0000", 2024-09-15T03:47:20.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095184099999999997 2024-09-15T03:47:20.803 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.803 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.803 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.804 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:197 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [delete] snapc 3=[3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e34)", 2024-09-15T03:47:20.804 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:20.804 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.655965299999998, 2024-09-15T03:47:20.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021934704999999999, 2024-09-15T03:47:20.804 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.805 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:20.805 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:20.805 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:20.805 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:20.805 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197 2024-09-15T03:47:20.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.805 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112266+0000", 2024-09-15T03:47:20.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5000000000000002e-07 2024-09-15T03:47:20.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112269+0000", 2024-09-15T03:47:20.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.066e-06 2024-09-15T03:47:20.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112276+0000", 2024-09-15T03:47:20.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-09-15T03:47:20.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112281+0000", 2024-09-15T03:47:20.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3970000000000004e-06 2024-09-15T03:47:20.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123281+0000", 2024-09-15T03:47:20.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011000575 2024-09-15T03:47:20.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:20.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123395+0000", 2024-09-15T03:47:20.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113398 2024-09-15T03:47:20.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123963+0000", 2024-09-15T03:47:20.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000568101 2024-09-15T03:47:20.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124220+0000", 2024-09-15T03:47:20.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025704900000000003 2024-09-15T03:47:20.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124229+0000", 2024-09-15T03:47:20.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4860000000000003e-06 2024-09-15T03:47:20.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124277+0000", 2024-09-15T03:47:20.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7574000000000002e-05 2024-09-15T03:47:20.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124312+0000", 2024-09-15T03:47:20.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5256000000000003e-05 2024-09-15T03:47:20.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:20.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125716+0000", 2024-09-15T03:47:20.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014043720000000001 2024-09-15T03:47:20.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.134200+0000", 2024-09-15T03:47:20.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084835120000000003 2024-09-15T03:47:20.811 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.811 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.811 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.812 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es4 34/28 ECSubWrite(tid=33, reqid=client.4521.0:195, at_version=34'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-15T03:47:20.812 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:47:20.812 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.650137940999997, 2024-09-15T03:47:20.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015907482000000001, 2024-09-15T03:47:20.813 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.813 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:20.813 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:47:20.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:47:20.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:47:20.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9400000000000004e-07 2024-09-15T03:47:20.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118299+0000", 2024-09-15T03:47:20.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000205677 2024-09-15T03:47:20.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118315+0000", 2024-09-15T03:47:20.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5429000000000001e-05 2024-09-15T03:47:20.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118322+0000", 2024-09-15T03:47:20.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8290000000000008e-06 2024-09-15T03:47:20.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118416+0000", 2024-09-15T03:47:20.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3140000000000006e-05 2024-09-15T03:47:20.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118431+0000", 2024-09-15T03:47:20.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5648e-05 2024-09-15T03:47:20.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118678+0000", 2024-09-15T03:47:20.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024701200000000001 2024-09-15T03:47:20.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118691+0000", 2024-09-15T03:47:20.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2653000000000001e-05 2024-09-15T03:47:20.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.119431+0000", 2024-09-15T03:47:20.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00074060399999999996 2024-09-15T03:47:20.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124280+0000", 2024-09-15T03:47:20.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0048484410000000002 2024-09-15T03:47:20.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.134000+0000", 2024-09-15T03:47:20.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0097201549999999994 2024-09-15T03:47:20.819 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.819 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.819 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.820 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:220 3.bs0 3:d99ddedc:::smithi00132172-44:head [copy-from ver 22] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:20.820 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:20.820 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.640585629999997, 2024-09-15T03:47:20.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024273429999999999, 2024-09-15T03:47:20.820 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.820 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:20.820 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:20.820 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:20.820 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:20.820 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 220 2024-09-15T03:47:20.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.821 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:20.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:20.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127647+0000", 2024-09-15T03:47:20.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1799999999999999e-06 2024-09-15T03:47:20.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127656+0000", 2024-09-15T03:47:20.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3759999999999997e-06 2024-09-15T03:47:20.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127674+0000", 2024-09-15T03:47:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7374e-05 2024-09-15T03:47:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127680+0000", 2024-09-15T03:47:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6949999999999997e-06 2024-09-15T03:47:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127736+0000", 2024-09-15T03:47:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6011999999999998e-05 2024-09-15T03:47:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127744+0000", 2024-09-15T03:47:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2870000000000004e-06 2024-09-15T03:47:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130338+0000", 2024-09-15T03:47:20.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025940249999999998 2024-09-15T03:47:20.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:20.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130423+0000", 2024-09-15T03:47:20.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5303000000000006e-05 2024-09-15T03:47:20.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.147840+0000", 2024-09-15T03:47:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017416580000000001 2024-09-15T03:47:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148318+0000", 2024-09-15T03:47:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047822899999999999 2024-09-15T03:47:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148329+0000", 2024-09-15T03:47:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0611000000000001e-05 2024-09-15T03:47:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148488+0000", 2024-09-15T03:47:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015938399999999999 2024-09-15T03:47:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148870+0000", 2024-09-15T03:47:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000382405 2024-09-15T03:47:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.151092+0000", 2024-09-15T03:47:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022212529999999999 2024-09-15T03:47:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.151918+0000", 2024-09-15T03:47:20.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082671600000000002 2024-09-15T03:47:20.829 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.829 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.829 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.829 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:221 3.bs0 3:d99ddedc:::smithi00132172-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:20.830 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:20.830 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.640541474999999, 2024-09-15T03:47:20.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021194474000000001, 2024-09-15T03:47:20.830 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.830 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:20.830 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:20.830 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:20.830 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:20.830 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 221 2024-09-15T03:47:20.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.831 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:20.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127690+0000", 2024-09-15T03:47:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:47:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127692+0000", 2024-09-15T03:47:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6589999999999999e-06 2024-09-15T03:47:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127700+0000", 2024-09-15T03:47:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4769999999999997e-06 2024-09-15T03:47:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127704+0000", 2024-09-15T03:47:20.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3819999999999997e-06 2024-09-15T03:47:20.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127805+0000", 2024-09-15T03:47:20.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100639 2024-09-15T03:47:20.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:20.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127809+0000", 2024-09-15T03:47:20.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8870000000000002e-06 2024-09-15T03:47:20.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130527+0000", 2024-09-15T03:47:20.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002718579 2024-09-15T03:47:20.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:20.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130558+0000", 2024-09-15T03:47:20.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0827999999999997e-05 2024-09-15T03:47:20.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148546+0000", 2024-09-15T03:47:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017988356 2024-09-15T03:47:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148683+0000", 2024-09-15T03:47:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013650899999999999 2024-09-15T03:47:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148884+0000", 2024-09-15T03:47:20.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020061499999999999 2024-09-15T03:47:20.837 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.837 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.837 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.837 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:252 3.bs0 3:d6f79b7b:::smithi00132172-29:head [copy-from ver 6] snapc 7=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:47:20.838 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.169146+0000", 2024-09-15T03:47:20.838 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 34.599084206000001, 2024-09-15T03:47:20.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016731068000000002, 2024-09-15T03:47:20.838 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.838 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:20.838 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:20.839 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:20.839 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:20.839 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 252 2024-09-15T03:47:20.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.839 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169146+0000", 2024-09-15T03:47:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169146+0000", 2024-09-15T03:47:20.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169147+0000", 2024-09-15T03:47:20.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1400000000000001e-07 2024-09-15T03:47:20.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169149+0000", 2024-09-15T03:47:20.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0279999999999999e-06 2024-09-15T03:47:20.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169152+0000", 2024-09-15T03:47:20.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0800000000000002e-06 2024-09-15T03:47:20.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169154+0000", 2024-09-15T03:47:20.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4679999999999999e-06 2024-09-15T03:47:20.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.174427+0000", 2024-09-15T03:47:20.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052724349999999998 2024-09-15T03:47:20.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:20.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.174543+0000", 2024-09-15T03:47:20.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116803 2024-09-15T03:47:20.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.178297+0000", 2024-09-15T03:47:20.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037531539999999999 2024-09-15T03:47:20.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179112+0000", 2024-09-15T03:47:20.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081548600000000003 2024-09-15T03:47:20.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179151+0000", 2024-09-15T03:47:20.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9158e-05 2024-09-15T03:47:20.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179272+0000", 2024-09-15T03:47:20.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120802 2024-09-15T03:47:20.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179531+0000", 2024-09-15T03:47:20.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000259351 2024-09-15T03:47:20.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:20.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179723+0000", 2024-09-15T03:47:20.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000191447 2024-09-15T03:47:20.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.185877+0000", 2024-09-15T03:47:20.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061546420000000001 2024-09-15T03:47:20.853 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.853 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.853 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.853 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:357 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.919189940999999, 2024-09-15T03:47:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.02017917, 2024-09-15T03:47:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 357 2024-09-15T03:47:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849043+0000", 2024-09-15T03:47:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.295e-06 2024-09-15T03:47:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849051+0000", 2024-09-15T03:47:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4079999999999998e-06 2024-09-15T03:47:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849069+0000", 2024-09-15T03:47:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7462999999999999e-05 2024-09-15T03:47:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849077+0000", 2024-09-15T03:47:20.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8869999999999992e-06 2024-09-15T03:47:20.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849194+0000", 2024-09-15T03:47:20.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011723899999999999 2024-09-15T03:47:20.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:20.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849296+0000", 2024-09-15T03:47:20.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102011 2024-09-15T03:47:20.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861106+0000", 2024-09-15T03:47:20.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0118097610000001 2024-09-15T03:47:20.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:20.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861222+0000", 2024-09-15T03:47:20.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116561 2024-09-15T03:47:20.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.869220+0000", 2024-09-15T03:47:20.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007997545 2024-09-15T03:47:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:358 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.919134995, 2024-09-15T03:47:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0228638050000001, 2024-09-15T03:47:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:20.860 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:20.860 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 358 2024-09-15T03:47:20.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.860 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:20.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:20.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4099999999999997e-07 2024-09-15T03:47:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849101+0000", 2024-09-15T03:47:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8180000000000003e-06 2024-09-15T03:47:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849110+0000", 2024-09-15T03:47:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1409999999999994e-06 2024-09-15T03:47:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849116+0000", 2024-09-15T03:47:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2390000000000002e-06 2024-09-15T03:47:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849326+0000", 2024-09-15T03:47:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000210029 2024-09-15T03:47:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849421+0000", 2024-09-15T03:47:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5775999999999999e-05 2024-09-15T03:47:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861647+0000", 2024-09-15T03:47:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0122257139999999 2024-09-15T03:47:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861777+0000", 2024-09-15T03:47:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130087 2024-09-15T03:47:20.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.871959+0000", 2024-09-15T03:47:20.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010182160000000001 2024-09-15T03:47:20.864 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.864 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.864 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.864 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:359 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:20.864 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.919109843000001, 2024-09-15T03:47:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0240468149999999, 2024-09-15T03:47:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 359 2024-09-15T03:47:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:47:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849124+0000", 2024-09-15T03:47:20.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2679999999999999e-06 2024-09-15T03:47:20.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849132+0000", 2024-09-15T03:47:20.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4030000000000001e-06 2024-09-15T03:47:20.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849136+0000", 2024-09-15T03:47:20.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2289999999999997e-06 2024-09-15T03:47:20.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849446+0000", 2024-09-15T03:47:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031043299999999999 2024-09-15T03:47:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849553+0000", 2024-09-15T03:47:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106415 2024-09-15T03:47:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.863404+0000", 2024-09-15T03:47:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0138511029999999 2024-09-15T03:47:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.863475+0000", 2024-09-15T03:47:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0814000000000003e-05 2024-09-15T03:47:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.873168+0000", 2024-09-15T03:47:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00969274 2024-09-15T03:47:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:474 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~786432] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.8931851680000005, 2024-09-15T03:47:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98604943700000003, 2024-09-15T03:47:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 474 2024-09-15T03:47:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875048+0000", 2024-09-15T03:47:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1600000000000001e-06 2024-09-15T03:47:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875844+0000", 2024-09-15T03:47:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079686299999999998 2024-09-15T03:47:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875857+0000", 2024-09-15T03:47:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2889e-05 2024-09-15T03:47:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875864+0000", 2024-09-15T03:47:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6139999999999997e-06 2024-09-15T03:47:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875916+0000", 2024-09-15T03:47:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1992e-05 2024-09-15T03:47:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875918+0000", 2024-09-15T03:47:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3300000000000001e-06 2024-09-15T03:47:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.845918+0000", 2024-09-15T03:47:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96999959199999997 2024-09-15T03:47:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.846229+0000", 2024-09-15T03:47:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000311601 2024-09-15T03:47:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848445+0000", 2024-09-15T03:47:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022160499999999998 2024-09-15T03:47:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848880+0000", 2024-09-15T03:47:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043432600000000002 2024-09-15T03:47:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848952+0000", 2024-09-15T03:47:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2509000000000007e-05 2024-09-15T03:47:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.849311+0000", 2024-09-15T03:47:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000358611 2024-09-15T03:47:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851131+0000", 2024-09-15T03:47:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001820115 2024-09-15T03:47:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861048+0000", 2024-09-15T03:47:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099164830000000002 2024-09-15T03:47:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861095+0000", 2024-09-15T03:47:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7302000000000002e-05 2024-09-15T03:47:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:475 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~491520] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.8904186969999994, 2024-09-15T03:47:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98407319000000004, 2024-09-15T03:47:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 475 2024-09-15T03:47:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877813+0000", 2024-09-15T03:47:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8800000000000002e-07 2024-09-15T03:47:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878260+0000", 2024-09-15T03:47:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044732099999999998 2024-09-15T03:47:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878270+0000", 2024-09-15T03:47:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9799999999999993e-06 2024-09-15T03:47:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878276+0000", 2024-09-15T03:47:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2269999999999998e-06 2024-09-15T03:47:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878313+0000", 2024-09-15T03:47:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6570999999999999e-05 2024-09-15T03:47:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878315+0000", 2024-09-15T03:47:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8729999999999999e-06 2024-09-15T03:47:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851316+0000", 2024-09-15T03:47:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97300154800000005 2024-09-15T03:47:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851440+0000", 2024-09-15T03:47:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124287 2024-09-15T03:47:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852732+0000", 2024-09-15T03:47:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001291174 2024-09-15T03:47:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852991+0000", 2024-09-15T03:47:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025905299999999997 2024-09-15T03:47:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853003+0000", 2024-09-15T03:47:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2908000000000001e-05 2024-09-15T03:47:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853104+0000", 2024-09-15T03:47:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010011800000000001 2024-09-15T03:47:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854342+0000", 2024-09-15T03:47:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012380189999999999 2024-09-15T03:47:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861206+0000", 2024-09-15T03:47:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068648499999999996 2024-09-15T03:47:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861885+0000", 2024-09-15T03:47:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00067857300000000002 2024-09-15T03:47:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:476 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~278528] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.8887916839999992, 2024-09-15T03:47:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98268067299999995, 2024-09-15T03:47:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 476 2024-09-15T03:47:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.13e-07 2024-09-15T03:47:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879706+0000", 2024-09-15T03:47:20.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026656100000000001 2024-09-15T03:47:20.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879715+0000", 2024-09-15T03:47:20.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9330000000000007e-06 2024-09-15T03:47:20.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879720+0000", 2024-09-15T03:47:20.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0880000000000002e-06 2024-09-15T03:47:20.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879743+0000", 2024-09-15T03:47:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2654e-05 2024-09-15T03:47:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879743+0000", 2024-09-15T03:47:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.99e-07 2024-09-15T03:47:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853170+0000", 2024-09-15T03:47:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97342687900000002 2024-09-15T03:47:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853297+0000", 2024-09-15T03:47:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127157 2024-09-15T03:47:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854150+0000", 2024-09-15T03:47:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085288900000000001 2024-09-15T03:47:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854292+0000", 2024-09-15T03:47:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014199300000000001 2024-09-15T03:47:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854559+0000", 2024-09-15T03:47:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026668999999999998 2024-09-15T03:47:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855998+0000", 2024-09-15T03:47:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001438714 2024-09-15T03:47:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856547+0000", 2024-09-15T03:47:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054880399999999996 2024-09-15T03:47:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861858+0000", 2024-09-15T03:47:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053110320000000003 2024-09-15T03:47:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.862120+0000", 2024-09-15T03:47:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026196699999999999 2024-09-15T03:47:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:477 3.bs0 3:d52faf55:::smithi00132172-40:head [setxattr _header (57) in=7b] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.8885037469999997, 2024-09-15T03:47:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98562198000000001, 2024-09-15T03:47:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 477 2024-09-15T03:47:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2800000000000002e-07 2024-09-15T03:47:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879730+0000", 2024-09-15T03:47:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0589999999999998e-06 2024-09-15T03:47:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879737+0000", 2024-09-15T03:47:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0419999999999997e-06 2024-09-15T03:47:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879741+0000", 2024-09-15T03:47:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9960000000000004e-06 2024-09-15T03:47:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879756+0000", 2024-09-15T03:47:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4681999999999999e-05 2024-09-15T03:47:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879756+0000", 2024-09-15T03:47:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.22e-07 2024-09-15T03:47:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854590+0000", 2024-09-15T03:47:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97483319000000002 2024-09-15T03:47:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854718+0000", 2024-09-15T03:47:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012842100000000001 2024-09-15T03:47:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855416+0000", 2024-09-15T03:47:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069759200000000003 2024-09-15T03:47:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855551+0000", 2024-09-15T03:47:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013517600000000001 2024-09-15T03:47:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856002+0000", 2024-09-15T03:47:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045169399999999999 2024-09-15T03:47:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856065+0000", 2024-09-15T03:47:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3028999999999996e-05 2024-09-15T03:47:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856554+0000", 2024-09-15T03:47:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000488486 2024-09-15T03:47:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865278+0000", 2024-09-15T03:47:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087241929999999999 2024-09-15T03:47:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865349+0000", 2024-09-15T03:47:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0669999999999999e-05 2024-09-15T03:47:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:478 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.8884849520000007, 2024-09-15T03:47:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99471090500000003, 2024-09-15T03:47:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 478 2024-09-15T03:47:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2599999999999998e-07 2024-09-15T03:47:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879748+0000", 2024-09-15T03:47:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9609999999999999e-06 2024-09-15T03:47:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879755+0000", 2024-09-15T03:47:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1879999999999996e-06 2024-09-15T03:47:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879760+0000", 2024-09-15T03:47:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2560000000000002e-06 2024-09-15T03:47:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879797+0000", 2024-09-15T03:47:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6452999999999998e-05 2024-09-15T03:47:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879799+0000", 2024-09-15T03:47:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6619999999999999e-06 2024-09-15T03:47:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855637+0000", 2024-09-15T03:47:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97583850999999999 2024-09-15T03:47:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855763+0000", 2024-09-15T03:47:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012612699999999999 2024-09-15T03:47:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865707+0000", 2024-09-15T03:47:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099439660000000003 2024-09-15T03:47:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865803+0000", 2024-09-15T03:47:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5946999999999995e-05 2024-09-15T03:47:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.874456+0000", 2024-09-15T03:47:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0086534090000000008 2024-09-15T03:47:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:486 3.bs0 3:df8b18a3:::smithi00132172-17:head [delete] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e64)", 2024-09-15T03:47:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.8856724729999996, 2024-09-15T03:47:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98362617100000005, 2024-09-15T03:47:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:20.908 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 486 2024-09-15T03:47:20.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.908 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:20.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:20.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882559+0000", 2024-09-15T03:47:20.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0359999999999999e-06 2024-09-15T03:47:20.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882564+0000", 2024-09-15T03:47:20.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6530000000000002e-06 2024-09-15T03:47:20.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882574+0000", 2024-09-15T03:47:20.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0464e-05 2024-09-15T03:47:20.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882578+0000", 2024-09-15T03:47:20.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2459999999999997e-06 2024-09-15T03:47:20.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882624+0000", 2024-09-15T03:47:20.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5698000000000002e-05 2024-09-15T03:47:20.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:20.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882627+0000", 2024-09-15T03:47:20.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3350000000000001e-06 2024-09-15T03:47:20.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856574+0000", 2024-09-15T03:47:20.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97394739399999997 2024-09-15T03:47:20.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:20.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856853+0000", 2024-09-15T03:47:20.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027918399999999998 2024-09-15T03:47:20.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.866184+0000", 2024-09-15T03:47:20.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093311609999999993 2024-09-15T03:47:20.912 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.912 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.912 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.912 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:495 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [copy-from ver 51] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-15T03:47:20.912 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:20.912 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.8797714239999994, 2024-09-15T03:47:20.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023555178, 2024-09-15T03:47:20.912 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.913 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:20.913 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:20.913 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:20.913 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:20.913 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 495 2024-09-15T03:47:20.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.913 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:20.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:20.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888461+0000", 2024-09-15T03:47:20.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.719e-06 2024-09-15T03:47:20.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888471+0000", 2024-09-15T03:47:20.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0161e-05 2024-09-15T03:47:20.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888500+0000", 2024-09-15T03:47:20.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8989e-05 2024-09-15T03:47:20.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888508+0000", 2024-09-15T03:47:20.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5299999999999999e-06 2024-09-15T03:47:20.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888554+0000", 2024-09-15T03:47:20.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6489000000000002e-05 2024-09-15T03:47:20.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:20.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888809+0000", 2024-09-15T03:47:20.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000255467 2024-09-15T03:47:20.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:20.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.905724+0000", 2024-09-15T03:47:20.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016914348999999999 2024-09-15T03:47:20.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:20.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906244+0000", 2024-09-15T03:47:20.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052054900000000003 2024-09-15T03:47:20.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:20.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906262+0000", 2024-09-15T03:47:20.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7499999999999998e-05 2024-09-15T03:47:20.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:20.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.907367+0000", 2024-09-15T03:47:20.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011049460000000001 2024-09-15T03:47:20.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:20.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.907384+0000", 2024-09-15T03:47:20.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7000999999999999e-05 2024-09-15T03:47:20.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:20.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.911965+0000", 2024-09-15T03:47:20.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045809470000000001 2024-09-15T03:47:20.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.912014+0000", 2024-09-15T03:47:20.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9530999999999997e-05 2024-09-15T03:47:20.919 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.919 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.919 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.919 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:496 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-15T03:47:20.919 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.888524+0000", 2024-09-15T03:47:20.919 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.8797063319999996, 2024-09-15T03:47:20.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01823953, 2024-09-15T03:47:20.919 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:20.919 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:20.919 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:20.920 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:20.920 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:20.920 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 496 2024-09-15T03:47:20.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.920 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:20.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:20.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888524+0000", 2024-09-15T03:47:20.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:20.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888524+0000", 2024-09-15T03:47:20.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:20.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:20.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888525+0000", 2024-09-15T03:47:20.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2699999999999999e-07 2024-09-15T03:47:20.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:20.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888529+0000", 2024-09-15T03:47:20.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7840000000000001e-06 2024-09-15T03:47:20.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:20.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888540+0000", 2024-09-15T03:47:20.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1477e-05 2024-09-15T03:47:20.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:20.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888546+0000", 2024-09-15T03:47:20.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4990000000000002e-06 2024-09-15T03:47:20.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888971+0000", 2024-09-15T03:47:20.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042491300000000002 2024-09-15T03:47:20.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:20.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.889024+0000", 2024-09-15T03:47:20.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3046e-05 2024-09-15T03:47:20.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:20.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906517+0000", 2024-09-15T03:47:20.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017492909000000001 2024-09-15T03:47:20.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:20.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906638+0000", 2024-09-15T03:47:20.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012138000000000001 2024-09-15T03:47:20.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:20.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:20.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:20.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906764+0000", 2024-09-15T03:47:20.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125795 2024-09-15T03:47:20.924 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.924 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.924 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.924 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:20.924 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:20.924 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:20.926 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:47:20.926 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:20.925+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:20.930 INFO:teuthology.orchestra.run.smithi001.stderr:set 3.a pg_upmap mapping to [4,10,8,9,5,6,2,0] 2024-09-15T03:47:20.931 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-15T03:47:20.931 INFO:tasks.rados.rados.0.smithi001.stdout:222: done (0 left) 2024-09-15T03:47:20.931 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 16 current snap is 28 2024-09-15T03:47:20.931 INFO:tasks.rados.rados.0.smithi001.stdout:223: seq_num 102 ranges {0=557056,557056=638976,1196032=524288} 2024-09-15T03:47:20.934 INFO:tasks.rados.rados.0.smithi001.stdout:223: writing smithi00132172-16 from 0 to 557056 tid 1 2024-09-15T03:47:20.938 INFO:tasks.rados.rados.0.smithi001.stdout:223: writing smithi00132172-16 from 557056 to 1196032 tid 2 2024-09-15T03:47:20.941 INFO:tasks.rados.rados.0.smithi001.stdout:223: writing smithi00132172-16 from 1196032 to 1720320 tid 3 2024-09-15T03:47:20.942 INFO:tasks.rados.rados.0.smithi001.stdout:224: rollback oid 21 current snap is 28 2024-09-15T03:47:20.942 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 21 to 1 2024-09-15T03:47:20.942 INFO:tasks.rados.rados.0.smithi001.stdout:225: rmattr oid 18 current snap is 28 2024-09-15T03:47:20.942 INFO:tasks.rados.rados.0.smithi001.stdout:226: read oid 15 snap 15 2024-09-15T03:47:20.942 INFO:tasks.rados.rados.0.smithi001.stdout:226: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-15T03:47:20.942 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 9 current snap is 28 2024-09-15T03:47:20.942 INFO:tasks.rados.rados.0.smithi001.stdout:227: seq_num 103 ranges {1654784=573440,2228224=786432,3014656=376832} 2024-09-15T03:47:20.956 INFO:tasks.rados.rados.0.smithi001.stdout:227: writing smithi00132172-9 from 1654784 to 2228224 tid 1 2024-09-15T03:47:20.960 INFO:tasks.rados.rados.0.smithi001.stdout:227: writing smithi00132172-9 from 2228224 to 3014656 tid 2 2024-09-15T03:47:20.963 INFO:tasks.rados.rados.0.smithi001.stdout:227: writing smithi00132172-9 from 3014656 to 3391488 tid 3 2024-09-15T03:47:20.963 INFO:tasks.rados.rados.0.smithi001.stdout:228: delete oid 37 current snap is 28 2024-09-15T03:47:20.963 INFO:tasks.rados.rados.0.smithi001.stdout:226: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-15T03:47:20.971 INFO:tasks.rados.rados.0.smithi001.stdout:223: finishing write tid 1 to smithi00132172-16 2024-09-15T03:47:20.971 INFO:tasks.rados.rados.0.smithi001.stdout:224: finishing rollback tid 0 to smithi00132172-21 2024-09-15T03:47:20.971 INFO:tasks.rados.rados.0.smithi001.stdout:223: finishing write tid 2 to smithi00132172-16 2024-09-15T03:47:20.971 INFO:tasks.rados.rados.0.smithi001.stdout:226: done (5 left) 2024-09-15T03:47:20.972 INFO:tasks.rados.rados.0.smithi001.stdout:228: done (4 left) 2024-09-15T03:47:20.972 INFO:tasks.rados.rados.0.smithi001.stdout:229: copy_from oid 4 from oid 34 current snap is 28 2024-09-15T03:47:20.972 INFO:tasks.rados.rados.0.smithi001.stdout:223: finishing write tid 3 to smithi00132172-16 2024-09-15T03:47:20.972 INFO:tasks.rados.rados.0.smithi001.stdout:224: finishing rollback tid 1 to smithi00132172-21 2024-09-15T03:47:20.972 INFO:tasks.rados.rados.0.smithi001.stdout:223: finishing write tid 4 to smithi00132172-16 2024-09-15T03:47:20.972 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 18 v 33 (ObjNum 90 snap 21 seq_num 90) dirty exists 2024-09-15T03:47:20.972 INFO:tasks.rados.rados.0.smithi001.stdout:224: finishing rollback tid 2 to smithi00132172-21 2024-09-15T03:47:20.972 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 21 v 34 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-15T03:47:20.972 INFO:tasks.rados.rados.0.smithi001.stdout:223: finishing write tid 5 to smithi00132172-16 2024-09-15T03:47:20.972 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 16 v 31 (ObjNum 102 snap 28 seq_num 102) dirty exists 2024-09-15T03:47:20.972 INFO:tasks.rados.rados.0.smithi001.stdout:223: left oid 16 (ObjNum 102 snap 28 seq_num 102) 2024-09-15T03:47:20.972 INFO:tasks.rados.rados.0.smithi001.stdout:227: finishing write tid 1 to smithi00132172-9 2024-09-15T03:47:20.972 INFO:tasks.rados.rados.0.smithi001.stdout:227: finishing write tid 2 to smithi00132172-9 2024-09-15T03:47:20.973 INFO:tasks.rados.rados.0.smithi001.stdout:227: finishing write tid 3 to smithi00132172-9 2024-09-15T03:47:20.973 INFO:tasks.rados.rados.0.smithi001.stdout:227: finishing write tid 4 to smithi00132172-9 2024-09-15T03:47:20.973 INFO:tasks.rados.rados.0.smithi001.stdout:227: finishing write tid 5 to smithi00132172-9 2024-09-15T03:47:20.973 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 9 v 79 (ObjNum 103 snap 28 seq_num 103) dirty exists 2024-09-15T03:47:20.973 INFO:tasks.rados.rados.0.smithi001.stdout:227: left oid 9 (ObjNum 103 snap 28 seq_num 103) 2024-09-15T03:47:20.973 INFO:tasks.rados.rados.0.smithi001.stdout:223: done (4 left) 2024-09-15T03:47:20.973 INFO:tasks.rados.rados.0.smithi001.stdout:224: done (3 left) 2024-09-15T03:47:20.973 INFO:tasks.rados.rados.0.smithi001.stdout:225: done (2 left) 2024-09-15T03:47:20.973 INFO:tasks.rados.rados.0.smithi001.stdout:227: done (1 left) 2024-09-15T03:47:20.973 INFO:tasks.rados.rados.0.smithi001.stdout:230: snap_create 2024-09-15T03:47:20.978 INFO:tasks.rados.rados.0.smithi001.stdout:229: finishing copy_from racing read to smithi00132172-4 2024-09-15T03:47:20.979 INFO:tasks.rados.rados.0.smithi001.stdout:229: finishing copy_from to smithi00132172-4 2024-09-15T03:47:20.979 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 4 v 8 (ObjNum 70 snap 8 seq_num 70) dirty exists 2024-09-15T03:47:21.026 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:21.026+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:21.044 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:21.044 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:21.044 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:21.044 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:21.052 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:47:21.127 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:21.126+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:21.151 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:21.151 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:21.151 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:21.151 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:21.151 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:21.159 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:47:21.228 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:21.226+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:21.258 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:21.258 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:21.258 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:21.258 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:47:21.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.259 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:47:21.259 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:21.259 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 63.353634616999997, 2024-09-15T03:47:21.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10988437500000001, 2024-09-15T03:47:21.259 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.259 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:21.259 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:21.259 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:47:21.259 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:47:21.259 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1 2024-09-15T03:47:21.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.260 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:21.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:21.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.261 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902355+0000", 2024-09-15T03:47:21.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.429e-06 2024-09-15T03:47:21.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.261 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.261 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902362+0000", 2024-09-15T03:47:21.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7290000000000003e-06 2024-09-15T03:47:21.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.261 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.261 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902373+0000", 2024-09-15T03:47:21.262 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0533999999999999e-05 2024-09-15T03:47:21.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.262 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.262 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902379+0000", 2024-09-15T03:47:21.262 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4350000000000002e-06 2024-09-15T03:47:21.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.262 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.262 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902428+0000", 2024-09-15T03:47:21.262 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8571e-05 2024-09-15T03:47:21.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.263 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:47:21.263 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902434+0000", 2024-09-15T03:47:21.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5610000000000004e-06 2024-09-15T03:47:21.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.263 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.263 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011307+0000", 2024-09-15T03:47:21.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.108872306 2024-09-15T03:47:21.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.264 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:21.264 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011448+0000", 2024-09-15T03:47:21.264 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014137199999999999 2024-09-15T03:47:21.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.264 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.264 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:47:21.264 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011569+0000", 2024-09-15T03:47:21.264 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120925 2024-09-15T03:47:21.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.264 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.264 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.264 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011695+0000", 2024-09-15T03:47:21.264 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012586800000000001 2024-09-15T03:47:21.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011703+0000", 2024-09-15T03:47:21.265 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4190000000000002e-06 2024-09-15T03:47:21.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011739+0000", 2024-09-15T03:47:21.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5474e-05 2024-09-15T03:47:21.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.266 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.266 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:47:21.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011940+0000", 2024-09-15T03:47:21.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020108699999999999 2024-09-15T03:47:21.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.267 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:47:21.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012179+0000", 2024-09-15T03:47:21.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023950299999999999 2024-09-15T03:47:21.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:21.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012215+0000", 2024-09-15T03:47:21.268 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5332e-05 2024-09-15T03:47:21.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.269 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.269 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.269 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012238+0000", 2024-09-15T03:47:21.269 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3830000000000001e-05 2024-09-15T03:47:21.269 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.269 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.269 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.269 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.269 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.270 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4263.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2024-09-15T03:47:21.270 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:21.270 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 57.212237930999997, 2024-09-15T03:47:21.270 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019495788999999999, 2024-09-15T03:47:21.270 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.270 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.270 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:21.270 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4263", 2024-09-15T03:47:21.270 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/646873808", 2024-09-15T03:47:21.270 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2 2024-09-15T03:47:21.270 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.270 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.271 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:21.271 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.271 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:21.271 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.271 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043752+0000", 2024-09-15T03:47:21.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.815e-06 2024-09-15T03:47:21.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043757+0000", 2024-09-15T03:47:21.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4240000000000003e-06 2024-09-15T03:47:21.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043766+0000", 2024-09-15T03:47:21.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8629999999999997e-06 2024-09-15T03:47:21.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043773+0000", 2024-09-15T03:47:21.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0099999999999998e-06 2024-09-15T03:47:21.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043830+0000", 2024-09-15T03:47:21.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7089999999999997e-05 2024-09-15T03:47:21.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:47:21.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043837+0000", 2024-09-15T03:47:21.274 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.08e-06 2024-09-15T03:47:21.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.274 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.274 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.274 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063103+0000", 2024-09-15T03:47:21.274 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019265619000000001 2024-09-15T03:47:21.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.274 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.274 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.274 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063246+0000", 2024-09-15T03:47:21.274 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000143888 2024-09-15T03:47:21.274 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.274 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.274 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.275 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.275 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/31 ECSubWrite(tid=38, reqid=client.4521.0:223, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:47:21.275 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:47:21.275 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 36.108781594, 2024-09-15T03:47:21.275 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010504493, 2024-09-15T03:47:21.275 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.275 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.275 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.275 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.275 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.275 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:47:21.275 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.276 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.276 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.276 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:47:21.276 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.276 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.276 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.276 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:47:21.276 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4999999999999999e-07 2024-09-15T03:47:21.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.276 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.276 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147648+0000", 2024-09-15T03:47:21.277 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00044030399999999998 2024-09-15T03:47:21.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.277 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.277 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147660+0000", 2024-09-15T03:47:21.277 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2517000000000001e-05 2024-09-15T03:47:21.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.277 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.277 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147667+0000", 2024-09-15T03:47:21.277 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.737e-06 2024-09-15T03:47:21.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.278 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.278 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.278 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147713+0000", 2024-09-15T03:47:21.278 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6405e-05 2024-09-15T03:47:21.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.278 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.278 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.278 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147719+0000", 2024-09-15T03:47:21.278 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0580000000000003e-06 2024-09-15T03:47:21.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.278 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.278 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.278 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147905+0000", 2024-09-15T03:47:21.278 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018574600000000001 2024-09-15T03:47:21.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.279 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.279 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147917+0000", 2024-09-15T03:47:21.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1749999999999999e-05 2024-09-15T03:47:21.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.279 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.279 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148022+0000", 2024-09-15T03:47:21.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104863 2024-09-15T03:47:21.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.279 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.279 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148037+0000", 2024-09-15T03:47:21.280 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5302999999999999e-05 2024-09-15T03:47:21.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.280 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.280 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.280 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157711+0000", 2024-09-15T03:47:21.280 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0096745600000000005 2024-09-15T03:47:21.280 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.280 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.280 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.280 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.280 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:47:21.280 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:21.280 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 35.085164542000001, 2024-09-15T03:47:21.281 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011878227, 2024-09-15T03:47:21.281 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.281 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.281 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.281 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.281 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.281 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:21.281 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.281 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.281 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.281 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.281 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:21.282 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.282 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.282 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170825+0000", 2024-09-15T03:47:21.282 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9599999999999999e-07 2024-09-15T03:47:21.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.282 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.282 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171100+0000", 2024-09-15T03:47:21.282 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027548500000000001 2024-09-15T03:47:21.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.282 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.282 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.283 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171112+0000", 2024-09-15T03:47:21.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2152e-05 2024-09-15T03:47:21.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.283 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.283 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.283 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171119+0000", 2024-09-15T03:47:21.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5050000000000004e-06 2024-09-15T03:47:21.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.283 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.283 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.283 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171162+0000", 2024-09-15T03:47:21.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2908000000000003e-05 2024-09-15T03:47:21.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.284 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.284 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.284 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171169+0000", 2024-09-15T03:47:21.284 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9739999999999996e-06 2024-09-15T03:47:21.284 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.284 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.284 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.284 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171398+0000", 2024-09-15T03:47:21.284 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022954799999999999 2024-09-15T03:47:21.284 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.284 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.284 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.284 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173640+0000", 2024-09-15T03:47:21.285 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022417000000000001 2024-09-15T03:47:21.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.285 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.285 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.285 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173705+0000", 2024-09-15T03:47:21.285 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4907999999999996e-05 2024-09-15T03:47:21.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.285 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.285 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.285 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173742+0000", 2024-09-15T03:47:21.285 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6940999999999997e-05 2024-09-15T03:47:21.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.285 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.285 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.286 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.182702+0000", 2024-09-15T03:47:21.286 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0089604100000000002 2024-09-15T03:47:21.286 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.286 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.286 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.286 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.286 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.286 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 39/31 ECSubWrite(tid=43, reqid=client.4521.0:252, at_version=39'27, trim_to=0'0, roll_forward_to=39'25))", 2024-09-15T03:47:21.286 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:21.286 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 35.077447223999997, 2024-09-15T03:47:21.286 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011286757, 2024-09-15T03:47:21.286 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.287 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.287 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.287 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.287 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:21.287 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.287 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.287 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:21.287 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.287 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.288 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.288 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178542+0000", 2024-09-15T03:47:21.288 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9400000000000004e-07 2024-09-15T03:47:21.288 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.288 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.288 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.288 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178754+0000", 2024-09-15T03:47:21.288 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021228800000000001 2024-09-15T03:47:21.288 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.288 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.288 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.288 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178765+0000", 2024-09-15T03:47:21.288 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0176e-05 2024-09-15T03:47:21.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.289 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.289 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.289 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178769+0000", 2024-09-15T03:47:21.289 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1359999999999999e-06 2024-09-15T03:47:21.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.289 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.289 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.289 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178810+0000", 2024-09-15T03:47:21.289 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0825999999999998e-05 2024-09-15T03:47:21.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.289 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.289 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178820+0000", 2024-09-15T03:47:21.290 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0237999999999999e-05 2024-09-15T03:47:21.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.290 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.290 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179033+0000", 2024-09-15T03:47:21.290 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021361500000000001 2024-09-15T03:47:21.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.290 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.290 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179040+0000", 2024-09-15T03:47:21.290 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0509999999999999e-06 2024-09-15T03:47:21.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.291 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.291 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179102+0000", 2024-09-15T03:47:21.291 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1977000000000006e-05 2024-09-15T03:47:21.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.291 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.291 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179114+0000", 2024-09-15T03:47:21.291 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1223000000000001e-05 2024-09-15T03:47:21.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.291 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.291 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.189828+0000", 2024-09-15T03:47:21.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010714433000000001 2024-09-15T03:47:21.292 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.292 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.292 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.292 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.292 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 39/31 ECSubWrite(tid=44, reqid=client.4521.0:252, at_version=0'0, trim_to=0'0, roll_forward_to=39'27))", 2024-09-15T03:47:21.292 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.179948+0000", 2024-09-15T03:47:21.292 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 35.076040124999999, 2024-09-15T03:47:21.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010068739, 2024-09-15T03:47:21.292 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.292 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.293 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.293 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.293 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.293 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179948+0000", 2024-09-15T03:47:21.293 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.293 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.293 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.293 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.293 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179948+0000", 2024-09-15T03:47:21.293 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.293 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.293 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.293 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.294 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179949+0000", 2024-09-15T03:47:21.294 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6199999999999998e-07 2024-09-15T03:47:21.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.294 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.294 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.294 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179960+0000", 2024-09-15T03:47:21.294 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1216000000000001e-05 2024-09-15T03:47:21.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.294 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.294 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.294 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179970+0000", 2024-09-15T03:47:21.294 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8980000000000008e-06 2024-09-15T03:47:21.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.294 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.295 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.295 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179975+0000", 2024-09-15T03:47:21.295 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5650000000000003e-06 2024-09-15T03:47:21.295 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.295 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.295 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.295 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180004+0000", 2024-09-15T03:47:21.295 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9224999999999999e-05 2024-09-15T03:47:21.295 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.295 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.295 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.295 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180009+0000", 2024-09-15T03:47:21.295 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9250000000000003e-06 2024-09-15T03:47:21.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.296 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.296 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.296 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180061+0000", 2024-09-15T03:47:21.296 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1758000000000001e-05 2024-09-15T03:47:21.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.296 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.296 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.296 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180067+0000", 2024-09-15T03:47:21.297 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3269999999999997e-06 2024-09-15T03:47:21.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.297 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.297 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.297 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180101+0000", 2024-09-15T03:47:21.297 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3701000000000001e-05 2024-09-15T03:47:21.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.297 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.297 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.297 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180112+0000", 2024-09-15T03:47:21.297 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1287000000000001e-05 2024-09-15T03:47:21.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.297 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.297 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.298 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.190017+0000", 2024-09-15T03:47:21.298 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0099052749999999998 2024-09-15T03:47:21.298 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.298 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.298 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.298 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.298 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 51/31 ECSubWrite(tid=67, reqid=client.4521.0:350, at_version=51'33, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:21.298 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:21.298 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 23.412147182999998, 2024-09-15T03:47:21.298 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.015569239, 2024-09-15T03:47:21.298 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.298 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.299 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.299 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.299 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.299 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:21.299 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.299 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.299 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.299 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:21.299 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.299 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.300 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.300 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843843+0000", 2024-09-15T03:47:21.300 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.195e-06 2024-09-15T03:47:21.300 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.300 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.300 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.300 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843880+0000", 2024-09-15T03:47:21.300 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7527999999999997e-05 2024-09-15T03:47:21.300 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.300 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.300 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.300 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843898+0000", 2024-09-15T03:47:21.300 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7696000000000001e-05 2024-09-15T03:47:21.301 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.301 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.301 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.301 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843907+0000", 2024-09-15T03:47:21.301 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8820000000000007e-06 2024-09-15T03:47:21.301 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.301 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.301 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.301 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843978+0000", 2024-09-15T03:47:21.301 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1537000000000003e-05 2024-09-15T03:47:21.301 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.301 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.301 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.302 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843989+0000", 2024-09-15T03:47:21.302 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0720000000000001e-05 2024-09-15T03:47:21.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.302 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.302 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.302 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844102+0000", 2024-09-15T03:47:21.302 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000113406 2024-09-15T03:47:21.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.302 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.302 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.302 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844131+0000", 2024-09-15T03:47:21.302 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8762e-05 2024-09-15T03:47:21.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.302 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.303 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.303 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844180+0000", 2024-09-15T03:47:21.303 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9329999999999997e-05 2024-09-15T03:47:21.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.303 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.303 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.303 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844201+0000", 2024-09-15T03:47:21.303 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0772999999999999e-05 2024-09-15T03:47:21.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.303 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.303 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.303 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.859411+0000", 2024-09-15T03:47:21.303 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01520941 2024-09-15T03:47:21.304 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.304 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.304 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.304 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.304 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.304 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:21.304 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:21.304 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 23.411268272000001, 2024-09-15T03:47:21.304 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012156452, 2024-09-15T03:47:21.304 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.304 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.304 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.304 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.305 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.305 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:21.305 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.305 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.305 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.305 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:21.305 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.305 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.305 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.305 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844721+0000", 2024-09-15T03:47:21.305 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1100000000000005e-07 2024-09-15T03:47:21.306 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.306 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.306 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.306 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844743+0000", 2024-09-15T03:47:21.306 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1999000000000001e-05 2024-09-15T03:47:21.306 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.306 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.306 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.306 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844758+0000", 2024-09-15T03:47:21.306 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4718999999999999e-05 2024-09-15T03:47:21.306 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.306 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.306 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.306 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844772+0000", 2024-09-15T03:47:21.307 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4637000000000001e-05 2024-09-15T03:47:21.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.307 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.307 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.307 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844842+0000", 2024-09-15T03:47:21.307 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9596000000000002e-05 2024-09-15T03:47:21.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.307 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.307 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.307 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844853+0000", 2024-09-15T03:47:21.307 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0912e-05 2024-09-15T03:47:21.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.307 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.307 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.308 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844966+0000", 2024-09-15T03:47:21.308 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011276 2024-09-15T03:47:21.308 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.308 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.308 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.308 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844977+0000", 2024-09-15T03:47:21.308 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1676e-05 2024-09-15T03:47:21.308 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.308 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.308 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.308 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847414+0000", 2024-09-15T03:47:21.308 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024363760000000001 2024-09-15T03:47:21.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.309 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.309 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.309 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847438+0000", 2024-09-15T03:47:21.309 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3737e-05 2024-09-15T03:47:21.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.309 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.309 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.309 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856877+0000", 2024-09-15T03:47:21.309 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094392290000000004 2024-09-15T03:47:21.309 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.309 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.309 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.310 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.310 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:21.310 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:21.310 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 23.411198877, 2024-09-15T03:47:21.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012258211, 2024-09-15T03:47:21.310 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.310 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.310 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.310 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.310 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.310 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:21.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.311 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:21.311 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.311 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:21.311 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.58e-07 2024-09-15T03:47:21.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.311 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844818+0000", 2024-09-15T03:47:21.312 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.796e-05 2024-09-15T03:47:21.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.312 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.312 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.312 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844829+0000", 2024-09-15T03:47:21.312 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0895e-05 2024-09-15T03:47:21.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.312 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.312 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.312 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844836+0000", 2024-09-15T03:47:21.312 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.618e-06 2024-09-15T03:47:21.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.312 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.312 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845023+0000", 2024-09-15T03:47:21.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018743500000000001 2024-09-15T03:47:21.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.313 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.313 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845035+0000", 2024-09-15T03:47:21.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1958e-05 2024-09-15T03:47:21.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.313 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.313 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845147+0000", 2024-09-15T03:47:21.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112281 2024-09-15T03:47:21.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.314 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.314 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.314 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847417+0000", 2024-09-15T03:47:21.314 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022693969999999998 2024-09-15T03:47:21.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.314 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.314 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.314 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847462+0000", 2024-09-15T03:47:21.314 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5466999999999997e-05 2024-09-15T03:47:21.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.314 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.314 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.314 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847503+0000", 2024-09-15T03:47:21.315 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1022999999999997e-05 2024-09-15T03:47:21.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.315 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.315 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.315 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.857048+0000", 2024-09-15T03:47:21.315 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095447190000000001 2024-09-15T03:47:21.315 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.315 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.315 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.315 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.315 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 51/31 ECSubWrite(tid=68, reqid=client.4521.0:351, at_version=51'34, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:21.315 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:21.315 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 23.410780759000001, 2024-09-15T03:47:21.316 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.014544793, 2024-09-15T03:47:21.316 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.316 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.316 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.316 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.316 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:21.316 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.316 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.316 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:21.316 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.317 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.317 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845209+0000", 2024-09-15T03:47:21.317 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5300000000000003e-07 2024-09-15T03:47:21.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.317 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.317 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845227+0000", 2024-09-15T03:47:21.317 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8759000000000001e-05 2024-09-15T03:47:21.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.317 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.317 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.318 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845241+0000", 2024-09-15T03:47:21.318 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3740000000000001e-05 2024-09-15T03:47:21.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.318 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.318 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.318 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845247+0000", 2024-09-15T03:47:21.318 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2779999999999999e-06 2024-09-15T03:47:21.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.318 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.318 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.318 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845308+0000", 2024-09-15T03:47:21.318 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0702000000000003e-05 2024-09-15T03:47:21.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.318 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.319 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.319 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845317+0000", 2024-09-15T03:47:21.319 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3770000000000002e-06 2024-09-15T03:47:21.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.319 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.319 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.319 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845413+0000", 2024-09-15T03:47:21.319 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5356e-05 2024-09-15T03:47:21.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.319 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.319 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.319 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847421+0000", 2024-09-15T03:47:21.319 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020077200000000002 2024-09-15T03:47:21.320 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.320 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.320 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.320 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847466+0000", 2024-09-15T03:47:21.320 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5754999999999998e-05 2024-09-15T03:47:21.320 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.320 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.320 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.320 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847572+0000", 2024-09-15T03:47:21.320 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010594000000000001 2024-09-15T03:47:21.320 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.320 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.320 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.321 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.859753+0000", 2024-09-15T03:47:21.321 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012180412999999999 2024-09-15T03:47:21.321 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.321 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.321 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.321 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.321 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.321 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 60/28 ECSubWrite(tid=73, reqid=client.4521.0:424, at_version=60'41, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:21.321 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:21.321 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.355577504999999, 2024-09-15T03:47:21.321 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01137208, 2024-09-15T03:47:21.321 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.321 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.321 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.322 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.322 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.322 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:21.322 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.322 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.322 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.322 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.322 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:21.322 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.322 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.322 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.322 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.322 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900412+0000", 2024-09-15T03:47:21.322 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7599999999999996e-07 2024-09-15T03:47:21.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.323 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.323 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.323 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:21.323 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7986999999999999e-05 2024-09-15T03:47:21.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.323 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.323 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.323 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900445+0000", 2024-09-15T03:47:21.323 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.468e-05 2024-09-15T03:47:21.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.323 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.323 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.324 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900451+0000", 2024-09-15T03:47:21.324 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5799999999999997e-06 2024-09-15T03:47:21.324 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.324 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.324 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.324 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900507+0000", 2024-09-15T03:47:21.324 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5855e-05 2024-09-15T03:47:21.324 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.324 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.324 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.324 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900530+0000", 2024-09-15T03:47:21.324 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2929e-05 2024-09-15T03:47:21.324 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.325 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.325 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.325 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900640+0000", 2024-09-15T03:47:21.325 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000110536 2024-09-15T03:47:21.325 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.325 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.325 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.325 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900648+0000", 2024-09-15T03:47:21.325 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9470000000000003e-06 2024-09-15T03:47:21.325 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.325 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.325 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.325 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900764+0000", 2024-09-15T03:47:21.326 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011518099999999999 2024-09-15T03:47:21.326 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.326 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.326 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.326 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900803+0000", 2024-09-15T03:47:21.326 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9941999999999998e-05 2024-09-15T03:47:21.326 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.326 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.326 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.326 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.911783+0000", 2024-09-15T03:47:21.326 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010979667 2024-09-15T03:47:21.326 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.326 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.327 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.327 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.327 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 60/28 ECSubWrite(tid=74, reqid=client.4521.0:425, at_version=60'42, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:21.327 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.901183+0000", 2024-09-15T03:47:21.327 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.354805507, 2024-09-15T03:47:21.327 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010721191999999999, 2024-09-15T03:47:21.327 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.327 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.327 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.327 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.327 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.327 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901183+0000", 2024-09-15T03:47:21.327 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.328 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.328 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901183+0000", 2024-09-15T03:47:21.328 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.328 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.328 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901184+0000", 2024-09-15T03:47:21.328 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6599999999999996e-07 2024-09-15T03:47:21.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.328 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.328 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901202+0000", 2024-09-15T03:47:21.329 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8658999999999999e-05 2024-09-15T03:47:21.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.329 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.329 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.329 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901217+0000", 2024-09-15T03:47:21.329 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4371e-05 2024-09-15T03:47:21.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.329 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.329 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.329 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901223+0000", 2024-09-15T03:47:21.329 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2489999999999998e-06 2024-09-15T03:47:21.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.329 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.329 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.330 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901274+0000", 2024-09-15T03:47:21.330 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0602000000000001e-05 2024-09-15T03:47:21.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.330 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.330 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.330 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901284+0000", 2024-09-15T03:47:21.330 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.995e-06 2024-09-15T03:47:21.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.330 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.330 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.330 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901372+0000", 2024-09-15T03:47:21.330 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8399000000000005e-05 2024-09-15T03:47:21.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.330 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.331 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901380+0000", 2024-09-15T03:47:21.331 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8040000000000002e-06 2024-09-15T03:47:21.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.331 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903553+0000", 2024-09-15T03:47:21.331 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021731580000000001 2024-09-15T03:47:21.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.331 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903570+0000", 2024-09-15T03:47:21.331 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7218999999999999e-05 2024-09-15T03:47:21.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.332 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.332 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.332 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.911904+0000", 2024-09-15T03:47:21.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0083340700000000007 2024-09-15T03:47:21.332 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.332 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.332 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.332 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.332 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:517 3.3s1 3:caaee79c:::smithi00132172-10:head [read 0~3145728,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:21.332 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.947050+0000", 2024-09-15T03:47:21.332 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.3089385719999997, 2024-09-15T03:47:21.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012110134, 2024-09-15T03:47:21.333 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.333 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:21.333 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:21.333 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:21.333 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:21.333 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 517 2024-09-15T03:47:21.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.333 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.333 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.333 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.333 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947050+0000", 2024-09-15T03:47:21.333 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.334 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.334 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947050+0000", 2024-09-15T03:47:21.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.334 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.334 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947052+0000", 2024-09-15T03:47:21.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9099999999999999e-06 2024-09-15T03:47:21.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.334 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.334 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947060+0000", 2024-09-15T03:47:21.335 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8529999999999992e-06 2024-09-15T03:47:21.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947074+0000", 2024-09-15T03:47:21.335 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4344e-05 2024-09-15T03:47:21.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947082+0000", 2024-09-15T03:47:21.335 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9249999999999995e-06 2024-09-15T03:47:21.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.336 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947138+0000", 2024-09-15T03:47:21.336 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6054000000000002e-05 2024-09-15T03:47:21.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.336 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.336 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:21.336 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947339+0000", 2024-09-15T03:47:21.336 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000200829 2024-09-15T03:47:21.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.336 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.336 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.336 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.959160+0000", 2024-09-15T03:47:21.336 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011821219000000001 2024-09-15T03:47:21.336 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.337 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.337 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.337 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.337 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.337 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:518 3.3s1 3:caaee79c:::smithi00132172-10:head [read 0~3145728,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:21.337 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.947091+0000", 2024-09-15T03:47:21.337 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.30889743, 2024-09-15T03:47:21.337 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017608233000000001, 2024-09-15T03:47:21.337 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.337 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:21.337 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:21.337 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:21.337 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:21.337 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 518 2024-09-15T03:47:21.338 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.338 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.338 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.338 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.338 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947091+0000", 2024-09-15T03:47:21.338 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.338 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.338 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.338 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.338 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947091+0000", 2024-09-15T03:47:21.338 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.338 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.338 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.339 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.339 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947092+0000", 2024-09-15T03:47:21.339 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7800000000000002e-07 2024-09-15T03:47:21.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.339 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.339 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.339 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947095+0000", 2024-09-15T03:47:21.339 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2959999999999999e-06 2024-09-15T03:47:21.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.339 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.339 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.339 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947102+0000", 2024-09-15T03:47:21.339 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7469999999999999e-06 2024-09-15T03:47:21.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947106+0000", 2024-09-15T03:47:21.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6550000000000003e-06 2024-09-15T03:47:21.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947622+0000", 2024-09-15T03:47:21.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000515727 2024-09-15T03:47:21.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:21.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947751+0000", 2024-09-15T03:47:21.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012920600000000001 2024-09-15T03:47:21.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.341 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.341 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.341 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.964699+0000", 2024-09-15T03:47:21.341 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.016948123999999998 2024-09-15T03:47:21.341 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.341 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.341 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.341 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.341 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:519 3.3s1 3:caaee79c:::smithi00132172-10:head [read 0~3145728,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:21.341 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:21.341 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.3088637240000001, 2024-09-15T03:47:21.342 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.021743243999999998, 2024-09-15T03:47:21.342 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.342 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:21.342 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:21.342 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:21.342 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:21.342 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 519 2024-09-15T03:47:21.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.342 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.342 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.342 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:21.342 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:21.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:21.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5999999999999999e-07 2024-09-15T03:47:21.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947130+0000", 2024-09-15T03:47:21.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7720000000000002e-06 2024-09-15T03:47:21.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947137+0000", 2024-09-15T03:47:21.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2370000000000003e-06 2024-09-15T03:47:21.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947141+0000", 2024-09-15T03:47:21.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8809999999999998e-06 2024-09-15T03:47:21.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.948008+0000", 2024-09-15T03:47:21.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00086656700000000005 2024-09-15T03:47:21.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:21.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.948113+0000", 2024-09-15T03:47:21.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104904 2024-09-15T03:47:21.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.968868+0000", 2024-09-15T03:47:21.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.020755422999999999 2024-09-15T03:47:21.345 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.345 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.345 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.346 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.3s1 69/68 ECSubReadReply(tid=162, attrs_read=0))", 2024-09-15T03:47:21.346 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.948615+0000", 2024-09-15T03:47:21.346 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.3073731430000004, 2024-09-15T03:47:21.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010628187000000001, 2024-09-15T03:47:21.346 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.346 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.346 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.948615+0000", 2024-09-15T03:47:21.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.948615+0000", 2024-09-15T03:47:21.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.948616+0000", 2024-09-15T03:47:21.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9699999999999996e-07 2024-09-15T03:47:21.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953599+0000", 2024-09-15T03:47:21.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049830919999999997 2024-09-15T03:47:21.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953642+0000", 2024-09-15T03:47:21.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3084999999999998e-05 2024-09-15T03:47:21.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953650+0000", 2024-09-15T03:47:21.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0930000000000003e-06 2024-09-15T03:47:21.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.954573+0000", 2024-09-15T03:47:21.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00092271499999999995 2024-09-15T03:47:21.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.959244+0000", 2024-09-15T03:47:21.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0046707049999999998 2024-09-15T03:47:21.350 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.350 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.350 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.350 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.3s1 69/68 ECSubReadReply(tid=163, attrs_read=0))", 2024-09-15T03:47:21.350 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:21.350 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.30228798, 2024-09-15T03:47:21.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011102266, 2024-09-15T03:47:21.350 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.350 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.350 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.350 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:21.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:21.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:21.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1199999999999998e-07 2024-09-15T03:47:21.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956300+0000", 2024-09-15T03:47:21.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025987459999999999 2024-09-15T03:47:21.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956314+0000", 2024-09-15T03:47:21.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4469e-05 2024-09-15T03:47:21.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956319+0000", 2024-09-15T03:47:21.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.746e-06 2024-09-15T03:47:21.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.959591+0000", 2024-09-15T03:47:21.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032721510000000001 2024-09-15T03:47:21.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.964803+0000", 2024-09-15T03:47:21.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0052117420000000001 2024-09-15T03:47:21.353 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.353 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.353 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.354 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.3s1 69/68 ECSubReadReply(tid=164, attrs_read=0))", 2024-09-15T03:47:21.354 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:21.354 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.2996635469999998, 2024-09-15T03:47:21.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012633276000000001, 2024-09-15T03:47:21.354 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.354 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.354 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.354 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:21.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:21.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:21.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5900000000000003e-07 2024-09-15T03:47:21.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957786+0000", 2024-09-15T03:47:21.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001460755 2024-09-15T03:47:21.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957807+0000", 2024-09-15T03:47:21.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0937e-05 2024-09-15T03:47:21.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957814+0000", 2024-09-15T03:47:21.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5989999999999998e-06 2024-09-15T03:47:21.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.964911+0000", 2024-09-15T03:47:21.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0070976370000000004 2024-09-15T03:47:21.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.968958+0000", 2024-09-15T03:47:21.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.004046989 2024-09-15T03:47:21.357 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.357 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.357 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.357 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:567 3.ds0 3:b166213b:::smithi00132172-21:head [rollback 3] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:21.358 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.947576+0000", 2024-09-15T03:47:21.358 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.308412619, 2024-09-15T03:47:21.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013871319999999999, 2024-09-15T03:47:21.358 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.358 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:21.358 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:21.358 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:21.358 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:21.358 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 567 2024-09-15T03:47:21.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.358 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947576+0000", 2024-09-15T03:47:21.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947576+0000", 2024-09-15T03:47:21.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947577+0000", 2024-09-15T03:47:21.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.37e-07 2024-09-15T03:47:21.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947580+0000", 2024-09-15T03:47:21.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5710000000000002e-06 2024-09-15T03:47:21.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947588+0000", 2024-09-15T03:47:21.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8939999999999993e-06 2024-09-15T03:47:21.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947593+0000", 2024-09-15T03:47:21.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7049999999999998e-06 2024-09-15T03:47:21.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948070+0000", 2024-09-15T03:47:21.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047685099999999998 2024-09-15T03:47:21.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:21.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948177+0000", 2024-09-15T03:47:21.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010765299999999999 2024-09-15T03:47:21.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948986+0000", 2024-09-15T03:47:21.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000808977 2024-09-15T03:47:21.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949117+0000", 2024-09-15T03:47:21.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000131032 2024-09-15T03:47:21.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949133+0000", 2024-09-15T03:47:21.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5798999999999999e-05 2024-09-15T03:47:21.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.363 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950936+0000", 2024-09-15T03:47:21.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00180316 2024-09-15T03:47:21.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.363 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950994+0000", 2024-09-15T03:47:21.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7884999999999999e-05 2024-09-15T03:47:21.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.363 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:21.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951532+0000", 2024-09-15T03:47:21.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00053787699999999997 2024-09-15T03:47:21.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.364 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.364 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961447+0000", 2024-09-15T03:47:21.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0099152790000000008 2024-09-15T03:47:21.364 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.364 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.364 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.364 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:568 3.ds0 3:b166213b:::smithi00132172-21:head [append 0~0] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:21.364 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.948048+0000", 2024-09-15T03:47:21.364 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.30794069400000001, 2024-09-15T03:47:21.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013579269, 2024-09-15T03:47:21.365 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.365 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:21.365 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:21.365 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:21.365 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:21.365 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 568 2024-09-15T03:47:21.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.365 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948048+0000", 2024-09-15T03:47:21.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.366 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948048+0000", 2024-09-15T03:47:21.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.366 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948049+0000", 2024-09-15T03:47:21.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7199999999999998e-07 2024-09-15T03:47:21.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.366 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948056+0000", 2024-09-15T03:47:21.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0280000000000003e-06 2024-09-15T03:47:21.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948066+0000", 2024-09-15T03:47:21.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0141e-05 2024-09-15T03:47:21.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948072+0000", 2024-09-15T03:47:21.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1519999999999997e-06 2024-09-15T03:47:21.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949221+0000", 2024-09-15T03:47:21.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001149125 2024-09-15T03:47:21.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:21.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949313+0000", 2024-09-15T03:47:21.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1469000000000001e-05 2024-09-15T03:47:21.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949818+0000", 2024-09-15T03:47:21.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00050549700000000004 2024-09-15T03:47:21.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950002+0000", 2024-09-15T03:47:21.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018414700000000001 2024-09-15T03:47:21.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950940+0000", 2024-09-15T03:47:21.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00093786100000000001 2024-09-15T03:47:21.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951004+0000", 2024-09-15T03:47:21.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3859e-05 2024-09-15T03:47:21.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951020+0000", 2024-09-15T03:47:21.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.628e-05 2024-09-15T03:47:21.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:21.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951753+0000", 2024-09-15T03:47:21.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00073276999999999999 2024-09-15T03:47:21.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961627+0000", 2024-09-15T03:47:21.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098741679999999991 2024-09-15T03:47:21.371 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.371 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.371 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.371 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.371 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.371 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:21.372 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:47:21.373 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:21.371+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:21.473 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:21.471+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:21.510 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:21.511 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:21.511 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:21.511 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:21.518 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:47:21.573 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:21.573+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:21.655 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:21.655 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:21.656 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:21.656 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:21.656 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:21.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:47:21.708 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:21.706+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:21.762 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:21.763 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:21.763 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:21.763 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:47:21.763 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.763 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:47:21.763 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:21.763 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 35.589597363000003, 2024-09-15T03:47:21.763 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012183955999999999, 2024-09-15T03:47:21.764 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.764 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.764 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.764 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.764 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.764 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:21.764 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.764 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.765 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.765 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.765 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:21.765 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.765 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.765 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.765 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:21.765 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.17e-07 2024-09-15T03:47:21.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.765 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.765 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.766 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171454+0000", 2024-09-15T03:47:21.766 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00035226700000000002 2024-09-15T03:47:21.766 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.766 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.766 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.766 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171465+0000", 2024-09-15T03:47:21.766 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0943e-05 2024-09-15T03:47:21.766 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.766 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.766 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.767 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171471+0000", 2024-09-15T03:47:21.767 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7400000000000001e-06 2024-09-15T03:47:21.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.767 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.767 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.767 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171508+0000", 2024-09-15T03:47:21.767 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7147999999999997e-05 2024-09-15T03:47:21.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.767 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.767 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.767 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171517+0000", 2024-09-15T03:47:21.767 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2560000000000008e-06 2024-09-15T03:47:21.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.768 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.768 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171716+0000", 2024-09-15T03:47:21.768 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019899699999999999 2024-09-15T03:47:21.768 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.768 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.768 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173686+0000", 2024-09-15T03:47:21.768 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019696829999999999 2024-09-15T03:47:21.768 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.768 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.768 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173777+0000", 2024-09-15T03:47:21.769 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0637999999999995e-05 2024-09-15T03:47:21.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.769 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.769 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173814+0000", 2024-09-15T03:47:21.769 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7499000000000001e-05 2024-09-15T03:47:21.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.769 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.769 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.183286+0000", 2024-09-15T03:47:21.769 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094716680000000008 2024-09-15T03:47:21.769 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.769 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.770 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.770 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.770 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 51/31 ECSubWrite(tid=67, reqid=client.4521.0:350, at_version=51'33, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:21.770 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.843852+0000", 2024-09-15T03:47:21.770 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 23.916847401999998, 2024-09-15T03:47:21.770 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010895471, 2024-09-15T03:47:21.770 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.770 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.770 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.770 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.770 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.771 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843852+0000", 2024-09-15T03:47:21.771 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.771 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.771 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.771 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.771 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843852+0000", 2024-09-15T03:47:21.771 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.771 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.771 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.771 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.771 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843853+0000", 2024-09-15T03:47:21.772 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.359e-06 2024-09-15T03:47:21.772 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.772 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.772 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.772 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843882+0000", 2024-09-15T03:47:21.772 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8711999999999999e-05 2024-09-15T03:47:21.772 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.772 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.772 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.772 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843901+0000", 2024-09-15T03:47:21.772 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9477e-05 2024-09-15T03:47:21.772 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.772 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.773 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843911+0000", 2024-09-15T03:47:21.773 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2590000000000006e-06 2024-09-15T03:47:21.773 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.773 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.773 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843973+0000", 2024-09-15T03:47:21.773 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2310999999999993e-05 2024-09-15T03:47:21.773 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.773 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.773 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843984+0000", 2024-09-15T03:47:21.773 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.132e-05 2024-09-15T03:47:21.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.774 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.774 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.774 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844108+0000", 2024-09-15T03:47:21.774 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012384399999999999 2024-09-15T03:47:21.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.774 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.774 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.774 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844139+0000", 2024-09-15T03:47:21.775 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1148000000000001e-05 2024-09-15T03:47:21.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.775 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.775 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.775 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844185+0000", 2024-09-15T03:47:21.775 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5388e-05 2024-09-15T03:47:21.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.775 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.775 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.775 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844202+0000", 2024-09-15T03:47:21.775 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7708999999999999e-05 2024-09-15T03:47:21.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.775 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.775 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.776 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.854747+0000", 2024-09-15T03:47:21.776 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010544944000000001 2024-09-15T03:47:21.776 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.776 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.776 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.777 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.777 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.777 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s7 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:21.777 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844746+0000", 2024-09-15T03:47:21.777 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 23.915952805, 2024-09-15T03:47:21.777 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095535919999999996, 2024-09-15T03:47:21.777 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.777 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.777 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.777 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.777 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.777 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844746+0000", 2024-09-15T03:47:21.778 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.778 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.778 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.778 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844746+0000", 2024-09-15T03:47:21.778 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.778 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.778 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.778 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844747+0000", 2024-09-15T03:47:21.778 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6899999999999996e-07 2024-09-15T03:47:21.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.778 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.778 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.779 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844769+0000", 2024-09-15T03:47:21.779 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1787999999999999e-05 2024-09-15T03:47:21.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.779 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.779 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.779 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844785+0000", 2024-09-15T03:47:21.779 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6212999999999999e-05 2024-09-15T03:47:21.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.779 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.779 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.779 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844808+0000", 2024-09-15T03:47:21.780 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2167e-05 2024-09-15T03:47:21.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.780 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.780 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.780 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844875+0000", 2024-09-15T03:47:21.780 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7683999999999998e-05 2024-09-15T03:47:21.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.780 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.780 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.780 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844893+0000", 2024-09-15T03:47:21.780 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7331000000000001e-05 2024-09-15T03:47:21.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.780 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.781 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.781 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844978+0000", 2024-09-15T03:47:21.781 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5711999999999995e-05 2024-09-15T03:47:21.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.781 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.781 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.781 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844989+0000", 2024-09-15T03:47:21.782 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1026000000000001e-05 2024-09-15T03:47:21.782 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.782 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.782 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.782 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847475+0000", 2024-09-15T03:47:21.782 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024854830000000001 2024-09-15T03:47:21.782 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.782 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.782 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.782 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847492+0000", 2024-09-15T03:47:21.782 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7189000000000001e-05 2024-09-15T03:47:21.783 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.783 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.783 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.783 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.854300+0000", 2024-09-15T03:47:21.783 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0068080299999999996 2024-09-15T03:47:21.783 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.784 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.784 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.784 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.784 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s7 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:21.785 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844831+0000", 2024-09-15T03:47:21.785 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 23.915867761000001, 2024-09-15T03:47:21.785 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095609189999999993, 2024-09-15T03:47:21.785 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.785 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.785 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.785 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.785 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.785 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844831+0000", 2024-09-15T03:47:21.785 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.786 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.786 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.786 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844831+0000", 2024-09-15T03:47:21.786 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.786 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.786 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.786 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.786 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844832+0000", 2024-09-15T03:47:21.786 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1399999999999997e-07 2024-09-15T03:47:21.786 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.787 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.787 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.787 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844855+0000", 2024-09-15T03:47:21.787 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2904e-05 2024-09-15T03:47:21.787 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.787 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.788 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.788 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844866+0000", 2024-09-15T03:47:21.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0691e-05 2024-09-15T03:47:21.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.788 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.788 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.788 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844872+0000", 2024-09-15T03:47:21.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.409e-06 2024-09-15T03:47:21.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.788 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.789 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845030+0000", 2024-09-15T03:47:21.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015832800000000001 2024-09-15T03:47:21.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.789 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.789 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845037+0000", 2024-09-15T03:47:21.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1099999999999997e-06 2024-09-15T03:47:21.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.789 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.789 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845118+0000", 2024-09-15T03:47:21.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0840000000000005e-05 2024-09-15T03:47:21.790 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.790 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.790 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.790 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847478+0000", 2024-09-15T03:47:21.790 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023595869999999998 2024-09-15T03:47:21.790 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.790 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.790 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.790 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847529+0000", 2024-09-15T03:47:21.790 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1585000000000002e-05 2024-09-15T03:47:21.790 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.790 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.790 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.791 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847544+0000", 2024-09-15T03:47:21.791 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4365000000000001e-05 2024-09-15T03:47:21.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.791 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.791 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.791 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.854392+0000", 2024-09-15T03:47:21.791 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0068485860000000003 2024-09-15T03:47:21.791 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.791 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.791 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.792 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.792 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 51/31 ECSubWrite(tid=68, reqid=client.4521.0:351, at_version=51'34, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:21.792 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:21.792 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 23.915483581, 2024-09-15T03:47:21.792 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098406970000000007, 2024-09-15T03:47:21.792 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.792 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.792 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.792 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.792 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.792 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:21.793 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.793 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.793 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:21.793 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.793 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.794 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:21.794 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1299999999999999e-07 2024-09-15T03:47:21.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.794 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.795 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.795 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845232+0000", 2024-09-15T03:47:21.795 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5366000000000002e-05 2024-09-15T03:47:21.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.796 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.796 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.796 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845242+0000", 2024-09-15T03:47:21.796 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0403e-05 2024-09-15T03:47:21.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.796 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.796 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.796 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845247+0000", 2024-09-15T03:47:21.796 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4870000000000004e-06 2024-09-15T03:47:21.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.797 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.797 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.797 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845292+0000", 2024-09-15T03:47:21.797 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5762e-05 2024-09-15T03:47:21.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.797 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.798 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.798 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845302+0000", 2024-09-15T03:47:21.798 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5969999999999998e-06 2024-09-15T03:47:21.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.798 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.798 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845378+0000", 2024-09-15T03:47:21.798 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5628000000000002e-05 2024-09-15T03:47:21.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.798 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.798 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847479+0000", 2024-09-15T03:47:21.798 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021009380000000001 2024-09-15T03:47:21.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.799 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847530+0000", 2024-09-15T03:47:21.799 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1548000000000002e-05 2024-09-15T03:47:21.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.799 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847544+0000", 2024-09-15T03:47:21.799 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4355000000000001e-05 2024-09-15T03:47:21.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.799 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.855056+0000", 2024-09-15T03:47:21.800 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075119000000000002 2024-09-15T03:47:21.800 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.800 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.800 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.800 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.800 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.800 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 51/31 ECSubWrite(tid=93, reqid=client.4521.0:352, at_version=51'24, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:21.800 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:21.800 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 23.913419647000001, 2024-09-15T03:47:21.801 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0096136880000000004, 2024-09-15T03:47:21.801 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.801 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.801 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.801 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.801 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.801 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:21.801 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.801 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.801 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.801 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.801 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:21.801 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.802 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.802 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.802 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.802 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:21.802 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9100000000000003e-07 2024-09-15T03:47:21.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.804 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847669+0000", 2024-09-15T03:47:21.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00038855999999999998 2024-09-15T03:47:21.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.804 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847682+0000", 2024-09-15T03:47:21.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3112000000000001e-05 2024-09-15T03:47:21.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847687+0000", 2024-09-15T03:47:21.805 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0880000000000002e-06 2024-09-15T03:47:21.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.805 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847745+0000", 2024-09-15T03:47:21.805 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8400999999999999e-05 2024-09-15T03:47:21.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.805 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.806 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847756+0000", 2024-09-15T03:47:21.806 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0325e-05 2024-09-15T03:47:21.806 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.806 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.806 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.806 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.848092+0000", 2024-09-15T03:47:21.807 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033636099999999997 2024-09-15T03:47:21.807 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.807 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.807 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.808 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.848101+0000", 2024-09-15T03:47:21.808 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6240000000000008e-06 2024-09-15T03:47:21.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.808 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.808 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.808 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850132+0000", 2024-09-15T03:47:21.809 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020309239999999999 2024-09-15T03:47:21.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850174+0000", 2024-09-15T03:47:21.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2630999999999999e-05 2024-09-15T03:47:21.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.810 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.810 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856893+0000", 2024-09-15T03:47:21.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0067188710000000004 2024-09-15T03:47:21.811 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.811 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.811 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.811 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 57/31 ECSubWrite(tid=90, reqid=client.4521.0:410, at_version=57'46, trim_to=0'0, roll_forward_to=56'44))", 2024-09-15T03:47:21.811 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:03.883223+0000", 2024-09-15T03:47:21.811 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 17.877476482999999, 2024-09-15T03:47:21.811 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092188510000000001, 2024-09-15T03:47:21.811 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.811 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.811 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.883223+0000", 2024-09-15T03:47:21.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.883223+0000", 2024-09-15T03:47:21.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.883223+0000", 2024-09-15T03:47:21.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1600000000000001e-07 2024-09-15T03:47:21.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884196+0000", 2024-09-15T03:47:21.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00097307999999999997 2024-09-15T03:47:21.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.814 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884210+0000", 2024-09-15T03:47:21.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4459e-05 2024-09-15T03:47:21.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.814 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884217+0000", 2024-09-15T03:47:21.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2870000000000001e-06 2024-09-15T03:47:21.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.815 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.815 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884281+0000", 2024-09-15T03:47:21.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4079999999999996e-05 2024-09-15T03:47:21.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.815 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.815 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884293+0000", 2024-09-15T03:47:21.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2081e-05 2024-09-15T03:47:21.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.815 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.815 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884681+0000", 2024-09-15T03:47:21.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00038795600000000001 2024-09-15T03:47:21.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.816 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884695+0000", 2024-09-15T03:47:21.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4497e-05 2024-09-15T03:47:21.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.816 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884962+0000", 2024-09-15T03:47:21.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00026705199999999999 2024-09-15T03:47:21.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.884982+0000", 2024-09-15T03:47:21.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9211999999999998e-05 2024-09-15T03:47:21.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:03.892442+0000", 2024-09-15T03:47:21.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0074600309999999998 2024-09-15T03:47:21.817 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.817 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.817 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.818 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 60/28 ECSubWrite(tid=73, reqid=client.4521.0:424, at_version=60'41, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:21.818 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.900505+0000", 2024-09-15T03:47:21.818 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.860193984, 2024-09-15T03:47:21.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011912449, 2024-09-15T03:47:21.818 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.818 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.818 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.818 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900505+0000", 2024-09-15T03:47:21.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900505+0000", 2024-09-15T03:47:21.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900506+0000", 2024-09-15T03:47:21.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0800000000000003e-07 2024-09-15T03:47:21.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900527+0000", 2024-09-15T03:47:21.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0682999999999998e-05 2024-09-15T03:47:21.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900541+0000", 2024-09-15T03:47:21.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4217e-05 2024-09-15T03:47:21.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900547+0000", 2024-09-15T03:47:21.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3010000000000003e-06 2024-09-15T03:47:21.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900596+0000", 2024-09-15T03:47:21.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8612000000000001e-05 2024-09-15T03:47:21.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.821 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900607+0000", 2024-09-15T03:47:21.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.063e-05 2024-09-15T03:47:21.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.821 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900714+0000", 2024-09-15T03:47:21.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000107321 2024-09-15T03:47:21.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.822 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900721+0000", 2024-09-15T03:47:21.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1709999999999996e-06 2024-09-15T03:47:21.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.822 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900753+0000", 2024-09-15T03:47:21.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1565e-05 2024-09-15T03:47:21.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.822 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900766+0000", 2024-09-15T03:47:21.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3406e-05 2024-09-15T03:47:21.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.912418+0000", 2024-09-15T03:47:21.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011651635 2024-09-15T03:47:21.823 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.823 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.823 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.823 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 60/28 ECSubWrite(tid=74, reqid=client.4521.0:425, at_version=60'42, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:21.823 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.901156+0000", 2024-09-15T03:47:21.823 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.859543232, 2024-09-15T03:47:21.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011338225, 2024-09-15T03:47:21.823 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.824 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.824 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901156+0000", 2024-09-15T03:47:21.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901156+0000", 2024-09-15T03:47:21.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901157+0000", 2024-09-15T03:47:21.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9400000000000005e-07 2024-09-15T03:47:21.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901168+0000", 2024-09-15T03:47:21.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1924e-05 2024-09-15T03:47:21.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901178+0000", 2024-09-15T03:47:21.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7040000000000006e-06 2024-09-15T03:47:21.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901182+0000", 2024-09-15T03:47:21.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3080000000000001e-06 2024-09-15T03:47:21.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901226+0000", 2024-09-15T03:47:21.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3436999999999998e-05 2024-09-15T03:47:21.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901235+0000", 2024-09-15T03:47:21.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9400000000000008e-06 2024-09-15T03:47:21.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901329+0000", 2024-09-15T03:47:21.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3686999999999999e-05 2024-09-15T03:47:21.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901337+0000", 2024-09-15T03:47:21.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0129999999999993e-06 2024-09-15T03:47:21.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903268+0000", 2024-09-15T03:47:21.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001931545 2024-09-15T03:47:21.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903279+0000", 2024-09-15T03:47:21.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0626e-05 2024-09-15T03:47:21.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.912494+0000", 2024-09-15T03:47:21.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092154469999999999 2024-09-15T03:47:21.829 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.829 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.829 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.829 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 60/31 ECSubWrite(tid=129, reqid=client.4521.0:422, at_version=60'41, trim_to=0'0, roll_forward_to=60'39))", 2024-09-15T03:47:21.829 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.901373+0000", 2024-09-15T03:47:21.829 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.859326055, 2024-09-15T03:47:21.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013085485000000001, 2024-09-15T03:47:21.830 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.830 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.830 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.830 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.830 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.830 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901373+0000", 2024-09-15T03:47:21.830 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.830 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.830 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.830 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901373+0000", 2024-09-15T03:47:21.830 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901374+0000", 2024-09-15T03:47:21.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5600000000000005e-07 2024-09-15T03:47:21.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901713+0000", 2024-09-15T03:47:21.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033933299999999998 2024-09-15T03:47:21.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901726+0000", 2024-09-15T03:47:21.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2843999999999999e-05 2024-09-15T03:47:21.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.832 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901730+0000", 2024-09-15T03:47:21.832 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4660000000000001e-06 2024-09-15T03:47:21.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.832 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901773+0000", 2024-09-15T03:47:21.832 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2542e-05 2024-09-15T03:47:21.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.832 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901783+0000", 2024-09-15T03:47:21.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.041e-05 2024-09-15T03:47:21.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.833 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906331+0000", 2024-09-15T03:47:21.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0045479049999999997 2024-09-15T03:47:21.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.833 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906340+0000", 2024-09-15T03:47:21.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8920000000000006e-06 2024-09-15T03:47:21.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.834 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.834 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.834 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906403+0000", 2024-09-15T03:47:21.834 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2793999999999995e-05 2024-09-15T03:47:21.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.834 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.834 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.834 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906413+0000", 2024-09-15T03:47:21.834 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0370999999999999e-05 2024-09-15T03:47:21.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.834 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.834 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.834 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.914459+0000", 2024-09-15T03:47:21.834 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0080452719999999991 2024-09-15T03:47:21.835 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.835 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.835 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.835 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.835 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.835 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.fs7 65/64 ECSubWrite(tid=108, reqid=client.4521.0:466, at_version=65'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:21.835 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:11.853993+0000", 2024-09-15T03:47:21.835 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.9067058669999994, 2024-09-15T03:47:21.835 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013335351, 2024-09-15T03:47:21.835 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.835 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.835 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.835 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.836 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.853993+0000", 2024-09-15T03:47:21.836 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.836 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.836 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.853993+0000", 2024-09-15T03:47:21.836 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.836 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.836 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.853994+0000", 2024-09-15T03:47:21.836 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9999999999999997e-07 2024-09-15T03:47:21.837 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.837 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.837 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.837 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854199+0000", 2024-09-15T03:47:21.837 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020490999999999999 2024-09-15T03:47:21.837 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.837 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.837 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.837 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854210+0000", 2024-09-15T03:47:21.837 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0813e-05 2024-09-15T03:47:21.837 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.837 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.837 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.837 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854215+0000", 2024-09-15T03:47:21.838 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8270000000000003e-06 2024-09-15T03:47:21.838 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.838 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.838 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.838 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854264+0000", 2024-09-15T03:47:21.838 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8477999999999999e-05 2024-09-15T03:47:21.838 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.838 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.838 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.838 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854274+0000", 2024-09-15T03:47:21.838 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5729999999999997e-06 2024-09-15T03:47:21.838 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.838 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.838 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854563+0000", 2024-09-15T03:47:21.839 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000289208 2024-09-15T03:47:21.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.839 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.839 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.855280+0000", 2024-09-15T03:47:21.839 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00071722500000000005 2024-09-15T03:47:21.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.839 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.839 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.855441+0000", 2024-09-15T03:47:21.839 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016084199999999999 2024-09-15T03:47:21.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.840 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.840 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.840 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.855456+0000", 2024-09-15T03:47:21.840 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5123000000000001e-05 2024-09-15T03:47:21.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.840 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.840 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.840 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.867329+0000", 2024-09-15T03:47:21.840 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011872752 2024-09-15T03:47:21.840 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.840 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.840 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.840 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.841 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.fs7 65/64 ECSubWrite(tid=111, reqid=client.4521.0:467, at_version=65'29, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:21.841 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:11.858337+0000", 2024-09-15T03:47:21.841 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.9023625450000008, 2024-09-15T03:47:21.841 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0091986949999999998, 2024-09-15T03:47:21.841 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.841 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.841 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.841 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.841 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.841 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858337+0000", 2024-09-15T03:47:21.841 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.841 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.841 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.842 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.842 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858337+0000", 2024-09-15T03:47:21.842 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.842 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.842 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.842 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858337+0000", 2024-09-15T03:47:21.842 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5899999999999995e-07 2024-09-15T03:47:21.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.842 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.842 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.842 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858383+0000", 2024-09-15T03:47:21.842 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5358999999999997e-05 2024-09-15T03:47:21.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.843 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.843 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.843 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858399+0000", 2024-09-15T03:47:21.843 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6047999999999999e-05 2024-09-15T03:47:21.843 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.843 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.843 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.843 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858408+0000", 2024-09-15T03:47:21.843 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7530000000000007e-06 2024-09-15T03:47:21.843 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.843 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.843 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.843 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858456+0000", 2024-09-15T03:47:21.843 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8081000000000003e-05 2024-09-15T03:47:21.844 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.844 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.844 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.844 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858465+0000", 2024-09-15T03:47:21.844 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0769999999999997e-06 2024-09-15T03:47:21.844 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.844 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.844 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.844 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858579+0000", 2024-09-15T03:47:21.844 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000114318 2024-09-15T03:47:21.844 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.844 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.844 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.845 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858648+0000", 2024-09-15T03:47:21.845 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9244000000000003e-05 2024-09-15T03:47:21.845 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.845 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.845 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.845 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858811+0000", 2024-09-15T03:47:21.845 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016291499999999999 2024-09-15T03:47:21.845 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.845 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.845 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.845 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858950+0000", 2024-09-15T03:47:21.845 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000139257 2024-09-15T03:47:21.845 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.846 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.846 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.846 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.867535+0000", 2024-09-15T03:47:21.846 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0085848839999999992 2024-09-15T03:47:21.846 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.846 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.846 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.846 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.846 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.846 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.fs7 65/64 ECSubWrite(tid=112, reqid=client.4521.0:468, at_version=65'30, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:21.846 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:11.858610+0000", 2024-09-15T03:47:21.846 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.9020893440000002, 2024-09-15T03:47:21.846 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092864230000000002, 2024-09-15T03:47:21.846 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.847 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.847 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.847 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.847 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.847 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858610+0000", 2024-09-15T03:47:21.847 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.847 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.847 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.847 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.847 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858610+0000", 2024-09-15T03:47:21.847 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.847 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.847 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.848 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.848 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858610+0000", 2024-09-15T03:47:21.848 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0999999999999999e-07 2024-09-15T03:47:21.848 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.848 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.848 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.848 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858645+0000", 2024-09-15T03:47:21.848 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4848000000000002e-05 2024-09-15T03:47:21.848 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.848 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.848 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.848 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858654+0000", 2024-09-15T03:47:21.848 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2909999999999996e-06 2024-09-15T03:47:21.848 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.849 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.849 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.849 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858660+0000", 2024-09-15T03:47:21.849 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7119999999999997e-06 2024-09-15T03:47:21.849 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.849 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.849 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.849 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858699+0000", 2024-09-15T03:47:21.849 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8949000000000003e-05 2024-09-15T03:47:21.849 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.849 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.849 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.850 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858711+0000", 2024-09-15T03:47:21.850 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1669e-05 2024-09-15T03:47:21.850 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.850 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.850 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.850 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858830+0000", 2024-09-15T03:47:21.850 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011890700000000001 2024-09-15T03:47:21.850 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.850 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.850 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.851 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858838+0000", 2024-09-15T03:47:21.851 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0479999999999999e-06 2024-09-15T03:47:21.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.851 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.851 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.851 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858903+0000", 2024-09-15T03:47:21.851 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4788000000000004e-05 2024-09-15T03:47:21.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.851 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.851 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.851 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.858975+0000", 2024-09-15T03:47:21.851 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2336000000000007e-05 2024-09-15T03:47:21.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.851 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.852 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.852 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.867896+0000", 2024-09-15T03:47:21.852 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0089213650000000005 2024-09-15T03:47:21.852 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.852 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.852 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.852 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.852 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.852 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.ds6 66/31 ECSubWrite(tid=152, reqid=client.4521.0:509, at_version=66'22, trim_to=0'0, roll_forward_to=63'20))", 2024-09-15T03:47:21.852 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:12.916387+0000", 2024-09-15T03:47:21.852 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 8.844311716, 2024-09-15T03:47:21.852 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0086842389999999998, 2024-09-15T03:47:21.853 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.853 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.853 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.853 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.853 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.853 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.916387+0000", 2024-09-15T03:47:21.853 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.853 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.853 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.853 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.853 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.916387+0000", 2024-09-15T03:47:21.853 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.853 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.854 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.854 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.854 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.916388+0000", 2024-09-15T03:47:21.854 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3300000000000001e-07 2024-09-15T03:47:21.854 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.854 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.854 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.854 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.916450+0000", 2024-09-15T03:47:21.854 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1907000000000002e-05 2024-09-15T03:47:21.854 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.854 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.854 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.854 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.916461+0000", 2024-09-15T03:47:21.855 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1260000000000001e-05 2024-09-15T03:47:21.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.855 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.855 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.916474+0000", 2024-09-15T03:47:21.855 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3135e-05 2024-09-15T03:47:21.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.855 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.855 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.916547+0000", 2024-09-15T03:47:21.855 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2627999999999996e-05 2024-09-15T03:47:21.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.855 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.916563+0000", 2024-09-15T03:47:21.856 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6123000000000001e-05 2024-09-15T03:47:21.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.856 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.856 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.916886+0000", 2024-09-15T03:47:21.856 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00032349100000000002 2024-09-15T03:47:21.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.856 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.856 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.916896+0000", 2024-09-15T03:47:21.856 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6649999999999998e-06 2024-09-15T03:47:21.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.857 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.857 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.857 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.917092+0000", 2024-09-15T03:47:21.857 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000196059 2024-09-15T03:47:21.857 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.857 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.857 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.857 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.917533+0000", 2024-09-15T03:47:21.857 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00044085700000000001 2024-09-15T03:47:21.857 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.857 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.857 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.857 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:12.925072+0000", 2024-09-15T03:47:21.858 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075388809999999999 2024-09-15T03:47:21.858 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.858 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.858 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.858 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.858 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.ds6 74/70 ECSubWrite(tid=198, reqid=client.4521.0:566, at_version=74'32, trim_to=0'0, roll_forward_to=73'30))", 2024-09-15T03:47:21.858 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.944959+0000", 2024-09-15T03:47:21.858 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.815740664, 2024-09-15T03:47:21.858 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010023647, 2024-09-15T03:47:21.858 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.858 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.858 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.858 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.859 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.859 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.944959+0000", 2024-09-15T03:47:21.859 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.859 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.859 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.859 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.859 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.944959+0000", 2024-09-15T03:47:21.859 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.859 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.859 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.859 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.859 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.944959+0000", 2024-09-15T03:47:21.859 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3099999999999998e-07 2024-09-15T03:47:21.859 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.860 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.860 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.860 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.944986+0000", 2024-09-15T03:47:21.860 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7297999999999999e-05 2024-09-15T03:47:21.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.860 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.860 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.860 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.944997+0000", 2024-09-15T03:47:21.860 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0441999999999999e-05 2024-09-15T03:47:21.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.860 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.860 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.860 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.945008+0000", 2024-09-15T03:47:21.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1687e-05 2024-09-15T03:47:21.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.861 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.861 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.861 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.945058+0000", 2024-09-15T03:47:21.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9410999999999998e-05 2024-09-15T03:47:21.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.861 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.861 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.861 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.945069+0000", 2024-09-15T03:47:21.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0812000000000001e-05 2024-09-15T03:47:21.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.861 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.861 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.862 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.945348+0000", 2024-09-15T03:47:21.862 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027976800000000001 2024-09-15T03:47:21.862 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.862 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.862 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.862 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.945362+0000", 2024-09-15T03:47:21.862 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3834e-05 2024-09-15T03:47:21.862 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.862 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.862 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.862 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.945658+0000", 2024-09-15T03:47:21.862 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029610200000000002 2024-09-15T03:47:21.862 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.862 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.863 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.863 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.945707+0000", 2024-09-15T03:47:21.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9029000000000001e-05 2024-09-15T03:47:21.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.863 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.863 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.863 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.954982+0000", 2024-09-15T03:47:21.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092748329999999997 2024-09-15T03:47:21.863 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.863 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.863 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.864 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.864 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.ds6 74/70 ECSubWrite(tid=200, reqid=client.4521.0:567, at_version=74'33, trim_to=0'0, roll_forward_to=74'32))", 2024-09-15T03:47:21.864 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.948998+0000", 2024-09-15T03:47:21.864 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.81170128100000005, 2024-09-15T03:47:21.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011947362, 2024-09-15T03:47:21.864 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.864 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.864 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.864 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.864 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948998+0000", 2024-09-15T03:47:21.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.865 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.865 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948998+0000", 2024-09-15T03:47:21.865 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.865 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.865 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948999+0000", 2024-09-15T03:47:21.865 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.71e-07 2024-09-15T03:47:21.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.865 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.865 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949020+0000", 2024-09-15T03:47:21.866 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1488000000000002e-05 2024-09-15T03:47:21.866 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.866 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.866 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.866 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949047+0000", 2024-09-15T03:47:21.866 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6409000000000001e-05 2024-09-15T03:47:21.866 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.866 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.866 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.866 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949054+0000", 2024-09-15T03:47:21.866 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3830000000000003e-06 2024-09-15T03:47:21.866 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.866 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.867 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.867 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949111+0000", 2024-09-15T03:47:21.867 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7074999999999998e-05 2024-09-15T03:47:21.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.867 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.867 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.867 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949120+0000", 2024-09-15T03:47:21.867 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1549999999999996e-06 2024-09-15T03:47:21.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.867 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.867 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.867 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949214+0000", 2024-09-15T03:47:21.867 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4046e-05 2024-09-15T03:47:21.868 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.868 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.868 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.868 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949225+0000", 2024-09-15T03:47:21.868 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0567999999999999e-05 2024-09-15T03:47:21.868 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.868 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.868 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.868 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949303+0000", 2024-09-15T03:47:21.868 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8065999999999994e-05 2024-09-15T03:47:21.868 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.868 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.868 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.869 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949317+0000", 2024-09-15T03:47:21.869 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4035e-05 2024-09-15T03:47:21.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.869 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.869 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.869 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.960945+0000", 2024-09-15T03:47:21.869 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011628266 2024-09-15T03:47:21.869 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.869 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.869 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.869 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.869 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.9s4 74/70 ECSubWrite(tid=108, reqid=client.4521.0:569, at_version=74'33, trim_to=0'0, roll_forward_to=74'28))", 2024-09-15T03:47:21.870 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.949550+0000", 2024-09-15T03:47:21.870 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.81114938299999995, 2024-09-15T03:47:21.870 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012286695, 2024-09-15T03:47:21.870 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.870 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.870 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.870 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.870 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.870 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949550+0000", 2024-09-15T03:47:21.870 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.870 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.870 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.871 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949550+0000", 2024-09-15T03:47:21.871 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.871 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.871 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949551+0000", 2024-09-15T03:47:21.871 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9999999999999996e-07 2024-09-15T03:47:21.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.871 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.871 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949590+0000", 2024-09-15T03:47:21.871 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9560000000000001e-05 2024-09-15T03:47:21.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.872 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.872 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.872 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949611+0000", 2024-09-15T03:47:21.872 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1229e-05 2024-09-15T03:47:21.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.872 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.872 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.872 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949620+0000", 2024-09-15T03:47:21.872 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5140000000000001e-06 2024-09-15T03:47:21.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.872 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.872 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.873 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949670+0000", 2024-09-15T03:47:21.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9545000000000001e-05 2024-09-15T03:47:21.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.873 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.873 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949678+0000", 2024-09-15T03:47:21.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9260000000000006e-06 2024-09-15T03:47:21.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.873 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.873 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949987+0000", 2024-09-15T03:47:21.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030801899999999998 2024-09-15T03:47:21.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.874 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949997+0000", 2024-09-15T03:47:21.874 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0930999999999999e-05 2024-09-15T03:47:21.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.874 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950049+0000", 2024-09-15T03:47:21.874 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1384000000000001e-05 2024-09-15T03:47:21.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.874 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950063+0000", 2024-09-15T03:47:21.875 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4664000000000001e-05 2024-09-15T03:47:21.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.875 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.875 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961837+0000", 2024-09-15T03:47:21.875 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011773023000000001 2024-09-15T03:47:21.875 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.875 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.875 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.875 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.ds6 74/70 ECSubWrite(tid=201, reqid=client.4521.0:568, at_version=74'34, trim_to=0'0, roll_forward_to=74'32))", 2024-09-15T03:47:21.875 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.950074+0000", 2024-09-15T03:47:21.876 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.81062491199999998, 2024-09-15T03:47:21.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011003285, 2024-09-15T03:47:21.876 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.876 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.876 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.876 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.876 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950074+0000", 2024-09-15T03:47:21.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.877 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.877 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950074+0000", 2024-09-15T03:47:21.877 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.877 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.877 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.877 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950075+0000", 2024-09-15T03:47:21.877 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0999999999999997e-07 2024-09-15T03:47:21.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.877 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.877 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.877 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950095+0000", 2024-09-15T03:47:21.877 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0256000000000001e-05 2024-09-15T03:47:21.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.878 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950109+0000", 2024-09-15T03:47:21.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3325e-05 2024-09-15T03:47:21.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.878 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950123+0000", 2024-09-15T03:47:21.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3893000000000001e-05 2024-09-15T03:47:21.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.878 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950170+0000", 2024-09-15T03:47:21.879 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7496000000000001e-05 2024-09-15T03:47:21.879 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.879 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.879 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.879 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950178+0000", 2024-09-15T03:47:21.879 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4190000000000002e-06 2024-09-15T03:47:21.879 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.879 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.879 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.879 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950271+0000", 2024-09-15T03:47:21.879 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2961999999999995e-05 2024-09-15T03:47:21.879 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.879 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.880 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.880 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950281+0000", 2024-09-15T03:47:21.880 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0960000000000007e-06 2024-09-15T03:47:21.880 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.880 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.880 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.880 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950317+0000", 2024-09-15T03:47:21.880 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6529999999999998e-05 2024-09-15T03:47:21.880 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.880 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.880 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.880 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950326+0000", 2024-09-15T03:47:21.880 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1419999999999999e-06 2024-09-15T03:47:21.881 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.881 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.881 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.881 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961078+0000", 2024-09-15T03:47:21.881 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010751356 2024-09-15T03:47:21.881 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.881 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.881 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.881 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.881 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.881 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.ds6 74/70 ECSubWrite(tid=202, reqid=client.4521.0:568, at_version=0'0, trim_to=0'0, roll_forward_to=74'34))", 2024-09-15T03:47:21.881 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.952147+0000", 2024-09-15T03:47:21.881 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.80855233500000001, 2024-09-15T03:47:21.882 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092748239999999992, 2024-09-15T03:47:21.882 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.882 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.882 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.882 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.882 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.882 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952147+0000", 2024-09-15T03:47:21.882 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.882 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.882 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.882 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.882 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952147+0000", 2024-09-15T03:47:21.882 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.882 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.883 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.883 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952148+0000", 2024-09-15T03:47:21.883 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.223e-06 2024-09-15T03:47:21.883 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.883 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.883 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952162+0000", 2024-09-15T03:47:21.883 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3835999999999999e-05 2024-09-15T03:47:21.883 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.883 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.883 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952183+0000", 2024-09-15T03:47:21.883 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1070999999999999e-05 2024-09-15T03:47:21.884 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.884 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.884 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.884 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952191+0000", 2024-09-15T03:47:21.884 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7989999999999994e-06 2024-09-15T03:47:21.884 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.884 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.884 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.884 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952249+0000", 2024-09-15T03:47:21.884 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8171000000000003e-05 2024-09-15T03:47:21.884 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.884 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.884 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.885 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952259+0000", 2024-09-15T03:47:21.885 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.03e-05 2024-09-15T03:47:21.885 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.885 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.885 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.885 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952382+0000", 2024-09-15T03:47:21.885 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012252300000000001 2024-09-15T03:47:21.885 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.885 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.885 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.885 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952401+0000", 2024-09-15T03:47:21.885 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9451000000000001e-05 2024-09-15T03:47:21.885 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.886 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.886 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.886 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952438+0000", 2024-09-15T03:47:21.886 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6990000000000003e-05 2024-09-15T03:47:21.886 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.886 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.886 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.886 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952466+0000", 2024-09-15T03:47:21.886 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7755000000000001e-05 2024-09-15T03:47:21.886 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.886 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.886 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.886 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961422+0000", 2024-09-15T03:47:21.886 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0089557049999999996 2024-09-15T03:47:21.887 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.887 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.887 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.887 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.887 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.887 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.9s4 74/70 ECSubWrite(tid=109, reqid=client.4521.0:569, at_version=0'0, trim_to=0'0, roll_forward_to=74'33))", 2024-09-15T03:47:21.887 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.952609+0000", 2024-09-15T03:47:21.887 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.80809033399999997, 2024-09-15T03:47:21.887 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095066119999999994, 2024-09-15T03:47:21.887 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:21.887 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:21.887 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:21.887 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.888 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:21.888 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952609+0000", 2024-09-15T03:47:21.888 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.888 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.888 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.888 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:21.888 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952609+0000", 2024-09-15T03:47:21.888 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:21.888 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.888 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.888 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:21.888 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952610+0000", 2024-09-15T03:47:21.888 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2699999999999998e-07 2024-09-15T03:47:21.889 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.889 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.889 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:21.889 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952623+0000", 2024-09-15T03:47:21.889 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2832e-05 2024-09-15T03:47:21.889 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.889 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.889 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:21.889 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952640+0000", 2024-09-15T03:47:21.889 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7839999999999999e-05 2024-09-15T03:47:21.889 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.889 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.889 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:21.890 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952648+0000", 2024-09-15T03:47:21.890 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7400000000000004e-06 2024-09-15T03:47:21.890 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.890 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.890 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:21.890 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952699+0000", 2024-09-15T03:47:21.890 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0995999999999998e-05 2024-09-15T03:47:21.890 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.890 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.890 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:21.890 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952709+0000", 2024-09-15T03:47:21.890 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0190000000000001e-05 2024-09-15T03:47:21.890 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:21.891 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952867+0000", 2024-09-15T03:47:21.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000157622 2024-09-15T03:47:21.891 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:21.891 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952879+0000", 2024-09-15T03:47:21.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1545e-05 2024-09-15T03:47:21.891 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:21.891 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.953766+0000", 2024-09-15T03:47:21.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00088788199999999999 2024-09-15T03:47:21.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:21.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.953784+0000", 2024-09-15T03:47:21.892 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7104000000000001e-05 2024-09-15T03:47:21.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:21.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:21.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:21.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.962115+0000", 2024-09-15T03:47:21.892 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0083319339999999992 2024-09-15T03:47:21.892 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.892 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.892 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.893 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:21.893 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:21.893 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:21.894 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:47:21.895 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:21.892+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:21.933 INFO:tasks.rados.rados.0.smithi001.stdout:229: done (1 left) 2024-09-15T03:47:21.933 INFO:tasks.rados.rados.0.smithi001.stdout:230: done (0 left) 2024-09-15T03:47:21.933 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 5 current snap is 29 2024-09-15T03:47:21.933 INFO:tasks.rados.rados.0.smithi001.stdout:231: seq_num 105 ranges {0=606208,606208=688128,1294336=475136} 2024-09-15T03:47:21.936 INFO:tasks.rados.rados.0.smithi001.stdout:231: writing smithi00132172-5 from 0 to 606208 tid 1 2024-09-15T03:47:21.939 INFO:tasks.rados.rados.0.smithi001.stdout:231: writing smithi00132172-5 from 606208 to 1294336 tid 2 2024-09-15T03:47:21.943 INFO:tasks.rados.rados.0.smithi001.stdout:231: writing smithi00132172-5 from 1294336 to 1769472 tid 3 2024-09-15T03:47:21.943 INFO:tasks.rados.rados.0.smithi001.stdout:232: rmattr oid 19 current snap is 29 2024-09-15T03:47:21.943 INFO:tasks.rados.rados.0.smithi001.stdout:232: done (1 left) 2024-09-15T03:47:21.943 INFO:tasks.rados.rados.0.smithi001.stdout:233: copy_from oid 6 from oid 8 current snap is 29 2024-09-15T03:47:21.943 INFO:tasks.rados.rados.0.smithi001.stdout:234: read oid 19 snap -1 2024-09-15T03:47:21.943 INFO:tasks.rados.rados.0.smithi001.stdout:234: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-15T03:47:21.943 INFO:tasks.rados.rados.0.smithi001.stdout:235: rollback oid 50 current snap is 29 2024-09-15T03:47:21.943 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 50 to 26 2024-09-15T03:47:21.944 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 40 current snap is 29 2024-09-15T03:47:21.944 INFO:tasks.rados.rados.0.smithi001.stdout:236: seq_num 106 ranges {3358720=622592,3981312=442368,4423680=524288,4947968=196608} 2024-09-15T03:47:21.965 INFO:tasks.rados.rados.0.smithi001.stdout:236: writing smithi00132172-40 from 3358720 to 3981312 tid 1 2024-09-15T03:47:21.968 INFO:tasks.rados.rados.0.smithi001.stdout:236: writing smithi00132172-40 from 3981312 to 4423680 tid 2 2024-09-15T03:47:21.971 INFO:tasks.rados.rados.0.smithi001.stdout:236: writing smithi00132172-40 from 4423680 to 4947968 tid 3 2024-09-15T03:47:21.972 INFO:tasks.rados.rados.0.smithi001.stdout:236: writing smithi00132172-40 from 4947968 to 5144576 tid 4 2024-09-15T03:47:21.972 INFO:tasks.rados.rados.0.smithi001.stdout:237: read oid 4 snap -1 2024-09-15T03:47:21.972 INFO:tasks.rados.rados.0.smithi001.stdout:237: expect (ObjNum 70 snap 8 seq_num 70) 2024-09-15T03:47:21.972 INFO:tasks.rados.rados.0.smithi001.stdout:238: delete oid 31 current snap is 29 2024-09-15T03:47:21.972 INFO:tasks.rados.rados.0.smithi001.stdout:231: finishing write tid 1 to smithi00132172-5 2024-09-15T03:47:21.973 INFO:tasks.rados.rados.0.smithi001.stdout:231: finishing write tid 2 to smithi00132172-5 2024-09-15T03:47:21.973 INFO:tasks.rados.rados.0.smithi001.stdout:231: finishing write tid 3 to smithi00132172-5 2024-09-15T03:47:21.973 INFO:tasks.rados.rados.0.smithi001.stdout:231: finishing write tid 4 to smithi00132172-5 2024-09-15T03:47:21.973 INFO:tasks.rados.rados.0.smithi001.stdout:235: finishing rollback tid 0 to smithi00132172-50 2024-09-15T03:47:21.973 INFO:tasks.rados.rados.0.smithi001.stdout:235: finishing rollback tid 1 to smithi00132172-50 2024-09-15T03:47:21.973 INFO:tasks.rados.rados.0.smithi001.stdout:235: finishing rollback tid 2 to smithi00132172-50 2024-09-15T03:47:21.973 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 50 v 54 (ObjNum 72 snap 12 seq_num 72) dirty exists 2024-09-15T03:47:21.973 INFO:tasks.rados.rados.0.smithi001.stdout:234: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-15T03:47:21.982 INFO:tasks.rados.rados.0.smithi001.stdout:233: finishing copy_from racing read to smithi00132172-6 2024-09-15T03:47:21.982 INFO:tasks.rados.rados.0.smithi001.stdout:233: finishing copy_from to smithi00132172-6 2024-09-15T03:47:21.982 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 6 v 18 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-15T03:47:21.983 INFO:tasks.rados.rados.0.smithi001.stdout:231: finishing write tid 5 to smithi00132172-5 2024-09-15T03:47:21.983 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 5 v 33 (ObjNum 105 snap 29 seq_num 105) dirty exists 2024-09-15T03:47:21.983 INFO:tasks.rados.rados.0.smithi001.stdout:231: left oid 5 (ObjNum 105 snap 29 seq_num 105) 2024-09-15T03:47:21.983 INFO:tasks.rados.rados.0.smithi001.stdout:236: finishing write tid 1 to smithi00132172-40 2024-09-15T03:47:21.983 INFO:tasks.rados.rados.0.smithi001.stdout:236: finishing write tid 2 to smithi00132172-40 2024-09-15T03:47:21.983 INFO:tasks.rados.rados.0.smithi001.stdout:236: finishing write tid 3 to smithi00132172-40 2024-09-15T03:47:21.983 INFO:tasks.rados.rados.0.smithi001.stdout:236: finishing write tid 4 to smithi00132172-40 2024-09-15T03:47:21.983 INFO:tasks.rados.rados.0.smithi001.stdout:236: finishing write tid 5 to smithi00132172-40 2024-09-15T03:47:21.983 INFO:tasks.rados.rados.0.smithi001.stdout:237: expect (ObjNum 70 snap 8 seq_num 70) 2024-09-15T03:47:21.995 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:21.994+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:22.013 INFO:tasks.rados.rados.0.smithi001.stdout:236: finishing write tid 6 to smithi00132172-40 2024-09-15T03:47:22.014 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 40 v 91 (ObjNum 106 snap 29 seq_num 106) dirty exists 2024-09-15T03:47:22.014 INFO:tasks.rados.rados.0.smithi001.stdout:236: left oid 40 (ObjNum 106 snap 29 seq_num 106) 2024-09-15T03:47:22.014 INFO:tasks.rados.rados.0.smithi001.stdout:231: done (6 left) 2024-09-15T03:47:22.014 INFO:tasks.rados.rados.0.smithi001.stdout:233: done (5 left) 2024-09-15T03:47:22.014 INFO:tasks.rados.rados.0.smithi001.stdout:234: done (4 left) 2024-09-15T03:47:22.014 INFO:tasks.rados.rados.0.smithi001.stdout:235: done (3 left) 2024-09-15T03:47:22.014 INFO:tasks.rados.rados.0.smithi001.stdout:236: done (2 left) 2024-09-15T03:47:22.014 INFO:tasks.rados.rados.0.smithi001.stdout:237: done (1 left) 2024-09-15T03:47:22.014 INFO:tasks.rados.rados.0.smithi001.stdout:238: done (0 left) 2024-09-15T03:47:22.014 INFO:tasks.rados.rados.0.smithi001.stdout:239: snap_remove snap 26 2024-09-15T03:47:22.039 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:22.039 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:22.039 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:22.039 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:22.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:47:22.095 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:22.094+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:22.144 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:22.144 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:22.144 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:22.144 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:22.144 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:22.152 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:47:22.196 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:22.195+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:22.291 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:22.291 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:22.291 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:22.292 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:47:22.292 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.292 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "pg_update_log_missing(3.fs7 epoch 69/68 rep_tid 131 entries 69'34 (0'0) error 3:f7b86273:::smithi00132172-37:head by client.4521.0:516 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-09-15T03:47:22.292 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.898080+0000", 2024-09-15T03:47:22.292 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.3915627480000001, 2024-09-15T03:47:22.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.047982374000000001, 2024-09-15T03:47:22.292 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:22.292 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:22.292 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:22.292 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.292 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:22.292 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898080+0000", 2024-09-15T03:47:22.293 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.293 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.293 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.293 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:22.293 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898080+0000", 2024-09-15T03:47:22.293 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.293 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.293 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.293 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:22.293 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898080+0000", 2024-09-15T03:47:22.293 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2399999999999999e-07 2024-09-15T03:47:22.293 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.293 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.294 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:22.294 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898086+0000", 2024-09-15T03:47:22.294 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8499999999999999e-06 2024-09-15T03:47:22.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.294 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.294 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:22.294 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898102+0000", 2024-09-15T03:47:22.294 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5233e-05 2024-09-15T03:47:22.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.294 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.294 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:22.294 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898111+0000", 2024-09-15T03:47:22.294 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3579999999999992e-06 2024-09-15T03:47:22.295 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.295 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.295 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.295 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898169+0000", 2024-09-15T03:47:22.295 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8563999999999997e-05 2024-09-15T03:47:22.295 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.295 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.295 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for flush", 2024-09-15T03:47:22.295 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898205+0000", 2024-09-15T03:47:22.295 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5833000000000001e-05 2024-09-15T03:47:22.295 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.295 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.295 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.295 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.945919+0000", 2024-09-15T03:47:22.296 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.047713658999999999 2024-09-15T03:47:22.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.296 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.296 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:22.296 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.946063+0000", 2024-09-15T03:47:22.296 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014365299999999999 2024-09-15T03:47:22.296 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.296 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:22.296 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.296 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.296 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(3.2s2 69/68 [PushOp(3:4cab1a9a:::smithi00132172-23:head, version: 32'6, data_included: [0~94208], data_size: 94208, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:4cab1a9a:::smithi00132172-23:head@32'6, size: 376832, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:376832, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T03:47:22.296 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.898323+0000", 2024-09-15T03:47:22.296 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.3913198790000001, 2024-09-15T03:47:22.297 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.027476691000000001, 2024-09-15T03:47:22.297 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:22.297 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:22.297 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:22.297 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.297 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:22.297 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898323+0000", 2024-09-15T03:47:22.297 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.297 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.297 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:22.297 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898323+0000", 2024-09-15T03:47:22.297 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.298 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.298 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:22.298 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898323+0000", 2024-09-15T03:47:22.298 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2800000000000003e-07 2024-09-15T03:47:22.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.298 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.298 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:22.298 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898437+0000", 2024-09-15T03:47:22.298 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000113899 2024-09-15T03:47:22.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.298 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.298 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:22.299 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898458+0000", 2024-09-15T03:47:22.299 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0738e-05 2024-09-15T03:47:22.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.299 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.299 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:22.299 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898469+0000", 2024-09-15T03:47:22.299 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0446e-05 2024-09-15T03:47:22.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.299 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.299 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.299 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898665+0000", 2024-09-15T03:47:22.299 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000196665 2024-09-15T03:47:22.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.300 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.300 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for flush", 2024-09-15T03:47:22.300 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.898674+0000", 2024-09-15T03:47:22.300 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1819999999999995e-06 2024-09-15T03:47:22.300 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.300 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.300 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.300 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.925633+0000", 2024-09-15T03:47:22.300 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.026958947 2024-09-15T03:47:22.300 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.300 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.300 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:22.300 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.925800+0000", 2024-09-15T03:47:22.301 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000166486 2024-09-15T03:47:22.301 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.301 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:22.301 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.301 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.301 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.301 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(3.bs5 69/68 [PushOp(3:dbbb10d0:::smithi00132172-15:head, version: 32'4, data_included: [0~61440], data_size: 61440, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:dbbb10d0:::smithi00132172-15:head@32'4, size: 245760, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:245760, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T03:47:22.301 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.899036+0000", 2024-09-15T03:47:22.301 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.3906072419999997, 2024-09-15T03:47:22.301 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.043768267999999999, 2024-09-15T03:47:22.301 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:22.301 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:22.301 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:22.302 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.302 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:22.302 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899036+0000", 2024-09-15T03:47:22.302 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.302 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.302 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:22.302 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899036+0000", 2024-09-15T03:47:22.302 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.302 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.302 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:22.303 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899036+0000", 2024-09-15T03:47:22.303 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8500000000000001e-07 2024-09-15T03:47:22.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.303 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.303 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:22.303 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899149+0000", 2024-09-15T03:47:22.303 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112902 2024-09-15T03:47:22.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.303 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.303 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:22.303 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899166+0000", 2024-09-15T03:47:22.303 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6939000000000002e-05 2024-09-15T03:47:22.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.304 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.304 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:22.304 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899176+0000", 2024-09-15T03:47:22.304 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8300000000000008e-06 2024-09-15T03:47:22.304 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.304 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.304 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.304 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899248+0000", 2024-09-15T03:47:22.304 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2231999999999996e-05 2024-09-15T03:47:22.304 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.304 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.304 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for flush", 2024-09-15T03:47:22.304 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899258+0000", 2024-09-15T03:47:22.305 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7690000000000008e-06 2024-09-15T03:47:22.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.305 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.305 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.305 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.942666+0000", 2024-09-15T03:47:22.305 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.043408407000000003 2024-09-15T03:47:22.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.305 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.305 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:22.305 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.942804+0000", 2024-09-15T03:47:22.305 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013760399999999999 2024-09-15T03:47:22.305 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.305 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:22.306 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.306 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.306 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.306 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(3.7s7 e69,68 3:e4b8e19d:::smithi00132172-41:head,32'13)", 2024-09-15T03:47:22.306 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.899363+0000", 2024-09-15T03:47:22.306 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.3902799149999998, 2024-09-15T03:47:22.306 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.046423364000000002, 2024-09-15T03:47:22.306 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:22.306 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:22.306 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:22.306 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.306 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:22.306 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899363+0000", 2024-09-15T03:47:22.307 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.307 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.307 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:22.307 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899363+0000", 2024-09-15T03:47:22.307 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.307 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.307 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:22.307 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899364+0000", 2024-09-15T03:47:22.307 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5899999999999996e-07 2024-09-15T03:47:22.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.307 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.307 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:22.308 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899369+0000", 2024-09-15T03:47:22.308 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0069999999999997e-06 2024-09-15T03:47:22.308 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.308 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.308 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:22.308 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899383+0000", 2024-09-15T03:47:22.308 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3766e-05 2024-09-15T03:47:22.308 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.308 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.308 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:22.308 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899389+0000", 2024-09-15T03:47:22.308 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.511e-06 2024-09-15T03:47:22.308 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.308 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.309 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.309 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899439+0000", 2024-09-15T03:47:22.309 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9910999999999997e-05 2024-09-15T03:47:22.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.309 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.309 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for flush", 2024-09-15T03:47:22.309 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.899447+0000", 2024-09-15T03:47:22.309 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1070000000000005e-06 2024-09-15T03:47:22.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.309 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.309 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.309 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.945757+0000", 2024-09-15T03:47:22.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.046310285 2024-09-15T03:47:22.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.310 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.310 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:22.310 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.945764+0000", 2024-09-15T03:47:22.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6069999999999996e-06 2024-09-15T03:47:22.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.310 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.310 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:22.310 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.945786+0000", 2024-09-15T03:47:22.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2510999999999998e-05 2024-09-15T03:47:22.310 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.310 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:22.310 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.311 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.311 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(3.6s7 69/68 [PushOp(3:68743546:::smithi00132172-8:head, version: 32'2, data_included: [0~32768], data_size: 32768, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:68743546:::smithi00132172-8:head@32'2, size: 131072, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131072, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T03:47:22.311 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.900159+0000", 2024-09-15T03:47:22.311 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.3894842000000001, 2024-09-15T03:47:22.311 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.042041885000000001, 2024-09-15T03:47:22.311 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:22.311 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:22.311 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:22.311 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:22.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900159+0000", 2024-09-15T03:47:22.311 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.312 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.312 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:22.312 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900159+0000", 2024-09-15T03:47:22.312 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.312 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.312 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:22.312 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900160+0000", 2024-09-15T03:47:22.312 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0299999999999998e-07 2024-09-15T03:47:22.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.312 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.313 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:22.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900200+0000", 2024-09-15T03:47:22.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0435000000000002e-05 2024-09-15T03:47:22.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.313 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.313 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:22.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900220+0000", 2024-09-15T03:47:22.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9752e-05 2024-09-15T03:47:22.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.313 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.313 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:22.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900230+0000", 2024-09-15T03:47:22.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0368e-05 2024-09-15T03:47:22.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.314 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.314 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.314 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900298+0000", 2024-09-15T03:47:22.314 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8399999999999996e-05 2024-09-15T03:47:22.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.314 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.314 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for flush", 2024-09-15T03:47:22.314 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900307+0000", 2024-09-15T03:47:22.314 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7499999999999992e-06 2024-09-15T03:47:22.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.314 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.314 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.314 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.942080+0000", 2024-09-15T03:47:22.315 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.041773235999999998 2024-09-15T03:47:22.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.315 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.315 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:22.315 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.942201+0000", 2024-09-15T03:47:22.315 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120241 2024-09-15T03:47:22.315 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.315 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:22.315 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.315 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.315 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(3.3s0 69/68 [PushOp(3:c325496f:::smithi00132172-2:head, version: 32'2, data_included: [0~8192], data_size: 8192, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:c325496f:::smithi00132172-2:head@32'2, size: 32768, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:32768, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T03:47:22.315 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.900588+0000", 2024-09-15T03:47:22.316 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.3890553380000004, 2024-09-15T03:47:22.316 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.046022875999999997, 2024-09-15T03:47:22.316 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:22.316 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:22.316 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:22.316 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.316 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:22.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900588+0000", 2024-09-15T03:47:22.316 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.316 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.316 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:22.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900588+0000", 2024-09-15T03:47:22.317 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.317 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.317 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:22.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900588+0000", 2024-09-15T03:47:22.317 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9399999999999999e-07 2024-09-15T03:47:22.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.317 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.317 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:22.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900618+0000", 2024-09-15T03:47:22.317 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9828999999999999e-05 2024-09-15T03:47:22.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.318 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.318 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:22.318 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900635+0000", 2024-09-15T03:47:22.318 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7402999999999999e-05 2024-09-15T03:47:22.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.318 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.318 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:22.318 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900648+0000", 2024-09-15T03:47:22.318 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2435999999999999e-05 2024-09-15T03:47:22.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.318 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.318 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.318 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900706+0000", 2024-09-15T03:47:22.319 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8322000000000002e-05 2024-09-15T03:47:22.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.319 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.319 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for flush", 2024-09-15T03:47:22.319 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900716+0000", 2024-09-15T03:47:22.319 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7030000000000001e-06 2024-09-15T03:47:22.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.319 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.319 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.319 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.946484+0000", 2024-09-15T03:47:22.319 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.045768640999999999 2024-09-15T03:47:22.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.319 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.319 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:22.320 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.946611+0000", 2024-09-15T03:47:22.320 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000126348 2024-09-15T03:47:22.320 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.320 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:22.320 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.320 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.320 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.320 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(3.ds4 69/68 [PushOp(3:b166213b:::smithi00132172-21:19, version: 62'17, data_included: [0~86016], data_size: 86016, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:b166213b:::smithi00132172-21:19@62'17, size: 344064, copy_subset: [], clone_subset: {}, snapset: 22=[]:{19=[18,17,16,12,11,c,3,2],22=[22,21,20]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:344064, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-15T03:47:22.320 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.900892+0000", 2024-09-15T03:47:22.320 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.3887508259999999, 2024-09-15T03:47:22.320 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.045623043000000002, 2024-09-15T03:47:22.320 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:22.320 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:22.321 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:22.321 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.321 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:22.321 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900892+0000", 2024-09-15T03:47:22.321 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.321 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.321 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.321 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:22.321 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900892+0000", 2024-09-15T03:47:22.321 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.321 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.321 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.321 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:22.322 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900893+0000", 2024-09-15T03:47:22.322 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5599999999999995e-07 2024-09-15T03:47:22.322 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.322 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.322 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:22.322 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900960+0000", 2024-09-15T03:47:22.322 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7058999999999996e-05 2024-09-15T03:47:22.322 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.322 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.322 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:22.322 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900978+0000", 2024-09-15T03:47:22.322 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8167000000000001e-05 2024-09-15T03:47:22.322 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.323 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.323 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:22.323 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.900990+0000", 2024-09-15T03:47:22.323 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1625e-05 2024-09-15T03:47:22.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.323 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.323 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.323 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.901037+0000", 2024-09-15T03:47:22.323 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7407000000000002e-05 2024-09-15T03:47:22.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.323 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.323 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for flush", 2024-09-15T03:47:22.323 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.901043+0000", 2024-09-15T03:47:22.324 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5459999999999999e-06 2024-09-15T03:47:22.324 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.324 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.324 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.324 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.946208+0000", 2024-09-15T03:47:22.324 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.045165023999999998 2024-09-15T03:47:22.324 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.324 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.324 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:22.324 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.946515+0000", 2024-09-15T03:47:22.324 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000307659 2024-09-15T03:47:22.324 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.324 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:22.324 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.325 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.325 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.325 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.ds4 74/70 ECSubWrite(tid=200, reqid=client.4521.0:567, at_version=74'33, trim_to=0'0, roll_forward_to=74'32))", 2024-09-15T03:47:22.325 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.949024+0000", 2024-09-15T03:47:22.325 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.3406190819999999, 2024-09-15T03:47:22.325 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012503234, 2024-09-15T03:47:22.325 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:22.325 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:22.325 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:22.325 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.325 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:22.325 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949024+0000", 2024-09-15T03:47:22.325 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.326 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.326 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.326 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:22.326 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949024+0000", 2024-09-15T03:47:22.326 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.326 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.326 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.326 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:22.326 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949025+0000", 2024-09-15T03:47:22.326 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9599999999999998e-07 2024-09-15T03:47:22.326 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.326 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.326 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:22.327 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949045+0000", 2024-09-15T03:47:22.327 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0463e-05 2024-09-15T03:47:22.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.327 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.327 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:22.327 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949062+0000", 2024-09-15T03:47:22.327 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6348999999999999e-05 2024-09-15T03:47:22.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.327 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.327 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:22.327 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949068+0000", 2024-09-15T03:47:22.327 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.759e-06 2024-09-15T03:47:22.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.328 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.328 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949138+0000", 2024-09-15T03:47:22.328 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9956999999999994e-05 2024-09-15T03:47:22.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.328 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.328 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:22.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949147+0000", 2024-09-15T03:47:22.328 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0680000000000003e-06 2024-09-15T03:47:22.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.328 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.328 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:22.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949237+0000", 2024-09-15T03:47:22.329 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9702000000000003e-05 2024-09-15T03:47:22.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.329 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.329 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:22.329 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949261+0000", 2024-09-15T03:47:22.329 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3864000000000001e-05 2024-09-15T03:47:22.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.329 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.329 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:22.329 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950935+0000", 2024-09-15T03:47:22.329 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0016745410000000001 2024-09-15T03:47:22.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.329 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.330 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:22.330 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950969+0000", 2024-09-15T03:47:22.330 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3393999999999999e-05 2024-09-15T03:47:22.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.330 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.330 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:22.330 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961527+0000", 2024-09-15T03:47:22.330 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010558341000000001 2024-09-15T03:47:22.330 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.330 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:22.330 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.330 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.330 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.9s2 74/70 ECSubWrite(tid=108, reqid=client.4521.0:569, at_version=74'33, trim_to=0'0, roll_forward_to=74'28))", 2024-09-15T03:47:22.331 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.949556+0000", 2024-09-15T03:47:22.331 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.3400866090000001, 2024-09-15T03:47:22.331 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012673649, 2024-09-15T03:47:22.331 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:22.331 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:22.331 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:22.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.331 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:22.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949556+0000", 2024-09-15T03:47:22.331 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.331 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:22.332 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949556+0000", 2024-09-15T03:47:22.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.332 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.332 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:22.332 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949557+0000", 2024-09-15T03:47:22.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0299999999999997e-07 2024-09-15T03:47:22.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.332 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.332 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:22.332 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949596+0000", 2024-09-15T03:47:22.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8436000000000003e-05 2024-09-15T03:47:22.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.333 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.333 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:22.333 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949642+0000", 2024-09-15T03:47:22.333 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6081000000000001e-05 2024-09-15T03:47:22.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.333 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.333 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:22.333 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949650+0000", 2024-09-15T03:47:22.333 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5990000000000002e-06 2024-09-15T03:47:22.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.333 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.333 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949703+0000", 2024-09-15T03:47:22.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2734e-05 2024-09-15T03:47:22.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.334 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.334 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:22.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949712+0000", 2024-09-15T03:47:22.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4300000000000006e-06 2024-09-15T03:47:22.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.334 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.334 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:22.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949977+0000", 2024-09-15T03:47:22.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00026508599999999999 2024-09-15T03:47:22.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.334 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:22.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950939+0000", 2024-09-15T03:47:22.335 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00096198599999999996 2024-09-15T03:47:22.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:22.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951015+0000", 2024-09-15T03:47:22.335 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6555000000000001e-05 2024-09-15T03:47:22.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:22.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951026+0000", 2024-09-15T03:47:22.335 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0798999999999999e-05 2024-09-15T03:47:22.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.336 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.336 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:22.336 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.962230+0000", 2024-09-15T03:47:22.336 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01120404 2024-09-15T03:47:22.336 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.336 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:22.336 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.336 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.336 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.ds4 74/70 ECSubWrite(tid=201, reqid=client.4521.0:568, at_version=74'34, trim_to=0'0, roll_forward_to=74'32))", 2024-09-15T03:47:22.336 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.950302+0000", 2024-09-15T03:47:22.336 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.339341009, 2024-09-15T03:47:22.337 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011370633999999999, 2024-09-15T03:47:22.337 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:22.337 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:22.337 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:22.337 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.337 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:22.337 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950302+0000", 2024-09-15T03:47:22.337 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.337 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.338 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.338 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:22.339 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950302+0000", 2024-09-15T03:47:22.339 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.339 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.339 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:22.339 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950303+0000", 2024-09-15T03:47:22.339 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1900000000000001e-07 2024-09-15T03:47:22.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.339 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.339 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:22.339 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950323+0000", 2024-09-15T03:47:22.339 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.039e-05 2024-09-15T03:47:22.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:22.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950340+0000", 2024-09-15T03:47:22.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6742999999999999e-05 2024-09-15T03:47:22.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:22.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950347+0000", 2024-09-15T03:47:22.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8129999999999998e-06 2024-09-15T03:47:22.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950409+0000", 2024-09-15T03:47:22.341 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.19e-05 2024-09-15T03:47:22.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.341 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.341 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:22.341 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950419+0000", 2024-09-15T03:47:22.341 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0509000000000001e-05 2024-09-15T03:47:22.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.341 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.341 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:22.341 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950518+0000", 2024-09-15T03:47:22.341 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8974000000000005e-05 2024-09-15T03:47:22.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.341 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.342 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:22.342 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950940+0000", 2024-09-15T03:47:22.342 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000421314 2024-09-15T03:47:22.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.342 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:22.342 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951016+0000", 2024-09-15T03:47:22.342 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6252000000000001e-05 2024-09-15T03:47:22.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.342 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:22.342 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951047+0000", 2024-09-15T03:47:22.342 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0994000000000002e-05 2024-09-15T03:47:22.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:22.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961673+0000", 2024-09-15T03:47:22.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010625826 2024-09-15T03:47:22.343 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.343 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:22.343 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.343 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.ds4 74/70 ECSubWrite(tid=202, reqid=client.4521.0:568, at_version=0'0, trim_to=0'0, roll_forward_to=74'34))", 2024-09-15T03:47:22.343 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.952210+0000", 2024-09-15T03:47:22.343 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.3374334399999999, 2024-09-15T03:47:22.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0097371300000000001, 2024-09-15T03:47:22.344 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:22.344 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:22.344 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:22.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:22.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952210+0000", 2024-09-15T03:47:22.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:22.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952210+0000", 2024-09-15T03:47:22.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:22.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952211+0000", 2024-09-15T03:47:22.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0160000000000001e-06 2024-09-15T03:47:22.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:22.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952233+0000", 2024-09-15T03:47:22.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2724000000000001e-05 2024-09-15T03:47:22.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:22.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952253+0000", 2024-09-15T03:47:22.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0001999999999999e-05 2024-09-15T03:47:22.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:22.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952268+0000", 2024-09-15T03:47:22.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4477e-05 2024-09-15T03:47:22.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952344+0000", 2024-09-15T03:47:22.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6149999999999994e-05 2024-09-15T03:47:22.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:22.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952354+0000", 2024-09-15T03:47:22.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5799999999999998e-06 2024-09-15T03:47:22.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:22.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952458+0000", 2024-09-15T03:47:22.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104022 2024-09-15T03:47:22.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:22.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952872+0000", 2024-09-15T03:47:22.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00041471800000000001 2024-09-15T03:47:22.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:22.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952919+0000", 2024-09-15T03:47:22.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7002000000000002e-05 2024-09-15T03:47:22.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:22.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952936+0000", 2024-09-15T03:47:22.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6818000000000001e-05 2024-09-15T03:47:22.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:22.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961947+0000", 2024-09-15T03:47:22.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0090106209999999999 2024-09-15T03:47:22.349 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.349 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:22.349 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.349 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.9s2 74/70 ECSubWrite(tid=109, reqid=client.4521.0:569, at_version=0'0, trim_to=0'0, roll_forward_to=74'33))", 2024-09-15T03:47:22.349 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.952331+0000", 2024-09-15T03:47:22.349 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.3373117130000001, 2024-09-15T03:47:22.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010396994, 2024-09-15T03:47:22.349 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:22.349 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:22.350 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:22.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.350 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:22.350 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952331+0000", 2024-09-15T03:47:22.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.350 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:22.350 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952331+0000", 2024-09-15T03:47:22.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.350 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:22.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952332+0000", 2024-09-15T03:47:22.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4799999999999997e-07 2024-09-15T03:47:22.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:22.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952346+0000", 2024-09-15T03:47:22.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.363e-05 2024-09-15T03:47:22.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:22.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952357+0000", 2024-09-15T03:47:22.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1224e-05 2024-09-15T03:47:22.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:22.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952368+0000", 2024-09-15T03:47:22.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0997e-05 2024-09-15T03:47:22.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952416+0000", 2024-09-15T03:47:22.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7577999999999997e-05 2024-09-15T03:47:22.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:22.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952425+0000", 2024-09-15T03:47:22.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4390000000000006e-06 2024-09-15T03:47:22.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:22.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952565+0000", 2024-09-15T03:47:22.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013950400000000001 2024-09-15T03:47:22.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:22.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952874+0000", 2024-09-15T03:47:22.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030934499999999999 2024-09-15T03:47:22.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.354 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:22.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952920+0000", 2024-09-15T03:47:22.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6031e-05 2024-09-15T03:47:22.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.354 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:22.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.952973+0000", 2024-09-15T03:47:22.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2617999999999997e-05 2024-09-15T03:47:22.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:22.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.962728+0000", 2024-09-15T03:47:22.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0097556799999999992 2024-09-15T03:47:22.355 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.355 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:22.355 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.355 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:573 3.3s0 3:cdb3c515:::smithi00132172-9:head [append 0~573440] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:22.355 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.955752+0000", 2024-09-15T03:47:22.355 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.3338906480000001, 2024-09-15T03:47:22.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0080949639999999996, 2024-09-15T03:47:22.355 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:22.356 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:22.356 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:22.356 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:22.356 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:22.356 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 573 2024-09-15T03:47:22.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.356 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:22.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:22.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.955752+0000", 2024-09-15T03:47:22.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:22.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.955752+0000", 2024-09-15T03:47:22.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:22.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.955755+0000", 2024-09-15T03:47:22.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.565e-06 2024-09-15T03:47:22.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:22.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.956366+0000", 2024-09-15T03:47:22.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000611441 2024-09-15T03:47:22.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:22.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.956389+0000", 2024-09-15T03:47:22.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2847999999999999e-05 2024-09-15T03:47:22.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:22.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.956398+0000", 2024-09-15T03:47:22.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2220000000000008e-06 2024-09-15T03:47:22.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.956449+0000", 2024-09-15T03:47:22.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0148000000000002e-05 2024-09-15T03:47:22.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:22.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.956623+0000", 2024-09-15T03:47:22.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017447200000000001 2024-09-15T03:47:22.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:22.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.957641+0000", 2024-09-15T03:47:22.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010183289999999999 2024-09-15T03:47:22.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:22.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.958217+0000", 2024-09-15T03:47:22.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00057589499999999999 2024-09-15T03:47:22.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:22.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.958231+0000", 2024-09-15T03:47:22.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3327e-05 2024-09-15T03:47:22.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:22.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.960462+0000", 2024-09-15T03:47:22.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022311589999999999 2024-09-15T03:47:22.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:22.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.960483+0000", 2024-09-15T03:47:22.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0869999999999998e-05 2024-09-15T03:47:22.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:22.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961825+0000", 2024-09-15T03:47:22.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001342843 2024-09-15T03:47:22.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:22.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.963847+0000", 2024-09-15T03:47:22.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002021845 2024-09-15T03:47:22.362 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.362 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:22.362 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.362 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s7 75/70 ECSubWrite(tid=198, reqid=client.4521.0:593, at_version=75'53, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:22.362 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:21.946494+0000", 2024-09-15T03:47:22.362 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.343149441, 2024-09-15T03:47:22.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010095843, 2024-09-15T03:47:22.362 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:22.362 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:22.362 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:22.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:22.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946494+0000", 2024-09-15T03:47:22.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.363 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:22.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946494+0000", 2024-09-15T03:47:22.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.363 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:22.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946494+0000", 2024-09-15T03:47:22.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.78e-07 2024-09-15T03:47:22.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.364 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:22.364 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946514+0000", 2024-09-15T03:47:22.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9531e-05 2024-09-15T03:47:22.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.364 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:22.364 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946528+0000", 2024-09-15T03:47:22.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4226999999999999e-05 2024-09-15T03:47:22.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.364 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:22.364 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946535+0000", 2024-09-15T03:47:22.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6599999999999998e-06 2024-09-15T03:47:22.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946585+0000", 2024-09-15T03:47:22.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0015000000000001e-05 2024-09-15T03:47:22.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:22.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946594+0000", 2024-09-15T03:47:22.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2599999999999994e-06 2024-09-15T03:47:22.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:22.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946708+0000", 2024-09-15T03:47:22.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000113812 2024-09-15T03:47:22.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.366 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:22.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946730+0000", 2024-09-15T03:47:22.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1702999999999999e-05 2024-09-15T03:47:22.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.366 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:22.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946761+0000", 2024-09-15T03:47:22.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0848e-05 2024-09-15T03:47:22.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:22.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946771+0000", 2024-09-15T03:47:22.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.027e-05 2024-09-15T03:47:22.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:22.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.956589+0000", 2024-09-15T03:47:22.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098186390000000005 2024-09-15T03:47:22.367 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.367 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:22.367 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.368 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s7 75/70 ECSubWrite(tid=200, reqid=client.4521.0:594, at_version=75'54, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:22.368 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:21.947430+0000", 2024-09-15T03:47:22.368 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.342213247, 2024-09-15T03:47:22.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095446720000000006, 2024-09-15T03:47:22.368 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:22.368 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:22.368 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:22.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:22.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947430+0000", 2024-09-15T03:47:22.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:22.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947430+0000", 2024-09-15T03:47:22.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:22.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947430+0000", 2024-09-15T03:47:22.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.99e-07 2024-09-15T03:47:22.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:22.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947446+0000", 2024-09-15T03:47:22.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5631999999999999e-05 2024-09-15T03:47:22.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:22.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947464+0000", 2024-09-15T03:47:22.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7563000000000001e-05 2024-09-15T03:47:22.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:22.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947470+0000", 2024-09-15T03:47:22.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7959999999999998e-06 2024-09-15T03:47:22.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947517+0000", 2024-09-15T03:47:22.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6159000000000003e-05 2024-09-15T03:47:22.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:22.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947526+0000", 2024-09-15T03:47:22.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2019999999999993e-06 2024-09-15T03:47:22.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:22.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947618+0000", 2024-09-15T03:47:22.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1736000000000004e-05 2024-09-15T03:47:22.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.372 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:22.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947624+0000", 2024-09-15T03:47:22.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7309999999999996e-06 2024-09-15T03:47:22.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.372 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:22.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948013+0000", 2024-09-15T03:47:22.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00038882000000000001 2024-09-15T03:47:22.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.372 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:22.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948029+0000", 2024-09-15T03:47:22.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5818e-05 2024-09-15T03:47:22.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:22.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.956974+0000", 2024-09-15T03:47:22.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0089455160000000006 2024-09-15T03:47:22.373 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.373 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:22.373 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.373 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s7 75/70 ECSubWrite(tid=203, reqid=client.4521.0:594, at_version=0'0, trim_to=0'0, roll_forward_to=75'54))", 2024-09-15T03:47:22.373 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:21.949408+0000", 2024-09-15T03:47:22.373 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.34023531099999998, 2024-09-15T03:47:22.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0078501809999999995, 2024-09-15T03:47:22.374 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:22.374 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:22.374 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:22.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:22.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949408+0000", 2024-09-15T03:47:22.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:22.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949408+0000", 2024-09-15T03:47:22.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:22.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949409+0000", 2024-09-15T03:47:22.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0999999999999997e-07 2024-09-15T03:47:22.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:22.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949424+0000", 2024-09-15T03:47:22.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5007999999999999e-05 2024-09-15T03:47:22.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:22.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949438+0000", 2024-09-15T03:47:22.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4551e-05 2024-09-15T03:47:22.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:22.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949444+0000", 2024-09-15T03:47:22.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3260000000000001e-06 2024-09-15T03:47:22.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949490+0000", 2024-09-15T03:47:22.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5537999999999997e-05 2024-09-15T03:47:22.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:22.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949499+0000", 2024-09-15T03:47:22.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0799999999999995e-06 2024-09-15T03:47:22.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:22.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949615+0000", 2024-09-15T03:47:22.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000115884 2024-09-15T03:47:22.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.378 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:22.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949622+0000", 2024-09-15T03:47:22.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3390000000000004e-06 2024-09-15T03:47:22.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.378 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:22.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949718+0000", 2024-09-15T03:47:22.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5816999999999994e-05 2024-09-15T03:47:22.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.378 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:22.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949731+0000", 2024-09-15T03:47:22.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3047999999999999e-05 2024-09-15T03:47:22.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:22.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.957258+0000", 2024-09-15T03:47:22.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075267800000000003 2024-09-15T03:47:22.379 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.379 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:22.379 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.379 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.0s7 75/70 ECSubWrite(tid=109, reqid=client.4521.0:587, at_version=75'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:22.379 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:21.950394+0000", 2024-09-15T03:47:22.379 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.33924902899999998, 2024-09-15T03:47:22.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011842180000000001, 2024-09-15T03:47:22.380 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:22.380 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:22.380 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:22.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:22.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950394+0000", 2024-09-15T03:47:22.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:22.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950394+0000", 2024-09-15T03:47:22.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:22.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950394+0000", 2024-09-15T03:47:22.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2599999999999998e-07 2024-09-15T03:47:22.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:22.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950536+0000", 2024-09-15T03:47:22.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014176400000000001 2024-09-15T03:47:22.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:22.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950548+0000", 2024-09-15T03:47:22.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1681e-05 2024-09-15T03:47:22.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:22.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950554+0000", 2024-09-15T03:47:22.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1580000000000002e-06 2024-09-15T03:47:22.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950602+0000", 2024-09-15T03:47:22.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8319000000000003e-05 2024-09-15T03:47:22.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:22.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950621+0000", 2024-09-15T03:47:22.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9069999999999999e-05 2024-09-15T03:47:22.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:22.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950930+0000", 2024-09-15T03:47:22.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030853400000000001 2024-09-15T03:47:22.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:22.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950941+0000", 2024-09-15T03:47:22.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1260999999999999e-05 2024-09-15T03:47:22.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:22.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.951021+0000", 2024-09-15T03:47:22.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9600999999999999e-05 2024-09-15T03:47:22.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:22.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.951041+0000", 2024-09-15T03:47:22.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0132e-05 2024-09-15T03:47:22.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:22.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.962236+0000", 2024-09-15T03:47:22.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011195333999999999 2024-09-15T03:47:22.385 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.385 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:22.385 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.385 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.0s7 75/70 ECSubWrite(tid=110, reqid=client.4521.0:587, at_version=0'0, trim_to=0'0, roll_forward_to=75'28))", 2024-09-15T03:47:22.385 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:21.953246+0000", 2024-09-15T03:47:22.385 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.336397483, 2024-09-15T03:47:22.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0091457559999999997, 2024-09-15T03:47:22.385 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:22.385 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:22.386 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:22.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:22.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953246+0000", 2024-09-15T03:47:22.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:22.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953246+0000", 2024-09-15T03:47:22.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:22.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953247+0000", 2024-09-15T03:47:22.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.102e-06 2024-09-15T03:47:22.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:22.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953263+0000", 2024-09-15T03:47:22.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6277999999999999e-05 2024-09-15T03:47:22.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:22.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953280+0000", 2024-09-15T03:47:22.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6668e-05 2024-09-15T03:47:22.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:22.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953288+0000", 2024-09-15T03:47:22.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1419999999999993e-06 2024-09-15T03:47:22.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953333+0000", 2024-09-15T03:47:22.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5189999999999999e-05 2024-09-15T03:47:22.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:22.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953340+0000", 2024-09-15T03:47:22.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5179999999999998e-06 2024-09-15T03:47:22.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:22.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953405+0000", 2024-09-15T03:47:22.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4068999999999999e-05 2024-09-15T03:47:22.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:22.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953414+0000", 2024-09-15T03:47:22.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4599999999999992e-06 2024-09-15T03:47:22.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:22.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953462+0000", 2024-09-15T03:47:22.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7741000000000003e-05 2024-09-15T03:47:22.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:22.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953475+0000", 2024-09-15T03:47:22.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3254e-05 2024-09-15T03:47:22.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.962391+0000", 2024-09-15T03:47:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0089163339999999997 2024-09-15T03:47:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs5 75/70 ECSubWrite(tid=160, reqid=client.4521.0:595, at_version=75'87, trim_to=0'0, roll_forward_to=73'85))", 2024-09-15T03:47:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:21.968372+0000", 2024-09-15T03:47:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.321271001, 2024-09-15T03:47:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094954359999999995, 2024-09-15T03:47:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.968372+0000", 2024-09-15T03:47:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.968372+0000", 2024-09-15T03:47:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.968372+0000", 2024-09-15T03:47:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9200000000000002e-07 2024-09-15T03:47:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969061+0000", 2024-09-15T03:47:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00068916400000000003 2024-09-15T03:47:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969074+0000", 2024-09-15T03:47:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.234e-05 2024-09-15T03:47:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969080+0000", 2024-09-15T03:47:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9490000000000001e-06 2024-09-15T03:47:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969130+0000", 2024-09-15T03:47:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0639000000000001e-05 2024-09-15T03:47:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969140+0000", 2024-09-15T03:47:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8560000000000002e-06 2024-09-15T03:47:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969423+0000", 2024-09-15T03:47:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00028318400000000002 2024-09-15T03:47:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969445+0000", 2024-09-15T03:47:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1997999999999998e-05 2024-09-15T03:47:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969544+0000", 2024-09-15T03:47:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9056999999999997e-05 2024-09-15T03:47:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969560+0000", 2024-09-15T03:47:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5891000000000001e-05 2024-09-15T03:47:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.977867+0000", 2024-09-15T03:47:22.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0083070660000000001 2024-09-15T03:47:22.397 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.397 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:22.397 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.397 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs5 75/70 ECSubWrite(tid=161, reqid=client.4521.0:596, at_version=75'88, trim_to=0'0, roll_forward_to=73'85))", 2024-09-15T03:47:22.397 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:21.970276+0000", 2024-09-15T03:47:22.397 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.31936652799999998, 2024-09-15T03:47:22.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0082187070000000004, 2024-09-15T03:47:22.397 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:22.397 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:22.397 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:22.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:22.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.970276+0000", 2024-09-15T03:47:22.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:22.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.970276+0000", 2024-09-15T03:47:22.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:22.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:22.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.970277+0000", 2024-09-15T03:47:22.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6700000000000002e-07 2024-09-15T03:47:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.970635+0000", 2024-09-15T03:47:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00035746600000000002 2024-09-15T03:47:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.970654+0000", 2024-09-15T03:47:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8848e-05 2024-09-15T03:47:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.970661+0000", 2024-09-15T03:47:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5279999999999998e-06 2024-09-15T03:47:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.970708+0000", 2024-09-15T03:47:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7286000000000001e-05 2024-09-15T03:47:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.970719+0000", 2024-09-15T03:47:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.076e-05 2024-09-15T03:47:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.970871+0000", 2024-09-15T03:47:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015181 2024-09-15T03:47:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.970933+0000", 2024-09-15T03:47:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1884000000000006e-05 2024-09-15T03:47:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.971072+0000", 2024-09-15T03:47:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013884100000000001 2024-09-15T03:47:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.971090+0000", 2024-09-15T03:47:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8700999999999999e-05 2024-09-15T03:47:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.978495+0000", 2024-09-15T03:47:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0074047160000000004 2024-09-15T03:47:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:22.403 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.403 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:22.403 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:22.403 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:22.404 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:47:22.405 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:22.403+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:22.451 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:47:22.501 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:22.501 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:22.502 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:22.502 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:22.505 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:22.504+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:22.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:47:22.582 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:22.583 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:22.597 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:22.597 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:22.605 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:22.605+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:22.608 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:22.608 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:22.608 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:22.608 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:22.608 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:22.611 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:22.612 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:22.616 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:47:22.625 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:22.626 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:22.640 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:22.641 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:22.655 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:22.656 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:22.671 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:22.671 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:22.685 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:22.686 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:22.700 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:22.700 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:22.706 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:22.706+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:22.715 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:22.715 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:22.731 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:47:22.732 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:47:22.746 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:22.746 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:22.755 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:22.756 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:22.764 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:47:22.808 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:22.807+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:22.904 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:22.904 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:22.909 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:22.908+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:22.913 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:47:22.939 INFO:tasks.rados.rados.0.smithi001.stdout:239: done (0 left) 2024-09-15T03:47:22.939 INFO:tasks.rados.rados.0.smithi001.stdout:240: delete oid 48 current snap is 29 2024-09-15T03:47:22.954 INFO:tasks.rados.rados.0.smithi001.stdout:240: done (0 left) 2024-09-15T03:47:22.954 INFO:tasks.rados.rados.0.smithi001.stdout:241: snap_create 2024-09-15T03:47:22.958 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T03:47:23.010 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:23.009+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:23.021 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:23.022 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:23.032 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:47:23.111 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:23.110+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:23.174 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:23.174 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:23.182 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:47:23.226 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:23.225+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:23.283 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:23.283 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:23.291 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:47:23.313 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:23.312+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:23.414 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:23.413+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:23.432 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:23.432 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:23.440 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:47:23.515 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:23.514+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:23.541 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:23.541 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:23.549 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:47:23.616 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:23.615+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:23.650 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:23.650 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:23.658 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:47:23.717 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:23.716+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:23.758 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:23.759 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:23.767 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:47:23.818 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:23.818+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:23.867 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:23.868 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:23.876 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:47:23.919 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:23.918+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:23.945 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-15T03:47:23.946 INFO:tasks.rados.rados.0.smithi001.stdout:241: done (0 left) 2024-09-15T03:47:23.946 INFO:tasks.rados.rados.0.smithi001.stdout:242: snap_remove snap 27 2024-09-15T03:47:23.979 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:23.979 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:23.987 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:47:24.020 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:24.019+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:24.121 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:24.120+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:24.128 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:24.128 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:24.136 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:47:24.221 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:24.220+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:24.237 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:24.237 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:24.245 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:47:24.322 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:24.322+0000 7f97769f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 29467) UID: 0 2024-09-15T03:47:24.359 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:24.359 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:24.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:47:24.423 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:24.423+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:24.469 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:24.470 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:24.477 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:47:24.524 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:24.523+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:24.619 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:24.619 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:24.625 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:24.624+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:24.627 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:47:24.726 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:24.726+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:24.758 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:47:24.768 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:24.768 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:24.777 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:47:24.827 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:24.827+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:24.878 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:24.879 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:24.887 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:47:24.888 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:24.888 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:24.901 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:24.902 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:24.916 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:24.917 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:24.928 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:24.928+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:24.930 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:24.931 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:24.946 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:24.947 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:24.952 INFO:tasks.rados.rados.0.smithi001.stdout:242: done (0 left) 2024-09-15T03:47:24.952 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 29 current snap is 30 2024-09-15T03:47:24.952 INFO:tasks.rados.rados.0.smithi001.stdout:243: seq_num 109 ranges {376832=671744,1048576=753664,1802240=409600} 2024-09-15T03:47:24.959 INFO:tasks.rados.rados.0.smithi001.stdout:243: writing smithi00132172-29 from 376832 to 1048576 tid 1 2024-09-15T03:47:24.966 INFO:tasks.rados.rados.0.smithi001.stdout:243: writing smithi00132172-29 from 1048576 to 1802240 tid 2 2024-09-15T03:47:24.969 INFO:tasks.rados.rados.0.smithi001.stdout:243: writing smithi00132172-29 from 1802240 to 2211840 tid 3 2024-09-15T03:47:24.969 INFO:tasks.rados.rados.0.smithi001.stdout:244: read oid 48 snap -1 2024-09-15T03:47:24.969 INFO:tasks.rados.rados.0.smithi001.stdout:244: expect deleted 2024-09-15T03:47:24.969 INFO:tasks.rados.rados.0.smithi001.stdout:245: read oid 41 snap 25 2024-09-15T03:47:24.969 INFO:tasks.rados.rados.0.smithi001.stdout:245: expect (ObjNum 81 snap 16 seq_num 81) 2024-09-15T03:47:24.970 INFO:tasks.rados.rados.0.smithi001.stdout:246: snap_create 2024-09-15T03:47:24.970 INFO:tasks.rados.rados.0.smithi001.stdout:243: finishing write tid 1 to smithi00132172-29 2024-09-15T03:47:24.971 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:24.972 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:24.973 INFO:tasks.rados.rados.0.smithi001.stdout:243: finishing write tid 2 to smithi00132172-29 2024-09-15T03:47:24.974 INFO:tasks.rados.rados.0.smithi001.stdout:243: finishing write tid 3 to smithi00132172-29 2024-09-15T03:47:24.975 INFO:tasks.rados.rados.0.smithi001.stdout:243: finishing write tid 4 to smithi00132172-29 2024-09-15T03:47:24.977 INFO:tasks.rados.rados.0.smithi001.stdout:243: finishing write tid 5 to smithi00132172-29 2024-09-15T03:47:24.977 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 29 v 102 (ObjNum 109 snap 30 seq_num 109) dirty exists 2024-09-15T03:47:24.978 INFO:tasks.rados.rados.0.smithi001.stdout:243: left oid 29 (ObjNum 109 snap 30 seq_num 109) 2024-09-15T03:47:24.983 INFO:tasks.rados.rados.0.smithi001.stdout:245: expect (ObjNum 81 snap 16 seq_num 81) 2024-09-15T03:47:24.996 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:24.996 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:25.002 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:25.003 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:25.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:25.003 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:25.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.003 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 38/28 ECSubWrite(tid=33, reqid=client.4521.0:222, at_version=38'14, trim_to=0'0, roll_forward_to=35'12))", 2024-09-15T03:47:25.003 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:47:25.003 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.870128811999997, 2024-09-15T03:47:25.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9949187450000001, 2024-09-15T03:47:25.003 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.003 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.003 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:47:25.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:47:25.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130996+0000", 2024-09-15T03:47:25.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8809999999999998e-06 2024-09-15T03:47:25.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131020+0000", 2024-09-15T03:47:25.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4190999999999999e-05 2024-09-15T03:47:25.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131039+0000", 2024-09-15T03:47:25.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9228999999999998e-05 2024-09-15T03:47:25.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131048+0000", 2024-09-15T03:47:25.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2380000000000003e-06 2024-09-15T03:47:25.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131122+0000", 2024-09-15T03:47:25.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3146999999999996e-05 2024-09-15T03:47:25.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131132+0000", 2024-09-15T03:47:25.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.08e-05 2024-09-15T03:47:25.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131381+0000", 2024-09-15T03:47:25.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000248936 2024-09-15T03:47:25.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131392+0000", 2024-09-15T03:47:25.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0987e-05 2024-09-15T03:47:25.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131444+0000", 2024-09-15T03:47:25.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1585000000000002e-05 2024-09-15T03:47:25.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131465+0000", 2024-09-15T03:47:25.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1365999999999998e-05 2024-09-15T03:47:25.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.125911+0000", 2024-09-15T03:47:25.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9944453850000001 2024-09-15T03:47:25.008 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.008 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.009 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.009 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 38/28 ECSubWrite(tid=34, reqid=client.4521.0:222, at_version=0'0, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T03:47:25.009 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:47:25.009 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.868729713, 2024-09-15T03:47:25.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.002752697, 2024-09-15T03:47:25.009 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.009 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.009 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:47:25.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:47:25.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132392+0000", 2024-09-15T03:47:25.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5899999999999995e-07 2024-09-15T03:47:25.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132415+0000", 2024-09-15T03:47:25.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3561e-05 2024-09-15T03:47:25.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132436+0000", 2024-09-15T03:47:25.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0879000000000001e-05 2024-09-15T03:47:25.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132444+0000", 2024-09-15T03:47:25.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9200000000000004e-06 2024-09-15T03:47:25.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132497+0000", 2024-09-15T03:47:25.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2299000000000002e-05 2024-09-15T03:47:25.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132508+0000", 2024-09-15T03:47:25.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1100999999999999e-05 2024-09-15T03:47:25.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132615+0000", 2024-09-15T03:47:25.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107546 2024-09-15T03:47:25.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132635+0000", 2024-09-15T03:47:25.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9374000000000001e-05 2024-09-15T03:47:25.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132677+0000", 2024-09-15T03:47:25.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2456999999999997e-05 2024-09-15T03:47:25.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132711+0000", 2024-09-15T03:47:25.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4118000000000001e-05 2024-09-15T03:47:25.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135144+0000", 2024-09-15T03:47:25.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.002432583 2024-09-15T03:47:25.014 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.014 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.014 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.014 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=45, reqid=client.4521.0:227, at_version=38'24, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:47:25.015 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:25.015 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.845803171, 2024-09-15T03:47:25.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.974765339, 2024-09-15T03:47:25.015 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.015 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.015 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:25.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:25.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:25.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.43e-07 2024-09-15T03:47:25.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156110+0000", 2024-09-15T03:47:25.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000792409 2024-09-15T03:47:25.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156139+0000", 2024-09-15T03:47:25.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9207e-05 2024-09-15T03:47:25.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156148+0000", 2024-09-15T03:47:25.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1269999999999992e-06 2024-09-15T03:47:25.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156248+0000", 2024-09-15T03:47:25.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9421999999999998e-05 2024-09-15T03:47:25.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156258+0000", 2024-09-15T03:47:25.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6989999999999998e-06 2024-09-15T03:47:25.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156674+0000", 2024-09-15T03:47:25.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041682600000000002 2024-09-15T03:47:25.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156705+0000", 2024-09-15T03:47:25.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0525000000000003e-05 2024-09-15T03:47:25.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157742+0000", 2024-09-15T03:47:25.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010374 2024-09-15T03:47:25.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157777+0000", 2024-09-15T03:47:25.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4878e-05 2024-09-15T03:47:25.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.130083+0000", 2024-09-15T03:47:25.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972305703 2024-09-15T03:47:25.020 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.020 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.021 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.021 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=37, reqid=client.4521.0:236, at_version=38'5, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:25.021 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:25.021 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.844189714999999, 2024-09-15T03:47:25.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9740836640000001, 2024-09-15T03:47:25.021 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.021 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.021 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:25.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:25.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:25.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1300000000000001e-07 2024-09-15T03:47:25.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156953+0000", 2024-09-15T03:47:25.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1585000000000001e-05 2024-09-15T03:47:25.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156974+0000", 2024-09-15T03:47:25.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0764e-05 2024-09-15T03:47:25.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156981+0000", 2024-09-15T03:47:25.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2760000000000003e-06 2024-09-15T03:47:25.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157061+0000", 2024-09-15T03:47:25.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9583e-05 2024-09-15T03:47:25.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157075+0000", 2024-09-15T03:47:25.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4019000000000001e-05 2024-09-15T03:47:25.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157351+0000", 2024-09-15T03:47:25.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027641100000000001 2024-09-15T03:47:25.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157746+0000", 2024-09-15T03:47:25.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039553199999999998 2024-09-15T03:47:25.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157838+0000", 2024-09-15T03:47:25.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1632000000000006e-05 2024-09-15T03:47:25.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157856+0000", 2024-09-15T03:47:25.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7873999999999999e-05 2024-09-15T03:47:25.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.131015+0000", 2024-09-15T03:47:25.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9731586750000001 2024-09-15T03:47:25.026 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.026 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.027 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.027 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=46, reqid=client.4521.0:228, at_version=38'25, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:47:25.027 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:25.027 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.844130933000002, 2024-09-15T03:47:25.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9732838510000001, 2024-09-15T03:47:25.027 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.027 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.027 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:25.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:25.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:25.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0999999999999998e-07 2024-09-15T03:47:25.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157149+0000", 2024-09-15T03:47:25.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000158763 2024-09-15T03:47:25.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157165+0000", 2024-09-15T03:47:25.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5883999999999999e-05 2024-09-15T03:47:25.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157175+0000", 2024-09-15T03:47:25.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4949999999999997e-06 2024-09-15T03:47:25.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157219+0000", 2024-09-15T03:47:25.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4069000000000001e-05 2024-09-15T03:47:25.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157228+0000", 2024-09-15T03:47:25.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7829999999999993e-06 2024-09-15T03:47:25.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157345+0000", 2024-09-15T03:47:25.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116655 2024-09-15T03:47:25.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157746+0000", 2024-09-15T03:47:25.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040045100000000002 2024-09-15T03:47:25.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157838+0000", 2024-09-15T03:47:25.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1955999999999998e-05 2024-09-15T03:47:25.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157888+0000", 2024-09-15T03:47:25.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0943000000000003e-05 2024-09-15T03:47:25.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.130274+0000", 2024-09-15T03:47:25.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972385142 2024-09-15T03:47:25.032 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.032 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.032 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.033 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=38, reqid=client.4521.0:237, at_version=38'6, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:25.033 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:47:25.033 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.842788362999997, 2024-09-15T03:47:25.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9764011529999999, 2024-09-15T03:47:25.033 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.033 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.033 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:47:25.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:47:25.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158333+0000", 2024-09-15T03:47:25.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8599999999999997e-07 2024-09-15T03:47:25.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158353+0000", 2024-09-15T03:47:25.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9504000000000002e-05 2024-09-15T03:47:25.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158368+0000", 2024-09-15T03:47:25.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4966999999999999e-05 2024-09-15T03:47:25.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158374+0000", 2024-09-15T03:47:25.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.798e-06 2024-09-15T03:47:25.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158430+0000", 2024-09-15T03:47:25.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5744e-05 2024-09-15T03:47:25.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158440+0000", 2024-09-15T03:47:25.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4560000000000006e-06 2024-09-15T03:47:25.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158529+0000", 2024-09-15T03:47:25.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9172999999999995e-05 2024-09-15T03:47:25.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158543+0000", 2024-09-15T03:47:25.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.395e-05 2024-09-15T03:47:25.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158654+0000", 2024-09-15T03:47:25.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011110700000000001 2024-09-15T03:47:25.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158671+0000", 2024-09-15T03:47:25.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7068999999999999e-05 2024-09-15T03:47:25.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134733+0000", 2024-09-15T03:47:25.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.976062599 2024-09-15T03:47:25.038 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.038 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.038 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.038 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=47, reqid=client.4521.0:229, at_version=38'26, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:47:25.039 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:47:25.039 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.842784401000003, 2024-09-15T03:47:25.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9727810969999999, 2024-09-15T03:47:25.039 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.039 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.039 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:47:25.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:47:25.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158337+0000", 2024-09-15T03:47:25.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.15e-07 2024-09-15T03:47:25.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158351+0000", 2024-09-15T03:47:25.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3871e-05 2024-09-15T03:47:25.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158372+0000", 2024-09-15T03:47:25.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1058999999999999e-05 2024-09-15T03:47:25.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158376+0000", 2024-09-15T03:47:25.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5249999999999999e-06 2024-09-15T03:47:25.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158420+0000", 2024-09-15T03:47:25.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3918000000000002e-05 2024-09-15T03:47:25.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158429+0000", 2024-09-15T03:47:25.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5150000000000006e-06 2024-09-15T03:47:25.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158514+0000", 2024-09-15T03:47:25.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5252000000000003e-05 2024-09-15T03:47:25.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158542+0000", 2024-09-15T03:47:25.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.798e-05 2024-09-15T03:47:25.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158653+0000", 2024-09-15T03:47:25.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111315 2024-09-15T03:47:25.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158701+0000", 2024-09-15T03:47:25.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8241000000000001e-05 2024-09-15T03:47:25.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.131117+0000", 2024-09-15T03:47:25.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972416006 2024-09-15T03:47:25.044 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.044 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.044 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.045 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=39, reqid=client.4521.0:238, at_version=38'7, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:25.045 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:47:25.045 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.842097244999998, 2024-09-15T03:47:25.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9758214509999998, 2024-09-15T03:47:25.045 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.045 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.045 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:47:25.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:47:25.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159024+0000", 2024-09-15T03:47:25.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6700000000000003e-07 2024-09-15T03:47:25.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159096+0000", 2024-09-15T03:47:25.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1386999999999999e-05 2024-09-15T03:47:25.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159113+0000", 2024-09-15T03:47:25.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7020999999999998e-05 2024-09-15T03:47:25.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159122+0000", 2024-09-15T03:47:25.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4720000000000001e-06 2024-09-15T03:47:25.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159168+0000", 2024-09-15T03:47:25.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5766000000000002e-05 2024-09-15T03:47:25.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159176+0000", 2024-09-15T03:47:25.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3070000000000003e-06 2024-09-15T03:47:25.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159258+0000", 2024-09-15T03:47:25.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1398999999999998e-05 2024-09-15T03:47:25.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159292+0000", 2024-09-15T03:47:25.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4480000000000002e-05 2024-09-15T03:47:25.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159342+0000", 2024-09-15T03:47:25.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9450999999999998e-05 2024-09-15T03:47:25.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159362+0000", 2024-09-15T03:47:25.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0633e-05 2024-09-15T03:47:25.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134845+0000", 2024-09-15T03:47:25.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.975482768 2024-09-15T03:47:25.051 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.051 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.051 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.052 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=48, reqid=client.4521.0:229, at_version=0'0, trim_to=0'0, roll_forward_to=38'26))", 2024-09-15T03:47:25.052 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:47:25.052 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.840730448999999, 2024-09-15T03:47:25.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.974485472, 2024-09-15T03:47:25.052 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.052 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.052 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:47:25.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:47:25.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160391+0000", 2024-09-15T03:47:25.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.136e-06 2024-09-15T03:47:25.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160405+0000", 2024-09-15T03:47:25.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3613e-05 2024-09-15T03:47:25.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160426+0000", 2024-09-15T03:47:25.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1234999999999999e-05 2024-09-15T03:47:25.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160434+0000", 2024-09-15T03:47:25.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.61e-06 2024-09-15T03:47:25.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160496+0000", 2024-09-15T03:47:25.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2384000000000004e-05 2024-09-15T03:47:25.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160507+0000", 2024-09-15T03:47:25.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1005000000000001e-05 2024-09-15T03:47:25.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160632+0000", 2024-09-15T03:47:25.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124566 2024-09-15T03:47:25.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160653+0000", 2024-09-15T03:47:25.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0707999999999999e-05 2024-09-15T03:47:25.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160701+0000", 2024-09-15T03:47:25.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8751999999999997e-05 2024-09-15T03:47:25.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160739+0000", 2024-09-15T03:47:25.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7486e-05 2024-09-15T03:47:25.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134876+0000", 2024-09-15T03:47:25.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9741369770000001 2024-09-15T03:47:25.057 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.057 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.057 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.058 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=40, reqid=client.4521.0:238, at_version=0'0, trim_to=0'0, roll_forward_to=38'7))", 2024-09-15T03:47:25.058 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:47:25.058 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.838884028000003, 2024-09-15T03:47:25.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9727862730000001, 2024-09-15T03:47:25.058 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.058 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.059 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:47:25.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:47:25.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162238+0000", 2024-09-15T03:47:25.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.672e-06 2024-09-15T03:47:25.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162253+0000", 2024-09-15T03:47:25.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4932e-05 2024-09-15T03:47:25.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162284+0000", 2024-09-15T03:47:25.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1211000000000003e-05 2024-09-15T03:47:25.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162296+0000", 2024-09-15T03:47:25.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1229999999999999e-05 2024-09-15T03:47:25.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162373+0000", 2024-09-15T03:47:25.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7146999999999998e-05 2024-09-15T03:47:25.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162387+0000", 2024-09-15T03:47:25.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4290999999999999e-05 2024-09-15T03:47:25.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162502+0000", 2024-09-15T03:47:25.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115054 2024-09-15T03:47:25.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162516+0000", 2024-09-15T03:47:25.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3597999999999999e-05 2024-09-15T03:47:25.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162558+0000", 2024-09-15T03:47:25.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2562999999999999e-05 2024-09-15T03:47:25.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162584+0000", 2024-09-15T03:47:25.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5738e-05 2024-09-15T03:47:25.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135023+0000", 2024-09-15T03:47:25.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9724388369999999 2024-09-15T03:47:25.064 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.064 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.064 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.064 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 39/28 ECSubWrite(tid=38, reqid=osd.11.0:38, at_version=39'16, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T03:47:25.064 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:25.064 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.857501728000003, 2024-09-15T03:47:25.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99210653800000004, 2024-09-15T03:47:25.065 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.065 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.065 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:25.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:25.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:25.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1800000000000002e-07 2024-09-15T03:47:25.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143648+0000", 2024-09-15T03:47:25.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.849e-05 2024-09-15T03:47:25.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143670+0000", 2024-09-15T03:47:25.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2036e-05 2024-09-15T03:47:25.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143680+0000", 2024-09-15T03:47:25.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8320000000000001e-06 2024-09-15T03:47:25.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143783+0000", 2024-09-15T03:47:25.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103569 2024-09-15T03:47:25.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143801+0000", 2024-09-15T03:47:25.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8077999999999999e-05 2024-09-15T03:47:25.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144077+0000", 2024-09-15T03:47:25.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027569500000000001 2024-09-15T03:47:25.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144459+0000", 2024-09-15T03:47:25.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038227299999999998 2024-09-15T03:47:25.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144519+0000", 2024-09-15T03:47:25.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9907000000000001e-05 2024-09-15T03:47:25.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144628+0000", 2024-09-15T03:47:25.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108958 2024-09-15T03:47:25.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135726+0000", 2024-09-15T03:47:25.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99109738199999997 2024-09-15T03:47:25.070 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.070 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.070 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.070 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 39/31 ECSubWrite(tid=41, reqid=osd.6.0:41, at_version=39'9, trim_to=0'0, roll_forward_to=38'7))", 2024-09-15T03:47:25.070 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:25.070 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.856851472999999, 2024-09-15T03:47:25.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99106653099999997, 2024-09-15T03:47:25.070 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.070 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.070 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:25.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:25.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:25.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.97e-07 2024-09-15T03:47:25.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144297+0000", 2024-09-15T03:47:25.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7815000000000001e-05 2024-09-15T03:47:25.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144314+0000", 2024-09-15T03:47:25.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6269e-05 2024-09-15T03:47:25.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144322+0000", 2024-09-15T03:47:25.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1880000000000002e-06 2024-09-15T03:47:25.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144767+0000", 2024-09-15T03:47:25.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044555300000000001 2024-09-15T03:47:25.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144777+0000", 2024-09-15T03:47:25.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5950000000000005e-06 2024-09-15T03:47:25.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144992+0000", 2024-09-15T03:47:25.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000215479 2024-09-15T03:47:25.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145142+0000", 2024-09-15T03:47:25.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149302 2024-09-15T03:47:25.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145249+0000", 2024-09-15T03:47:25.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106971 2024-09-15T03:47:25.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145299+0000", 2024-09-15T03:47:25.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0413e-05 2024-09-15T03:47:25.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135336+0000", 2024-09-15T03:47:25.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99003674900000005 2024-09-15T03:47:25.076 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.076 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.076 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.076 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 39/28 ECSubWrite(tid=50, reqid=osd.2.0:50, at_version=39'28, trim_to=0'0, roll_forward_to=38'26))", 2024-09-15T03:47:25.076 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:25.076 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.856553990999998, 2024-09-15T03:47:25.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99095576299999999, 2024-09-15T03:47:25.076 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.076 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.076 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:25.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:25.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:25.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.03e-07 2024-09-15T03:47:25.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144593+0000", 2024-09-15T03:47:25.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6520999999999999e-05 2024-09-15T03:47:25.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144611+0000", 2024-09-15T03:47:25.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7963000000000001e-05 2024-09-15T03:47:25.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144619+0000", 2024-09-15T03:47:25.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7409999999999992e-06 2024-09-15T03:47:25.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144821+0000", 2024-09-15T03:47:25.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000201652 2024-09-15T03:47:25.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144832+0000", 2024-09-15T03:47:25.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0827e-05 2024-09-15T03:47:25.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145099+0000", 2024-09-15T03:47:25.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026688299999999999 2024-09-15T03:47:25.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145143+0000", 2024-09-15T03:47:25.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.481e-05 2024-09-15T03:47:25.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145249+0000", 2024-09-15T03:47:25.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010618799999999999 2024-09-15T03:47:25.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145340+0000", 2024-09-15T03:47:25.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0340000000000006e-05 2024-09-15T03:47:25.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135522+0000", 2024-09-15T03:47:25.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99018263500000003 2024-09-15T03:47:25.083 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.083 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.083 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.084 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=43, reqid=client.4521.0:240, at_version=39'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:25.084 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:47:25.084 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.853213271000001, 2024-09-15T03:47:25.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99104975500000003, 2024-09-15T03:47:25.084 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.084 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.084 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:47:25.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:47:25.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147908+0000", 2024-09-15T03:47:25.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.05e-07 2024-09-15T03:47:25.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147931+0000", 2024-09-15T03:47:25.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3621e-05 2024-09-15T03:47:25.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147945+0000", 2024-09-15T03:47:25.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3648999999999999e-05 2024-09-15T03:47:25.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147951+0000", 2024-09-15T03:47:25.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0510000000000002e-06 2024-09-15T03:47:25.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148055+0000", 2024-09-15T03:47:25.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104421 2024-09-15T03:47:25.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148068+0000", 2024-09-15T03:47:25.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3019e-05 2024-09-15T03:47:25.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148375+0000", 2024-09-15T03:47:25.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030683400000000002 2024-09-15T03:47:25.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148399+0000", 2024-09-15T03:47:25.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3760999999999998e-05 2024-09-15T03:47:25.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148453+0000", 2024-09-15T03:47:25.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3851999999999999e-05 2024-09-15T03:47:25.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148481+0000", 2024-09-15T03:47:25.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8637999999999999e-05 2024-09-15T03:47:25.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.138957+0000", 2024-09-15T03:47:25.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99047580400000002 2024-09-15T03:47:25.089 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.089 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.089 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.089 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 39/28 ECSubWrite(tid=39, reqid=osd.11.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:47:25.089 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:47:25.089 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.853080781999999, 2024-09-15T03:47:25.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99452509, 2024-09-15T03:47:25.089 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.090 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.090 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:47:25.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:47:25.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148041+0000", 2024-09-15T03:47:25.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0999999999999998e-07 2024-09-15T03:47:25.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148052+0000", 2024-09-15T03:47:25.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.129e-05 2024-09-15T03:47:25.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148064+0000", 2024-09-15T03:47:25.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173e-05 2024-09-15T03:47:25.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148068+0000", 2024-09-15T03:47:25.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5680000000000001e-06 2024-09-15T03:47:25.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148136+0000", 2024-09-15T03:47:25.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7508999999999993e-05 2024-09-15T03:47:25.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148148+0000", 2024-09-15T03:47:25.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2002e-05 2024-09-15T03:47:25.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148265+0000", 2024-09-15T03:47:25.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117669 2024-09-15T03:47:25.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148289+0000", 2024-09-15T03:47:25.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3672e-05 2024-09-15T03:47:25.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148348+0000", 2024-09-15T03:47:25.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8838000000000001e-05 2024-09-15T03:47:25.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148368+0000", 2024-09-15T03:47:25.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9942999999999999e-05 2024-09-15T03:47:25.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142565+0000", 2024-09-15T03:47:25.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99419715900000005 2024-09-15T03:47:25.095 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.095 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.095 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.095 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=45, reqid=client.4521.0:241, at_version=39'17, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:25.095 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:47:25.095 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.850077759000001, 2024-09-15T03:47:25.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99118202200000005, 2024-09-15T03:47:25.095 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.096 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.096 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:47:25.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:47:25.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151044+0000", 2024-09-15T03:47:25.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1340000000000001e-06 2024-09-15T03:47:25.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151073+0000", 2024-09-15T03:47:25.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8894000000000002e-05 2024-09-15T03:47:25.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151095+0000", 2024-09-15T03:47:25.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1889e-05 2024-09-15T03:47:25.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151107+0000", 2024-09-15T03:47:25.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2136999999999999e-05 2024-09-15T03:47:25.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151189+0000", 2024-09-15T03:47:25.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1883000000000002e-05 2024-09-15T03:47:25.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151203+0000", 2024-09-15T03:47:25.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3825999999999999e-05 2024-09-15T03:47:25.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151305+0000", 2024-09-15T03:47:25.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101816 2024-09-15T03:47:25.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151337+0000", 2024-09-15T03:47:25.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2703000000000002e-05 2024-09-15T03:47:25.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151387+0000", 2024-09-15T03:47:25.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9366000000000001e-05 2024-09-15T03:47:25.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151405+0000", 2024-09-15T03:47:25.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8748000000000001e-05 2024-09-15T03:47:25.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142225+0000", 2024-09-15T03:47:25.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99081962599999995 2024-09-15T03:47:25.101 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.101 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.101 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.102 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 39/31 ECSubWrite(tid=42, reqid=osd.6.0:41, at_version=0'0, trim_to=0'0, roll_forward_to=39'9))", 2024-09-15T03:47:25.102 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:47:25.102 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.849781718000003, 2024-09-15T03:47:25.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99092639999999999, 2024-09-15T03:47:25.102 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.102 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.102 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:47:25.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:47:25.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151340+0000", 2024-09-15T03:47:25.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8299999999999996e-07 2024-09-15T03:47:25.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151350+0000", 2024-09-15T03:47:25.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0675e-05 2024-09-15T03:47:25.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151362+0000", 2024-09-15T03:47:25.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2048999999999999e-05 2024-09-15T03:47:25.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151369+0000", 2024-09-15T03:47:25.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3230000000000003e-06 2024-09-15T03:47:25.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151425+0000", 2024-09-15T03:47:25.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5964000000000002e-05 2024-09-15T03:47:25.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151435+0000", 2024-09-15T03:47:25.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9240000000000002e-06 2024-09-15T03:47:25.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151530+0000", 2024-09-15T03:47:25.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5649000000000005e-05 2024-09-15T03:47:25.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151558+0000", 2024-09-15T03:47:25.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7976000000000001e-05 2024-09-15T03:47:25.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151599+0000", 2024-09-15T03:47:25.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1152000000000002e-05 2024-09-15T03:47:25.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151617+0000", 2024-09-15T03:47:25.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7676000000000002e-05 2024-09-15T03:47:25.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142265+0000", 2024-09-15T03:47:25.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99064822900000005 2024-09-15T03:47:25.108 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.108 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.108 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.108 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=46, reqid=client.4521.0:242, at_version=39'18, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:25.108 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:25.108 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.849756081999999, 2024-09-15T03:47:25.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.991000204, 2024-09-15T03:47:25.108 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.108 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.109 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:25.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:25.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:25.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3399999999999999e-07 2024-09-15T03:47:25.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151393+0000", 2024-09-15T03:47:25.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8149000000000001e-05 2024-09-15T03:47:25.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151405+0000", 2024-09-15T03:47:25.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2063e-05 2024-09-15T03:47:25.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151415+0000", 2024-09-15T03:47:25.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4579999999999999e-06 2024-09-15T03:47:25.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151472+0000", 2024-09-15T03:47:25.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7018999999999997e-05 2024-09-15T03:47:25.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151480+0000", 2024-09-15T03:47:25.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1410000000000005e-06 2024-09-15T03:47:25.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151571+0000", 2024-09-15T03:47:25.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1243999999999996e-05 2024-09-15T03:47:25.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151602+0000", 2024-09-15T03:47:25.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0834000000000003e-05 2024-09-15T03:47:25.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151632+0000", 2024-09-15T03:47:25.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9527e-05 2024-09-15T03:47:25.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151685+0000", 2024-09-15T03:47:25.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3599e-05 2024-09-15T03:47:25.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142365+0000", 2024-09-15T03:47:25.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99067953600000003 2024-09-15T03:47:25.115 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.115 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.115 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.115 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 39/28 ECSubWrite(tid=51, reqid=osd.2.0:50, at_version=0'0, trim_to=0'0, roll_forward_to=39'28))", 2024-09-15T03:47:25.115 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:47:25.115 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.846392643999998, 2024-09-15T03:47:25.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98766307399999997, 2024-09-15T03:47:25.115 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.115 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.115 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:47:25.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:47:25.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154729+0000", 2024-09-15T03:47:25.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3799999999999996e-07 2024-09-15T03:47:25.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154742+0000", 2024-09-15T03:47:25.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2909e-05 2024-09-15T03:47:25.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154755+0000", 2024-09-15T03:47:25.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3251999999999999e-05 2024-09-15T03:47:25.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154761+0000", 2024-09-15T03:47:25.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6559999999999997e-06 2024-09-15T03:47:25.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154829+0000", 2024-09-15T03:47:25.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8549999999999999e-05 2024-09-15T03:47:25.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154840+0000", 2024-09-15T03:47:25.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0159999999999999e-05 2024-09-15T03:47:25.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154943+0000", 2024-09-15T03:47:25.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010347400000000001 2024-09-15T03:47:25.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154955+0000", 2024-09-15T03:47:25.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1935e-05 2024-09-15T03:47:25.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156442+0000", 2024-09-15T03:47:25.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014867859999999999 2024-09-15T03:47:25.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156466+0000", 2024-09-15T03:47:25.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4077999999999999e-05 2024-09-15T03:47:25.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142391+0000", 2024-09-15T03:47:25.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98592533599999999 2024-09-15T03:47:25.120 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.121 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.121 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.121 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=48, reqid=client.4521.0:242, at_version=0'0, trim_to=0'0, roll_forward_to=39'18))", 2024-09-15T03:47:25.121 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:47:25.121 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.844505624999996, 2024-09-15T03:47:25.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98592495099999999, 2024-09-15T03:47:25.121 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.121 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.121 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:47:25.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:47:25.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156616+0000", 2024-09-15T03:47:25.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1999999999999998e-07 2024-09-15T03:47:25.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156629+0000", 2024-09-15T03:47:25.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2870999999999999e-05 2024-09-15T03:47:25.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156649+0000", 2024-09-15T03:47:25.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9786e-05 2024-09-15T03:47:25.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156657+0000", 2024-09-15T03:47:25.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5110000000000003e-06 2024-09-15T03:47:25.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156723+0000", 2024-09-15T03:47:25.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6209000000000003e-05 2024-09-15T03:47:25.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156734+0000", 2024-09-15T03:47:25.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0475999999999999e-05 2024-09-15T03:47:25.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156868+0000", 2024-09-15T03:47:25.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013402999999999999 2024-09-15T03:47:25.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156889+0000", 2024-09-15T03:47:25.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1146e-05 2024-09-15T03:47:25.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156937+0000", 2024-09-15T03:47:25.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8031999999999997e-05 2024-09-15T03:47:25.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156970+0000", 2024-09-15T03:47:25.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2462999999999997e-05 2024-09-15T03:47:25.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142540+0000", 2024-09-15T03:47:25.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98557050700000004 2024-09-15T03:47:25.126 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.126 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.126 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.126 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.126 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.126 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:25.127 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:25.128 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:25.128 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:25.129 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:25.129 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:47:25.130 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:25.130 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:25.130 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:25.127+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:25.228 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:25.228+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:25.230 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:25.230 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:25.231 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:25.231 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:25.239 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:47:25.261 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:47:25.261 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:47:25.277 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:25.277 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:25.329 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:25.329+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:25.381 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:25.381 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:25.381 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:25.381 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:25.381 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:25.388 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:47:25.430 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:25.430+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:25.531 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:25.531 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:25.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:25.531 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:25.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.531 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=21, reqid=client.4521.0:184, at_version=33'6, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:25.531 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:25.531 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.422259521999997, 2024-09-15T03:47:25.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010901256999999999, 2024-09-15T03:47:25.531 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.532 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.532 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:25.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:25.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:25.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0200000000000001e-07 2024-09-15T03:47:25.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106529+0000", 2024-09-15T03:47:25.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9539e-05 2024-09-15T03:47:25.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106542+0000", 2024-09-15T03:47:25.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2731999999999999e-05 2024-09-15T03:47:25.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106549+0000", 2024-09-15T03:47:25.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3339999999999996e-06 2024-09-15T03:47:25.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106613+0000", 2024-09-15T03:47:25.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4527000000000001e-05 2024-09-15T03:47:25.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106622+0000", 2024-09-15T03:47:25.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.755e-06 2024-09-15T03:47:25.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106841+0000", 2024-09-15T03:47:25.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021906200000000001 2024-09-15T03:47:25.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106869+0000", 2024-09-15T03:47:25.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.739e-05 2024-09-15T03:47:25.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106948+0000", 2024-09-15T03:47:25.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9422e-05 2024-09-15T03:47:25.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106965+0000", 2024-09-15T03:47:25.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7010000000000001e-05 2024-09-15T03:47:25.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.117410+0000", 2024-09-15T03:47:25.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010445284000000001 2024-09-15T03:47:25.537 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.537 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.537 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.537 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:25.537 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:25.537 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.420346035000001, 2024-09-15T03:47:25.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015296944, 2024-09-15T03:47:25.537 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.537 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.538 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:25.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:25.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:25.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5499999999999998e-07 2024-09-15T03:47:25.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108447+0000", 2024-09-15T03:47:25.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3314000000000001e-05 2024-09-15T03:47:25.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108460+0000", 2024-09-15T03:47:25.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3444e-05 2024-09-15T03:47:25.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108464+0000", 2024-09-15T03:47:25.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3610000000000003e-06 2024-09-15T03:47:25.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108494+0000", 2024-09-15T03:47:25.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9688000000000001e-05 2024-09-15T03:47:25.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108502+0000", 2024-09-15T03:47:25.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.678e-06 2024-09-15T03:47:25.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108601+0000", 2024-09-15T03:47:25.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9748999999999997e-05 2024-09-15T03:47:25.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108610+0000", 2024-09-15T03:47:25.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2490000000000001e-06 2024-09-15T03:47:25.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109977+0000", 2024-09-15T03:47:25.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013669450000000001 2024-09-15T03:47:25.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109993+0000", 2024-09-15T03:47:25.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6688e-05 2024-09-15T03:47:25.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123720+0000", 2024-09-15T03:47:25.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013726172999999999 2024-09-15T03:47:25.543 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.543 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.543 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.543 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:25.543 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:25.544 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.420296374000003, 2024-09-15T03:47:25.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015105210000000001, 2024-09-15T03:47:25.544 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.544 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.544 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:25.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:25.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108473+0000", 2024-09-15T03:47:25.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.84e-07 2024-09-15T03:47:25.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108486+0000", 2024-09-15T03:47:25.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3845999999999999e-05 2024-09-15T03:47:25.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108491+0000", 2024-09-15T03:47:25.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4719999999999997e-06 2024-09-15T03:47:25.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108495+0000", 2024-09-15T03:47:25.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.822e-06 2024-09-15T03:47:25.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108546+0000", 2024-09-15T03:47:25.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0853000000000003e-05 2024-09-15T03:47:25.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108555+0000", 2024-09-15T03:47:25.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3319999999999998e-06 2024-09-15T03:47:25.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108638+0000", 2024-09-15T03:47:25.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3580999999999998e-05 2024-09-15T03:47:25.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109983+0000", 2024-09-15T03:47:25.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001344392 2024-09-15T03:47:25.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110023+0000", 2024-09-15T03:47:25.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9895999999999999e-05 2024-09-15T03:47:25.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110040+0000", 2024-09-15T03:47:25.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.747e-05 2024-09-15T03:47:25.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123577+0000", 2024-09-15T03:47:25.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013537262 2024-09-15T03:47:25.549 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.549 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.549 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.549 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:25.549 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:25.549 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.420262495000003, 2024-09-15T03:47:25.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015311313999999999, 2024-09-15T03:47:25.550 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.550 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.550 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:25.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:25.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:25.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8500000000000002e-07 2024-09-15T03:47:25.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108517+0000", 2024-09-15T03:47:25.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0543000000000001e-05 2024-09-15T03:47:25.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108524+0000", 2024-09-15T03:47:25.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7170000000000002e-06 2024-09-15T03:47:25.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108528+0000", 2024-09-15T03:47:25.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3970000000000004e-06 2024-09-15T03:47:25.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108632+0000", 2024-09-15T03:47:25.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103687 2024-09-15T03:47:25.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108637+0000", 2024-09-15T03:47:25.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4389999999999999e-06 2024-09-15T03:47:25.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108713+0000", 2024-09-15T03:47:25.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6223000000000005e-05 2024-09-15T03:47:25.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109984+0000", 2024-09-15T03:47:25.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012700719999999999 2024-09-15T03:47:25.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110023+0000", 2024-09-15T03:47:25.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9864999999999999e-05 2024-09-15T03:47:25.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110041+0000", 2024-09-15T03:47:25.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8040999999999999e-05 2024-09-15T03:47:25.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123817+0000", 2024-09-15T03:47:25.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013776045000000001 2024-09-15T03:47:25.556 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.556 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.556 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.556 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:25.556 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:25.556 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.419768886999996, 2024-09-15T03:47:25.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014864502, 2024-09-15T03:47:25.556 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.556 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.557 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:25.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:25.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:25.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:47:25.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109016+0000", 2024-09-15T03:47:25.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5889999999999999e-05 2024-09-15T03:47:25.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109029+0000", 2024-09-15T03:47:25.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2812e-05 2024-09-15T03:47:25.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109035+0000", 2024-09-15T03:47:25.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0109999999999997e-06 2024-09-15T03:47:25.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109073+0000", 2024-09-15T03:47:25.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8260000000000003e-05 2024-09-15T03:47:25.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109080+0000", 2024-09-15T03:47:25.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0190000000000001e-06 2024-09-15T03:47:25.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109165+0000", 2024-09-15T03:47:25.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4957000000000007e-05 2024-09-15T03:47:25.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109984+0000", 2024-09-15T03:47:25.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081866399999999998 2024-09-15T03:47:25.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110024+0000", 2024-09-15T03:47:25.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9644999999999997e-05 2024-09-15T03:47:25.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110072+0000", 2024-09-15T03:47:25.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8418999999999998e-05 2024-09-15T03:47:25.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123864+0000", 2024-09-15T03:47:25.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013792234 2024-09-15T03:47:25.562 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.562 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.562 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.562 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=27, reqid=client.4521.0:189, at_version=0'0, trim_to=0'0, roll_forward_to=33'10))", 2024-09-15T03:47:25.562 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.114624+0000", 2024-09-15T03:47:25.562 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.414144898000004, 2024-09-15T03:47:25.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092946850000000004, 2024-09-15T03:47:25.562 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.563 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.563 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114624+0000", 2024-09-15T03:47:25.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114624+0000", 2024-09-15T03:47:25.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114624+0000", 2024-09-15T03:47:25.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3600000000000004e-07 2024-09-15T03:47:25.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114633+0000", 2024-09-15T03:47:25.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1579999999999994e-06 2024-09-15T03:47:25.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114644+0000", 2024-09-15T03:47:25.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.04e-05 2024-09-15T03:47:25.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114649+0000", 2024-09-15T03:47:25.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8760000000000004e-06 2024-09-15T03:47:25.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114687+0000", 2024-09-15T03:47:25.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8438e-05 2024-09-15T03:47:25.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114697+0000", 2024-09-15T03:47:25.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6479999999999998e-06 2024-09-15T03:47:25.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114818+0000", 2024-09-15T03:47:25.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121116 2024-09-15T03:47:25.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114830+0000", 2024-09-15T03:47:25.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1844e-05 2024-09-15T03:47:25.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114921+0000", 2024-09-15T03:47:25.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1461999999999999e-05 2024-09-15T03:47:25.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114934+0000", 2024-09-15T03:47:25.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2857000000000001e-05 2024-09-15T03:47:25.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123918+0000", 2024-09-15T03:47:25.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089843500000000003 2024-09-15T03:47:25.569 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.569 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.569 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.570 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=34, reqid=client.4521.0:197, at_version=34'21, trim_to=0'0, roll_forward_to=34'18))", 2024-09-15T03:47:25.570 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:25.570 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.404757007000001, 2024-09-15T03:47:25.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012992399, 2024-09-15T03:47:25.570 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.570 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.570 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:25.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:25.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124013+0000", 2024-09-15T03:47:25.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6299999999999993e-07 2024-09-15T03:47:25.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124044+0000", 2024-09-15T03:47:25.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0995999999999999e-05 2024-09-15T03:47:25.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124062+0000", 2024-09-15T03:47:25.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8413000000000001e-05 2024-09-15T03:47:25.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124069+0000", 2024-09-15T03:47:25.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3930000000000002e-06 2024-09-15T03:47:25.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124150+0000", 2024-09-15T03:47:25.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1097e-05 2024-09-15T03:47:25.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124161+0000", 2024-09-15T03:47:25.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0720000000000001e-05 2024-09-15T03:47:25.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124402+0000", 2024-09-15T03:47:25.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000240442 2024-09-15T03:47:25.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125141+0000", 2024-09-15T03:47:25.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00073915500000000002 2024-09-15T03:47:25.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125191+0000", 2024-09-15T03:47:25.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9951999999999999e-05 2024-09-15T03:47:25.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125251+0000", 2024-09-15T03:47:25.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9938000000000001e-05 2024-09-15T03:47:25.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.137004+0000", 2024-09-15T03:47:25.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011753329999999999 2024-09-15T03:47:25.575 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.575 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.575 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.576 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=35, reqid=client.4521.0:197, at_version=0'0, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:47:25.576 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:25.576 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.402816962999999, 2024-09-15T03:47:25.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011202456, 2024-09-15T03:47:25.576 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.576 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.576 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:25.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:25.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125953+0000", 2024-09-15T03:47:25.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9100000000000002e-07 2024-09-15T03:47:25.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125964+0000", 2024-09-15T03:47:25.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1399999999999999e-05 2024-09-15T03:47:25.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125979+0000", 2024-09-15T03:47:25.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4810000000000001e-05 2024-09-15T03:47:25.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126004+0000", 2024-09-15T03:47:25.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4913e-05 2024-09-15T03:47:25.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126059+0000", 2024-09-15T03:47:25.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5668999999999998e-05 2024-09-15T03:47:25.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126071+0000", 2024-09-15T03:47:25.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1219999999999999e-05 2024-09-15T03:47:25.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126151+0000", 2024-09-15T03:47:25.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0572e-05 2024-09-15T03:47:25.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126163+0000", 2024-09-15T03:47:25.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2101e-05 2024-09-15T03:47:25.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.127945+0000", 2024-09-15T03:47:25.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001781803 2024-09-15T03:47:25.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.127982+0000", 2024-09-15T03:47:25.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7129999999999999e-05 2024-09-15T03:47:25.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.137154+0000", 2024-09-15T03:47:25.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0091719469999999997 2024-09-15T03:47:25.582 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.582 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.582 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.583 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:243 3.3s0 3:c7c3ef16:::smithi00132172-25:head [create,setxattr iZnWfebab`ptjURR\\f`psW]tpS`qpUaeWkY[Oqep (1330) in=40b,setxattr f[mZ_qQmcZejQ`fbdeic_tkPUaTYcnm[WprjVmc] (12488) in=40b,setxattr tkZgQ]S_pdlhZnqrVaZRa`oQQr_USmUfhado^dXT (1121) in=40b,setxattr srWjTqXhcth`hfZmQY_fg]UqNjWofa^dfgOV`sYP (455) in=40b,setxattr ^_mdiRroj_TtQ^gcNj`XtkW[ooqY`PWXqYmcOk_n (138) in=40b,setxattr jQlajdtcalhkcfYaakZfNb[^_TjkfcS`Vrk]q_d[ (806) in=40b,setxattr ZfYgkVcrsXcnpchgfcbldNrUTasnQNmfZW]Nbh\\o (561) in=40b,setxattr _hkYOqjPNhmhUlSNktfeQpsqSWjO\\fbf[fpTlZiO (1227) in=40b,setxattr ]mdV^Xdc\\sViXdkXNU[js_Qhh[b[sbfnZVXkdids (11788) in=40b,setxattr tb^mbOrNhfrhtce^g (455) in=17b,setxattr koTkVgrfU\\m`StnZ`Rirjhbhg`PhVqm^a^Xgaaqn (35) in=40b,setxattr aUtZPUPXPnQmT]\\\\n\\lZpnXakdNkTldVQoQhPUf] (559) in=40b,setxattr mrhR]^U`Tteo[Topmql`SNma\\pqfk`if]OppnYt` (770) in=40b,setxattr Oetk]RrnWeXbVctVtVrhXf_RfTdelqX^lWp[sTkp (1226) in=40b,setxattr WtedXoOd`VqnbksOqQkatk`gdTnW^lXbUtrlYgkl (3576) in=40b,setxattr dddOWN\\j\\XXg^fr[rZcskeVX]jOQ^ZmoWepk[]\\a (17824) in=40b,setxattr TeRO^djPgNZiemQ_aRXgP[]jgUZa\\^\\iUgesaqaN (208) in=40b,setxattr teqm]mUobZbWjcdQ_jTa[f[k[Sd\\lh]n\\osigqkN (7436) in=40b,setxattr dphaol[XkeitbkomWYaQctWfataUhbSiNntkWo`P (1893) in=40b,setxattr oQsnhO]ZklSVYQ[WUpaelY[p[Qie^UepOt`Z[q (455) in=38b] snapc 7=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:47:25.583 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:25.583 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.359853074999997, 2024-09-15T03:47:25.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012365956000000001, 2024-09-15T03:47:25.583 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.583 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:25.583 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:25.583 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:25.583 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:25.583 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 243 2024-09-15T03:47:25.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.583 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:25.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:25.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168917+0000", 2024-09-15T03:47:25.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.725e-06 2024-09-15T03:47:25.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169030+0000", 2024-09-15T03:47:25.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113035 2024-09-15T03:47:25.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169047+0000", 2024-09-15T03:47:25.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6701000000000002e-05 2024-09-15T03:47:25.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169054+0000", 2024-09-15T03:47:25.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1369999999999996e-06 2024-09-15T03:47:25.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169140+0000", 2024-09-15T03:47:25.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6069999999999994e-05 2024-09-15T03:47:25.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:25.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169270+0000", 2024-09-15T03:47:25.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130071 2024-09-15T03:47:25.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.170661+0000", 2024-09-15T03:47:25.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001390666 2024-09-15T03:47:25.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171751+0000", 2024-09-15T03:47:25.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010899079999999999 2024-09-15T03:47:25.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171812+0000", 2024-09-15T03:47:25.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0797000000000001e-05 2024-09-15T03:47:25.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172279+0000", 2024-09-15T03:47:25.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046704500000000003 2024-09-15T03:47:25.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172299+0000", 2024-09-15T03:47:25.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0387999999999999e-05 2024-09-15T03:47:25.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:25.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.176110+0000", 2024-09-15T03:47:25.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038112340000000001 2024-09-15T03:47:25.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.181282+0000", 2024-09-15T03:47:25.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051711789999999997 2024-09-15T03:47:25.590 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.591 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.591 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.591 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:258 3.3s0 3:c9b663a2:::smithi00132172-7:head [delete] snapc 7=[6,5,3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e40)", 2024-09-15T03:47:25.591 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:47.146116+0000", 2024-09-15T03:47:25.592 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.382652256999997, 2024-09-15T03:47:25.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096428380000000008, 2024-09-15T03:47:25.592 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.592 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:25.592 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:25.592 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:25.592 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:25.592 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 258 2024-09-15T03:47:25.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.592 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146116+0000", 2024-09-15T03:47:25.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146116+0000", 2024-09-15T03:47:25.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146119+0000", 2024-09-15T03:47:25.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.227e-06 2024-09-15T03:47:25.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146127+0000", 2024-09-15T03:47:25.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1529999999999997e-06 2024-09-15T03:47:25.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146146+0000", 2024-09-15T03:47:25.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9318999999999999e-05 2024-09-15T03:47:25.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146154+0000", 2024-09-15T03:47:25.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7200000000000006e-06 2024-09-15T03:47:25.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146233+0000", 2024-09-15T03:47:25.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8725000000000003e-05 2024-09-15T03:47:25.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:25.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146244+0000", 2024-09-15T03:47:25.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1725e-05 2024-09-15T03:47:25.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.149207+0000", 2024-09-15T03:47:25.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029627999999999998 2024-09-15T03:47:25.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:25.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.149326+0000", 2024-09-15T03:47:25.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011869199999999999 2024-09-15T03:47:25.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.150266+0000", 2024-09-15T03:47:25.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00094048500000000002 2024-09-15T03:47:25.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.150626+0000", 2024-09-15T03:47:25.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035946699999999998 2024-09-15T03:47:25.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.150751+0000", 2024-09-15T03:47:25.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012544800000000001 2024-09-15T03:47:25.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.151656+0000", 2024-09-15T03:47:25.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00090447499999999996 2024-09-15T03:47:25.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.153489+0000", 2024-09-15T03:47:25.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001833115 2024-09-15T03:47:25.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:25.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.155171+0000", 2024-09-15T03:47:25.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001682455 2024-09-15T03:47:25.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.155759+0000", 2024-09-15T03:47:25.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00058803200000000005 2024-09-15T03:47:25.600 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.600 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.600 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.600 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/28 ECSubWrite(tid=15, reqid=client.4521.0:337, at_version=49'10, trim_to=0'0, roll_forward_to=43'8))", 2024-09-15T03:47:25.600 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:25.600 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 29.670863166, 2024-09-15T03:47:25.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010500893000000001, 2024-09-15T03:47:25.600 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.600 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.600 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:25.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:25.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857906+0000", 2024-09-15T03:47:25.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.63e-07 2024-09-15T03:47:25.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857928+0000", 2024-09-15T03:47:25.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2155999999999999e-05 2024-09-15T03:47:25.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857937+0000", 2024-09-15T03:47:25.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2359999999999993e-06 2024-09-15T03:47:25.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857941+0000", 2024-09-15T03:47:25.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0960000000000003e-06 2024-09-15T03:47:25.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857999+0000", 2024-09-15T03:47:25.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7407000000000001e-05 2024-09-15T03:47:25.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858011+0000", 2024-09-15T03:47:25.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1759999999999999e-05 2024-09-15T03:47:25.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858240+0000", 2024-09-15T03:47:25.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022949299999999999 2024-09-15T03:47:25.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858251+0000", 2024-09-15T03:47:25.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1168999999999999e-05 2024-09-15T03:47:25.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858332+0000", 2024-09-15T03:47:25.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1204000000000004e-05 2024-09-15T03:47:25.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858360+0000", 2024-09-15T03:47:25.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7767999999999999e-05 2024-09-15T03:47:25.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.868406+0000", 2024-09-15T03:47:25.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010046141 2024-09-15T03:47:25.606 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.606 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.606 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.606 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/28 ECSubWrite(tid=16, reqid=client.4521.0:337, at_version=0'0, trim_to=0'0, roll_forward_to=49'10))", 2024-09-15T03:47:25.606 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:47:25.606 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 29.669507138, 2024-09-15T03:47:25.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093024639999999999, 2024-09-15T03:47:25.606 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.607 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.607 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:47:25.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:47:25.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859263+0000", 2024-09-15T03:47:25.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1969999999999999e-06 2024-09-15T03:47:25.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859275+0000", 2024-09-15T03:47:25.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2153999999999999e-05 2024-09-15T03:47:25.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859294+0000", 2024-09-15T03:47:25.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8875000000000001e-05 2024-09-15T03:47:25.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859301+0000", 2024-09-15T03:47:25.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9360000000000002e-06 2024-09-15T03:47:25.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859397+0000", 2024-09-15T03:47:25.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6831999999999997e-05 2024-09-15T03:47:25.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859407+0000", 2024-09-15T03:47:25.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0939999999999997e-06 2024-09-15T03:47:25.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859500+0000", 2024-09-15T03:47:25.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3338000000000007e-05 2024-09-15T03:47:25.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859528+0000", 2024-09-15T03:47:25.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8039999999999999e-05 2024-09-15T03:47:25.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859568+0000", 2024-09-15T03:47:25.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0221e-05 2024-09-15T03:47:25.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859583+0000", 2024-09-15T03:47:25.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5217e-05 2024-09-15T03:47:25.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.868564+0000", 2024-09-15T03:47:25.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089805600000000003 2024-09-15T03:47:25.612 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.612 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.612 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.612 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:349 3.3s0 3:c7c3ef16:::smithi00132172-25:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:25.612 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:25.612 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.690287139999999, 2024-09-15T03:47:25.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005145406, 2024-09-15T03:47:25.613 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.613 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:25.613 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:25.613 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:25.613 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:25.613 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 349 2024-09-15T03:47:25.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.613 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:25.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:25.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838485+0000", 2024-09-15T03:47:25.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0620000000000001e-06 2024-09-15T03:47:25.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838494+0000", 2024-09-15T03:47:25.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3179999999999996e-06 2024-09-15T03:47:25.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:25.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3689999999999998e-05 2024-09-15T03:47:25.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838812+0000", 2024-09-15T03:47:25.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029437300000000001 2024-09-15T03:47:25.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838905+0000", 2024-09-15T03:47:25.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3196999999999996e-05 2024-09-15T03:47:25.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:25.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838914+0000", 2024-09-15T03:47:25.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0890000000000006e-06 2024-09-15T03:47:25.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840773+0000", 2024-09-15T03:47:25.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018584820000000001 2024-09-15T03:47:25.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:25.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840889+0000", 2024-09-15T03:47:25.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115986 2024-09-15T03:47:25.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842016+0000", 2024-09-15T03:47:25.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001126878 2024-09-15T03:47:25.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842320+0000", 2024-09-15T03:47:25.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030409799999999999 2024-09-15T03:47:25.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842360+0000", 2024-09-15T03:47:25.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0750000000000001e-05 2024-09-15T03:47:25.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842448+0000", 2024-09-15T03:47:25.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7842000000000004e-05 2024-09-15T03:47:25.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844580+0000", 2024-09-15T03:47:25.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021313719999999999 2024-09-15T03:47:25.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:25.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843574+0000", 2024-09-15T03:47:25.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99899468199999997 2024-09-15T03:47:25.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843627+0000", 2024-09-15T03:47:25.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2587000000000003e-05 2024-09-15T03:47:25.619 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.619 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.620 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.620 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:350 3.3s0 3:c7c3ef16:::smithi00132172-25:head [rollback 14] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:25.620 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:25.620 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.689935478999999, 2024-09-15T03:47:25.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0050383839999999, 2024-09-15T03:47:25.620 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.620 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:25.620 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:25.620 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:25.620 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:25.620 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 350 2024-09-15T03:47:25.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.621 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:25.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:25.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838834+0000", 2024-09-15T03:47:25.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0200000000000001e-07 2024-09-15T03:47:25.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838839+0000", 2024-09-15T03:47:25.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9790000000000001e-06 2024-09-15T03:47:25.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838852+0000", 2024-09-15T03:47:25.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3138e-05 2024-09-15T03:47:25.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838858+0000", 2024-09-15T03:47:25.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8320000000000002e-06 2024-09-15T03:47:25.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839003+0000", 2024-09-15T03:47:25.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014513500000000001 2024-09-15T03:47:25.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:25.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839011+0000", 2024-09-15T03:47:25.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7719999999999994e-06 2024-09-15T03:47:25.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842395+0000", 2024-09-15T03:47:25.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033840110000000001 2024-09-15T03:47:25.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:25.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842580+0000", 2024-09-15T03:47:25.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000185067 2024-09-15T03:47:25.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843394+0000", 2024-09-15T03:47:25.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081377299999999995 2024-09-15T03:47:25.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843510+0000", 2024-09-15T03:47:25.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0001165 2024-09-15T03:47:25.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843524+0000", 2024-09-15T03:47:25.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4022999999999999e-05 2024-09-15T03:47:25.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843579+0000", 2024-09-15T03:47:25.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4654999999999998e-05 2024-09-15T03:47:25.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844583+0000", 2024-09-15T03:47:25.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010042429999999999 2024-09-15T03:47:25.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:25.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843772+0000", 2024-09-15T03:47:25.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99918911200000005 2024-09-15T03:47:25.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843872+0000", 2024-09-15T03:47:25.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9341999999999999e-05 2024-09-15T03:47:25.627 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.627 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.627 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.627 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:351 3.3s0 3:c7c3ef16:::smithi00132172-25:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:25.627 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:25.628 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.689877846000002, 2024-09-15T03:47:25.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0051167679999999, 2024-09-15T03:47:25.628 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.628 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:25.628 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:25.628 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:25.628 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:25.628 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 351 2024-09-15T03:47:25.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.628 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:25.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:25.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:25.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0800000000000004e-07 2024-09-15T03:47:25.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838895+0000", 2024-09-15T03:47:25.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.241e-06 2024-09-15T03:47:25.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838904+0000", 2024-09-15T03:47:25.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9549999999999998e-06 2024-09-15T03:47:25.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838909+0000", 2024-09-15T03:47:25.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7869999999999998e-06 2024-09-15T03:47:25.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839045+0000", 2024-09-15T03:47:25.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013556399999999999 2024-09-15T03:47:25.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:25.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839052+0000", 2024-09-15T03:47:25.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8199999999999999e-06 2024-09-15T03:47:25.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843587+0000", 2024-09-15T03:47:25.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045349739999999998 2024-09-15T03:47:25.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:25.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843708+0000", 2024-09-15T03:47:25.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120776 2024-09-15T03:47:25.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844246+0000", 2024-09-15T03:47:25.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053838200000000001 2024-09-15T03:47:25.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844335+0000", 2024-09-15T03:47:25.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9240999999999995e-05 2024-09-15T03:47:25.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844347+0000", 2024-09-15T03:47:25.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2187e-05 2024-09-15T03:47:25.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844460+0000", 2024-09-15T03:47:25.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112571 2024-09-15T03:47:25.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844584+0000", 2024-09-15T03:47:25.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012403800000000001 2024-09-15T03:47:25.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:25.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843972+0000", 2024-09-15T03:47:25.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99938787299999998 2024-09-15T03:47:25.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844008+0000", 2024-09-15T03:47:25.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5750999999999997e-05 2024-09-15T03:47:25.635 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.635 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.635 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.635 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:25.635 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:25.635 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.682743895000002, 2024-09-15T03:47:25.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013010173, 2024-09-15T03:47:25.636 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.636 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.636 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:25.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:25.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846026+0000", 2024-09-15T03:47:25.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1230000000000001e-06 2024-09-15T03:47:25.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846047+0000", 2024-09-15T03:47:25.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0774000000000001e-05 2024-09-15T03:47:25.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846064+0000", 2024-09-15T03:47:25.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7560000000000001e-05 2024-09-15T03:47:25.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846072+0000", 2024-09-15T03:47:25.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5020000000000003e-06 2024-09-15T03:47:25.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846187+0000", 2024-09-15T03:47:25.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115717 2024-09-15T03:47:25.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846201+0000", 2024-09-15T03:47:25.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3329999999999999e-05 2024-09-15T03:47:25.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846303+0000", 2024-09-15T03:47:25.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101807 2024-09-15T03:47:25.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846312+0000", 2024-09-15T03:47:25.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5529999999999998e-06 2024-09-15T03:47:25.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846382+0000", 2024-09-15T03:47:25.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0146000000000002e-05 2024-09-15T03:47:25.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846520+0000", 2024-09-15T03:47:25.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013814099999999999 2024-09-15T03:47:25.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.859035+0000", 2024-09-15T03:47:25.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012514519999999999 2024-09-15T03:47:25.641 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.641 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.641 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.641 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:25.641 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:25.641 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.682647148000001, 2024-09-15T03:47:25.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013044694000000001, 2024-09-15T03:47:25.641 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.642 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.642 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:25.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:25.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846122+0000", 2024-09-15T03:47:25.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0399999999999996e-07 2024-09-15T03:47:25.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846174+0000", 2024-09-15T03:47:25.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1672999999999998e-05 2024-09-15T03:47:25.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846184+0000", 2024-09-15T03:47:25.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0514e-05 2024-09-15T03:47:25.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846190+0000", 2024-09-15T03:47:25.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2820000000000002e-06 2024-09-15T03:47:25.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846341+0000", 2024-09-15T03:47:25.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150803 2024-09-15T03:47:25.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846351+0000", 2024-09-15T03:47:25.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3899999999999999e-06 2024-09-15T03:47:25.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846450+0000", 2024-09-15T03:47:25.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9839000000000004e-05 2024-09-15T03:47:25.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846482+0000", 2024-09-15T03:47:25.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1089e-05 2024-09-15T03:47:25.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846615+0000", 2024-09-15T03:47:25.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013305899999999999 2024-09-15T03:47:25.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846634+0000", 2024-09-15T03:47:25.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8899999999999999e-05 2024-09-15T03:47:25.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.859166+0000", 2024-09-15T03:47:25.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012532641000000001 2024-09-15T03:47:25.647 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.647 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.647 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.647 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 60/31 ECSubWrite(tid=70, reqid=client.4521.0:421, at_version=60'22, trim_to=0'0, roll_forward_to=47'20))", 2024-09-15T03:47:25.648 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:06.837638+0000", 2024-09-15T03:47:25.648 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.691131074000001, 2024-09-15T03:47:25.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092785220000000009, 2024-09-15T03:47:25.648 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.648 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.648 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837638+0000", 2024-09-15T03:47:25.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837638+0000", 2024-09-15T03:47:25.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837638+0000", 2024-09-15T03:47:25.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6399999999999998e-07 2024-09-15T03:47:25.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837664+0000", 2024-09-15T03:47:25.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5788000000000001e-05 2024-09-15T03:47:25.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837675+0000", 2024-09-15T03:47:25.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1269e-05 2024-09-15T03:47:25.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837682+0000", 2024-09-15T03:47:25.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5120000000000002e-06 2024-09-15T03:47:25.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837771+0000", 2024-09-15T03:47:25.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8802999999999997e-05 2024-09-15T03:47:25.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837793+0000", 2024-09-15T03:47:25.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2141e-05 2024-09-15T03:47:25.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.838114+0000", 2024-09-15T03:47:25.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000320453 2024-09-15T03:47:25.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.838243+0000", 2024-09-15T03:47:25.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012918599999999999 2024-09-15T03:47:25.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.838298+0000", 2024-09-15T03:47:25.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5164999999999998e-05 2024-09-15T03:47:25.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.838665+0000", 2024-09-15T03:47:25.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036705199999999998 2024-09-15T03:47:25.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.846916+0000", 2024-09-15T03:47:25.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082508890000000008 2024-09-15T03:47:25.653 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.653 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.653 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.653 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 60/28 ECSubWrite(tid=73, reqid=client.4521.0:424, at_version=60'41, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:25.654 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:25.654 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.628338300999999, 2024-09-15T03:47:25.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011124821, 2024-09-15T03:47:25.654 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.654 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.654 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:25.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:25.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900431+0000", 2024-09-15T03:47:25.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2900000000000003e-07 2024-09-15T03:47:25.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900451+0000", 2024-09-15T03:47:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0066999999999999e-05 2024-09-15T03:47:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900466+0000", 2024-09-15T03:47:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5062000000000001e-05 2024-09-15T03:47:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900474+0000", 2024-09-15T03:47:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6310000000000003e-06 2024-09-15T03:47:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900524+0000", 2024-09-15T03:47:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0037000000000002e-05 2024-09-15T03:47:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900535+0000", 2024-09-15T03:47:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1209000000000001e-05 2024-09-15T03:47:25.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900659+0000", 2024-09-15T03:47:25.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012441000000000001 2024-09-15T03:47:25.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900971+0000", 2024-09-15T03:47:25.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031183799999999999 2024-09-15T03:47:25.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901144+0000", 2024-09-15T03:47:25.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172261 2024-09-15T03:47:25.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901734+0000", 2024-09-15T03:47:25.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059069700000000005 2024-09-15T03:47:25.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.911555+0000", 2024-09-15T03:47:25.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0098208800000000006 2024-09-15T03:47:25.660 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.660 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.660 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.660 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 60/28 ECSubWrite(tid=74, reqid=client.4521.0:425, at_version=60'42, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:25.660 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:06.901240+0000", 2024-09-15T03:47:25.660 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.627528722000001, 2024-09-15T03:47:25.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010429678, 2024-09-15T03:47:25.660 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:25.661 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:25.661 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:25.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:25.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901240+0000", 2024-09-15T03:47:25.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:25.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901240+0000", 2024-09-15T03:47:25.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:25.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:25.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901241+0000", 2024-09-15T03:47:25.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7999999999999993e-07 2024-09-15T03:47:25.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:25.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901273+0000", 2024-09-15T03:47:25.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2087e-05 2024-09-15T03:47:25.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:25.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901302+0000", 2024-09-15T03:47:25.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8705e-05 2024-09-15T03:47:25.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:25.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901311+0000", 2024-09-15T03:47:25.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9190000000000005e-06 2024-09-15T03:47:25.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:25.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901797+0000", 2024-09-15T03:47:25.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048606499999999999 2024-09-15T03:47:25.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:25.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901808+0000", 2024-09-15T03:47:25.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1075e-05 2024-09-15T03:47:25.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:25.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901934+0000", 2024-09-15T03:47:25.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012674299999999999 2024-09-15T03:47:25.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:25.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901964+0000", 2024-09-15T03:47:25.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9833999999999999e-05 2024-09-15T03:47:25.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:25.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.902006+0000", 2024-09-15T03:47:25.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1823999999999998e-05 2024-09-15T03:47:25.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:25.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.902020+0000", 2024-09-15T03:47:25.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.363e-05 2024-09-15T03:47:25.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:25.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:25.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:25.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.911670+0000", 2024-09-15T03:47:25.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096498160000000003 2024-09-15T03:47:25.666 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.666 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.666 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.666 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:25.666 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:25.666 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:25.668 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:47:25.668 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:25.667+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:25.768 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:25.768+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:25.769 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:25.769 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:25.769 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:25.769 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:25.777 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:47:25.869 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:25.868+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:25.919 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:25.919 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:25.919 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:25.919 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:25.919 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:25.927 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:47:25.956 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-15T03:47:25.956 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T03:47:25.956 INFO:tasks.thrashosds.thrasher:Killing osd 10, live_osds are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-15T03:47:25.957 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-09-15T03:47:25.957 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:47:25.958 INFO:tasks.rados.rados.0.smithi001.stdout:243: done (3 left) 2024-09-15T03:47:25.958 INFO:tasks.rados.rados.0.smithi001.stdout:244: done (2 left) 2024-09-15T03:47:25.958 INFO:tasks.rados.rados.0.smithi001.stdout:245: done (1 left) 2024-09-15T03:47:25.959 INFO:tasks.rados.rados.0.smithi001.stdout:246: done (0 left) 2024-09-15T03:47:25.959 INFO:tasks.rados.rados.0.smithi001.stdout:247: rollback oid 21 current snap is 31 2024-09-15T03:47:25.959 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 21 to 11 2024-09-15T03:47:25.959 INFO:tasks.rados.rados.0.smithi001.stdout:248: snap_remove snap 15 2024-09-15T03:47:25.960 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T03:47:25.974 INFO:tasks.ceph.osd.10:Stopped 2024-09-15T03:47:25.974 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.10 2024-09-15T03:47:25.975 INFO:tasks.thrashosds.thrasher:remote for osd 10 is ubuntu@smithi190.front.sepia.ceph.com 2024-09-15T03:47:25.975 DEBUG:teuthology.orchestra.run.smithi190:> type ceph-objectstore-tool 2024-09-15T03:47:26.005 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:26.004+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:26.032 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-15T03:47:26.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --op list-pgs 2024-09-15T03:47:26.071 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:26.071 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:26.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:26.071 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:26.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.071 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs3 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:26.071 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108456+0000", 2024-09-15T03:47:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.960390373000003, 2024-09-15T03:47:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015075257, 2024-09-15T03:47:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108456+0000", 2024-09-15T03:47:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108456+0000", 2024-09-15T03:47:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108457+0000", 2024-09-15T03:47:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9999999999999997e-07 2024-09-15T03:47:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108477+0000", 2024-09-15T03:47:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0344e-05 2024-09-15T03:47:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108497+0000", 2024-09-15T03:47:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0143999999999999e-05 2024-09-15T03:47:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108504+0000", 2024-09-15T03:47:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8499999999999996e-06 2024-09-15T03:47:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108554+0000", 2024-09-15T03:47:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9567999999999997e-05 2024-09-15T03:47:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108568+0000", 2024-09-15T03:47:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4326e-05 2024-09-15T03:47:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108654+0000", 2024-09-15T03:47:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6193999999999995e-05 2024-09-15T03:47:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108663+0000", 2024-09-15T03:47:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7479999999999999e-06 2024-09-15T03:47:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111234+0000", 2024-09-15T03:47:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025709079999999998 2024-09-15T03:47:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111259+0000", 2024-09-15T03:47:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4797999999999999e-05 2024-09-15T03:47:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123531+0000", 2024-09-15T03:47:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012272777 2024-09-15T03:47:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.078 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as2 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:26.078 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:47:26.078 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.960161718999998, 2024-09-15T03:47:26.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0149108, 2024-09-15T03:47:26.078 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:26.078 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:26.078 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:26.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:47:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:47:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108685+0000", 2024-09-15T03:47:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1699999999999999e-07 2024-09-15T03:47:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108710+0000", 2024-09-15T03:47:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5032999999999999e-05 2024-09-15T03:47:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108725+0000", 2024-09-15T03:47:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4788e-05 2024-09-15T03:47:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108733+0000", 2024-09-15T03:47:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4610000000000008e-06 2024-09-15T03:47:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108778+0000", 2024-09-15T03:47:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4224999999999998e-05 2024-09-15T03:47:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108800+0000", 2024-09-15T03:47:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2189999999999999e-05 2024-09-15T03:47:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108884+0000", 2024-09-15T03:47:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4569999999999998e-05 2024-09-15T03:47:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111237+0000", 2024-09-15T03:47:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002352313 2024-09-15T03:47:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114139+0000", 2024-09-15T03:47:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002902057 2024-09-15T03:47:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114173+0000", 2024-09-15T03:47:26.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4461000000000001e-05 2024-09-15T03:47:26.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:26.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123596+0000", 2024-09-15T03:47:26.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094222850000000007 2024-09-15T03:47:26.084 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.084 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:26.084 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.084 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs3 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:26.085 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:47:26.085 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.960102440999997, 2024-09-15T03:47:26.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014886043, 2024-09-15T03:47:26.085 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:26.085 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:26.085 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:26.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:26.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:47:26.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:26.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:47:26.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:26.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108744+0000", 2024-09-15T03:47:26.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5100000000000001e-07 2024-09-15T03:47:26.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:26.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108759+0000", 2024-09-15T03:47:26.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4945e-05 2024-09-15T03:47:26.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:26.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108768+0000", 2024-09-15T03:47:26.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2309999999999996e-06 2024-09-15T03:47:26.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108795+0000", 2024-09-15T03:47:26.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7829000000000001e-05 2024-09-15T03:47:26.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108841+0000", 2024-09-15T03:47:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5330999999999997e-05 2024-09-15T03:47:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108850+0000", 2024-09-15T03:47:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9959999999999999e-06 2024-09-15T03:47:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108940+0000", 2024-09-15T03:47:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9823999999999999e-05 2024-09-15T03:47:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111237+0000", 2024-09-15T03:47:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002297927 2024-09-15T03:47:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114139+0000", 2024-09-15T03:47:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029020029999999998 2024-09-15T03:47:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114212+0000", 2024-09-15T03:47:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2177999999999999e-05 2024-09-15T03:47:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123630+0000", 2024-09-15T03:47:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094184279999999995 2024-09-15T03:47:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as2 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.109397+0000", 2024-09-15T03:47:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.959449382999999, 2024-09-15T03:47:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014322056, 2024-09-15T03:47:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109397+0000", 2024-09-15T03:47:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109397+0000", 2024-09-15T03:47:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109398+0000", 2024-09-15T03:47:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1600000000000001e-07 2024-09-15T03:47:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:26.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109410+0000", 2024-09-15T03:47:26.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2294e-05 2024-09-15T03:47:26.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:26.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109420+0000", 2024-09-15T03:47:26.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.764e-06 2024-09-15T03:47:26.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109426+0000", 2024-09-15T03:47:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9309999999999996e-06 2024-09-15T03:47:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109457+0000", 2024-09-15T03:47:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1047000000000003e-05 2024-09-15T03:47:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109462+0000", 2024-09-15T03:47:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1499999999999998e-06 2024-09-15T03:47:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109513+0000", 2024-09-15T03:47:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0722e-05 2024-09-15T03:47:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.111238+0000", 2024-09-15T03:47:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017253380000000001 2024-09-15T03:47:26.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114140+0000", 2024-09-15T03:47:26.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029017370000000002 2024-09-15T03:47:26.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114201+0000", 2024-09-15T03:47:26.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1047e-05 2024-09-15T03:47:26.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:26.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123719+0000", 2024-09-15T03:47:26.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095184099999999997 2024-09-15T03:47:26.097 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:197 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [delete] snapc 3=[3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e34)", 2024-09-15T03:47:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.956581206999999, 2024-09-15T03:47:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021934704999999999, 2024-09-15T03:47:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197 2024-09-15T03:47:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112266+0000", 2024-09-15T03:47:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5000000000000002e-07 2024-09-15T03:47:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112269+0000", 2024-09-15T03:47:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.066e-06 2024-09-15T03:47:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112276+0000", 2024-09-15T03:47:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-09-15T03:47:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112281+0000", 2024-09-15T03:47:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3970000000000004e-06 2024-09-15T03:47:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123281+0000", 2024-09-15T03:47:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011000575 2024-09-15T03:47:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123395+0000", 2024-09-15T03:47:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113398 2024-09-15T03:47:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123963+0000", 2024-09-15T03:47:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000568101 2024-09-15T03:47:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124220+0000", 2024-09-15T03:47:26.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025704900000000003 2024-09-15T03:47:26.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124229+0000", 2024-09-15T03:47:26.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4860000000000003e-06 2024-09-15T03:47:26.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124277+0000", 2024-09-15T03:47:26.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7574000000000002e-05 2024-09-15T03:47:26.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124312+0000", 2024-09-15T03:47:26.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5256000000000003e-05 2024-09-15T03:47:26.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:26.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125716+0000", 2024-09-15T03:47:26.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014043720000000001 2024-09-15T03:47:26.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:26.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.134200+0000", 2024-09-15T03:47:26.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084835120000000003 2024-09-15T03:47:26.107 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.107 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:26.107 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.107 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es4 34/28 ECSubWrite(tid=33, reqid=client.4521.0:195, at_version=34'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-15T03:47:26.107 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:47:26.107 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.950753847999998, 2024-09-15T03:47:26.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015907482000000001, 2024-09-15T03:47:26.107 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:26.107 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:26.108 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:26.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:26.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:47:26.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:26.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:47:26.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:26.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:47:26.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9400000000000004e-07 2024-09-15T03:47:26.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:26.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118299+0000", 2024-09-15T03:47:26.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000205677 2024-09-15T03:47:26.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:26.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118315+0000", 2024-09-15T03:47:26.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5429000000000001e-05 2024-09-15T03:47:26.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118322+0000", 2024-09-15T03:47:26.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8290000000000008e-06 2024-09-15T03:47:26.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118416+0000", 2024-09-15T03:47:26.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3140000000000006e-05 2024-09-15T03:47:26.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118431+0000", 2024-09-15T03:47:26.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5648e-05 2024-09-15T03:47:26.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118678+0000", 2024-09-15T03:47:26.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024701200000000001 2024-09-15T03:47:26.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118691+0000", 2024-09-15T03:47:26.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2653000000000001e-05 2024-09-15T03:47:26.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.119431+0000", 2024-09-15T03:47:26.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00074060399999999996 2024-09-15T03:47:26.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124280+0000", 2024-09-15T03:47:26.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0048484410000000002 2024-09-15T03:47:26.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:26.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.134000+0000", 2024-09-15T03:47:26.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0097201549999999994 2024-09-15T03:47:26.113 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.113 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:26.113 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.113 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:220 3.bs0 3:d99ddedc:::smithi00132172-44:head [copy-from ver 22] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:26.113 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:26.113 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.941201536999998, 2024-09-15T03:47:26.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024273429999999999, 2024-09-15T03:47:26.113 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:26.113 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:26.113 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:26.113 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:26.114 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.114 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 220 2024-09-15T03:47:26.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.114 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:26.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:26.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:26.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:26.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:26.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:26.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127647+0000", 2024-09-15T03:47:26.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1799999999999999e-06 2024-09-15T03:47:26.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:26.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127656+0000", 2024-09-15T03:47:26.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3759999999999997e-06 2024-09-15T03:47:26.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:26.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127674+0000", 2024-09-15T03:47:26.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7374e-05 2024-09-15T03:47:26.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127680+0000", 2024-09-15T03:47:26.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6949999999999997e-06 2024-09-15T03:47:26.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127736+0000", 2024-09-15T03:47:26.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6011999999999998e-05 2024-09-15T03:47:26.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:26.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127744+0000", 2024-09-15T03:47:26.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2870000000000004e-06 2024-09-15T03:47:26.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130338+0000", 2024-09-15T03:47:26.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025940249999999998 2024-09-15T03:47:26.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:26.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130423+0000", 2024-09-15T03:47:26.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5303000000000006e-05 2024-09-15T03:47:26.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.147840+0000", 2024-09-15T03:47:26.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017416580000000001 2024-09-15T03:47:26.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148318+0000", 2024-09-15T03:47:26.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047822899999999999 2024-09-15T03:47:26.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148329+0000", 2024-09-15T03:47:26.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0611000000000001e-05 2024-09-15T03:47:26.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148488+0000", 2024-09-15T03:47:26.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015938399999999999 2024-09-15T03:47:26.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148870+0000", 2024-09-15T03:47:26.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000382405 2024-09-15T03:47:26.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:26.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.151092+0000", 2024-09-15T03:47:26.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022212529999999999 2024-09-15T03:47:26.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:26.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.151918+0000", 2024-09-15T03:47:26.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082671600000000002 2024-09-15T03:47:26.120 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.120 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:26.120 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.121 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:221 3.bs0 3:d99ddedc:::smithi00132172-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:26.121 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:26.121 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.941157382, 2024-09-15T03:47:26.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021194474000000001, 2024-09-15T03:47:26.121 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:26.121 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:26.121 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:26.121 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:26.121 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.121 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 221 2024-09-15T03:47:26.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.121 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:26.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:26.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:26.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:26.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:26.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:26.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127690+0000", 2024-09-15T03:47:26.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:47:26.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:26.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127692+0000", 2024-09-15T03:47:26.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6589999999999999e-06 2024-09-15T03:47:26.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:26.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127700+0000", 2024-09-15T03:47:26.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4769999999999997e-06 2024-09-15T03:47:26.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127704+0000", 2024-09-15T03:47:26.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3819999999999997e-06 2024-09-15T03:47:26.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127805+0000", 2024-09-15T03:47:26.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100639 2024-09-15T03:47:26.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:26.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127809+0000", 2024-09-15T03:47:26.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8870000000000002e-06 2024-09-15T03:47:26.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130527+0000", 2024-09-15T03:47:26.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002718579 2024-09-15T03:47:26.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:26.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130558+0000", 2024-09-15T03:47:26.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0827999999999997e-05 2024-09-15T03:47:26.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148546+0000", 2024-09-15T03:47:26.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017988356 2024-09-15T03:47:26.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:26.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148683+0000", 2024-09-15T03:47:26.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013650899999999999 2024-09-15T03:47:26.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:26.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148884+0000", 2024-09-15T03:47:26.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020061499999999999 2024-09-15T03:47:26.127 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.127 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:26.127 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.127 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:252 3.bs0 3:d6f79b7b:::smithi00132172-29:head [copy-from ver 6] snapc 7=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:47:26.127 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.169146+0000", 2024-09-15T03:47:26.127 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.899700113000002, 2024-09-15T03:47:26.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016731068000000002, 2024-09-15T03:47:26.127 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:26.127 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:26.127 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:26.128 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:26.128 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.128 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 252 2024-09-15T03:47:26.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.128 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:26.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:26.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169146+0000", 2024-09-15T03:47:26.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:26.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169146+0000", 2024-09-15T03:47:26.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:26.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169147+0000", 2024-09-15T03:47:26.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1400000000000001e-07 2024-09-15T03:47:26.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:26.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169149+0000", 2024-09-15T03:47:26.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0279999999999999e-06 2024-09-15T03:47:26.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:26.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169152+0000", 2024-09-15T03:47:26.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0800000000000002e-06 2024-09-15T03:47:26.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169154+0000", 2024-09-15T03:47:26.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4679999999999999e-06 2024-09-15T03:47:26.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.174427+0000", 2024-09-15T03:47:26.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052724349999999998 2024-09-15T03:47:26.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:26.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.174543+0000", 2024-09-15T03:47:26.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116803 2024-09-15T03:47:26.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.178297+0000", 2024-09-15T03:47:26.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037531539999999999 2024-09-15T03:47:26.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179112+0000", 2024-09-15T03:47:26.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081548600000000003 2024-09-15T03:47:26.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179151+0000", 2024-09-15T03:47:26.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9158e-05 2024-09-15T03:47:26.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179272+0000", 2024-09-15T03:47:26.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120802 2024-09-15T03:47:26.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179531+0000", 2024-09-15T03:47:26.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000259351 2024-09-15T03:47:26.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:26.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179723+0000", 2024-09-15T03:47:26.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000191447 2024-09-15T03:47:26.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:26.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.185877+0000", 2024-09-15T03:47:26.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061546420000000001 2024-09-15T03:47:26.133 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.133 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:26.134 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.134 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:357 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:26.134 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:26.134 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.219805848, 2024-09-15T03:47:26.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.02017917, 2024-09-15T03:47:26.134 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:26.134 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:26.134 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:26.134 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:26.134 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.134 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 357 2024-09-15T03:47:26.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.135 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:26.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:26.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:26.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:26.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:26.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:26.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849043+0000", 2024-09-15T03:47:26.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.295e-06 2024-09-15T03:47:26.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:26.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849051+0000", 2024-09-15T03:47:26.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4079999999999998e-06 2024-09-15T03:47:26.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:26.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849069+0000", 2024-09-15T03:47:26.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7462999999999999e-05 2024-09-15T03:47:26.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849077+0000", 2024-09-15T03:47:26.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8869999999999992e-06 2024-09-15T03:47:26.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849194+0000", 2024-09-15T03:47:26.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011723899999999999 2024-09-15T03:47:26.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:26.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849296+0000", 2024-09-15T03:47:26.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102011 2024-09-15T03:47:26.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861106+0000", 2024-09-15T03:47:26.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0118097610000001 2024-09-15T03:47:26.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:26.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861222+0000", 2024-09-15T03:47:26.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116561 2024-09-15T03:47:26.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:26.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.869220+0000", 2024-09-15T03:47:26.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007997545 2024-09-15T03:47:26.139 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.139 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:26.139 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.139 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:358 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:26.139 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:26.139 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.219750902000001, 2024-09-15T03:47:26.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0228638050000001, 2024-09-15T03:47:26.140 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:26.140 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:26.140 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:26.140 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:26.140 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.140 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 358 2024-09-15T03:47:26.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.140 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:26.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:26.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:26.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:26.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:26.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:26.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:26.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4099999999999997e-07 2024-09-15T03:47:26.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:26.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849101+0000", 2024-09-15T03:47:26.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8180000000000003e-06 2024-09-15T03:47:26.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:26.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849110+0000", 2024-09-15T03:47:26.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1409999999999994e-06 2024-09-15T03:47:26.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849116+0000", 2024-09-15T03:47:26.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2390000000000002e-06 2024-09-15T03:47:26.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849326+0000", 2024-09-15T03:47:26.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000210029 2024-09-15T03:47:26.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:26.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849421+0000", 2024-09-15T03:47:26.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5775999999999999e-05 2024-09-15T03:47:26.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861647+0000", 2024-09-15T03:47:26.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0122257139999999 2024-09-15T03:47:26.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:26.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861777+0000", 2024-09-15T03:47:26.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130087 2024-09-15T03:47:26.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:26.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.871959+0000", 2024-09-15T03:47:26.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010182160000000001 2024-09-15T03:47:26.144 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.144 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:26.144 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.145 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:359 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:26.145 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:26.145 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.219725749999998, 2024-09-15T03:47:26.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0240468149999999, 2024-09-15T03:47:26.145 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:26.145 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:26.145 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:26.145 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:26.145 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.145 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 359 2024-09-15T03:47:26.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.146 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:26.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:26.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:26.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:26.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:26.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:26.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:26.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:47:26.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:26.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849124+0000", 2024-09-15T03:47:26.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2679999999999999e-06 2024-09-15T03:47:26.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:26.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849132+0000", 2024-09-15T03:47:26.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4030000000000001e-06 2024-09-15T03:47:26.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849136+0000", 2024-09-15T03:47:26.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2289999999999997e-06 2024-09-15T03:47:26.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849446+0000", 2024-09-15T03:47:26.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031043299999999999 2024-09-15T03:47:26.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:26.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849553+0000", 2024-09-15T03:47:26.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106415 2024-09-15T03:47:26.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.863404+0000", 2024-09-15T03:47:26.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0138511029999999 2024-09-15T03:47:26.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:26.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.863475+0000", 2024-09-15T03:47:26.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0814000000000003e-05 2024-09-15T03:47:26.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:26.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.873168+0000", 2024-09-15T03:47:26.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00969274 2024-09-15T03:47:26.150 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.150 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:26.150 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.150 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:474 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~786432] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:26.150 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:26.150 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.193801075, 2024-09-15T03:47:26.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98604943700000003, 2024-09-15T03:47:26.150 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:26.150 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:26.150 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:26.150 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:26.151 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.151 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 474 2024-09-15T03:47:26.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.151 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:26.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:26.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:26.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:26.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:26.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:26.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875048+0000", 2024-09-15T03:47:26.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1600000000000001e-06 2024-09-15T03:47:26.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:26.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875844+0000", 2024-09-15T03:47:26.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079686299999999998 2024-09-15T03:47:26.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:26.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875857+0000", 2024-09-15T03:47:26.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2889e-05 2024-09-15T03:47:26.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875864+0000", 2024-09-15T03:47:26.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6139999999999997e-06 2024-09-15T03:47:26.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875916+0000", 2024-09-15T03:47:26.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1992e-05 2024-09-15T03:47:26.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:26.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875918+0000", 2024-09-15T03:47:26.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3300000000000001e-06 2024-09-15T03:47:26.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.845918+0000", 2024-09-15T03:47:26.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96999959199999997 2024-09-15T03:47:26.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:26.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.846229+0000", 2024-09-15T03:47:26.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000311601 2024-09-15T03:47:26.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848445+0000", 2024-09-15T03:47:26.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022160499999999998 2024-09-15T03:47:26.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848880+0000", 2024-09-15T03:47:26.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043432600000000002 2024-09-15T03:47:26.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848952+0000", 2024-09-15T03:47:26.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2509000000000007e-05 2024-09-15T03:47:26.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.849311+0000", 2024-09-15T03:47:26.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000358611 2024-09-15T03:47:26.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851131+0000", 2024-09-15T03:47:26.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001820115 2024-09-15T03:47:26.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:26.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861048+0000", 2024-09-15T03:47:26.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099164830000000002 2024-09-15T03:47:26.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:26.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861095+0000", 2024-09-15T03:47:26.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7302000000000002e-05 2024-09-15T03:47:26.157 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.157 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:26.157 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.158 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:475 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~491520] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:26.158 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:26.158 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.191034604, 2024-09-15T03:47:26.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98407319000000004, 2024-09-15T03:47:26.158 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:26.158 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:26.158 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:26.158 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:26.158 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.158 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 475 2024-09-15T03:47:26.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.158 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:26.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:26.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:26.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:26.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:26.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:26.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877813+0000", 2024-09-15T03:47:26.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8800000000000002e-07 2024-09-15T03:47:26.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:26.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878260+0000", 2024-09-15T03:47:26.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044732099999999998 2024-09-15T03:47:26.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:26.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878270+0000", 2024-09-15T03:47:26.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9799999999999993e-06 2024-09-15T03:47:26.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878276+0000", 2024-09-15T03:47:26.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2269999999999998e-06 2024-09-15T03:47:26.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878313+0000", 2024-09-15T03:47:26.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6570999999999999e-05 2024-09-15T03:47:26.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:26.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878315+0000", 2024-09-15T03:47:26.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8729999999999999e-06 2024-09-15T03:47:26.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851316+0000", 2024-09-15T03:47:26.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97300154800000005 2024-09-15T03:47:26.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:26.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851440+0000", 2024-09-15T03:47:26.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124287 2024-09-15T03:47:26.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852732+0000", 2024-09-15T03:47:26.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001291174 2024-09-15T03:47:26.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852991+0000", 2024-09-15T03:47:26.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025905299999999997 2024-09-15T03:47:26.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853003+0000", 2024-09-15T03:47:26.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2908000000000001e-05 2024-09-15T03:47:26.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853104+0000", 2024-09-15T03:47:26.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010011800000000001 2024-09-15T03:47:26.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854342+0000", 2024-09-15T03:47:26.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012380189999999999 2024-09-15T03:47:26.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:26.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861206+0000", 2024-09-15T03:47:26.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068648499999999996 2024-09-15T03:47:26.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:26.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861885+0000", 2024-09-15T03:47:26.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00067857300000000002 2024-09-15T03:47:26.166 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.166 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:26.166 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.166 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:476 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~278528] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:26.166 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:26.166 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.189407591, 2024-09-15T03:47:26.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98268067299999995, 2024-09-15T03:47:26.166 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:26.167 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:26.167 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:26.167 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:26.167 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.167 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 476 2024-09-15T03:47:26.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.167 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:26.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:26.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:26.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:26.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:26.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:26.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:26.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.13e-07 2024-09-15T03:47:26.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:26.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879706+0000", 2024-09-15T03:47:26.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026656100000000001 2024-09-15T03:47:26.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:26.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879715+0000", 2024-09-15T03:47:26.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9330000000000007e-06 2024-09-15T03:47:26.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879720+0000", 2024-09-15T03:47:26.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0880000000000002e-06 2024-09-15T03:47:26.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879743+0000", 2024-09-15T03:47:26.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2654e-05 2024-09-15T03:47:26.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:26.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879743+0000", 2024-09-15T03:47:26.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.99e-07 2024-09-15T03:47:26.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853170+0000", 2024-09-15T03:47:26.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97342687900000002 2024-09-15T03:47:26.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:26.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853297+0000", 2024-09-15T03:47:26.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127157 2024-09-15T03:47:26.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854150+0000", 2024-09-15T03:47:26.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085288900000000001 2024-09-15T03:47:26.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854292+0000", 2024-09-15T03:47:26.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014199300000000001 2024-09-15T03:47:26.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854559+0000", 2024-09-15T03:47:26.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026668999999999998 2024-09-15T03:47:26.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855998+0000", 2024-09-15T03:47:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001438714 2024-09-15T03:47:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856547+0000", 2024-09-15T03:47:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054880399999999996 2024-09-15T03:47:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861858+0000", 2024-09-15T03:47:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053110320000000003 2024-09-15T03:47:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.862120+0000", 2024-09-15T03:47:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026196699999999999 2024-09-15T03:47:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:477 3.bs0 3:d52faf55:::smithi00132172-40:head [setxattr _header (57) in=7b] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.189119654000001, 2024-09-15T03:47:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98562198000000001, 2024-09-15T03:47:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 477 2024-09-15T03:47:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2800000000000002e-07 2024-09-15T03:47:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879730+0000", 2024-09-15T03:47:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0589999999999998e-06 2024-09-15T03:47:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879737+0000", 2024-09-15T03:47:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0419999999999997e-06 2024-09-15T03:47:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879741+0000", 2024-09-15T03:47:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9960000000000004e-06 2024-09-15T03:47:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879756+0000", 2024-09-15T03:47:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4681999999999999e-05 2024-09-15T03:47:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879756+0000", 2024-09-15T03:47:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.22e-07 2024-09-15T03:47:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854590+0000", 2024-09-15T03:47:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97483319000000002 2024-09-15T03:47:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854718+0000", 2024-09-15T03:47:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012842100000000001 2024-09-15T03:47:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855416+0000", 2024-09-15T03:47:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069759200000000003 2024-09-15T03:47:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855551+0000", 2024-09-15T03:47:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013517600000000001 2024-09-15T03:47:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856002+0000", 2024-09-15T03:47:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045169399999999999 2024-09-15T03:47:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856065+0000", 2024-09-15T03:47:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3028999999999996e-05 2024-09-15T03:47:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856554+0000", 2024-09-15T03:47:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000488486 2024-09-15T03:47:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865278+0000", 2024-09-15T03:47:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087241929999999999 2024-09-15T03:47:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865349+0000", 2024-09-15T03:47:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0669999999999999e-05 2024-09-15T03:47:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:478 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.189100859, 2024-09-15T03:47:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99471090500000003, 2024-09-15T03:47:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 478 2024-09-15T03:47:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:26.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:26.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:26.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2599999999999998e-07 2024-09-15T03:47:26.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:26.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879748+0000", 2024-09-15T03:47:26.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9609999999999999e-06 2024-09-15T03:47:26.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:26.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879755+0000", 2024-09-15T03:47:26.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1879999999999996e-06 2024-09-15T03:47:26.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879760+0000", 2024-09-15T03:47:26.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2560000000000002e-06 2024-09-15T03:47:26.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879797+0000", 2024-09-15T03:47:26.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6452999999999998e-05 2024-09-15T03:47:26.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:26.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879799+0000", 2024-09-15T03:47:26.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6619999999999999e-06 2024-09-15T03:47:26.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855637+0000", 2024-09-15T03:47:26.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97583850999999999 2024-09-15T03:47:26.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:26.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855763+0000", 2024-09-15T03:47:26.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012612699999999999 2024-09-15T03:47:26.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865707+0000", 2024-09-15T03:47:26.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099439660000000003 2024-09-15T03:47:26.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:26.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865803+0000", 2024-09-15T03:47:26.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5946999999999995e-05 2024-09-15T03:47:26.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:26.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.874456+0000", 2024-09-15T03:47:26.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0086534090000000008 2024-09-15T03:47:26.188 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.188 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:26.188 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.188 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:486 3.bs0 3:df8b18a3:::smithi00132172-17:head [delete] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e64)", 2024-09-15T03:47:26.188 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:26.189 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.186288380000001, 2024-09-15T03:47:26.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98362617100000005, 2024-09-15T03:47:26.189 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:26.189 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:26.189 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:26.189 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:26.189 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.189 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 486 2024-09-15T03:47:26.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.189 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:26.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:26.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:26.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:26.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:26.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:26.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882559+0000", 2024-09-15T03:47:26.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0359999999999999e-06 2024-09-15T03:47:26.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:26.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882564+0000", 2024-09-15T03:47:26.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6530000000000002e-06 2024-09-15T03:47:26.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:26.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882574+0000", 2024-09-15T03:47:26.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0464e-05 2024-09-15T03:47:26.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882578+0000", 2024-09-15T03:47:26.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2459999999999997e-06 2024-09-15T03:47:26.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882624+0000", 2024-09-15T03:47:26.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5698000000000002e-05 2024-09-15T03:47:26.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:26.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882627+0000", 2024-09-15T03:47:26.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3350000000000001e-06 2024-09-15T03:47:26.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856574+0000", 2024-09-15T03:47:26.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97394739399999997 2024-09-15T03:47:26.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:26.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856853+0000", 2024-09-15T03:47:26.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027918399999999998 2024-09-15T03:47:26.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:26.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.866184+0000", 2024-09-15T03:47:26.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093311609999999993 2024-09-15T03:47:26.194 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.194 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:26.194 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.194 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:495 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [copy-from ver 51] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-15T03:47:26.194 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:26.194 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.180387331, 2024-09-15T03:47:26.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023555178, 2024-09-15T03:47:26.195 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:26.195 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:26.195 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:26.195 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:26.195 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.195 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 495 2024-09-15T03:47:26.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.195 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:26.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:26.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:26.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:26.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:26.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:26.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888461+0000", 2024-09-15T03:47:26.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.719e-06 2024-09-15T03:47:26.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:26.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888471+0000", 2024-09-15T03:47:26.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0161e-05 2024-09-15T03:47:26.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:26.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888500+0000", 2024-09-15T03:47:26.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8989e-05 2024-09-15T03:47:26.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888508+0000", 2024-09-15T03:47:26.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5299999999999999e-06 2024-09-15T03:47:26.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888554+0000", 2024-09-15T03:47:26.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6489000000000002e-05 2024-09-15T03:47:26.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:26.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888809+0000", 2024-09-15T03:47:26.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000255467 2024-09-15T03:47:26.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.905724+0000", 2024-09-15T03:47:26.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016914348999999999 2024-09-15T03:47:26.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906244+0000", 2024-09-15T03:47:26.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052054900000000003 2024-09-15T03:47:26.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906262+0000", 2024-09-15T03:47:26.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7499999999999998e-05 2024-09-15T03:47:26.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.907367+0000", 2024-09-15T03:47:26.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011049460000000001 2024-09-15T03:47:26.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.907384+0000", 2024-09-15T03:47:26.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7000999999999999e-05 2024-09-15T03:47:26.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:26.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.911965+0000", 2024-09-15T03:47:26.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045809470000000001 2024-09-15T03:47:26.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:26.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.912014+0000", 2024-09-15T03:47:26.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9530999999999997e-05 2024-09-15T03:47:26.202 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.202 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:26.202 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.202 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:496 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-15T03:47:26.203 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.888524+0000", 2024-09-15T03:47:26.203 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.180322239000001, 2024-09-15T03:47:26.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01823953, 2024-09-15T03:47:26.203 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:26.203 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:26.203 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:26.203 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:26.203 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.203 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 496 2024-09-15T03:47:26.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.203 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:26.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:26.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888524+0000", 2024-09-15T03:47:26.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:26.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888524+0000", 2024-09-15T03:47:26.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:26.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:26.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888525+0000", 2024-09-15T03:47:26.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2699999999999999e-07 2024-09-15T03:47:26.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:26.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888529+0000", 2024-09-15T03:47:26.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7840000000000001e-06 2024-09-15T03:47:26.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:26.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888540+0000", 2024-09-15T03:47:26.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1477e-05 2024-09-15T03:47:26.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888546+0000", 2024-09-15T03:47:26.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4990000000000002e-06 2024-09-15T03:47:26.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888971+0000", 2024-09-15T03:47:26.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042491300000000002 2024-09-15T03:47:26.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:26.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.889024+0000", 2024-09-15T03:47:26.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3046e-05 2024-09-15T03:47:26.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:26.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906517+0000", 2024-09-15T03:47:26.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017492909000000001 2024-09-15T03:47:26.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:26.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906638+0000", 2024-09-15T03:47:26.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012138000000000001 2024-09-15T03:47:26.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:26.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:26.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:26.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906764+0000", 2024-09-15T03:47:26.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125795 2024-09-15T03:47:26.208 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.208 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:26.208 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.208 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:26.208 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:26.208 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:26.209 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:26.070+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:26.210 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:47:26.211 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:26.209+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:26.269 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:47:26.268+0000 7f632aed8200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:47:26.280 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:47:26.279+0000 7f632aed8200 -1 journal do_read_entry(30416896): bad header magic 2024-09-15T03:47:26.280 INFO:teuthology.orchestra.run.smithi190.stdout:3.5s0 2024-09-15T03:47:26.280 INFO:teuthology.orchestra.run.smithi190.stdout:3.3s0 2024-09-15T03:47:26.281 INFO:teuthology.orchestra.run.smithi190.stdout:3.7s7 2024-09-15T03:47:26.281 INFO:teuthology.orchestra.run.smithi190.stdout:3.2s2 2024-09-15T03:47:26.281 INFO:teuthology.orchestra.run.smithi190.stdout:3.bs5 2024-09-15T03:47:26.281 INFO:teuthology.orchestra.run.smithi190.stdout:3.1s3 2024-09-15T03:47:26.281 INFO:teuthology.orchestra.run.smithi190.stdout:3.fs7 2024-09-15T03:47:26.281 INFO:teuthology.orchestra.run.smithi190.stdout:3.ds4 2024-09-15T03:47:26.281 INFO:teuthology.orchestra.run.smithi190.stdout:3.cs5 2024-09-15T03:47:26.281 INFO:teuthology.orchestra.run.smithi190.stdout:3.8s7 2024-09-15T03:47:26.281 INFO:teuthology.orchestra.run.smithi190.stdout:3.6s7 2024-09-15T03:47:26.281 INFO:teuthology.orchestra.run.smithi190.stdout:3.9s2 2024-09-15T03:47:26.281 INFO:teuthology.orchestra.run.smithi190.stdout:3.4s7 2024-09-15T03:47:26.281 INFO:teuthology.orchestra.run.smithi190.stdout:3.0s7 2024-09-15T03:47:26.285 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --op export --pgid 3.5s0 --file /var/log/ceph/exp.3.5s0.10 2024-09-15T03:47:26.311 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:26.310+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:26.322 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:26.322 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:47:26.322 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.322 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:621 3.ds0 3:b166213b:::smithi00132172-21:head [append 0~0] snapc 2d=[2d,2b,29,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-15T03:47:26.323 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:25.956700+0000", 2024-09-15T03:47:26.323 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.36518375800000002, 2024-09-15T03:47:26.323 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.365214645, 2024-09-15T03:47:26.323 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.323 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:26.323 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:26.323 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:26.323 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.323 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 621 2024-09-15T03:47:26.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.323 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.323 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.324 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.324 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956700+0000", 2024-09-15T03:47:26.324 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.324 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.324 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.324 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.324 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956700+0000", 2024-09-15T03:47:26.324 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.324 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.324 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.324 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.325 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956701+0000", 2024-09-15T03:47:26.325 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5519999999999999e-06 2024-09-15T03:47:26.325 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.325 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.325 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.325 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956707+0000", 2024-09-15T03:47:26.325 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.037e-06 2024-09-15T03:47:26.325 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.325 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.325 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.325 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956722+0000", 2024-09-15T03:47:26.325 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5102e-05 2024-09-15T03:47:26.325 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.326 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.326 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.326 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956729+0000", 2024-09-15T03:47:26.326 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0700000000000001e-06 2024-09-15T03:47:26.326 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.326 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.326 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.326 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956782+0000", 2024-09-15T03:47:26.326 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2132000000000002e-05 2024-09-15T03:47:26.326 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.326 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.326 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-09-15T03:47:26.327 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956788+0000", 2024-09-15T03:47:26.327 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0569999999999998e-06 2024-09-15T03:47:26.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.327 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.327 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.327 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.958401+0000", 2024-09-15T03:47:26.327 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0016133090000000001 2024-09-15T03:47:26.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.327 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.327 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:26.327 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.958529+0000", 2024-09-15T03:47:26.327 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000128372 2024-09-15T03:47:26.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.328 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.328 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.959212+0000", 2024-09-15T03:47:26.328 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00068307199999999995 2024-09-15T03:47:26.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.328 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.328 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.959528+0000", 2024-09-15T03:47:26.328 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00031523 2024-09-15T03:47:26.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.328 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.328 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.329 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.959541+0000", 2024-09-15T03:47:26.329 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3456e-05 2024-09-15T03:47:26.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.329 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.329 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.329 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.959598+0000", 2024-09-15T03:47:26.329 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6628000000000001e-05 2024-09-15T03:47:26.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.329 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.329 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.329 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.962043+0000", 2024-09-15T03:47:26.329 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002445483 2024-09-15T03:47:26.330 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.330 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.330 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.330 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.330 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:622 3.ds0 3:b166213b:::smithi00132172-21:head [rollback 12] snapc 2d=[2d,2b,29,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-15T03:47:26.330 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:25.956738+0000", 2024-09-15T03:47:26.330 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.36514524900000001, 2024-09-15T03:47:26.330 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.36524189600000001, 2024-09-15T03:47:26.330 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.330 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:26.331 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:26.331 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:26.331 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.331 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 622 2024-09-15T03:47:26.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.331 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.331 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956738+0000", 2024-09-15T03:47:26.331 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.332 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.332 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956738+0000", 2024-09-15T03:47:26.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.332 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.332 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.332 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956738+0000", 2024-09-15T03:47:26.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3999999999999997e-07 2024-09-15T03:47:26.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.332 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.332 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.332 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956740+0000", 2024-09-15T03:47:26.333 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8419999999999999e-06 2024-09-15T03:47:26.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.333 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.333 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.333 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956745+0000", 2024-09-15T03:47:26.333 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1030000000000001e-06 2024-09-15T03:47:26.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.333 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.333 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.333 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956748+0000", 2024-09-15T03:47:26.333 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6850000000000001e-06 2024-09-15T03:47:26.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.333 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.333 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956854+0000", 2024-09-15T03:47:26.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000106319 2024-09-15T03:47:26.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.334 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.334 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:26.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956859+0000", 2024-09-15T03:47:26.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5859999999999998e-06 2024-09-15T03:47:26.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.334 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.334 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.959626+0000", 2024-09-15T03:47:26.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002766892 2024-09-15T03:47:26.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.334 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:26.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.960424+0000", 2024-09-15T03:47:26.335 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00079811499999999996 2024-09-15T03:47:26.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.961075+0000", 2024-09-15T03:47:26.335 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00065067 2024-09-15T03:47:26.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.336 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.961185+0000", 2024-09-15T03:47:26.336 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000110293 2024-09-15T03:47:26.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.336 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.336 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.336 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.961198+0000", 2024-09-15T03:47:26.336 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2756e-05 2024-09-15T03:47:26.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.336 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.336 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.336 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.961239+0000", 2024-09-15T03:47:26.336 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1661e-05 2024-09-15T03:47:26.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.336 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.337 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.337 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.962050+0000", 2024-09-15T03:47:26.337 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00081059800000000005 2024-09-15T03:47:26.337 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.337 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.337 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.337 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.337 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.337 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:623 3.ds0 3:b166213b:::smithi00132172-21:head [append 0~0] snapc 2d=[2d,2b,29,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-15T03:47:26.337 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:25.956778+0000", 2024-09-15T03:47:26.337 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.36510488899999999, 2024-09-15T03:47:26.337 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.36525873199999997, 2024-09-15T03:47:26.337 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.338 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:26.338 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:26.338 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:26.338 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.338 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 623 2024-09-15T03:47:26.338 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.338 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.338 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.338 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.338 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956778+0000", 2024-09-15T03:47:26.338 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.339 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.339 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.339 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956778+0000", 2024-09-15T03:47:26.339 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956779+0000", 2024-09-15T03:47:26.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9799999999999999e-07 2024-09-15T03:47:26.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956781+0000", 2024-09-15T03:47:26.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2199999999999999e-06 2024-09-15T03:47:26.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.341 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956785+0000", 2024-09-15T03:47:26.342 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3490000000000002e-06 2024-09-15T03:47:26.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.342 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.342 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956790+0000", 2024-09-15T03:47:26.342 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4839999999999998e-06 2024-09-15T03:47:26.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956892+0000", 2024-09-15T03:47:26.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010189 2024-09-15T03:47:26.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:26.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.956896+0000", 2024-09-15T03:47:26.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4379999999999997e-06 2024-09-15T03:47:26.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.961256+0000", 2024-09-15T03:47:26.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0043601630000000002 2024-09-15T03:47:26.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:26.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.961360+0000", 2024-09-15T03:47:26.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000103908 2024-09-15T03:47:26.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.961843+0000", 2024-09-15T03:47:26.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00048291799999999999 2024-09-15T03:47:26.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.961951+0000", 2024-09-15T03:47:26.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010779499999999999 2024-09-15T03:47:26.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.961969+0000", 2024-09-15T03:47:26.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7737e-05 2024-09-15T03:47:26.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.962717+0000", 2024-09-15T03:47:26.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00074830000000000003 2024-09-15T03:47:26.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:25.962759+0000", 2024-09-15T03:47:26.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2218999999999997e-05 2024-09-15T03:47:26.347 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.347 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.347 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.347 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.347 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2024-09-15T03:47:26.347 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 3 2024-09-15T03:47:26.347 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:26.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:47:26.392 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:47:26.392+0000 7f2643a10200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:47:26.393 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:47:26.393+0000 7f2643a10200 -1 journal do_read_entry(30416896): bad header magic 2024-09-15T03:47:26.393 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:47:26.393+0000 7f2643a10200 -1 journal do_read_entry(30416896): bad header magic 2024-09-15T03:47:26.394 INFO:teuthology.orchestra.run.smithi190.stderr:Exporting 3.5s0 info 3.5s0( v 65'35 (0'0,65'35] local-lis/les=70/71 n=4 ec=28/28 lis/c=70/70 les/c/f=71/71/0 sis=70) 2024-09-15T03:47:26.395 INFO:teuthology.orchestra.run.smithi190.stderr:Read 0#3:af1dd10e:::smithi00132172-27:4# 2024-09-15T03:47:26.395 INFO:teuthology.orchestra.run.smithi190.stderr:Read 0#3:af1dd10e:::smithi00132172-27:12# 2024-09-15T03:47:26.395 INFO:teuthology.orchestra.run.smithi190.stderr:Read 0#3:af1dd10e:::smithi00132172-27:19# 2024-09-15T03:47:26.396 INFO:teuthology.orchestra.run.smithi190.stderr:Read 0#3:af1dd10e:::smithi00132172-27:head# 2024-09-15T03:47:26.397 INFO:teuthology.orchestra.run.smithi190.stderr:Export successful 2024-09-15T03:47:26.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --force --op remove --pgid 3.5s0 2024-09-15T03:47:26.443 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:26.442+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:26.487 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:47:26.486+0000 7ff80bcd7200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:47:26.487 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:47:26.486+0000 7ff80bcd7200 -1 journal do_read_entry(30416896): bad header magic 2024-09-15T03:47:26.487 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:47:26.486+0000 7ff80bcd7200 -1 journal do_read_entry(30416896): bad header magic 2024-09-15T03:47:26.487 INFO:teuthology.orchestra.run.smithi190.stdout: marking collection for removal 2024-09-15T03:47:26.488 INFO:teuthology.orchestra.run.smithi190.stdout:setting '_remove' omap key 2024-09-15T03:47:26.488 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:26.488 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:26.488 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:26.488 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:26.488 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:26.489 INFO:teuthology.orchestra.run.smithi190.stdout:finish_remove_pgs 3.5s0_head removing 3.5s0 2024-09-15T03:47:26.489 INFO:teuthology.orchestra.run.smithi190.stdout:Remove successful 2024-09-15T03:47:26.493 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.5s0.10 2024-09-15T03:47:26.497 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:47:26.538 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:26.537+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:26.542 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:47:26.541+0000 7f85af713200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:47:26.542 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:47:26.541+0000 7f85af713200 -1 journal do_read_entry(30429184): bad header magic 2024-09-15T03:47:26.542 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-15T03:47:26.541+0000 7f85af713200 -1 journal do_read_entry(30429184): bad header magic 2024-09-15T03:47:26.543 INFO:teuthology.orchestra.run.smithi190.stdout:get_pg_num_history pg_num_history pg_num_history(e78 pg_nums {1={18=1},2={21=8},3={28=16}} deleted_pools ) 2024-09-15T03:47:26.543 INFO:teuthology.orchestra.run.smithi190.stdout:Importing pgid 3.5s0 2024-09-15T03:47:26.543 INFO:teuthology.orchestra.run.smithi190.stdout:Write 0#3:af1dd10e:::smithi00132172-27:4# 2024-09-15T03:47:26.544 INFO:teuthology.orchestra.run.smithi190.stdout:Write 0#3:af1dd10e:::smithi00132172-27:12# 2024-09-15T03:47:26.545 INFO:teuthology.orchestra.run.smithi190.stdout:Write 0#3:af1dd10e:::smithi00132172-27:19# 2024-09-15T03:47:26.548 INFO:teuthology.orchestra.run.smithi190.stdout:Write 0#3:af1dd10e:::smithi00132172-27:head# 2024-09-15T03:47:26.548 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 19=[]:{4=[3,2],12=[12,11,c],19=[18,17,16]} 2024-09-15T03:47:26.550 INFO:teuthology.orchestra.run.smithi190.stdout:write_pg epoch 78 info 3.5s0( v 65'35 (0'0,65'35] local-lis/les=70/71 n=4 ec=28/28 lis/c=70/70 les/c/f=71/71/0 sis=70) 2024-09-15T03:47:26.551 INFO:teuthology.orchestra.run.smithi190.stdout:Import successful 2024-09-15T03:47:26.557 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /var/log/ceph/exp.3.5s0.10 2024-09-15T03:47:26.614 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:26.614+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:26.623 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:47:26.638 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:26.638 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:26.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:26.638 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:47:26.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.638 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:47:26.639 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:26.639 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 68.734340638000006, 2024-09-15T03:47:26.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10988437500000001, 2024-09-15T03:47:26.639 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.639 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:26.639 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:26.639 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:47:26.639 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:47:26.639 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1 2024-09-15T03:47:26.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.639 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:26.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:26.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902355+0000", 2024-09-15T03:47:26.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.429e-06 2024-09-15T03:47:26.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902362+0000", 2024-09-15T03:47:26.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7290000000000003e-06 2024-09-15T03:47:26.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902373+0000", 2024-09-15T03:47:26.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0533999999999999e-05 2024-09-15T03:47:26.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902379+0000", 2024-09-15T03:47:26.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4350000000000002e-06 2024-09-15T03:47:26.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902428+0000", 2024-09-15T03:47:26.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8571e-05 2024-09-15T03:47:26.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:47:26.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902434+0000", 2024-09-15T03:47:26.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5610000000000004e-06 2024-09-15T03:47:26.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011307+0000", 2024-09-15T03:47:26.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.108872306 2024-09-15T03:47:26.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:26.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011448+0000", 2024-09-15T03:47:26.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014137199999999999 2024-09-15T03:47:26.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:47:26.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011569+0000", 2024-09-15T03:47:26.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120925 2024-09-15T03:47:26.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011695+0000", 2024-09-15T03:47:26.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012586800000000001 2024-09-15T03:47:26.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011703+0000", 2024-09-15T03:47:26.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4190000000000002e-06 2024-09-15T03:47:26.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011739+0000", 2024-09-15T03:47:26.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5474e-05 2024-09-15T03:47:26.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:47:26.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011940+0000", 2024-09-15T03:47:26.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020108699999999999 2024-09-15T03:47:26.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:47:26.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012179+0000", 2024-09-15T03:47:26.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023950299999999999 2024-09-15T03:47:26.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:26.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012215+0000", 2024-09-15T03:47:26.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5332e-05 2024-09-15T03:47:26.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:26.647 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012238+0000", 2024-09-15T03:47:26.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3830000000000001e-05 2024-09-15T03:47:26.647 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.647 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.647 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.647 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4263.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2024-09-15T03:47:26.647 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:26.647 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 62.592943951999999, 2024-09-15T03:47:26.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019495788999999999, 2024-09-15T03:47:26.647 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.647 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:26.648 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:26.648 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4263", 2024-09-15T03:47:26.648 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/646873808", 2024-09-15T03:47:26.648 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2 2024-09-15T03:47:26.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.648 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:26.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:26.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043752+0000", 2024-09-15T03:47:26.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.815e-06 2024-09-15T03:47:26.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043757+0000", 2024-09-15T03:47:26.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4240000000000003e-06 2024-09-15T03:47:26.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043766+0000", 2024-09-15T03:47:26.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8629999999999997e-06 2024-09-15T03:47:26.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043773+0000", 2024-09-15T03:47:26.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0099999999999998e-06 2024-09-15T03:47:26.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043830+0000", 2024-09-15T03:47:26.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7089999999999997e-05 2024-09-15T03:47:26.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.651 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:47:26.651 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043837+0000", 2024-09-15T03:47:26.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.08e-06 2024-09-15T03:47:26.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.651 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.651 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063103+0000", 2024-09-15T03:47:26.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019265619000000001 2024-09-15T03:47:26.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:26.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063246+0000", 2024-09-15T03:47:26.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000143888 2024-09-15T03:47:26.652 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.652 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.652 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.653 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/31 ECSubWrite(tid=38, reqid=client.4521.0:223, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:47:26.653 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:47:26.653 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 41.489487615000002, 2024-09-15T03:47:26.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010504493, 2024-09-15T03:47:26.653 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.653 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:26.653 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:47:26.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:47:26.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147207+0000", 2024-09-15T03:47:26.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4999999999999999e-07 2024-09-15T03:47:26.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147648+0000", 2024-09-15T03:47:26.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00044030399999999998 2024-09-15T03:47:26.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147660+0000", 2024-09-15T03:47:26.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2517000000000001e-05 2024-09-15T03:47:26.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147667+0000", 2024-09-15T03:47:26.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.737e-06 2024-09-15T03:47:26.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147713+0000", 2024-09-15T03:47:26.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6405e-05 2024-09-15T03:47:26.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147719+0000", 2024-09-15T03:47:26.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0580000000000003e-06 2024-09-15T03:47:26.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147905+0000", 2024-09-15T03:47:26.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018574600000000001 2024-09-15T03:47:26.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.147917+0000", 2024-09-15T03:47:26.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1749999999999999e-05 2024-09-15T03:47:26.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148022+0000", 2024-09-15T03:47:26.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104863 2024-09-15T03:47:26.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.148037+0000", 2024-09-15T03:47:26.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5302999999999999e-05 2024-09-15T03:47:26.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:26.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:45.157711+0000", 2024-09-15T03:47:26.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0096745600000000005 2024-09-15T03:47:26.658 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.658 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.658 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.658 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:47:26.658 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:26.659 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 40.465870563000003, 2024-09-15T03:47:26.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011878227, 2024-09-15T03:47:26.659 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.659 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:26.659 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.659 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:26.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.659 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:26.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.660 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.660 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170825+0000", 2024-09-15T03:47:26.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9599999999999999e-07 2024-09-15T03:47:26.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.660 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.660 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171100+0000", 2024-09-15T03:47:26.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027548500000000001 2024-09-15T03:47:26.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.660 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171112+0000", 2024-09-15T03:47:26.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2152e-05 2024-09-15T03:47:26.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171119+0000", 2024-09-15T03:47:26.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5050000000000004e-06 2024-09-15T03:47:26.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171162+0000", 2024-09-15T03:47:26.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2908000000000003e-05 2024-09-15T03:47:26.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171169+0000", 2024-09-15T03:47:26.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9739999999999996e-06 2024-09-15T03:47:26.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171398+0000", 2024-09-15T03:47:26.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022954799999999999 2024-09-15T03:47:26.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173640+0000", 2024-09-15T03:47:26.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022417000000000001 2024-09-15T03:47:26.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173705+0000", 2024-09-15T03:47:26.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4907999999999996e-05 2024-09-15T03:47:26.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173742+0000", 2024-09-15T03:47:26.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6940999999999997e-05 2024-09-15T03:47:26.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.664 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:26.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.182702+0000", 2024-09-15T03:47:26.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0089604100000000002 2024-09-15T03:47:26.664 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.664 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.664 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.664 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 39/31 ECSubWrite(tid=43, reqid=client.4521.0:252, at_version=39'27, trim_to=0'0, roll_forward_to=39'25))", 2024-09-15T03:47:26.664 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:26.664 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 40.458153244999998, 2024-09-15T03:47:26.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011286757, 2024-09-15T03:47:26.664 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.665 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:26.665 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.665 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:26.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.665 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:26.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178542+0000", 2024-09-15T03:47:26.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9400000000000004e-07 2024-09-15T03:47:26.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178754+0000", 2024-09-15T03:47:26.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021228800000000001 2024-09-15T03:47:26.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178765+0000", 2024-09-15T03:47:26.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0176e-05 2024-09-15T03:47:26.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178769+0000", 2024-09-15T03:47:26.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1359999999999999e-06 2024-09-15T03:47:26.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178810+0000", 2024-09-15T03:47:26.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0825999999999998e-05 2024-09-15T03:47:26.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178820+0000", 2024-09-15T03:47:26.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0237999999999999e-05 2024-09-15T03:47:26.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179033+0000", 2024-09-15T03:47:26.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021361500000000001 2024-09-15T03:47:26.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179040+0000", 2024-09-15T03:47:26.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0509999999999999e-06 2024-09-15T03:47:26.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179102+0000", 2024-09-15T03:47:26.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1977000000000006e-05 2024-09-15T03:47:26.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179114+0000", 2024-09-15T03:47:26.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1223000000000001e-05 2024-09-15T03:47:26.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:26.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.189828+0000", 2024-09-15T03:47:26.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010714433000000001 2024-09-15T03:47:26.670 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.670 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.670 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.670 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.670 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 39/31 ECSubWrite(tid=44, reqid=client.4521.0:252, at_version=0'0, trim_to=0'0, roll_forward_to=39'27))", 2024-09-15T03:47:26.670 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.179948+0000", 2024-09-15T03:47:26.670 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 40.456746146, 2024-09-15T03:47:26.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010068739, 2024-09-15T03:47:26.670 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.670 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:26.670 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.670 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.671 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179948+0000", 2024-09-15T03:47:26.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.671 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179948+0000", 2024-09-15T03:47:26.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.671 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179949+0000", 2024-09-15T03:47:26.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6199999999999998e-07 2024-09-15T03:47:26.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179960+0000", 2024-09-15T03:47:26.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1216000000000001e-05 2024-09-15T03:47:26.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179970+0000", 2024-09-15T03:47:26.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8980000000000008e-06 2024-09-15T03:47:26.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179975+0000", 2024-09-15T03:47:26.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5650000000000003e-06 2024-09-15T03:47:26.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.673 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180004+0000", 2024-09-15T03:47:26.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9224999999999999e-05 2024-09-15T03:47:26.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.673 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180009+0000", 2024-09-15T03:47:26.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9250000000000003e-06 2024-09-15T03:47:26.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180061+0000", 2024-09-15T03:47:26.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1758000000000001e-05 2024-09-15T03:47:26.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180067+0000", 2024-09-15T03:47:26.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3269999999999997e-06 2024-09-15T03:47:26.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180101+0000", 2024-09-15T03:47:26.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3701000000000001e-05 2024-09-15T03:47:26.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.675 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.180112+0000", 2024-09-15T03:47:26.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1287000000000001e-05 2024-09-15T03:47:26.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.675 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:26.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.190017+0000", 2024-09-15T03:47:26.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0099052749999999998 2024-09-15T03:47:26.675 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.675 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.675 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.676 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 51/31 ECSubWrite(tid=67, reqid=client.4521.0:350, at_version=51'33, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:26.676 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:26.676 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 28.792853204, 2024-09-15T03:47:26.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.015569239, 2024-09-15T03:47:26.676 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.676 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:26.676 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.676 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:26.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:26.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843843+0000", 2024-09-15T03:47:26.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.195e-06 2024-09-15T03:47:26.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843880+0000", 2024-09-15T03:47:26.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7527999999999997e-05 2024-09-15T03:47:26.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843898+0000", 2024-09-15T03:47:26.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7696000000000001e-05 2024-09-15T03:47:26.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843907+0000", 2024-09-15T03:47:26.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8820000000000007e-06 2024-09-15T03:47:26.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843978+0000", 2024-09-15T03:47:26.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1537000000000003e-05 2024-09-15T03:47:26.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843989+0000", 2024-09-15T03:47:26.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0720000000000001e-05 2024-09-15T03:47:26.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844102+0000", 2024-09-15T03:47:26.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000113406 2024-09-15T03:47:26.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844131+0000", 2024-09-15T03:47:26.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8762e-05 2024-09-15T03:47:26.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844180+0000", 2024-09-15T03:47:26.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9329999999999997e-05 2024-09-15T03:47:26.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844201+0000", 2024-09-15T03:47:26.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0772999999999999e-05 2024-09-15T03:47:26.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.681 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:26.681 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.859411+0000", 2024-09-15T03:47:26.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01520941 2024-09-15T03:47:26.681 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.681 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.681 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.681 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:26.681 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:26.681 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 28.791974292999999, 2024-09-15T03:47:26.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012156452, 2024-09-15T03:47:26.682 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.682 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:26.682 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.682 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.682 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.682 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:26.682 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.682 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.682 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.682 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:26.682 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844721+0000", 2024-09-15T03:47:26.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1100000000000005e-07 2024-09-15T03:47:26.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844743+0000", 2024-09-15T03:47:26.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1999000000000001e-05 2024-09-15T03:47:26.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844758+0000", 2024-09-15T03:47:26.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4718999999999999e-05 2024-09-15T03:47:26.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.684 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.684 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844772+0000", 2024-09-15T03:47:26.684 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4637000000000001e-05 2024-09-15T03:47:26.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.684 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.684 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844842+0000", 2024-09-15T03:47:26.684 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9596000000000002e-05 2024-09-15T03:47:26.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.684 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.684 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844853+0000", 2024-09-15T03:47:26.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0912e-05 2024-09-15T03:47:26.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.685 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844966+0000", 2024-09-15T03:47:26.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011276 2024-09-15T03:47:26.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.685 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844977+0000", 2024-09-15T03:47:26.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1676e-05 2024-09-15T03:47:26.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.686 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.686 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847414+0000", 2024-09-15T03:47:26.686 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024363760000000001 2024-09-15T03:47:26.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.686 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.686 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847438+0000", 2024-09-15T03:47:26.686 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3737e-05 2024-09-15T03:47:26.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.686 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:26.686 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856877+0000", 2024-09-15T03:47:26.686 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094392290000000004 2024-09-15T03:47:26.687 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.687 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.687 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.687 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.687 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.687 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:26.687 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:26.687 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 28.791904897999999, 2024-09-15T03:47:26.687 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012258211, 2024-09-15T03:47:26.687 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.687 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:26.688 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.688 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.688 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.688 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:26.688 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.688 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.688 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.688 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:26.688 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.688 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.688 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:26.689 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.58e-07 2024-09-15T03:47:26.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.689 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844818+0000", 2024-09-15T03:47:26.689 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.796e-05 2024-09-15T03:47:26.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.689 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844829+0000", 2024-09-15T03:47:26.689 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0895e-05 2024-09-15T03:47:26.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.690 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.690 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844836+0000", 2024-09-15T03:47:26.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.618e-06 2024-09-15T03:47:26.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.690 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.690 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.690 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845023+0000", 2024-09-15T03:47:26.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018743500000000001 2024-09-15T03:47:26.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.690 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.690 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.690 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845035+0000", 2024-09-15T03:47:26.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1958e-05 2024-09-15T03:47:26.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.691 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.691 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.691 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845147+0000", 2024-09-15T03:47:26.691 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112281 2024-09-15T03:47:26.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.691 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.691 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.691 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847417+0000", 2024-09-15T03:47:26.691 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022693969999999998 2024-09-15T03:47:26.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847462+0000", 2024-09-15T03:47:26.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5466999999999997e-05 2024-09-15T03:47:26.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847503+0000", 2024-09-15T03:47:26.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1022999999999997e-05 2024-09-15T03:47:26.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:26.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.857048+0000", 2024-09-15T03:47:26.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095447190000000001 2024-09-15T03:47:26.693 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.693 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.693 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.694 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 51/31 ECSubWrite(tid=68, reqid=client.4521.0:351, at_version=51'34, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:26.694 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:26.694 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 28.79148678, 2024-09-15T03:47:26.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.014544793, 2024-09-15T03:47:26.694 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.694 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:26.694 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.694 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:26.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:26.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845209+0000", 2024-09-15T03:47:26.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5300000000000003e-07 2024-09-15T03:47:26.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845227+0000", 2024-09-15T03:47:26.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8759000000000001e-05 2024-09-15T03:47:26.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845241+0000", 2024-09-15T03:47:26.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3740000000000001e-05 2024-09-15T03:47:26.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845247+0000", 2024-09-15T03:47:26.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2779999999999999e-06 2024-09-15T03:47:26.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845308+0000", 2024-09-15T03:47:26.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0702000000000003e-05 2024-09-15T03:47:26.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845317+0000", 2024-09-15T03:47:26.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3770000000000002e-06 2024-09-15T03:47:26.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845413+0000", 2024-09-15T03:47:26.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5356e-05 2024-09-15T03:47:26.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847421+0000", 2024-09-15T03:47:26.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020077200000000002 2024-09-15T03:47:26.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847466+0000", 2024-09-15T03:47:26.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5754999999999998e-05 2024-09-15T03:47:26.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847572+0000", 2024-09-15T03:47:26.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010594000000000001 2024-09-15T03:47:26.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:26.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.859753+0000", 2024-09-15T03:47:26.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012180412999999999 2024-09-15T03:47:26.699 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.699 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.699 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.700 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 60/28 ECSubWrite(tid=73, reqid=client.4521.0:424, at_version=60'41, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:26.700 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:26.700 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 19.736283526000001, 2024-09-15T03:47:26.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01137208, 2024-09-15T03:47:26.700 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.700 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:26.700 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:26.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:26.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900412+0000", 2024-09-15T03:47:26.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7599999999999996e-07 2024-09-15T03:47:26.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:26.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7986999999999999e-05 2024-09-15T03:47:26.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900445+0000", 2024-09-15T03:47:26.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.468e-05 2024-09-15T03:47:26.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900451+0000", 2024-09-15T03:47:26.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5799999999999997e-06 2024-09-15T03:47:26.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900507+0000", 2024-09-15T03:47:26.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5855e-05 2024-09-15T03:47:26.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900530+0000", 2024-09-15T03:47:26.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2929e-05 2024-09-15T03:47:26.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900640+0000", 2024-09-15T03:47:26.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000110536 2024-09-15T03:47:26.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900648+0000", 2024-09-15T03:47:26.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9470000000000003e-06 2024-09-15T03:47:26.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900764+0000", 2024-09-15T03:47:26.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011518099999999999 2024-09-15T03:47:26.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900803+0000", 2024-09-15T03:47:26.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9941999999999998e-05 2024-09-15T03:47:26.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:26.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.911783+0000", 2024-09-15T03:47:26.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010979667 2024-09-15T03:47:26.705 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.705 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.705 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.705 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 60/28 ECSubWrite(tid=74, reqid=client.4521.0:425, at_version=60'42, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:26.705 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.901183+0000", 2024-09-15T03:47:26.705 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 19.735511528, 2024-09-15T03:47:26.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010721191999999999, 2024-09-15T03:47:26.706 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.706 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:26.706 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901183+0000", 2024-09-15T03:47:26.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901183+0000", 2024-09-15T03:47:26.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901184+0000", 2024-09-15T03:47:26.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6599999999999996e-07 2024-09-15T03:47:26.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901202+0000", 2024-09-15T03:47:26.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8658999999999999e-05 2024-09-15T03:47:26.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901217+0000", 2024-09-15T03:47:26.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4371e-05 2024-09-15T03:47:26.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.708 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901223+0000", 2024-09-15T03:47:26.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2489999999999998e-06 2024-09-15T03:47:26.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.708 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901274+0000", 2024-09-15T03:47:26.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0602000000000001e-05 2024-09-15T03:47:26.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.708 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901284+0000", 2024-09-15T03:47:26.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.995e-06 2024-09-15T03:47:26.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901372+0000", 2024-09-15T03:47:26.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8399000000000005e-05 2024-09-15T03:47:26.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901380+0000", 2024-09-15T03:47:26.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8040000000000002e-06 2024-09-15T03:47:26.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903553+0000", 2024-09-15T03:47:26.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021731580000000001 2024-09-15T03:47:26.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903570+0000", 2024-09-15T03:47:26.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7218999999999999e-05 2024-09-15T03:47:26.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.712 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:26.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.911904+0000", 2024-09-15T03:47:26.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0083340700000000007 2024-09-15T03:47:26.712 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.712 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.712 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.712 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:517 3.3s1 3:caaee79c:::smithi00132172-10:head [read 0~3145728,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:26.712 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.947050+0000", 2024-09-15T03:47:26.713 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.689644593000001, 2024-09-15T03:47:26.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012110134, 2024-09-15T03:47:26.713 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.713 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:26.713 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:26.713 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:26.713 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.713 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 517 2024-09-15T03:47:26.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.713 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.713 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947050+0000", 2024-09-15T03:47:26.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.714 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947050+0000", 2024-09-15T03:47:26.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.714 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947052+0000", 2024-09-15T03:47:26.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9099999999999999e-06 2024-09-15T03:47:26.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947060+0000", 2024-09-15T03:47:26.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8529999999999992e-06 2024-09-15T03:47:26.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947074+0000", 2024-09-15T03:47:26.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4344e-05 2024-09-15T03:47:26.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947082+0000", 2024-09-15T03:47:26.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9249999999999995e-06 2024-09-15T03:47:26.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947138+0000", 2024-09-15T03:47:26.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6054000000000002e-05 2024-09-15T03:47:26.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:26.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947339+0000", 2024-09-15T03:47:26.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000200829 2024-09-15T03:47:26.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:26.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.959160+0000", 2024-09-15T03:47:26.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011821219000000001 2024-09-15T03:47:26.717 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.717 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.717 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.717 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:518 3.3s1 3:caaee79c:::smithi00132172-10:head [read 0~3145728,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:26.717 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.947091+0000", 2024-09-15T03:47:26.717 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.689603451, 2024-09-15T03:47:26.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017608233000000001, 2024-09-15T03:47:26.717 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.717 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:26.718 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:26.718 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:26.718 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.718 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 518 2024-09-15T03:47:26.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.718 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947091+0000", 2024-09-15T03:47:26.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947091+0000", 2024-09-15T03:47:26.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.719 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.719 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947092+0000", 2024-09-15T03:47:26.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7800000000000002e-07 2024-09-15T03:47:26.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.719 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.719 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.720 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947095+0000", 2024-09-15T03:47:26.720 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2959999999999999e-06 2024-09-15T03:47:26.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.720 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.720 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947102+0000", 2024-09-15T03:47:26.720 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7469999999999999e-06 2024-09-15T03:47:26.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.720 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.720 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947106+0000", 2024-09-15T03:47:26.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6550000000000003e-06 2024-09-15T03:47:26.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947622+0000", 2024-09-15T03:47:26.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000515727 2024-09-15T03:47:26.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:26.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947751+0000", 2024-09-15T03:47:26.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012920600000000001 2024-09-15T03:47:26.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:26.722 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.964699+0000", 2024-09-15T03:47:26.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.016948123999999998 2024-09-15T03:47:26.722 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.722 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.722 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.722 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:519 3.3s1 3:caaee79c:::smithi00132172-10:head [read 0~3145728,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:26.722 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:26.722 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.689569745, 2024-09-15T03:47:26.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.021743243999999998, 2024-09-15T03:47:26.722 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.722 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:26.722 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:26.723 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:26.723 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.723 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 519 2024-09-15T03:47:26.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.723 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:26.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:26.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:26.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5999999999999999e-07 2024-09-15T03:47:26.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947130+0000", 2024-09-15T03:47:26.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7720000000000002e-06 2024-09-15T03:47:26.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947137+0000", 2024-09-15T03:47:26.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2370000000000003e-06 2024-09-15T03:47:26.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947141+0000", 2024-09-15T03:47:26.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8809999999999998e-06 2024-09-15T03:47:26.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.948008+0000", 2024-09-15T03:47:26.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00086656700000000005 2024-09-15T03:47:26.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.726 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:26.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.948113+0000", 2024-09-15T03:47:26.726 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104904 2024-09-15T03:47:26.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.726 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:26.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.968868+0000", 2024-09-15T03:47:26.726 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.020755422999999999 2024-09-15T03:47:26.726 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.726 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.726 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.727 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.3s1 69/68 ECSubReadReply(tid=162, attrs_read=0))", 2024-09-15T03:47:26.727 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.948615+0000", 2024-09-15T03:47:26.727 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.688079163999999, 2024-09-15T03:47:26.727 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010628187000000001, 2024-09-15T03:47:26.727 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.727 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:26.727 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.948615+0000", 2024-09-15T03:47:26.727 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.728 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.948615+0000", 2024-09-15T03:47:26.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.728 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.728 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.728 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.948616+0000", 2024-09-15T03:47:26.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9699999999999996e-07 2024-09-15T03:47:26.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.728 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.728 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.728 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953599+0000", 2024-09-15T03:47:26.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049830919999999997 2024-09-15T03:47:26.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.729 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.729 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953642+0000", 2024-09-15T03:47:26.729 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3084999999999998e-05 2024-09-15T03:47:26.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.729 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.729 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953650+0000", 2024-09-15T03:47:26.729 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0930000000000003e-06 2024-09-15T03:47:26.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.729 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.729 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.954573+0000", 2024-09-15T03:47:26.730 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00092271499999999995 2024-09-15T03:47:26.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.730 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.730 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:26.730 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.959244+0000", 2024-09-15T03:47:26.730 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0046707049999999998 2024-09-15T03:47:26.730 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.730 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.730 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.730 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.730 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.3s1 69/68 ECSubReadReply(tid=163, attrs_read=0))", 2024-09-15T03:47:26.730 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:26.731 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.682994001000001, 2024-09-15T03:47:26.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011102266, 2024-09-15T03:47:26.731 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.731 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:26.731 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:26.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:26.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.732 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:26.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1199999999999998e-07 2024-09-15T03:47:26.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.732 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956300+0000", 2024-09-15T03:47:26.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025987459999999999 2024-09-15T03:47:26.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956314+0000", 2024-09-15T03:47:26.733 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4469e-05 2024-09-15T03:47:26.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956319+0000", 2024-09-15T03:47:26.733 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.746e-06 2024-09-15T03:47:26.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.959591+0000", 2024-09-15T03:47:26.733 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032721510000000001 2024-09-15T03:47:26.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.734 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.734 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:26.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.964803+0000", 2024-09-15T03:47:26.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0052117420000000001 2024-09-15T03:47:26.734 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.734 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.734 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.734 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.734 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.3s1 69/68 ECSubReadReply(tid=164, attrs_read=0))", 2024-09-15T03:47:26.734 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:26.734 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.680369568, 2024-09-15T03:47:26.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012633276000000001, 2024-09-15T03:47:26.735 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.735 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:26.735 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.735 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.735 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.735 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:26.735 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.735 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.735 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.735 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.735 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:26.735 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.735 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.736 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.736 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.736 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:26.736 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5900000000000003e-07 2024-09-15T03:47:26.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.736 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.736 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.736 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957786+0000", 2024-09-15T03:47:26.736 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001460755 2024-09-15T03:47:26.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.736 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.736 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.736 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957807+0000", 2024-09-15T03:47:26.736 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0937e-05 2024-09-15T03:47:26.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.737 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.737 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957814+0000", 2024-09-15T03:47:26.737 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5989999999999998e-06 2024-09-15T03:47:26.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.737 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.737 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.964911+0000", 2024-09-15T03:47:26.737 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0070976370000000004 2024-09-15T03:47:26.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.737 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.737 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:26.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.968958+0000", 2024-09-15T03:47:26.738 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.004046989 2024-09-15T03:47:26.738 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.738 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.738 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.738 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:567 3.ds0 3:b166213b:::smithi00132172-21:head [rollback 3] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:26.738 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.947576+0000", 2024-09-15T03:47:26.738 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.6891186400000002, 2024-09-15T03:47:26.738 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013871319999999999, 2024-09-15T03:47:26.738 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.738 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:26.738 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:26.739 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:26.739 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.739 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 567 2024-09-15T03:47:26.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.739 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.739 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.739 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.739 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947576+0000", 2024-09-15T03:47:26.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.739 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.739 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.739 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947576+0000", 2024-09-15T03:47:26.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.740 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947577+0000", 2024-09-15T03:47:26.740 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.37e-07 2024-09-15T03:47:26.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.740 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947580+0000", 2024-09-15T03:47:26.740 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5710000000000002e-06 2024-09-15T03:47:26.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.740 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947588+0000", 2024-09-15T03:47:26.741 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8939999999999993e-06 2024-09-15T03:47:26.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.741 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.741 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.741 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947593+0000", 2024-09-15T03:47:26.741 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7049999999999998e-06 2024-09-15T03:47:26.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.741 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.741 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.741 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948070+0000", 2024-09-15T03:47:26.741 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047685099999999998 2024-09-15T03:47:26.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.741 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.741 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:26.742 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948177+0000", 2024-09-15T03:47:26.742 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010765299999999999 2024-09-15T03:47:26.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.742 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.742 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.742 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948986+0000", 2024-09-15T03:47:26.742 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000808977 2024-09-15T03:47:26.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.742 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.742 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.742 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949117+0000", 2024-09-15T03:47:26.742 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000131032 2024-09-15T03:47:26.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.742 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949133+0000", 2024-09-15T03:47:26.743 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5798999999999999e-05 2024-09-15T03:47:26.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.743 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950936+0000", 2024-09-15T03:47:26.743 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00180316 2024-09-15T03:47:26.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.743 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950994+0000", 2024-09-15T03:47:26.743 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7884999999999999e-05 2024-09-15T03:47:26.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.744 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.744 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:26.744 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951532+0000", 2024-09-15T03:47:26.744 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00053787699999999997 2024-09-15T03:47:26.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.744 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.744 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:26.744 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961447+0000", 2024-09-15T03:47:26.744 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0099152790000000008 2024-09-15T03:47:26.744 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.744 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.744 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.744 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.745 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:568 3.ds0 3:b166213b:::smithi00132172-21:head [append 0~0] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:26.745 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.948048+0000", 2024-09-15T03:47:26.745 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.688646715, 2024-09-15T03:47:26.745 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013579269, 2024-09-15T03:47:26.745 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:26.745 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:26.745 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:26.745 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:26.745 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:26.745 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 568 2024-09-15T03:47:26.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.745 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:26.745 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.745 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:26.746 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948048+0000", 2024-09-15T03:47:26.746 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.746 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.746 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:26.746 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948048+0000", 2024-09-15T03:47:26.746 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:26.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.746 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.746 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:26.746 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948049+0000", 2024-09-15T03:47:26.746 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7199999999999998e-07 2024-09-15T03:47:26.747 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.747 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.747 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:26.747 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948056+0000", 2024-09-15T03:47:26.747 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0280000000000003e-06 2024-09-15T03:47:26.747 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.747 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.747 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:26.747 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948066+0000", 2024-09-15T03:47:26.748 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0141e-05 2024-09-15T03:47:26.748 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.748 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.748 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:26.748 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948072+0000", 2024-09-15T03:47:26.748 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1519999999999997e-06 2024-09-15T03:47:26.748 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.748 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.748 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:26.748 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949221+0000", 2024-09-15T03:47:26.748 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001149125 2024-09-15T03:47:26.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.749 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.749 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:26.749 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949313+0000", 2024-09-15T03:47:26.749 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1469000000000001e-05 2024-09-15T03:47:26.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.749 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.749 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:26.749 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949818+0000", 2024-09-15T03:47:26.749 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00050549700000000004 2024-09-15T03:47:26.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.749 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.749 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:26.750 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950002+0000", 2024-09-15T03:47:26.750 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018414700000000001 2024-09-15T03:47:26.750 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.750 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.750 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:26.750 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950940+0000", 2024-09-15T03:47:26.750 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00093786100000000001 2024-09-15T03:47:26.750 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.750 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.750 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:26.750 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951004+0000", 2024-09-15T03:47:26.750 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3859e-05 2024-09-15T03:47:26.750 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.750 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.750 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:26.751 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951020+0000", 2024-09-15T03:47:26.751 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.628e-05 2024-09-15T03:47:26.751 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.751 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.751 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:26.751 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951753+0000", 2024-09-15T03:47:26.751 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00073276999999999999 2024-09-15T03:47:26.751 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:26.751 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:26.751 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:26.751 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961627+0000", 2024-09-15T03:47:26.752 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098741679999999991 2024-09-15T03:47:26.752 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.752 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.752 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.752 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:26.752 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:26.752 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:26.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:47:26.754 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:26.753+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:26.854 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:26.854+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:26.891 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:26.891 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:26.891 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:26.891 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:26.899 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:47:26.904 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:47:26.905 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":79,"fsid":"f27b540e-e5e3-4e95-afe8-c95ee66b0a62","created":"2024-09-15T03:46:09.843094+0000","modified":"2024-09-15T03:47:25.951035+0000","last_up_change":"2024-09-15T03:46:15.871985+0000","last_in_change":"2024-09-15T03:47:13.897657+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:46:16.185530+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:46:22.284406+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":24,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-09-15T03:46:34.105566+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":8,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":45,"snap_epoch":79,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lrcprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"cd433a6a-c875-4e6e-b89c-938caecbae3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":28908},{"type":"v1","addr":"172.21.15.1:6801","nonce":28908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":28908},{"type":"v1","addr":"172.21.15.1:6803","nonce":28908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":28908},{"type":"v1","addr":"172.21.15.1:6807","nonce":28908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":28908},{"type":"v1","addr":"172.21.15.1:6805","nonce":28908}]},"public_addr":"172.21.15.1:6801/28908","cluster_addr":"172.21.15.1:6803/28908","heartbeat_back_addr":"172.21.15.1:6807/28908","heartbeat_front_addr":"172.21.15.1:6805/28908","state":["exists","up"]},{"osd":1,"uuid":"789930f9-42bf-476b-9f09-a21bffe33a79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":28913},{"type":"v1","addr":"172.21.15.1:6809","nonce":28913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":28913},{"type":"v1","addr":"172.21.15.1:6811","nonce":28913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":28913},{"type":"v1","addr":"172.21.15.1:6815","nonce":28913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":28913},{"type":"v1","addr":"172.21.15.1:6813","nonce":28913}]},"public_addr":"172.21.15.1:6809/28913","cluster_addr":"172.21.15.1:6811/28913","heartbeat_back_addr":"172.21.15.1:6815/28913","heartbeat_front_addr":"172.21.15.1:6813/28913","state":["exists","up"]},{"osd":2,"uuid":"9a83fec5-0d63-461c-b516-289443b3882b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":28911},{"type":"v1","addr":"172.21.15.1:6823","nonce":28911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":28911},{"type":"v1","addr":"172.21.15.1:6827","nonce":28911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":28911},{"type":"v1","addr":"172.21.15.1:6831","nonce":28911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":28911},{"type":"v1","addr":"172.21.15.1:6829","nonce":28911}]},"public_addr":"172.21.15.1:6823/28911","cluster_addr":"172.21.15.1:6827/28911","heartbeat_back_addr":"172.21.15.1:6831/28911","heartbeat_front_addr":"172.21.15.1:6829/28911","state":["exists","up"]},{"osd":3,"uuid":"d944bf89-e183-496a-9ca3-7949741b4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":28914},{"type":"v1","addr":"172.21.15.1:6817","nonce":28914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":28914},{"type":"v1","addr":"172.21.15.1:6819","nonce":28914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":28914},{"type":"v1","addr":"172.21.15.1:6826","nonce":28914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":28914},{"type":"v1","addr":"172.21.15.1:6822","nonce":28914}]},"public_addr":"172.21.15.1:6817/28914","cluster_addr":"172.21.15.1:6819/28914","heartbeat_back_addr":"172.21.15.1:6826/28914","heartbeat_front_addr":"172.21.15.1:6822/28914","state":["exists","up"]},{"osd":4,"uuid":"15ec2dbd-d099-4aff-94b2-1bf9dbb00b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":28468},{"type":"v1","addr":"172.21.15.67:6805","nonce":28468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6807","nonce":28468},{"type":"v1","addr":"172.21.15.67:6809","nonce":28468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":28468},{"type":"v1","addr":"172.21.15.67:6815","nonce":28468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":28468},{"type":"v1","addr":"172.21.15.67:6813","nonce":28468}]},"public_addr":"172.21.15.67:6805/28468","cluster_addr":"172.21.15.67:6809/28468","heartbeat_back_addr":"172.21.15.67:6815/28468","heartbeat_front_addr":"172.21.15.67:6813/28468","state":["exists","up"]},{"osd":5,"uuid":"1e45de07-746b-4612-9a52-d05509cae9a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":28495},{"type":"v1","addr":"172.21.15.67:6825","nonce":28495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":28495},{"type":"v1","addr":"172.21.15.67:6827","nonce":28495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":28495},{"type":"v1","addr":"172.21.15.67:6831","nonce":28495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":28495},{"type":"v1","addr":"172.21.15.67:6829","nonce":28495}]},"public_addr":"172.21.15.67:6825/28495","cluster_addr":"172.21.15.67:6827/28495","heartbeat_back_addr":"172.21.15.67:6831/28495","heartbeat_front_addr":"172.21.15.67:6829/28495","state":["exists","up"]},{"osd":6,"uuid":"ccf785f8-751d-473e-b17c-3df9346be9f8","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":28469},{"type":"v1","addr":"172.21.15.67:6801","nonce":28469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":28469},{"type":"v1","addr":"172.21.15.67:6804","nonce":28469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":28469},{"type":"v1","addr":"172.21.15.67:6812","nonce":28469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":28469},{"type":"v1","addr":"172.21.15.67:6808","nonce":28469}]},"public_addr":"172.21.15.67:6801/28469","cluster_addr":"172.21.15.67:6804/28469","heartbeat_back_addr":"172.21.15.67:6812/28469","heartbeat_front_addr":"172.21.15.67:6808/28469","state":["exists","up"]},{"osd":7,"uuid":"f9f59987-869d-42eb-9eb6-08c39e3afe6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":28470},{"type":"v1","addr":"172.21.15.67:6817","nonce":28470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":28470},{"type":"v1","addr":"172.21.15.67:6819","nonce":28470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":28470},{"type":"v1","addr":"172.21.15.67:6823","nonce":28470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":28470},{"type":"v1","addr":"172.21.15.67:6821","nonce":28470}]},"public_addr":"172.21.15.67:6817/28470","cluster_addr":"172.21.15.67:6819/28470","heartbeat_back_addr":"172.21.15.67:6823/28470","heartbeat_front_addr":"172.21.15.67:6821/28470","state":["exists","up"]},{"osd":8,"uuid":"69ddfe52-11df-4608-9a10-e8907c4cd87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":29472},{"type":"v1","addr":"172.21.15.190:6809","nonce":29472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":29472},{"type":"v1","addr":"172.21.15.190:6811","nonce":29472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":29472},{"type":"v1","addr":"172.21.15.190:6815","nonce":29472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":29472},{"type":"v1","addr":"172.21.15.190:6813","nonce":29472}]},"public_addr":"172.21.15.190:6809/29472","cluster_addr":"172.21.15.190:6811/29472","heartbeat_back_addr":"172.21.15.190:6815/29472","heartbeat_front_addr":"172.21.15.190:6813/29472","state":["exists","up"]},{"osd":9,"uuid":"f92e1609-2a1a-4814-9ac0-44a73373d229","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":29473},{"type":"v1","addr":"172.21.15.190:6801","nonce":29473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":29473},{"type":"v1","addr":"172.21.15.190:6803","nonce":29473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":29473},{"type":"v1","addr":"172.21.15.190:6807","nonce":29473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":29473},{"type":"v1","addr":"172.21.15.190:6805","nonce":29473}]},"public_addr":"172.21.15.190:6801/29473","cluster_addr":"172.21.15.190:6803/29473","heartbeat_back_addr":"172.21.15.190:6807/29473","heartbeat_front_addr":"172.21.15.190:6805/29473","state":["exists","up"]},{"osd":10,"uuid":"5f2e8ca3-8c65-4595-8e82-27dc3768ab99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":29474},{"type":"v1","addr":"172.21.15.190:6817","nonce":29474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":29474},{"type":"v1","addr":"172.21.15.190:6819","nonce":29474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":29474},{"type":"v1","addr":"172.21.15.190:6823","nonce":29474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":29474},{"type":"v1","addr":"172.21.15.190:6821","nonce":29474}]},"public_addr":"172.21.15.190:6817/29474","cluster_addr":"172.21.15.190:6819/29474","heartbeat_back_addr":"172.21.15.190:6823/29474","heartbeat_front_addr":"172.21.15.190:6821/29474","state":["exists","up"]},{"osd":11,"uuid":"cedf7df4-f875-42f5-a8cf-13cb82f05cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":29475},{"type":"v1","addr":"172.21.15.190:6825","nonce":29475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":29475},{"type":"v1","addr":"172.21.15.190:6827","nonce":29475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":29475},{"type":"v1","addr":"172.21.15.190:6831","nonce":29475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":29475},{"type":"v1","addr":"172.21.15.190:6829","nonce":29475}]},"public_addr":"172.21.15.190:6825/29475","cluster_addr":"172.21.15.190:6827/29475","heartbeat_back_addr":"172.21.15.190:6831/29475","heartbeat_front_addr":"172.21.15.190:6829/29475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.943165+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:18.975241+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.981045+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:14.019932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:25.908552+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.982429+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-15T03:46:13.953833+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:18.853134+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.996438+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.926618+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:13.993934+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:46:14.022484+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.a","osds":[4,10,8,9,5,6,2,0]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lrcprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","l":"3","m":"2","name":"lrcprofile","plugin":"lrc"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":39,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:47:26.917 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-15T03:46:16.185530+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-15T03:46:22.284406+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 24, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-15T03:46:34.105566+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 8, 'min_size': 5, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 45, 'snap_epoch': 79, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'lrcprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 16384, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}] 2024-09-15T03:47:26.917 DEBUG:teuthology.orchestra.run.smithi190:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --op apply-layout-settings --pool .mgr 2024-09-15T03:47:26.964 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-15T03:47:26.965 INFO:tasks.rados.rados.0.smithi001.stdout:248: done (1 left) 2024-09-15T03:47:26.965 INFO:tasks.rados.rados.0.smithi001.stdout:249: setattr oid 10 current snap is 31 2024-09-15T03:47:26.965 INFO:tasks.rados.rados.0.smithi001.stdout:250: read oid 27 snap 14 2024-09-15T03:47:26.965 INFO:tasks.rados.rados.0.smithi001.stdout:250: expect (ObjNum 77 snap 12 seq_num 77) 2024-09-15T03:47:26.965 INFO:tasks.rados.rados.0.smithi001.stdout:251: copy_from oid 20 from oid 28 current snap is 31 2024-09-15T03:47:26.965 INFO:tasks.rados.rados.0.smithi001.stdout:252: copy_from oid 8 from oid 2 current snap is 31 2024-09-15T03:47:26.966 INFO:tasks.rados.rados.0.smithi001.stdout:253: delete oid 49 current snap is 31 2024-09-15T03:47:26.987 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:26.986+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:26.995 INFO:teuthology.orchestra.run.smithi190.stderr:Finished 0/0 collections 2024-09-15T03:47:26.999 DEBUG:teuthology.orchestra.run.smithi190:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --op apply-layout-settings --pool rbd 2024-09-15T03:47:27.007 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:27.007 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:27.007 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:27.008 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:27.008 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:27.016 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:47:27.052 INFO:teuthology.orchestra.run.smithi190.stderr:Finished 0/0 collections 2024-09-15T03:47:27.056 DEBUG:teuthology.orchestra.run.smithi190:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --op apply-layout-settings --pool unique_pool_0 2024-09-15T03:47:27.056 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:27.055+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:27.143 INFO:teuthology.orchestra.run.smithi190.stderr:Finished 0/14 collections Finished 1/14 collections Finished 2/14 collections Finished 3/14 collections Finished 4/14 collections Finished 5/14 collections Finished 6/14 collections Finished 7/14 collections Finished 8/14 collections Finished 9/14 collections Finished 10/14 collections Finished 11/14 collections Finished 12/14 collections Finished 13/14 collections Finished 14/14 collections 2024-09-15T03:47:27.154 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:27.154 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:27.154 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:27.154 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:47:27.155 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.155 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:47:27.155 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:27.155 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 40.981523168000002, 2024-09-15T03:47:27.155 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012183955999999999, 2024-09-15T03:47:27.155 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:27.155 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:27.155 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:27.155 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.155 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:27.155 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:27.155 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.156 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:27.156 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:27.156 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.156 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:27.156 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:27.156 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.17e-07 2024-09-15T03:47:27.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.156 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:27.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171454+0000", 2024-09-15T03:47:27.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00035226700000000002 2024-09-15T03:47:27.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:27.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171465+0000", 2024-09-15T03:47:27.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0943e-05 2024-09-15T03:47:27.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:27.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171471+0000", 2024-09-15T03:47:27.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7400000000000001e-06 2024-09-15T03:47:27.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.158 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:27.158 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171508+0000", 2024-09-15T03:47:27.158 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7147999999999997e-05 2024-09-15T03:47:27.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.158 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.158 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:27.158 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171517+0000", 2024-09-15T03:47:27.158 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2560000000000008e-06 2024-09-15T03:47:27.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.158 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.158 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:27.158 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171716+0000", 2024-09-15T03:47:27.159 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019899699999999999 2024-09-15T03:47:27.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.159 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.159 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:27.159 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173686+0000", 2024-09-15T03:47:27.159 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019696829999999999 2024-09-15T03:47:27.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.159 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.159 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:27.159 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173777+0000", 2024-09-15T03:47:27.159 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0637999999999995e-05 2024-09-15T03:47:27.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.159 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.160 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:27.160 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173814+0000", 2024-09-15T03:47:27.160 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7499000000000001e-05 2024-09-15T03:47:27.160 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.160 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.160 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:27.160 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.183286+0000", 2024-09-15T03:47:27.160 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094716680000000008 2024-09-15T03:47:27.160 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.160 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:27.160 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.160 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.160 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.161 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 51/31 ECSubWrite(tid=67, reqid=client.4521.0:350, at_version=51'33, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:27.161 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.843852+0000", 2024-09-15T03:47:27.161 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 29.308773207000002, 2024-09-15T03:47:27.161 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010895471, 2024-09-15T03:47:27.161 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:27.161 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:27.161 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:27.161 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.161 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:27.161 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843852+0000", 2024-09-15T03:47:27.161 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.161 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.162 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:27.162 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843852+0000", 2024-09-15T03:47:27.162 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.162 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.162 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:27.162 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843853+0000", 2024-09-15T03:47:27.162 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.359e-06 2024-09-15T03:47:27.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.162 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.162 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:27.162 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843882+0000", 2024-09-15T03:47:27.162 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8711999999999999e-05 2024-09-15T03:47:27.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.163 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:27.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843901+0000", 2024-09-15T03:47:27.163 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9477e-05 2024-09-15T03:47:27.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.163 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:27.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843911+0000", 2024-09-15T03:47:27.163 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2590000000000006e-06 2024-09-15T03:47:27.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.163 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:27.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843973+0000", 2024-09-15T03:47:27.164 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2310999999999993e-05 2024-09-15T03:47:27.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.164 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:27.164 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843984+0000", 2024-09-15T03:47:27.164 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.132e-05 2024-09-15T03:47:27.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.164 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:27.164 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844108+0000", 2024-09-15T03:47:27.164 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012384399999999999 2024-09-15T03:47:27.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.165 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:27.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844139+0000", 2024-09-15T03:47:27.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1148000000000001e-05 2024-09-15T03:47:27.165 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.165 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.165 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:27.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844185+0000", 2024-09-15T03:47:27.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5388e-05 2024-09-15T03:47:27.165 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.165 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.165 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:27.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844202+0000", 2024-09-15T03:47:27.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7708999999999999e-05 2024-09-15T03:47:27.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.166 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:27.166 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.854747+0000", 2024-09-15T03:47:27.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010544944000000001 2024-09-15T03:47:27.166 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.166 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:27.166 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.166 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s7 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:27.166 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844831+0000", 2024-09-15T03:47:27.166 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 29.307793566000001, 2024-09-15T03:47:27.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095609189999999993, 2024-09-15T03:47:27.167 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:27.167 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:27.167 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:27.167 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.167 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:27.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844831+0000", 2024-09-15T03:47:27.167 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.167 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.167 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.167 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:27.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844831+0000", 2024-09-15T03:47:27.167 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.167 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.167 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.168 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:27.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844832+0000", 2024-09-15T03:47:27.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1399999999999997e-07 2024-09-15T03:47:27.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.168 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:27.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844855+0000", 2024-09-15T03:47:27.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2904e-05 2024-09-15T03:47:27.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.168 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:27.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844866+0000", 2024-09-15T03:47:27.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0691e-05 2024-09-15T03:47:27.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:27.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844872+0000", 2024-09-15T03:47:27.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.409e-06 2024-09-15T03:47:27.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:27.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845030+0000", 2024-09-15T03:47:27.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015832800000000001 2024-09-15T03:47:27.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:27.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845037+0000", 2024-09-15T03:47:27.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1099999999999997e-06 2024-09-15T03:47:27.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.170 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.170 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:27.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845118+0000", 2024-09-15T03:47:27.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0840000000000005e-05 2024-09-15T03:47:27.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.170 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.170 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:27.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847478+0000", 2024-09-15T03:47:27.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023595869999999998 2024-09-15T03:47:27.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.171 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:27.171 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847529+0000", 2024-09-15T03:47:27.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1585000000000002e-05 2024-09-15T03:47:27.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.171 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:27.171 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847544+0000", 2024-09-15T03:47:27.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4365000000000001e-05 2024-09-15T03:47:27.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.171 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:27.171 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.854392+0000", 2024-09-15T03:47:27.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0068485860000000003 2024-09-15T03:47:27.172 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.172 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:27.172 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.172 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.172 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 51/31 ECSubWrite(tid=68, reqid=client.4521.0:351, at_version=51'34, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:27.172 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:27.172 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 29.307409386, 2024-09-15T03:47:27.172 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098406970000000007, 2024-09-15T03:47:27.172 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:27.172 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:27.172 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:27.172 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.173 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:27.173 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:27.173 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.173 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.173 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:27.173 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:27.173 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.173 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.173 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:27.173 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:27.173 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1299999999999999e-07 2024-09-15T03:47:27.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.174 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.174 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:27.174 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845232+0000", 2024-09-15T03:47:27.174 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5366000000000002e-05 2024-09-15T03:47:27.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.174 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.174 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:27.174 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845242+0000", 2024-09-15T03:47:27.174 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0403e-05 2024-09-15T03:47:27.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.174 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.174 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:27.174 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845247+0000", 2024-09-15T03:47:27.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4870000000000004e-06 2024-09-15T03:47:27.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:27.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845292+0000", 2024-09-15T03:47:27.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5762e-05 2024-09-15T03:47:27.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:27.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845302+0000", 2024-09-15T03:47:27.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5969999999999998e-06 2024-09-15T03:47:27.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:27.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845378+0000", 2024-09-15T03:47:27.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5628000000000002e-05 2024-09-15T03:47:27.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:27.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847479+0000", 2024-09-15T03:47:27.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021009380000000001 2024-09-15T03:47:27.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:27.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847530+0000", 2024-09-15T03:47:27.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1548000000000002e-05 2024-09-15T03:47:27.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.177 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:27.177 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847544+0000", 2024-09-15T03:47:27.177 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4355000000000001e-05 2024-09-15T03:47:27.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.177 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:27.177 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.855056+0000", 2024-09-15T03:47:27.177 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075119000000000002 2024-09-15T03:47:27.177 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.177 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:27.177 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.178 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.178 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 51/31 ECSubWrite(tid=93, reqid=client.4521.0:352, at_version=51'24, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:27.178 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:27.178 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 29.305345452000001, 2024-09-15T03:47:27.178 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0096136880000000004, 2024-09-15T03:47:27.178 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:27.178 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:27.178 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:27.178 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.178 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:27.178 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:27.178 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.178 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.179 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.179 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:27.179 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:27.179 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.179 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.179 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.179 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:27.179 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:27.179 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9100000000000003e-07 2024-09-15T03:47:27.179 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.179 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.179 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:27.179 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847669+0000", 2024-09-15T03:47:27.179 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00038855999999999998 2024-09-15T03:47:27.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.180 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.180 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:27.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847682+0000", 2024-09-15T03:47:27.180 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3112000000000001e-05 2024-09-15T03:47:27.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.180 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.180 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:27.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847687+0000", 2024-09-15T03:47:27.180 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0880000000000002e-06 2024-09-15T03:47:27.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.180 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.180 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:27.181 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847745+0000", 2024-09-15T03:47:27.181 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8400999999999999e-05 2024-09-15T03:47:27.181 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.181 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:27.181 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847756+0000", 2024-09-15T03:47:27.181 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0325e-05 2024-09-15T03:47:27.181 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.182 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:27.182 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.848092+0000", 2024-09-15T03:47:27.182 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033636099999999997 2024-09-15T03:47:27.182 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.182 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.182 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:27.182 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.848101+0000", 2024-09-15T03:47:27.182 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6240000000000008e-06 2024-09-15T03:47:27.182 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.182 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.182 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:27.182 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850132+0000", 2024-09-15T03:47:27.182 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020309239999999999 2024-09-15T03:47:27.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.183 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.183 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:27.183 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850174+0000", 2024-09-15T03:47:27.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2630999999999999e-05 2024-09-15T03:47:27.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.183 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.183 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:27.183 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856893+0000", 2024-09-15T03:47:27.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0067188710000000004 2024-09-15T03:47:27.183 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.183 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:27.183 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.184 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 60/28 ECSubWrite(tid=73, reqid=client.4521.0:424, at_version=60'41, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:27.184 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.900505+0000", 2024-09-15T03:47:27.184 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 20.252119788999998, 2024-09-15T03:47:27.184 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011912449, 2024-09-15T03:47:27.184 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:27.184 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:27.184 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:27.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.184 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:27.184 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900505+0000", 2024-09-15T03:47:27.184 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.185 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.185 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:27.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900505+0000", 2024-09-15T03:47:27.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.185 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.185 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:27.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900506+0000", 2024-09-15T03:47:27.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0800000000000003e-07 2024-09-15T03:47:27.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.185 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.185 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:27.186 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900527+0000", 2024-09-15T03:47:27.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0682999999999998e-05 2024-09-15T03:47:27.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.186 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.186 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:27.186 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900541+0000", 2024-09-15T03:47:27.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4217e-05 2024-09-15T03:47:27.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.186 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.186 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:27.186 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900547+0000", 2024-09-15T03:47:27.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3010000000000003e-06 2024-09-15T03:47:27.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.186 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:27.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900596+0000", 2024-09-15T03:47:27.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8612000000000001e-05 2024-09-15T03:47:27.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:27.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900607+0000", 2024-09-15T03:47:27.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.063e-05 2024-09-15T03:47:27.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:27.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900714+0000", 2024-09-15T03:47:27.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000107321 2024-09-15T03:47:27.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:27.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900721+0000", 2024-09-15T03:47:27.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1709999999999996e-06 2024-09-15T03:47:27.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:27.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900753+0000", 2024-09-15T03:47:27.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1565e-05 2024-09-15T03:47:27.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.189 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:27.189 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900766+0000", 2024-09-15T03:47:27.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3406e-05 2024-09-15T03:47:27.189 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.189 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.189 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:27.189 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.912418+0000", 2024-09-15T03:47:27.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011651635 2024-09-15T03:47:27.189 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.189 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:27.189 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.189 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.189 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.190 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 60/28 ECSubWrite(tid=74, reqid=client.4521.0:425, at_version=60'42, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:27.190 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.901156+0000", 2024-09-15T03:47:27.190 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 20.251469037, 2024-09-15T03:47:27.190 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011338225, 2024-09-15T03:47:27.190 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:27.190 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:27.190 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:27.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.190 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:27.190 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901156+0000", 2024-09-15T03:47:27.190 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.190 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.190 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:27.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901156+0000", 2024-09-15T03:47:27.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:27.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901157+0000", 2024-09-15T03:47:27.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9400000000000005e-07 2024-09-15T03:47:27.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:27.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901168+0000", 2024-09-15T03:47:27.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1924e-05 2024-09-15T03:47:27.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:27.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901178+0000", 2024-09-15T03:47:27.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7040000000000006e-06 2024-09-15T03:47:27.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:27.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901182+0000", 2024-09-15T03:47:27.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3080000000000001e-06 2024-09-15T03:47:27.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:27.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901226+0000", 2024-09-15T03:47:27.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3436999999999998e-05 2024-09-15T03:47:27.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:27.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901235+0000", 2024-09-15T03:47:27.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9400000000000008e-06 2024-09-15T03:47:27.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:27.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901329+0000", 2024-09-15T03:47:27.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3686999999999999e-05 2024-09-15T03:47:27.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:27.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901337+0000", 2024-09-15T03:47:27.194 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0129999999999993e-06 2024-09-15T03:47:27.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.194 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:27.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903268+0000", 2024-09-15T03:47:27.194 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001931545 2024-09-15T03:47:27.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.194 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:27.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903279+0000", 2024-09-15T03:47:27.194 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0626e-05 2024-09-15T03:47:27.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.195 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.195 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:27.195 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.912494+0000", 2024-09-15T03:47:27.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092154469999999999 2024-09-15T03:47:27.195 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.195 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:27.195 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.195 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.195 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 60/31 ECSubWrite(tid=129, reqid=client.4521.0:422, at_version=60'41, trim_to=0'0, roll_forward_to=60'39))", 2024-09-15T03:47:27.195 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.901373+0000", 2024-09-15T03:47:27.195 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 20.25125186, 2024-09-15T03:47:27.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013085485000000001, 2024-09-15T03:47:27.195 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:27.196 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:27.196 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:27.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.196 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:27.196 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901373+0000", 2024-09-15T03:47:27.196 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.196 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:27.196 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901373+0000", 2024-09-15T03:47:27.196 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:27.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901374+0000", 2024-09-15T03:47:27.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5600000000000005e-07 2024-09-15T03:47:27.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:27.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901713+0000", 2024-09-15T03:47:27.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033933299999999998 2024-09-15T03:47:27.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:27.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901726+0000", 2024-09-15T03:47:27.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2843999999999999e-05 2024-09-15T03:47:27.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:27.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901730+0000", 2024-09-15T03:47:27.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4660000000000001e-06 2024-09-15T03:47:27.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:27.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901773+0000", 2024-09-15T03:47:27.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2542e-05 2024-09-15T03:47:27.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:27.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901783+0000", 2024-09-15T03:47:27.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.041e-05 2024-09-15T03:47:27.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.199 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:27.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906331+0000", 2024-09-15T03:47:27.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0045479049999999997 2024-09-15T03:47:27.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.199 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:27.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906340+0000", 2024-09-15T03:47:27.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8920000000000006e-06 2024-09-15T03:47:27.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.200 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:27.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906403+0000", 2024-09-15T03:47:27.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2793999999999995e-05 2024-09-15T03:47:27.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.200 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:27.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906413+0000", 2024-09-15T03:47:27.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0370999999999999e-05 2024-09-15T03:47:27.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.200 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:27.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.914459+0000", 2024-09-15T03:47:27.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0080452719999999991 2024-09-15T03:47:27.201 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.201 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:27.201 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.201 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.201 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.201 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.fs7 65/64 ECSubWrite(tid=108, reqid=client.4521.0:466, at_version=65'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:27.201 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:11.853993+0000", 2024-09-15T03:47:27.201 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.298631672000001, 2024-09-15T03:47:27.201 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013335351, 2024-09-15T03:47:27.201 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:27.201 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:27.201 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:27.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:27.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.853993+0000", 2024-09-15T03:47:27.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:27.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.853993+0000", 2024-09-15T03:47:27.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:27.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.853994+0000", 2024-09-15T03:47:27.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9999999999999997e-07 2024-09-15T03:47:27.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:27.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854199+0000", 2024-09-15T03:47:27.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020490999999999999 2024-09-15T03:47:27.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:27.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854210+0000", 2024-09-15T03:47:27.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0813e-05 2024-09-15T03:47:27.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:27.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854215+0000", 2024-09-15T03:47:27.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8270000000000003e-06 2024-09-15T03:47:27.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:27.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854264+0000", 2024-09-15T03:47:27.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8477999999999999e-05 2024-09-15T03:47:27.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:27.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854274+0000", 2024-09-15T03:47:27.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5729999999999997e-06 2024-09-15T03:47:27.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:27.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854563+0000", 2024-09-15T03:47:27.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000289208 2024-09-15T03:47:27.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:27.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.855280+0000", 2024-09-15T03:47:27.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00071722500000000005 2024-09-15T03:47:27.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:27.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.855441+0000", 2024-09-15T03:47:27.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016084199999999999 2024-09-15T03:47:27.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:27.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.855456+0000", 2024-09-15T03:47:27.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5123000000000001e-05 2024-09-15T03:47:27.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:27.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.867329+0000", 2024-09-15T03:47:27.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011872752 2024-09-15T03:47:27.206 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.206 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:27.206 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.207 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.ds6 74/70 ECSubWrite(tid=198, reqid=client.4521.0:566, at_version=74'32, trim_to=0'0, roll_forward_to=73'30))", 2024-09-15T03:47:27.207 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.944959+0000", 2024-09-15T03:47:27.207 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.2076664690000003, 2024-09-15T03:47:27.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010023647, 2024-09-15T03:47:27.207 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:27.207 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:27.207 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:27.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.207 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:27.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.944959+0000", 2024-09-15T03:47:27.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.208 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:27.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.944959+0000", 2024-09-15T03:47:27.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.208 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:27.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.944959+0000", 2024-09-15T03:47:27.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3099999999999998e-07 2024-09-15T03:47:27.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.208 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:27.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.944986+0000", 2024-09-15T03:47:27.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7297999999999999e-05 2024-09-15T03:47:27.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:27.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.944997+0000", 2024-09-15T03:47:27.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0441999999999999e-05 2024-09-15T03:47:27.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:27.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.945008+0000", 2024-09-15T03:47:27.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1687e-05 2024-09-15T03:47:27.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:27.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.945058+0000", 2024-09-15T03:47:27.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9410999999999998e-05 2024-09-15T03:47:27.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:27.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.945069+0000", 2024-09-15T03:47:27.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0812000000000001e-05 2024-09-15T03:47:27.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:27.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.945348+0000", 2024-09-15T03:47:27.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027976800000000001 2024-09-15T03:47:27.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:27.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.945362+0000", 2024-09-15T03:47:27.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3834e-05 2024-09-15T03:47:27.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:27.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.945658+0000", 2024-09-15T03:47:27.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029610200000000002 2024-09-15T03:47:27.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:27.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.945707+0000", 2024-09-15T03:47:27.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9029000000000001e-05 2024-09-15T03:47:27.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.212 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.212 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:27.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.954982+0000", 2024-09-15T03:47:27.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092748329999999997 2024-09-15T03:47:27.212 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.212 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:27.212 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.212 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.212 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.ds6 74/70 ECSubWrite(tid=200, reqid=client.4521.0:567, at_version=74'33, trim_to=0'0, roll_forward_to=74'32))", 2024-09-15T03:47:27.212 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.948998+0000", 2024-09-15T03:47:27.213 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.203627086, 2024-09-15T03:47:27.213 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011947362, 2024-09-15T03:47:27.213 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:27.213 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:27.213 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:27.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.213 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:27.213 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948998+0000", 2024-09-15T03:47:27.213 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.213 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.213 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:27.213 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948998+0000", 2024-09-15T03:47:27.213 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:27.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948999+0000", 2024-09-15T03:47:27.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.71e-07 2024-09-15T03:47:27.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:27.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949020+0000", 2024-09-15T03:47:27.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1488000000000002e-05 2024-09-15T03:47:27.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:27.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949047+0000", 2024-09-15T03:47:27.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6409000000000001e-05 2024-09-15T03:47:27.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.215 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.215 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:27.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949054+0000", 2024-09-15T03:47:27.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3830000000000003e-06 2024-09-15T03:47:27.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.215 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.215 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:27.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949111+0000", 2024-09-15T03:47:27.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7074999999999998e-05 2024-09-15T03:47:27.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.215 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:27.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949120+0000", 2024-09-15T03:47:27.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1549999999999996e-06 2024-09-15T03:47:27.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:27.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949214+0000", 2024-09-15T03:47:27.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4046e-05 2024-09-15T03:47:27.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:27.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949225+0000", 2024-09-15T03:47:27.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0567999999999999e-05 2024-09-15T03:47:27.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:27.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949303+0000", 2024-09-15T03:47:27.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8065999999999994e-05 2024-09-15T03:47:27.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:27.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949317+0000", 2024-09-15T03:47:27.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4035e-05 2024-09-15T03:47:27.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:27.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.960945+0000", 2024-09-15T03:47:27.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011628266 2024-09-15T03:47:27.218 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.218 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:27.218 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.218 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.218 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.9s4 74/70 ECSubWrite(tid=108, reqid=client.4521.0:569, at_version=74'33, trim_to=0'0, roll_forward_to=74'28))", 2024-09-15T03:47:27.218 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.949550+0000", 2024-09-15T03:47:27.218 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.2030751879999997, 2024-09-15T03:47:27.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012286695, 2024-09-15T03:47:27.218 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:27.218 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:27.219 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:27.219 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.219 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:27.219 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949550+0000", 2024-09-15T03:47:27.219 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.219 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.219 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:27.219 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949550+0000", 2024-09-15T03:47:27.219 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.219 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.219 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:27.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949551+0000", 2024-09-15T03:47:27.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9999999999999996e-07 2024-09-15T03:47:27.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:27.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949590+0000", 2024-09-15T03:47:27.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9560000000000001e-05 2024-09-15T03:47:27.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:27.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949611+0000", 2024-09-15T03:47:27.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1229e-05 2024-09-15T03:47:27.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.221 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.221 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:27.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949620+0000", 2024-09-15T03:47:27.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5140000000000001e-06 2024-09-15T03:47:27.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.221 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.221 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:27.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949670+0000", 2024-09-15T03:47:27.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9545000000000001e-05 2024-09-15T03:47:27.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.221 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.221 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:27.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949678+0000", 2024-09-15T03:47:27.222 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9260000000000006e-06 2024-09-15T03:47:27.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.222 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:27.222 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949987+0000", 2024-09-15T03:47:27.222 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030801899999999998 2024-09-15T03:47:27.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.222 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:27.222 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949997+0000", 2024-09-15T03:47:27.222 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0930999999999999e-05 2024-09-15T03:47:27.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.222 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:27.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950049+0000", 2024-09-15T03:47:27.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1384000000000001e-05 2024-09-15T03:47:27.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:27.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950063+0000", 2024-09-15T03:47:27.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4664000000000001e-05 2024-09-15T03:47:27.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:27.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961837+0000", 2024-09-15T03:47:27.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011773023000000001 2024-09-15T03:47:27.223 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.224 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:27.224 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.224 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.ds6 74/70 ECSubWrite(tid=201, reqid=client.4521.0:568, at_version=74'34, trim_to=0'0, roll_forward_to=74'32))", 2024-09-15T03:47:27.224 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.950074+0000", 2024-09-15T03:47:27.224 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.2025507170000003, 2024-09-15T03:47:27.224 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011003285, 2024-09-15T03:47:27.224 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:27.224 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:27.224 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:27.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:27.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950074+0000", 2024-09-15T03:47:27.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:27.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950074+0000", 2024-09-15T03:47:27.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:27.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950075+0000", 2024-09-15T03:47:27.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0999999999999997e-07 2024-09-15T03:47:27.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.226 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:27.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950095+0000", 2024-09-15T03:47:27.226 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0256000000000001e-05 2024-09-15T03:47:27.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.226 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:27.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950109+0000", 2024-09-15T03:47:27.226 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3325e-05 2024-09-15T03:47:27.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.226 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:27.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950123+0000", 2024-09-15T03:47:27.226 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3893000000000001e-05 2024-09-15T03:47:27.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:27.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950170+0000", 2024-09-15T03:47:27.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7496000000000001e-05 2024-09-15T03:47:27.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:27.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950178+0000", 2024-09-15T03:47:27.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4190000000000002e-06 2024-09-15T03:47:27.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:27.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950271+0000", 2024-09-15T03:47:27.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2961999999999995e-05 2024-09-15T03:47:27.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:27.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950281+0000", 2024-09-15T03:47:27.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0960000000000007e-06 2024-09-15T03:47:27.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:27.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950317+0000", 2024-09-15T03:47:27.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6529999999999998e-05 2024-09-15T03:47:27.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:27.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950326+0000", 2024-09-15T03:47:27.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1419999999999999e-06 2024-09-15T03:47:27.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:27.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961078+0000", 2024-09-15T03:47:27.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010751356 2024-09-15T03:47:27.229 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.229 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:27.229 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.230 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 75/70 ECSubWrite(tid=198, reqid=client.4521.0:593, at_version=75'53, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:27.230 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:21.946647+0000", 2024-09-15T03:47:27.230 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.205977517, 2024-09-15T03:47:27.230 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012484423, 2024-09-15T03:47:27.230 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:27.230 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:27.230 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:27.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.230 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:27.230 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946647+0000", 2024-09-15T03:47:27.230 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.230 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:27.231 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946647+0000", 2024-09-15T03:47:27.231 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.231 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.231 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.231 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:27.231 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946648+0000", 2024-09-15T03:47:27.231 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6000000000000003e-07 2024-09-15T03:47:27.231 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.231 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.231 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:27.231 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946690+0000", 2024-09-15T03:47:27.231 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1430000000000001e-05 2024-09-15T03:47:27.231 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.232 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.232 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:27.232 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946705+0000", 2024-09-15T03:47:27.232 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4851000000000001e-05 2024-09-15T03:47:27.232 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.232 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.232 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:27.232 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946711+0000", 2024-09-15T03:47:27.232 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1380000000000004e-06 2024-09-15T03:47:27.232 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.232 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.232 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:27.232 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946755+0000", 2024-09-15T03:47:27.233 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4512999999999999e-05 2024-09-15T03:47:27.233 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.233 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.233 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:27.233 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946764+0000", 2024-09-15T03:47:27.233 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.337e-06 2024-09-15T03:47:27.233 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.233 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.233 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:27.233 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946900+0000", 2024-09-15T03:47:27.233 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013671500000000001 2024-09-15T03:47:27.233 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.233 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.234 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:27.234 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946922+0000", 2024-09-15T03:47:27.234 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1848000000000002e-05 2024-09-15T03:47:27.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.234 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.234 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:27.234 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946994+0000", 2024-09-15T03:47:27.234 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1790000000000002e-05 2024-09-15T03:47:27.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.234 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.234 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:27.234 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947046+0000", 2024-09-15T03:47:27.234 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2565000000000002e-05 2024-09-15T03:47:27.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.235 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.235 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:27.235 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.959132+0000", 2024-09-15T03:47:27.235 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012085476 2024-09-15T03:47:27.235 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.235 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:27.235 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.235 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.235 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.235 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 75/70 ECSubWrite(tid=200, reqid=client.4521.0:594, at_version=75'54, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:27.235 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:21.947454+0000", 2024-09-15T03:47:27.235 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.2051706869999999, 2024-09-15T03:47:27.235 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012079137, 2024-09-15T03:47:27.236 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:27.236 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:27.236 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:27.236 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.236 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:27.236 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947454+0000", 2024-09-15T03:47:27.236 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.236 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.236 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.236 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:27.236 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947454+0000", 2024-09-15T03:47:27.236 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.236 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.237 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.237 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:27.237 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947455+0000", 2024-09-15T03:47:27.237 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9800000000000003e-07 2024-09-15T03:47:27.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.237 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.237 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:27.237 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947472+0000", 2024-09-15T03:47:27.237 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7084000000000001e-05 2024-09-15T03:47:27.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.237 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.237 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:27.237 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947486+0000", 2024-09-15T03:47:27.237 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3684e-05 2024-09-15T03:47:27.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.238 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.238 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:27.238 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947492+0000", 2024-09-15T03:47:27.238 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5830000000000003e-06 2024-09-15T03:47:27.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.238 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.238 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:27.238 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947534+0000", 2024-09-15T03:47:27.238 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2107000000000002e-05 2024-09-15T03:47:27.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.238 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.238 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:27.239 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947543+0000", 2024-09-15T03:47:27.239 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0850000000000003e-06 2024-09-15T03:47:27.239 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.239 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.239 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:27.239 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947657+0000", 2024-09-15T03:47:27.239 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000113465 2024-09-15T03:47:27.239 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.239 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.239 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:27.239 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947664+0000", 2024-09-15T03:47:27.239 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.525e-06 2024-09-15T03:47:27.239 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.239 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.240 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:27.240 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948013+0000", 2024-09-15T03:47:27.240 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034869399999999998 2024-09-15T03:47:27.240 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.240 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.240 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:27.240 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948031+0000", 2024-09-15T03:47:27.240 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7571000000000002e-05 2024-09-15T03:47:27.240 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.240 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.240 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:27.240 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.959533+0000", 2024-09-15T03:47:27.240 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011502741 2024-09-15T03:47:27.240 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.241 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:27.241 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.241 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.241 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.241 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s5 75/70 ECSubWrite(tid=201, reqid=client.4521.0:585, at_version=0'0, trim_to=0'0, roll_forward_to=75'33))", 2024-09-15T03:47:27.241 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:21.948682+0000", 2024-09-15T03:47:27.241 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.2039433610000003, 2024-09-15T03:47:27.241 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01048613, 2024-09-15T03:47:27.241 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:27.241 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:27.241 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:27.241 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.241 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:27.242 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948682+0000", 2024-09-15T03:47:27.242 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.242 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.242 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:27.242 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948682+0000", 2024-09-15T03:47:27.242 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.242 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.242 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:27.242 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948682+0000", 2024-09-15T03:47:27.242 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2200000000000004e-07 2024-09-15T03:47:27.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.243 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.243 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:27.243 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948690+0000", 2024-09-15T03:47:27.243 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9170000000000006e-06 2024-09-15T03:47:27.243 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.243 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.243 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:27.243 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948701+0000", 2024-09-15T03:47:27.243 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0414999999999999e-05 2024-09-15T03:47:27.243 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.243 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.243 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:27.243 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948708+0000", 2024-09-15T03:47:27.244 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2169999999999997e-06 2024-09-15T03:47:27.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.244 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.244 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:27.244 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948755+0000", 2024-09-15T03:47:27.244 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6968999999999997e-05 2024-09-15T03:47:27.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.244 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.244 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:27.244 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948760+0000", 2024-09-15T03:47:27.244 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.78e-06 2024-09-15T03:47:27.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.244 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.244 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:27.245 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948838+0000", 2024-09-15T03:47:27.245 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8143e-05 2024-09-15T03:47:27.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.245 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.245 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:27.245 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948853+0000", 2024-09-15T03:47:27.245 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5449999999999999e-05 2024-09-15T03:47:27.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.245 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.245 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:27.245 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948905+0000", 2024-09-15T03:47:27.245 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1422000000000003e-05 2024-09-15T03:47:27.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.246 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.246 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:27.246 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948919+0000", 2024-09-15T03:47:27.246 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4448e-05 2024-09-15T03:47:27.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.246 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.246 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:27.246 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.959168+0000", 2024-09-15T03:47:27.246 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010248747000000001 2024-09-15T03:47:27.246 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.246 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:27.246 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.247 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.247 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 75/70 ECSubWrite(tid=203, reqid=client.4521.0:594, at_version=0'0, trim_to=0'0, roll_forward_to=75'54))", 2024-09-15T03:47:27.247 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:21.949416+0000", 2024-09-15T03:47:27.247 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.2032093509999999, 2024-09-15T03:47:27.247 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010405688999999999, 2024-09-15T03:47:27.247 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:27.247 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:27.247 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:27.247 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.247 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:27.247 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949416+0000", 2024-09-15T03:47:27.247 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.247 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.247 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.248 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:27.248 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949416+0000", 2024-09-15T03:47:27.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.248 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.248 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:27.248 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949416+0000", 2024-09-15T03:47:27.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9200000000000001e-07 2024-09-15T03:47:27.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.248 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.248 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:27.248 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949426+0000", 2024-09-15T03:47:27.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0169999999999999e-05 2024-09-15T03:47:27.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.249 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.249 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:27.249 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949441+0000", 2024-09-15T03:47:27.249 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4168000000000001e-05 2024-09-15T03:47:27.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.249 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.249 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:27.249 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949451+0000", 2024-09-15T03:47:27.249 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0003000000000001e-05 2024-09-15T03:47:27.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.249 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.249 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:27.250 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949485+0000", 2024-09-15T03:47:27.250 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4489999999999997e-05 2024-09-15T03:47:27.250 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.250 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.250 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:27.250 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949492+0000", 2024-09-15T03:47:27.250 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4139999999999997e-06 2024-09-15T03:47:27.250 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.250 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.250 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:27.250 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949600+0000", 2024-09-15T03:47:27.250 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000107219 2024-09-15T03:47:27.250 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.250 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.251 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:27.251 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949614+0000", 2024-09-15T03:47:27.251 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4586999999999999e-05 2024-09-15T03:47:27.251 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.251 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.251 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:27.251 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949645+0000", 2024-09-15T03:47:27.251 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1000999999999997e-05 2024-09-15T03:47:27.251 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.251 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.251 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:27.251 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949660+0000", 2024-09-15T03:47:27.251 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4843e-05 2024-09-15T03:47:27.252 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.252 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.252 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:27.252 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.959821+0000", 2024-09-15T03:47:27.252 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010161201999999999 2024-09-15T03:47:27.252 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.252 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:27.252 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.252 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.252 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.252 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.0s2 75/70 ECSubWrite(tid=109, reqid=client.4521.0:587, at_version=75'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:27.252 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:21.950286+0000", 2024-09-15T03:47:27.252 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.202338557, 2024-09-15T03:47:27.253 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013522467999999999, 2024-09-15T03:47:27.253 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:27.253 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:27.253 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:27.253 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.253 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:27.253 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950286+0000", 2024-09-15T03:47:27.253 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.253 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.253 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.253 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:27.253 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950286+0000", 2024-09-15T03:47:27.253 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.254 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.254 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:27.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950287+0000", 2024-09-15T03:47:27.254 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.54e-07 2024-09-15T03:47:27.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.254 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.254 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:27.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950503+0000", 2024-09-15T03:47:27.254 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000216725 2024-09-15T03:47:27.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.254 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.254 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:27.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950517+0000", 2024-09-15T03:47:27.255 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3988999999999999e-05 2024-09-15T03:47:27.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.255 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:27.255 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950527+0000", 2024-09-15T03:47:27.255 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.302e-06 2024-09-15T03:47:27.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.255 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:27.255 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950577+0000", 2024-09-15T03:47:27.255 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0395000000000001e-05 2024-09-15T03:47:27.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.256 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:27.256 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950586+0000", 2024-09-15T03:47:27.256 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5080000000000005e-06 2024-09-15T03:47:27.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.256 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.256 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:27.256 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950919+0000", 2024-09-15T03:47:27.256 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033313599999999999 2024-09-15T03:47:27.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.256 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.256 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:27.256 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950927+0000", 2024-09-15T03:47:27.256 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2279999999999998e-06 2024-09-15T03:47:27.257 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.257 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.257 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:27.257 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.951002+0000", 2024-09-15T03:47:27.257 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4975000000000006e-05 2024-09-15T03:47:27.257 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.257 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.257 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:27.257 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.951026+0000", 2024-09-15T03:47:27.257 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4258000000000001e-05 2024-09-15T03:47:27.257 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.257 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.257 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:27.258 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.963809+0000", 2024-09-15T03:47:27.258 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012782798 2024-09-15T03:47:27.258 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.258 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:27.258 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.258 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.258 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.258 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.0s2 75/70 ECSubWrite(tid=110, reqid=client.4521.0:587, at_version=0'0, trim_to=0'0, roll_forward_to=75'28))", 2024-09-15T03:47:27.258 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:21.953245+0000", 2024-09-15T03:47:27.258 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.1993798990000002, 2024-09-15T03:47:27.258 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010716737, 2024-09-15T03:47:27.258 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:27.258 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:27.258 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:27.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:27.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953245+0000", 2024-09-15T03:47:27.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:27.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953245+0000", 2024-09-15T03:47:27.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:27.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953246+0000", 2024-09-15T03:47:27.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.096e-06 2024-09-15T03:47:27.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:27.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953256+0000", 2024-09-15T03:47:27.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.026e-05 2024-09-15T03:47:27.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:27.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953270+0000", 2024-09-15T03:47:27.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3293e-05 2024-09-15T03:47:27.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.261 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:27.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953275+0000", 2024-09-15T03:47:27.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4770000000000002e-06 2024-09-15T03:47:27.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.261 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.261 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:27.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953322+0000", 2024-09-15T03:47:27.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7108999999999999e-05 2024-09-15T03:47:27.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.261 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.261 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:27.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953332+0000", 2024-09-15T03:47:27.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9559999999999992e-06 2024-09-15T03:47:27.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.262 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.262 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:27.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953403+0000", 2024-09-15T03:47:27.262 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0653000000000003e-05 2024-09-15T03:47:27.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.262 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.262 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:27.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953410+0000", 2024-09-15T03:47:27.262 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1969999999999999e-06 2024-09-15T03:47:27.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.262 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.262 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:27.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953456+0000", 2024-09-15T03:47:27.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6220000000000001e-05 2024-09-15T03:47:27.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.263 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:27.263 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953471+0000", 2024-09-15T03:47:27.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5057e-05 2024-09-15T03:47:27.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.263 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:27.263 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.963962+0000", 2024-09-15T03:47:27.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010490419000000001 2024-09-15T03:47:27.263 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.263 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:27.263 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.264 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.264 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs7 75/70 ECSubWrite(tid=160, reqid=client.4521.0:595, at_version=75'87, trim_to=0'0, roll_forward_to=73'85))", 2024-09-15T03:47:27.264 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:21.968440+0000", 2024-09-15T03:47:27.264 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.1841847579999998, 2024-09-15T03:47:27.264 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010079088999999999, 2024-09-15T03:47:27.264 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:27.264 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:27.264 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:27.264 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.264 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:27.264 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.968440+0000", 2024-09-15T03:47:27.264 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:27.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.968440+0000", 2024-09-15T03:47:27.265 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:27.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:27.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.968440+0000", 2024-09-15T03:47:27.265 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1500000000000001e-07 2024-09-15T03:47:27.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:27.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969096+0000", 2024-09-15T03:47:27.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00065533300000000002 2024-09-15T03:47:27.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.266 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.266 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:27.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969112+0000", 2024-09-15T03:47:27.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5943999999999998e-05 2024-09-15T03:47:27.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.266 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.266 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:27.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969118+0000", 2024-09-15T03:47:27.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3659999999999997e-06 2024-09-15T03:47:27.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.266 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:27.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969171+0000", 2024-09-15T03:47:27.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2951000000000002e-05 2024-09-15T03:47:27.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.267 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:27.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969184+0000", 2024-09-15T03:47:27.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2536e-05 2024-09-15T03:47:27.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.267 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:27.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969454+0000", 2024-09-15T03:47:27.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027025600000000002 2024-09-15T03:47:27.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:27.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969463+0000", 2024-09-15T03:47:27.268 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6569999999999992e-06 2024-09-15T03:47:27.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:27.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969584+0000", 2024-09-15T03:47:27.268 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012004 2024-09-15T03:47:27.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:27.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969604+0000", 2024-09-15T03:47:27.269 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0678000000000001e-05 2024-09-15T03:47:27.269 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:27.269 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:27.269 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:27.269 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.978519+0000", 2024-09-15T03:47:27.269 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0089151130000000006 2024-09-15T03:47:27.269 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.269 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:27.269 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.269 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:27.269 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:27.269 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:27.271 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:47:27.271 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:27.270+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:27.288 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:47:27.371 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:27.371+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:27.407 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:27.408 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:27.408 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:27.409 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:27.409 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:27.409 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:27.416 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:47:27.422 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:27.422 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:27.436 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:27.436 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:27.450 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:27.451 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:27.465 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:27.466 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:27.472 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:27.471+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:27.480 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:27.481 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:27.496 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:27.496 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:27.511 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:27.511 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:27.528 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:27.528 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:27.542 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:27.542 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:27.542 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-15T03:47:27.554 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:27.554 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:27.555 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:27.555 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:27.555 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:27.557 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:27.557 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:27.560 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T03:47:27.561 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T03:47:27.563 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:47:27.607 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:27.606+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:27.659 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:27.659 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:27.668 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:47:27.712 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:27.711+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:27.769 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:27.769 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:27.774 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:27.773+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:27.777 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:47:27.875 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:27.874+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:27.919 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:27.919 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:27.927 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:47:27.977 INFO:tasks.rados.rados.0.smithi001.stdout:247: finishing rollback tid 0 to smithi00132172-21 2024-09-15T03:47:27.977 INFO:tasks.rados.rados.0.smithi001.stdout:247: finishing rollback tid 1 to smithi00132172-21 2024-09-15T03:47:27.977 INFO:tasks.rados.rados.0.smithi001.stdout:247: finishing rollback tid 2 to smithi00132172-21 2024-09-15T03:47:27.977 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 21 v 44 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-15T03:47:27.978 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:27.977+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:27.982 INFO:tasks.rados.rados.0.smithi001.stdout:251: finishing copy_from to smithi00132172-20 2024-09-15T03:47:27.982 INFO:tasks.rados.rados.0.smithi001.stdout:251: got expected ENOENT (src dne) 2024-09-15T03:47:27.982 INFO:tasks.rados.rados.0.smithi001.stdout:251: finishing copy_from racing read to smithi00132172-20 2024-09-15T03:47:27.984 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 10 v 85 (ObjNum 96 snap 21 seq_num 96) dirty exists 2024-09-15T03:47:27.985 INFO:tasks.rados.rados.0.smithi001.stdout:247: done (5 left) 2024-09-15T03:47:27.985 INFO:tasks.rados.rados.0.smithi001.stdout:249: done (4 left) 2024-09-15T03:47:27.985 INFO:tasks.rados.rados.0.smithi001.stdout:251: done (3 left) 2024-09-15T03:47:27.985 INFO:tasks.rados.rados.0.smithi001.stdout:253: done (2 left) 2024-09-15T03:47:27.985 INFO:tasks.rados.rados.0.smithi001.stdout:254: read oid 11 snap 14 2024-09-15T03:47:27.985 INFO:tasks.rados.rados.0.smithi001.stdout:254: expect (ObjNum 78 snap 13 seq_num 78) 2024-09-15T03:47:27.985 INFO:tasks.rados.rados.0.smithi001.stdout:255: snap_create 2024-09-15T03:47:27.991 INFO:tasks.rados.rados.0.smithi001.stdout:252: finishing copy_from racing read to smithi00132172-8 2024-09-15T03:47:27.992 INFO:tasks.rados.rados.0.smithi001.stdout:252: finishing copy_from to smithi00132172-8 2024-09-15T03:47:27.992 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 8 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-15T03:47:27.998 INFO:tasks.rados.rados.0.smithi001.stdout:254: expect (ObjNum 78 snap 13 seq_num 78) 2024-09-15T03:47:28.044 INFO:tasks.rados.rados.0.smithi001.stdout:250: expect (ObjNum 77 snap 12 seq_num 77) 2024-09-15T03:47:28.076 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:28.076+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:28.084 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:28.084 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:28.093 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:47:28.178 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:28.177+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:28.235 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:28.235 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:28.243 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:47:28.285 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:28.284+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:28.380 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:28.380+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:28.383 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:28.384 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:28.392 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:47:28.481 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:28.481+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:28.534 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:28.534 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:28.542 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:47:28.582 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:28.582+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:28.683 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:28.683 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:28.683 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:28.682+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:28.691 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:47:28.784 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:28.783+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:28.791 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:28.791 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:28.800 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:47:28.884 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:28.884+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:28.941 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:28.941 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:28.949 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:47:28.968 INFO:tasks.rados.rados.0.smithi001.stdout:250: done (3 left) 2024-09-15T03:47:28.969 INFO:tasks.rados.rados.0.smithi001.stdout:252: done (2 left) 2024-09-15T03:47:28.969 INFO:tasks.rados.rados.0.smithi001.stdout:254: done (1 left) 2024-09-15T03:47:28.969 INFO:tasks.rados.rados.0.smithi001.stdout:255: done (0 left) 2024-09-15T03:47:28.969 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 37 current snap is 32 2024-09-15T03:47:28.969 INFO:tasks.rados.rados.0.smithi001.stdout:256: seq_num 111 ranges {0=704512,704512=655360,1359872=507904} 2024-09-15T03:47:28.972 INFO:tasks.rados.rados.0.smithi001.stdout:256: writing smithi00132172-37 from 0 to 704512 tid 1 2024-09-15T03:47:28.976 INFO:tasks.rados.rados.0.smithi001.stdout:256: writing smithi00132172-37 from 704512 to 1359872 tid 2 2024-09-15T03:47:28.980 INFO:tasks.rados.rados.0.smithi001.stdout:256: writing smithi00132172-37 from 1359872 to 1867776 tid 3 2024-09-15T03:47:28.980 INFO:tasks.rados.rados.0.smithi001.stdout:257: setattr oid 48 current snap is 32 2024-09-15T03:47:28.980 INFO:tasks.rados.rados.0.smithi001.stdout:256: finishing write tid 1 to smithi00132172-37 2024-09-15T03:47:28.981 INFO:tasks.rados.rados.0.smithi001.stdout:258: read oid 7 snap -1 2024-09-15T03:47:28.981 INFO:tasks.rados.rados.0.smithi001.stdout:258: expect deleted 2024-09-15T03:47:28.982 INFO:tasks.rados.rados.0.smithi001.stdout:259: rollback oid 2 current snap is 32 2024-09-15T03:47:28.982 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 2 to 11 2024-09-15T03:47:28.982 INFO:tasks.rados.rados.0.smithi001.stdout:260: read oid 33 snap 10 2024-09-15T03:47:28.982 INFO:tasks.rados.rados.0.smithi001.stdout:260: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-15T03:47:28.982 INFO:tasks.rados.rados.0.smithi001.stdout:261: rollback oid 35 current snap is 32 2024-09-15T03:47:28.982 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 35 to 16 2024-09-15T03:47:28.982 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 22 current snap is 32 2024-09-15T03:47:28.982 INFO:tasks.rados.rados.0.smithi001.stdout:262: seq_num 112 ranges {360448=720896,1081344=802816,1884160=360448} 2024-09-15T03:47:28.986 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:28.985+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:28.987 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T03:47:28.989 INFO:tasks.rados.rados.0.smithi001.stdout:262: writing smithi00132172-22 from 360448 to 1081344 tid 1 2024-09-15T03:47:28.994 INFO:tasks.rados.rados.0.smithi001.stdout:262: writing smithi00132172-22 from 1081344 to 1884160 tid 2 2024-09-15T03:47:28.997 INFO:tasks.rados.rados.0.smithi001.stdout:262: writing smithi00132172-22 from 1884160 to 2244608 tid 3 2024-09-15T03:47:28.997 INFO:tasks.rados.rados.0.smithi001.stdout:263: setattr oid 6 current snap is 32 2024-09-15T03:47:28.997 INFO:tasks.rados.rados.0.smithi001.stdout:256: finishing write tid 2 to smithi00132172-37 2024-09-15T03:47:28.997 INFO:tasks.rados.rados.0.smithi001.stdout:256: finishing write tid 3 to smithi00132172-37 2024-09-15T03:47:28.997 INFO:tasks.rados.rados.0.smithi001.stdout:256: finishing write tid 4 to smithi00132172-37 2024-09-15T03:47:28.997 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 48 v 63 (ObjNum 161 snap 0 seq_num 47) dirty exists 2024-09-15T03:47:28.997 INFO:tasks.rados.rados.0.smithi001.stdout:259: finishing rollback tid 0 to smithi00132172-2 2024-09-15T03:47:28.998 INFO:tasks.rados.rados.0.smithi001.stdout:259: finishing rollback tid 1 to smithi00132172-2 2024-09-15T03:47:28.998 INFO:tasks.rados.rados.0.smithi001.stdout:259: finishing rollback tid 2 to smithi00132172-2 2024-09-15T03:47:28.998 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 2 v 89 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-15T03:47:28.998 INFO:tasks.rados.rados.0.smithi001.stdout:261: finishing rollback tid 0 to smithi00132172-35 2024-09-15T03:47:28.998 INFO:tasks.rados.rados.0.smithi001.stdout:256: finishing write tid 5 to smithi00132172-37 2024-09-15T03:47:28.998 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 37 v 39 (ObjNum 111 snap 32 seq_num 111) dirty exists 2024-09-15T03:47:28.998 INFO:tasks.rados.rados.0.smithi001.stdout:256: left oid 37 (ObjNum 111 snap 32 seq_num 111) 2024-09-15T03:47:28.998 INFO:tasks.rados.rados.0.smithi001.stdout:260: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-15T03:47:29.013 INFO:tasks.rados.rados.0.smithi001.stdout:262: finishing write tid 1 to smithi00132172-22 2024-09-15T03:47:29.013 INFO:tasks.rados.rados.0.smithi001.stdout:261: finishing rollback tid 1 to smithi00132172-35 2024-09-15T03:47:29.013 INFO:tasks.rados.rados.0.smithi001.stdout:261: finishing rollback tid 2 to smithi00132172-35 2024-09-15T03:47:29.013 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 35 v 43 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-15T03:47:29.014 INFO:tasks.rados.rados.0.smithi001.stdout:262: finishing write tid 2 to smithi00132172-22 2024-09-15T03:47:29.014 INFO:tasks.rados.rados.0.smithi001.stdout:262: finishing write tid 3 to smithi00132172-22 2024-09-15T03:47:29.014 INFO:tasks.rados.rados.0.smithi001.stdout:262: finishing write tid 4 to smithi00132172-22 2024-09-15T03:47:29.014 INFO:tasks.rados.rados.0.smithi001.stdout:262: finishing write tid 5 to smithi00132172-22 2024-09-15T03:47:29.014 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 22 v 107 (ObjNum 112 snap 32 seq_num 112) dirty exists 2024-09-15T03:47:29.014 INFO:tasks.rados.rados.0.smithi001.stdout:262: left oid 22 (ObjNum 112 snap 32 seq_num 112) 2024-09-15T03:47:29.014 INFO:tasks.rados.rados.0.smithi001.stdout:256: done (7 left) 2024-09-15T03:47:29.014 INFO:tasks.rados.rados.0.smithi001.stdout:257: done (6 left) 2024-09-15T03:47:29.014 INFO:tasks.rados.rados.0.smithi001.stdout:258: done (5 left) 2024-09-15T03:47:29.014 INFO:tasks.rados.rados.0.smithi001.stdout:259: done (4 left) 2024-09-15T03:47:29.014 INFO:tasks.rados.rados.0.smithi001.stdout:260: done (3 left) 2024-09-15T03:47:29.014 INFO:tasks.rados.rados.0.smithi001.stdout:261: done (2 left) 2024-09-15T03:47:29.014 INFO:tasks.rados.rados.0.smithi001.stdout:262: done (1 left) 2024-09-15T03:47:29.014 INFO:tasks.rados.rados.0.smithi001.stdout:264: snap_remove snap 31 2024-09-15T03:47:29.063 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:29.063 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:29.072 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:47:29.113 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:29.112+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:29.187 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:29.186+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:29.214 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:29.214 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:29.222 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:47:29.288 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:29.287+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:29.322 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:29.322 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:29.331 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:47:29.389 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:29.388+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:29.445 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:29.445 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:29.454 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:47:29.490 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:29.489+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:29.556 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:29.556 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:29.562 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:47:29.564 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:47:29.607 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:29.606+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:29.666 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:29.666 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:29.675 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:47:29.679 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:29.680 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:29.691 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:29.690+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:29.693 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:29.694 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:29.708 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:29.708 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:29.722 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:29.722 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:29.737 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:29.737 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:29.751 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:29.752 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:29.767 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:29.777 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:29.777 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:29.777 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:29.777 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:29.782 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:29.782 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:29.789 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:47:29.792 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:29.791+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:29.797 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:29.797 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:29.812 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:29.812 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:29.813 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-15T03:47:29.828 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:29.828 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:29.831 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T03:47:29.832 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T03:47:29.889 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:29.890 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:29.890 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:29.890 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:29.890 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:29.893 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:29.892+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:29.898 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:47:29.978 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-15T03:47:29.979 INFO:tasks.rados.rados.0.smithi001.stdout:264: done (1 left) 2024-09-15T03:47:29.979 INFO:tasks.rados.rados.0.smithi001.stdout:265: read oid 11 snap -1 2024-09-15T03:47:29.979 INFO:tasks.rados.rados.0.smithi001.stdout:265: expect (ObjNum 78 snap 13 seq_num 78) 2024-09-15T03:47:29.979 INFO:tasks.rados.rados.0.smithi001.stdout:266: read oid 45 snap -1 2024-09-15T03:47:29.979 INFO:tasks.rados.rados.0.smithi001.stdout:266: expect (ObjNum 79 snap 16 seq_num 79) 2024-09-15T03:47:29.979 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 7 current snap is 32 2024-09-15T03:47:29.980 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 6 v 34 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-15T03:47:29.980 INFO:tasks.rados.rados.0.smithi001.stdout:267: seq_num 113 ranges {0=737280,737280=557056,1294336=606208} 2024-09-15T03:47:29.983 INFO:tasks.rados.rados.0.smithi001.stdout:267: writing smithi00132172-7 from 0 to 737280 tid 1 2024-09-15T03:47:29.987 INFO:tasks.rados.rados.0.smithi001.stdout:267: writing smithi00132172-7 from 737280 to 1294336 tid 2 2024-09-15T03:47:29.991 INFO:tasks.rados.rados.0.smithi001.stdout:267: writing smithi00132172-7 from 1294336 to 1900544 tid 3 2024-09-15T03:47:29.991 INFO:tasks.rados.rados.0.smithi001.stdout:263: done (3 left) 2024-09-15T03:47:29.991 INFO:tasks.rados.rados.0.smithi001.stdout:268: delete oid 28 current snap is 32 2024-09-15T03:47:29.991 INFO:tasks.rados.rados.0.smithi001.stdout:266: expect (ObjNum 79 snap 16 seq_num 79) 2024-09-15T03:47:29.994 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:29.994+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:30.045 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:30.045 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:30.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:30.045 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:30.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.045 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 38/28 ECSubWrite(tid=33, reqid=client.4521.0:222, at_version=38'14, trim_to=0'0, roll_forward_to=35'12))", 2024-09-15T03:47:30.045 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:47:30.045 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.911889072999998, 2024-09-15T03:47:30.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9949187450000001, 2024-09-15T03:47:30.045 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.045 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.046 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:47:30.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:47:30.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130996+0000", 2024-09-15T03:47:30.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8809999999999998e-06 2024-09-15T03:47:30.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131020+0000", 2024-09-15T03:47:30.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4190999999999999e-05 2024-09-15T03:47:30.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131039+0000", 2024-09-15T03:47:30.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9228999999999998e-05 2024-09-15T03:47:30.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131048+0000", 2024-09-15T03:47:30.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2380000000000003e-06 2024-09-15T03:47:30.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131122+0000", 2024-09-15T03:47:30.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3146999999999996e-05 2024-09-15T03:47:30.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131132+0000", 2024-09-15T03:47:30.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.08e-05 2024-09-15T03:47:30.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131381+0000", 2024-09-15T03:47:30.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000248936 2024-09-15T03:47:30.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131392+0000", 2024-09-15T03:47:30.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0987e-05 2024-09-15T03:47:30.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131444+0000", 2024-09-15T03:47:30.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1585000000000002e-05 2024-09-15T03:47:30.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131465+0000", 2024-09-15T03:47:30.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1365999999999998e-05 2024-09-15T03:47:30.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.125911+0000", 2024-09-15T03:47:30.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9944453850000001 2024-09-15T03:47:30.051 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.051 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.051 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.052 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 38/28 ECSubWrite(tid=34, reqid=client.4521.0:222, at_version=0'0, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T03:47:30.052 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:47:30.052 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.910489974000001, 2024-09-15T03:47:30.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.002752697, 2024-09-15T03:47:30.052 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.052 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.052 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:47:30.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:47:30.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132392+0000", 2024-09-15T03:47:30.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5899999999999995e-07 2024-09-15T03:47:30.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132415+0000", 2024-09-15T03:47:30.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3561e-05 2024-09-15T03:47:30.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132436+0000", 2024-09-15T03:47:30.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0879000000000001e-05 2024-09-15T03:47:30.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132444+0000", 2024-09-15T03:47:30.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9200000000000004e-06 2024-09-15T03:47:30.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132497+0000", 2024-09-15T03:47:30.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2299000000000002e-05 2024-09-15T03:47:30.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132508+0000", 2024-09-15T03:47:30.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1100999999999999e-05 2024-09-15T03:47:30.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132615+0000", 2024-09-15T03:47:30.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107546 2024-09-15T03:47:30.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132635+0000", 2024-09-15T03:47:30.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9374000000000001e-05 2024-09-15T03:47:30.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132677+0000", 2024-09-15T03:47:30.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2456999999999997e-05 2024-09-15T03:47:30.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132711+0000", 2024-09-15T03:47:30.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4118000000000001e-05 2024-09-15T03:47:30.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135144+0000", 2024-09-15T03:47:30.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.002432583 2024-09-15T03:47:30.057 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.058 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.058 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.058 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=45, reqid=client.4521.0:227, at_version=38'24, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:47:30.058 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:30.058 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.887563432, 2024-09-15T03:47:30.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.974765339, 2024-09-15T03:47:30.058 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.058 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.058 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:30.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:30.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:30.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.43e-07 2024-09-15T03:47:30.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156110+0000", 2024-09-15T03:47:30.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000792409 2024-09-15T03:47:30.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156139+0000", 2024-09-15T03:47:30.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9207e-05 2024-09-15T03:47:30.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156148+0000", 2024-09-15T03:47:30.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1269999999999992e-06 2024-09-15T03:47:30.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156248+0000", 2024-09-15T03:47:30.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9421999999999998e-05 2024-09-15T03:47:30.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156258+0000", 2024-09-15T03:47:30.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6989999999999998e-06 2024-09-15T03:47:30.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156674+0000", 2024-09-15T03:47:30.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041682600000000002 2024-09-15T03:47:30.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156705+0000", 2024-09-15T03:47:30.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0525000000000003e-05 2024-09-15T03:47:30.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157742+0000", 2024-09-15T03:47:30.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010374 2024-09-15T03:47:30.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157777+0000", 2024-09-15T03:47:30.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4878e-05 2024-09-15T03:47:30.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.130083+0000", 2024-09-15T03:47:30.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972305703 2024-09-15T03:47:30.064 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.064 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.064 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.065 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=37, reqid=client.4521.0:236, at_version=38'5, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:30.065 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:30.065 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.885949975999999, 2024-09-15T03:47:30.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9740836640000001, 2024-09-15T03:47:30.065 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.065 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.065 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:30.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:30.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:30.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1300000000000001e-07 2024-09-15T03:47:30.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156953+0000", 2024-09-15T03:47:30.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1585000000000001e-05 2024-09-15T03:47:30.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156974+0000", 2024-09-15T03:47:30.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0764e-05 2024-09-15T03:47:30.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156981+0000", 2024-09-15T03:47:30.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2760000000000003e-06 2024-09-15T03:47:30.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157061+0000", 2024-09-15T03:47:30.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9583e-05 2024-09-15T03:47:30.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157075+0000", 2024-09-15T03:47:30.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4019000000000001e-05 2024-09-15T03:47:30.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157351+0000", 2024-09-15T03:47:30.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027641100000000001 2024-09-15T03:47:30.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157746+0000", 2024-09-15T03:47:30.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039553199999999998 2024-09-15T03:47:30.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157838+0000", 2024-09-15T03:47:30.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1632000000000006e-05 2024-09-15T03:47:30.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157856+0000", 2024-09-15T03:47:30.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7873999999999999e-05 2024-09-15T03:47:30.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.131015+0000", 2024-09-15T03:47:30.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9731586750000001 2024-09-15T03:47:30.071 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.071 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.071 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=46, reqid=client.4521.0:228, at_version=38'25, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:47:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.885891194000003, 2024-09-15T03:47:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9732838510000001, 2024-09-15T03:47:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0999999999999998e-07 2024-09-15T03:47:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157149+0000", 2024-09-15T03:47:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000158763 2024-09-15T03:47:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157165+0000", 2024-09-15T03:47:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5883999999999999e-05 2024-09-15T03:47:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157175+0000", 2024-09-15T03:47:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4949999999999997e-06 2024-09-15T03:47:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157219+0000", 2024-09-15T03:47:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4069000000000001e-05 2024-09-15T03:47:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157228+0000", 2024-09-15T03:47:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7829999999999993e-06 2024-09-15T03:47:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157345+0000", 2024-09-15T03:47:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116655 2024-09-15T03:47:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157746+0000", 2024-09-15T03:47:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040045100000000002 2024-09-15T03:47:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157838+0000", 2024-09-15T03:47:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1955999999999998e-05 2024-09-15T03:47:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157888+0000", 2024-09-15T03:47:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0943000000000003e-05 2024-09-15T03:47:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.130274+0000", 2024-09-15T03:47:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972385142 2024-09-15T03:47:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=38, reqid=client.4521.0:237, at_version=38'6, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:47:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.884548623999997, 2024-09-15T03:47:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9764011529999999, 2024-09-15T03:47:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:47:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:47:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158333+0000", 2024-09-15T03:47:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8599999999999997e-07 2024-09-15T03:47:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158353+0000", 2024-09-15T03:47:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9504000000000002e-05 2024-09-15T03:47:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158368+0000", 2024-09-15T03:47:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4966999999999999e-05 2024-09-15T03:47:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158374+0000", 2024-09-15T03:47:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.798e-06 2024-09-15T03:47:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158430+0000", 2024-09-15T03:47:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5744e-05 2024-09-15T03:47:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158440+0000", 2024-09-15T03:47:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4560000000000006e-06 2024-09-15T03:47:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158529+0000", 2024-09-15T03:47:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9172999999999995e-05 2024-09-15T03:47:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158543+0000", 2024-09-15T03:47:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.395e-05 2024-09-15T03:47:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158654+0000", 2024-09-15T03:47:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011110700000000001 2024-09-15T03:47:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158671+0000", 2024-09-15T03:47:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7068999999999999e-05 2024-09-15T03:47:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134733+0000", 2024-09-15T03:47:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.976062599 2024-09-15T03:47:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=47, reqid=client.4521.0:229, at_version=38'26, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:47:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:47:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.884544662000003, 2024-09-15T03:47:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9727810969999999, 2024-09-15T03:47:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:47:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:47:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158337+0000", 2024-09-15T03:47:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.15e-07 2024-09-15T03:47:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158351+0000", 2024-09-15T03:47:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3871e-05 2024-09-15T03:47:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158372+0000", 2024-09-15T03:47:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1058999999999999e-05 2024-09-15T03:47:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158376+0000", 2024-09-15T03:47:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5249999999999999e-06 2024-09-15T03:47:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158420+0000", 2024-09-15T03:47:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3918000000000002e-05 2024-09-15T03:47:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158429+0000", 2024-09-15T03:47:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5150000000000006e-06 2024-09-15T03:47:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158514+0000", 2024-09-15T03:47:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5252000000000003e-05 2024-09-15T03:47:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158542+0000", 2024-09-15T03:47:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.798e-05 2024-09-15T03:47:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158653+0000", 2024-09-15T03:47:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111315 2024-09-15T03:47:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158701+0000", 2024-09-15T03:47:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8241000000000001e-05 2024-09-15T03:47:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.131117+0000", 2024-09-15T03:47:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972416006 2024-09-15T03:47:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=39, reqid=client.4521.0:238, at_version=38'7, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:47:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.883857505999998, 2024-09-15T03:47:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9758214509999998, 2024-09-15T03:47:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:47:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:47:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159024+0000", 2024-09-15T03:47:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6700000000000003e-07 2024-09-15T03:47:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159096+0000", 2024-09-15T03:47:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1386999999999999e-05 2024-09-15T03:47:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159113+0000", 2024-09-15T03:47:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7020999999999998e-05 2024-09-15T03:47:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159122+0000", 2024-09-15T03:47:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4720000000000001e-06 2024-09-15T03:47:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159168+0000", 2024-09-15T03:47:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5766000000000002e-05 2024-09-15T03:47:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159176+0000", 2024-09-15T03:47:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3070000000000003e-06 2024-09-15T03:47:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159258+0000", 2024-09-15T03:47:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1398999999999998e-05 2024-09-15T03:47:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159292+0000", 2024-09-15T03:47:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4480000000000002e-05 2024-09-15T03:47:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159342+0000", 2024-09-15T03:47:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9450999999999998e-05 2024-09-15T03:47:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159362+0000", 2024-09-15T03:47:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0633e-05 2024-09-15T03:47:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134845+0000", 2024-09-15T03:47:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.975482768 2024-09-15T03:47:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=48, reqid=client.4521.0:229, at_version=0'0, trim_to=0'0, roll_forward_to=38'26))", 2024-09-15T03:47:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:47:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.882490709999999, 2024-09-15T03:47:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.974485472, 2024-09-15T03:47:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:47:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:47:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160391+0000", 2024-09-15T03:47:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.136e-06 2024-09-15T03:47:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160405+0000", 2024-09-15T03:47:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3613e-05 2024-09-15T03:47:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160426+0000", 2024-09-15T03:47:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1234999999999999e-05 2024-09-15T03:47:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160434+0000", 2024-09-15T03:47:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.61e-06 2024-09-15T03:47:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160496+0000", 2024-09-15T03:47:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2384000000000004e-05 2024-09-15T03:47:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160507+0000", 2024-09-15T03:47:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1005000000000001e-05 2024-09-15T03:47:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160632+0000", 2024-09-15T03:47:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124566 2024-09-15T03:47:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160653+0000", 2024-09-15T03:47:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0707999999999999e-05 2024-09-15T03:47:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160701+0000", 2024-09-15T03:47:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8751999999999997e-05 2024-09-15T03:47:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160739+0000", 2024-09-15T03:47:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7486e-05 2024-09-15T03:47:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134876+0000", 2024-09-15T03:47:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9741369770000001 2024-09-15T03:47:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=40, reqid=client.4521.0:238, at_version=0'0, trim_to=0'0, roll_forward_to=38'7))", 2024-09-15T03:47:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:47:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.880644289000003, 2024-09-15T03:47:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9727862730000001, 2024-09-15T03:47:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:47:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:47:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162238+0000", 2024-09-15T03:47:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.672e-06 2024-09-15T03:47:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162253+0000", 2024-09-15T03:47:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4932e-05 2024-09-15T03:47:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162284+0000", 2024-09-15T03:47:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1211000000000003e-05 2024-09-15T03:47:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162296+0000", 2024-09-15T03:47:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1229999999999999e-05 2024-09-15T03:47:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162373+0000", 2024-09-15T03:47:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7146999999999998e-05 2024-09-15T03:47:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162387+0000", 2024-09-15T03:47:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4290999999999999e-05 2024-09-15T03:47:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162502+0000", 2024-09-15T03:47:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115054 2024-09-15T03:47:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162516+0000", 2024-09-15T03:47:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3597999999999999e-05 2024-09-15T03:47:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162558+0000", 2024-09-15T03:47:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2562999999999999e-05 2024-09-15T03:47:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162584+0000", 2024-09-15T03:47:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5738e-05 2024-09-15T03:47:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135023+0000", 2024-09-15T03:47:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9724388369999999 2024-09-15T03:47:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 39/28 ECSubWrite(tid=38, reqid=osd.11.0:38, at_version=39'16, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T03:47:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.899261989000003, 2024-09-15T03:47:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99210653800000004, 2024-09-15T03:47:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1800000000000002e-07 2024-09-15T03:47:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143648+0000", 2024-09-15T03:47:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.849e-05 2024-09-15T03:47:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143670+0000", 2024-09-15T03:47:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2036e-05 2024-09-15T03:47:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143680+0000", 2024-09-15T03:47:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8320000000000001e-06 2024-09-15T03:47:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143783+0000", 2024-09-15T03:47:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103569 2024-09-15T03:47:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143801+0000", 2024-09-15T03:47:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8077999999999999e-05 2024-09-15T03:47:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144077+0000", 2024-09-15T03:47:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027569500000000001 2024-09-15T03:47:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144459+0000", 2024-09-15T03:47:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038227299999999998 2024-09-15T03:47:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144519+0000", 2024-09-15T03:47:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9907000000000001e-05 2024-09-15T03:47:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144628+0000", 2024-09-15T03:47:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108958 2024-09-15T03:47:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135726+0000", 2024-09-15T03:47:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99109738199999997 2024-09-15T03:47:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 39/31 ECSubWrite(tid=41, reqid=osd.6.0:41, at_version=39'9, trim_to=0'0, roll_forward_to=38'7))", 2024-09-15T03:47:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.898611733999999, 2024-09-15T03:47:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99106653099999997, 2024-09-15T03:47:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.97e-07 2024-09-15T03:47:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144297+0000", 2024-09-15T03:47:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7815000000000001e-05 2024-09-15T03:47:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144314+0000", 2024-09-15T03:47:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6269e-05 2024-09-15T03:47:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144322+0000", 2024-09-15T03:47:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1880000000000002e-06 2024-09-15T03:47:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144767+0000", 2024-09-15T03:47:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044555300000000001 2024-09-15T03:47:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144777+0000", 2024-09-15T03:47:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5950000000000005e-06 2024-09-15T03:47:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144992+0000", 2024-09-15T03:47:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000215479 2024-09-15T03:47:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145142+0000", 2024-09-15T03:47:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149302 2024-09-15T03:47:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145249+0000", 2024-09-15T03:47:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106971 2024-09-15T03:47:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145299+0000", 2024-09-15T03:47:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0413e-05 2024-09-15T03:47:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135336+0000", 2024-09-15T03:47:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99003674900000005 2024-09-15T03:47:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 39/28 ECSubWrite(tid=50, reqid=osd.2.0:50, at_version=39'28, trim_to=0'0, roll_forward_to=38'26))", 2024-09-15T03:47:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.898314251999999, 2024-09-15T03:47:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99095576299999999, 2024-09-15T03:47:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.03e-07 2024-09-15T03:47:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144593+0000", 2024-09-15T03:47:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6520999999999999e-05 2024-09-15T03:47:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144611+0000", 2024-09-15T03:47:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7963000000000001e-05 2024-09-15T03:47:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144619+0000", 2024-09-15T03:47:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7409999999999992e-06 2024-09-15T03:47:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144821+0000", 2024-09-15T03:47:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000201652 2024-09-15T03:47:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144832+0000", 2024-09-15T03:47:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0827e-05 2024-09-15T03:47:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145099+0000", 2024-09-15T03:47:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026688299999999999 2024-09-15T03:47:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145143+0000", 2024-09-15T03:47:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.481e-05 2024-09-15T03:47:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145249+0000", 2024-09-15T03:47:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010618799999999999 2024-09-15T03:47:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145340+0000", 2024-09-15T03:47:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0340000000000006e-05 2024-09-15T03:47:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135522+0000", 2024-09-15T03:47:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99018263500000003 2024-09-15T03:47:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=43, reqid=client.4521.0:240, at_version=39'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:47:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.894973532000002, 2024-09-15T03:47:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99104975500000003, 2024-09-15T03:47:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:47:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:47:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147908+0000", 2024-09-15T03:47:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.05e-07 2024-09-15T03:47:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147931+0000", 2024-09-15T03:47:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3621e-05 2024-09-15T03:47:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147945+0000", 2024-09-15T03:47:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3648999999999999e-05 2024-09-15T03:47:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147951+0000", 2024-09-15T03:47:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0510000000000002e-06 2024-09-15T03:47:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148055+0000", 2024-09-15T03:47:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104421 2024-09-15T03:47:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148068+0000", 2024-09-15T03:47:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3019e-05 2024-09-15T03:47:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148375+0000", 2024-09-15T03:47:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030683400000000002 2024-09-15T03:47:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148399+0000", 2024-09-15T03:47:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3760999999999998e-05 2024-09-15T03:47:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148453+0000", 2024-09-15T03:47:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3851999999999999e-05 2024-09-15T03:47:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148481+0000", 2024-09-15T03:47:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8637999999999999e-05 2024-09-15T03:47:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.138957+0000", 2024-09-15T03:47:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99047580400000002 2024-09-15T03:47:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 39/28 ECSubWrite(tid=39, reqid=osd.11.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:47:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:47:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.894841043, 2024-09-15T03:47:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99452509, 2024-09-15T03:47:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:47:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:47:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148041+0000", 2024-09-15T03:47:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0999999999999998e-07 2024-09-15T03:47:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148052+0000", 2024-09-15T03:47:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.129e-05 2024-09-15T03:47:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148064+0000", 2024-09-15T03:47:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173e-05 2024-09-15T03:47:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148068+0000", 2024-09-15T03:47:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5680000000000001e-06 2024-09-15T03:47:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148136+0000", 2024-09-15T03:47:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7508999999999993e-05 2024-09-15T03:47:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148148+0000", 2024-09-15T03:47:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2002e-05 2024-09-15T03:47:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148265+0000", 2024-09-15T03:47:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117669 2024-09-15T03:47:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148289+0000", 2024-09-15T03:47:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3672e-05 2024-09-15T03:47:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148348+0000", 2024-09-15T03:47:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8838000000000001e-05 2024-09-15T03:47:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148368+0000", 2024-09-15T03:47:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9942999999999999e-05 2024-09-15T03:47:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142565+0000", 2024-09-15T03:47:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99419715900000005 2024-09-15T03:47:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=45, reqid=client.4521.0:241, at_version=39'17, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:47:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.891838020000002, 2024-09-15T03:47:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99118202200000005, 2024-09-15T03:47:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:47:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:47:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151044+0000", 2024-09-15T03:47:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1340000000000001e-06 2024-09-15T03:47:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151073+0000", 2024-09-15T03:47:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8894000000000002e-05 2024-09-15T03:47:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151095+0000", 2024-09-15T03:47:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1889e-05 2024-09-15T03:47:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151107+0000", 2024-09-15T03:47:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2136999999999999e-05 2024-09-15T03:47:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151189+0000", 2024-09-15T03:47:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1883000000000002e-05 2024-09-15T03:47:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151203+0000", 2024-09-15T03:47:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3825999999999999e-05 2024-09-15T03:47:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151305+0000", 2024-09-15T03:47:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101816 2024-09-15T03:47:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151337+0000", 2024-09-15T03:47:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2703000000000002e-05 2024-09-15T03:47:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151387+0000", 2024-09-15T03:47:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9366000000000001e-05 2024-09-15T03:47:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151405+0000", 2024-09-15T03:47:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8748000000000001e-05 2024-09-15T03:47:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142225+0000", 2024-09-15T03:47:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99081962599999995 2024-09-15T03:47:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 39/31 ECSubWrite(tid=42, reqid=osd.6.0:41, at_version=0'0, trim_to=0'0, roll_forward_to=39'9))", 2024-09-15T03:47:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:47:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.891541979000003, 2024-09-15T03:47:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99092639999999999, 2024-09-15T03:47:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:47:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:47:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151340+0000", 2024-09-15T03:47:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8299999999999996e-07 2024-09-15T03:47:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151350+0000", 2024-09-15T03:47:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0675e-05 2024-09-15T03:47:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151362+0000", 2024-09-15T03:47:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2048999999999999e-05 2024-09-15T03:47:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151369+0000", 2024-09-15T03:47:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3230000000000003e-06 2024-09-15T03:47:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151425+0000", 2024-09-15T03:47:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5964000000000002e-05 2024-09-15T03:47:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151435+0000", 2024-09-15T03:47:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9240000000000002e-06 2024-09-15T03:47:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151530+0000", 2024-09-15T03:47:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5649000000000005e-05 2024-09-15T03:47:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151558+0000", 2024-09-15T03:47:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7976000000000001e-05 2024-09-15T03:47:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151599+0000", 2024-09-15T03:47:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1152000000000002e-05 2024-09-15T03:47:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151617+0000", 2024-09-15T03:47:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7676000000000002e-05 2024-09-15T03:47:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142265+0000", 2024-09-15T03:47:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99064822900000005 2024-09-15T03:47:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=46, reqid=client.4521.0:242, at_version=39'18, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.891516342999999, 2024-09-15T03:47:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.991000204, 2024-09-15T03:47:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3399999999999999e-07 2024-09-15T03:47:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151393+0000", 2024-09-15T03:47:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8149000000000001e-05 2024-09-15T03:47:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151405+0000", 2024-09-15T03:47:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2063e-05 2024-09-15T03:47:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151415+0000", 2024-09-15T03:47:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4579999999999999e-06 2024-09-15T03:47:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151472+0000", 2024-09-15T03:47:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7018999999999997e-05 2024-09-15T03:47:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151480+0000", 2024-09-15T03:47:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1410000000000005e-06 2024-09-15T03:47:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151571+0000", 2024-09-15T03:47:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1243999999999996e-05 2024-09-15T03:47:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151602+0000", 2024-09-15T03:47:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0834000000000003e-05 2024-09-15T03:47:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151632+0000", 2024-09-15T03:47:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9527e-05 2024-09-15T03:47:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151685+0000", 2024-09-15T03:47:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3599e-05 2024-09-15T03:47:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142365+0000", 2024-09-15T03:47:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99067953600000003 2024-09-15T03:47:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 39/28 ECSubWrite(tid=51, reqid=osd.2.0:50, at_version=0'0, trim_to=0'0, roll_forward_to=39'28))", 2024-09-15T03:47:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:47:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.888152904999998, 2024-09-15T03:47:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98766307399999997, 2024-09-15T03:47:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:47:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:47:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154729+0000", 2024-09-15T03:47:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3799999999999996e-07 2024-09-15T03:47:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154742+0000", 2024-09-15T03:47:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2909e-05 2024-09-15T03:47:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154755+0000", 2024-09-15T03:47:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3251999999999999e-05 2024-09-15T03:47:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154761+0000", 2024-09-15T03:47:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6559999999999997e-06 2024-09-15T03:47:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154829+0000", 2024-09-15T03:47:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8549999999999999e-05 2024-09-15T03:47:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154840+0000", 2024-09-15T03:47:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0159999999999999e-05 2024-09-15T03:47:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154943+0000", 2024-09-15T03:47:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010347400000000001 2024-09-15T03:47:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154955+0000", 2024-09-15T03:47:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1935e-05 2024-09-15T03:47:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156442+0000", 2024-09-15T03:47:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014867859999999999 2024-09-15T03:47:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156466+0000", 2024-09-15T03:47:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4077999999999999e-05 2024-09-15T03:47:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142391+0000", 2024-09-15T03:47:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98592533599999999 2024-09-15T03:47:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=48, reqid=client.4521.0:242, at_version=0'0, trim_to=0'0, roll_forward_to=39'18))", 2024-09-15T03:47:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:47:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.886265885999997, 2024-09-15T03:47:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98592495099999999, 2024-09-15T03:47:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:47:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:47:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156616+0000", 2024-09-15T03:47:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1999999999999998e-07 2024-09-15T03:47:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156629+0000", 2024-09-15T03:47:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2870999999999999e-05 2024-09-15T03:47:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156649+0000", 2024-09-15T03:47:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9786e-05 2024-09-15T03:47:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156657+0000", 2024-09-15T03:47:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5110000000000003e-06 2024-09-15T03:47:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156723+0000", 2024-09-15T03:47:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6209000000000003e-05 2024-09-15T03:47:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156734+0000", 2024-09-15T03:47:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0475999999999999e-05 2024-09-15T03:47:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156868+0000", 2024-09-15T03:47:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013402999999999999 2024-09-15T03:47:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156889+0000", 2024-09-15T03:47:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1146e-05 2024-09-15T03:47:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156937+0000", 2024-09-15T03:47:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8031999999999997e-05 2024-09-15T03:47:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156970+0000", 2024-09-15T03:47:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2462999999999997e-05 2024-09-15T03:47:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142540+0000", 2024-09-15T03:47:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98557050700000004 2024-09-15T03:47:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.165 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:30.168 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:47:30.169 INFO:tasks.rados.rados.0.smithi001.stdout:265: expect (ObjNum 78 snap 13 seq_num 78) 2024-09-15T03:47:30.169 INFO:tasks.rados.rados.0.smithi001.stdout:267: finishing write tid 1 to smithi00132172-7 2024-09-15T03:47:30.170 INFO:tasks.rados.rados.0.smithi001.stdout:267: finishing write tid 2 to smithi00132172-7 2024-09-15T03:47:30.170 INFO:tasks.rados.rados.0.smithi001.stdout:267: finishing write tid 3 to smithi00132172-7 2024-09-15T03:47:30.170 INFO:tasks.rados.rados.0.smithi001.stdout:267: finishing write tid 4 to smithi00132172-7 2024-09-15T03:47:30.170 INFO:tasks.rados.rados.0.smithi001.stdout:267: finishing write tid 5 to smithi00132172-7 2024-09-15T03:47:30.170 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 7 v 93 (ObjNum 113 snap 32 seq_num 113) dirty exists 2024-09-15T03:47:30.170 INFO:tasks.rados.rados.0.smithi001.stdout:267: left oid 7 (ObjNum 113 snap 32 seq_num 113) 2024-09-15T03:47:30.170 INFO:tasks.rados.rados.0.smithi001.stdout:265: done (3 left) 2024-09-15T03:47:30.170 INFO:tasks.rados.rados.0.smithi001.stdout:266: done (2 left) 2024-09-15T03:47:30.170 INFO:tasks.rados.rados.0.smithi001.stdout:267: done (1 left) 2024-09-15T03:47:30.170 INFO:tasks.rados.rados.0.smithi001.stdout:268: done (0 left) 2024-09-15T03:47:30.170 INFO:tasks.rados.rados.0.smithi001.stdout:269: copy_from oid 27 from oid 3 current snap is 32 2024-09-15T03:47:30.170 INFO:tasks.rados.rados.0.smithi001.stdout:270: copy_from oid 10 from oid 16 current snap is 32 2024-09-15T03:47:30.170 INFO:tasks.rados.rados.0.smithi001.stdout:271: read oid 6 snap 1 2024-09-15T03:47:30.170 INFO:tasks.rados.rados.0.smithi001.stdout:271: expect (ObjNum 53 snap 0 seq_num 53) 2024-09-15T03:47:30.171 INFO:tasks.rados.rados.0.smithi001.stdout:272: read oid 44 snap -1 2024-09-15T03:47:30.171 INFO:tasks.rados.rados.0.smithi001.stdout:272: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-15T03:47:30.171 INFO:tasks.rados.rados.0.smithi001.stdout:273: read oid 23 snap -1 2024-09-15T03:47:30.171 INFO:tasks.rados.rados.0.smithi001.stdout:273: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-15T03:47:30.171 INFO:tasks.rados.rados.0.smithi001.stdout:274: setattr oid 25 current snap is 32 2024-09-15T03:47:30.171 INFO:tasks.rados.rados.0.smithi001.stdout:275: read oid 40 snap -1 2024-09-15T03:47:30.171 INFO:tasks.rados.rados.0.smithi001.stdout:275: expect (ObjNum 106 snap 29 seq_num 106) 2024-09-15T03:47:30.171 INFO:tasks.rados.rados.0.smithi001.stdout:276: copy_from oid 18 from oid 12 current snap is 32 2024-09-15T03:47:30.171 INFO:tasks.rados.rados.0.smithi001.stdout:277: rollback oid 19 current snap is 32 2024-09-15T03:47:30.171 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 19 to 19 2024-09-15T03:47:30.171 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 37 current snap is 32 2024-09-15T03:47:30.171 INFO:tasks.rados.rados.0.smithi001.stdout:278: seq_num 115 ranges {1867776=770048,2637824=704512,3342336=458752} 2024-09-15T03:47:30.171 INFO:tasks.rados.rados.0.smithi001.stdout:278: writing smithi00132172-37 from 1867776 to 2637824 tid 1 2024-09-15T03:47:30.172 INFO:tasks.rados.rados.0.smithi001.stdout:278: writing smithi00132172-37 from 2637824 to 3342336 tid 2 2024-09-15T03:47:30.172 INFO:tasks.rados.rados.0.smithi001.stdout:278: writing smithi00132172-37 from 3342336 to 3801088 tid 3 2024-09-15T03:47:30.172 INFO:tasks.rados.rados.0.smithi001.stdout:279: snap_remove snap 28 2024-09-15T03:47:30.172 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:30.166+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:30.268 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:30.267+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:30.309 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:30.309 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:30.309 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:30.309 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:30.317 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:47:30.369 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:30.368+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:30.418 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:30.418 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:30.418 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:30.418 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:30.418 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:30.426 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:47:30.470 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:30.469+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:30.569 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:30.569 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:30.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:30.569 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:30.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=21, reqid=client.4521.0:184, at_version=33'6, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.460719363000003, 2024-09-15T03:47:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010901256999999999, 2024-09-15T03:47:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0200000000000001e-07 2024-09-15T03:47:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106529+0000", 2024-09-15T03:47:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9539e-05 2024-09-15T03:47:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106542+0000", 2024-09-15T03:47:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2731999999999999e-05 2024-09-15T03:47:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106549+0000", 2024-09-15T03:47:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3339999999999996e-06 2024-09-15T03:47:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106613+0000", 2024-09-15T03:47:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4527000000000001e-05 2024-09-15T03:47:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106622+0000", 2024-09-15T03:47:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.755e-06 2024-09-15T03:47:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106841+0000", 2024-09-15T03:47:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021906200000000001 2024-09-15T03:47:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106869+0000", 2024-09-15T03:47:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.739e-05 2024-09-15T03:47:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106948+0000", 2024-09-15T03:47:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9422e-05 2024-09-15T03:47:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106965+0000", 2024-09-15T03:47:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7010000000000001e-05 2024-09-15T03:47:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.117410+0000", 2024-09-15T03:47:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010445284000000001 2024-09-15T03:47:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.458805876, 2024-09-15T03:47:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015296944, 2024-09-15T03:47:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5499999999999998e-07 2024-09-15T03:47:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108447+0000", 2024-09-15T03:47:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3314000000000001e-05 2024-09-15T03:47:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108460+0000", 2024-09-15T03:47:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3444e-05 2024-09-15T03:47:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108464+0000", 2024-09-15T03:47:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3610000000000003e-06 2024-09-15T03:47:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108494+0000", 2024-09-15T03:47:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9688000000000001e-05 2024-09-15T03:47:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108502+0000", 2024-09-15T03:47:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.678e-06 2024-09-15T03:47:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108601+0000", 2024-09-15T03:47:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9748999999999997e-05 2024-09-15T03:47:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108610+0000", 2024-09-15T03:47:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2490000000000001e-06 2024-09-15T03:47:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109977+0000", 2024-09-15T03:47:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013669450000000001 2024-09-15T03:47:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109993+0000", 2024-09-15T03:47:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6688e-05 2024-09-15T03:47:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123720+0000", 2024-09-15T03:47:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013726172999999999 2024-09-15T03:47:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.458756215000001, 2024-09-15T03:47:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015105210000000001, 2024-09-15T03:47:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108473+0000", 2024-09-15T03:47:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.84e-07 2024-09-15T03:47:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108486+0000", 2024-09-15T03:47:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3845999999999999e-05 2024-09-15T03:47:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108491+0000", 2024-09-15T03:47:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4719999999999997e-06 2024-09-15T03:47:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108495+0000", 2024-09-15T03:47:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.822e-06 2024-09-15T03:47:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108546+0000", 2024-09-15T03:47:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0853000000000003e-05 2024-09-15T03:47:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108555+0000", 2024-09-15T03:47:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3319999999999998e-06 2024-09-15T03:47:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108638+0000", 2024-09-15T03:47:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3580999999999998e-05 2024-09-15T03:47:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109983+0000", 2024-09-15T03:47:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001344392 2024-09-15T03:47:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110023+0000", 2024-09-15T03:47:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9895999999999999e-05 2024-09-15T03:47:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110040+0000", 2024-09-15T03:47:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.747e-05 2024-09-15T03:47:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123577+0000", 2024-09-15T03:47:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013537262 2024-09-15T03:47:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.458722336000001, 2024-09-15T03:47:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015311313999999999, 2024-09-15T03:47:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8500000000000002e-07 2024-09-15T03:47:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108517+0000", 2024-09-15T03:47:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0543000000000001e-05 2024-09-15T03:47:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108524+0000", 2024-09-15T03:47:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7170000000000002e-06 2024-09-15T03:47:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108528+0000", 2024-09-15T03:47:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3970000000000004e-06 2024-09-15T03:47:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108632+0000", 2024-09-15T03:47:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103687 2024-09-15T03:47:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108637+0000", 2024-09-15T03:47:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4389999999999999e-06 2024-09-15T03:47:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108713+0000", 2024-09-15T03:47:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6223000000000005e-05 2024-09-15T03:47:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109984+0000", 2024-09-15T03:47:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012700719999999999 2024-09-15T03:47:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110023+0000", 2024-09-15T03:47:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9864999999999999e-05 2024-09-15T03:47:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110041+0000", 2024-09-15T03:47:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8040999999999999e-05 2024-09-15T03:47:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123817+0000", 2024-09-15T03:47:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013776045000000001 2024-09-15T03:47:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.458228728000002, 2024-09-15T03:47:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014864502, 2024-09-15T03:47:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:47:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109016+0000", 2024-09-15T03:47:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5889999999999999e-05 2024-09-15T03:47:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109029+0000", 2024-09-15T03:47:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2812e-05 2024-09-15T03:47:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109035+0000", 2024-09-15T03:47:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0109999999999997e-06 2024-09-15T03:47:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109073+0000", 2024-09-15T03:47:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8260000000000003e-05 2024-09-15T03:47:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109080+0000", 2024-09-15T03:47:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0190000000000001e-06 2024-09-15T03:47:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109165+0000", 2024-09-15T03:47:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4957000000000007e-05 2024-09-15T03:47:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109984+0000", 2024-09-15T03:47:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081866399999999998 2024-09-15T03:47:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110024+0000", 2024-09-15T03:47:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9644999999999997e-05 2024-09-15T03:47:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110072+0000", 2024-09-15T03:47:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8418999999999998e-05 2024-09-15T03:47:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123864+0000", 2024-09-15T03:47:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013792234 2024-09-15T03:47:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=27, reqid=client.4521.0:189, at_version=0'0, trim_to=0'0, roll_forward_to=33'10))", 2024-09-15T03:47:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.114624+0000", 2024-09-15T03:47:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.452604739000002, 2024-09-15T03:47:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092946850000000004, 2024-09-15T03:47:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114624+0000", 2024-09-15T03:47:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114624+0000", 2024-09-15T03:47:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114624+0000", 2024-09-15T03:47:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3600000000000004e-07 2024-09-15T03:47:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114633+0000", 2024-09-15T03:47:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1579999999999994e-06 2024-09-15T03:47:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114644+0000", 2024-09-15T03:47:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.04e-05 2024-09-15T03:47:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114649+0000", 2024-09-15T03:47:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8760000000000004e-06 2024-09-15T03:47:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114687+0000", 2024-09-15T03:47:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8438e-05 2024-09-15T03:47:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114697+0000", 2024-09-15T03:47:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6479999999999998e-06 2024-09-15T03:47:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114818+0000", 2024-09-15T03:47:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121116 2024-09-15T03:47:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114830+0000", 2024-09-15T03:47:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1844e-05 2024-09-15T03:47:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114921+0000", 2024-09-15T03:47:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1461999999999999e-05 2024-09-15T03:47:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.114934+0000", 2024-09-15T03:47:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2857000000000001e-05 2024-09-15T03:47:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123918+0000", 2024-09-15T03:47:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089843500000000003 2024-09-15T03:47:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=34, reqid=client.4521.0:197, at_version=34'21, trim_to=0'0, roll_forward_to=34'18))", 2024-09-15T03:47:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 49.443216847999999, 2024-09-15T03:47:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012992399, 2024-09-15T03:47:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124013+0000", 2024-09-15T03:47:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6299999999999993e-07 2024-09-15T03:47:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124044+0000", 2024-09-15T03:47:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0995999999999999e-05 2024-09-15T03:47:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124062+0000", 2024-09-15T03:47:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8413000000000001e-05 2024-09-15T03:47:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124069+0000", 2024-09-15T03:47:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3930000000000002e-06 2024-09-15T03:47:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124150+0000", 2024-09-15T03:47:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1097e-05 2024-09-15T03:47:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124161+0000", 2024-09-15T03:47:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0720000000000001e-05 2024-09-15T03:47:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124402+0000", 2024-09-15T03:47:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000240442 2024-09-15T03:47:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125141+0000", 2024-09-15T03:47:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00073915500000000002 2024-09-15T03:47:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125191+0000", 2024-09-15T03:47:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9951999999999999e-05 2024-09-15T03:47:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125251+0000", 2024-09-15T03:47:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9938000000000001e-05 2024-09-15T03:47:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.137004+0000", 2024-09-15T03:47:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011753329999999999 2024-09-15T03:47:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=35, reqid=client.4521.0:197, at_version=0'0, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:47:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 49.441276803999997, 2024-09-15T03:47:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011202456, 2024-09-15T03:47:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125953+0000", 2024-09-15T03:47:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9100000000000002e-07 2024-09-15T03:47:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125964+0000", 2024-09-15T03:47:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1399999999999999e-05 2024-09-15T03:47:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125979+0000", 2024-09-15T03:47:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4810000000000001e-05 2024-09-15T03:47:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126004+0000", 2024-09-15T03:47:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4913e-05 2024-09-15T03:47:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126059+0000", 2024-09-15T03:47:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5668999999999998e-05 2024-09-15T03:47:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126071+0000", 2024-09-15T03:47:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1219999999999999e-05 2024-09-15T03:47:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126151+0000", 2024-09-15T03:47:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0572e-05 2024-09-15T03:47:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126163+0000", 2024-09-15T03:47:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2101e-05 2024-09-15T03:47:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.127945+0000", 2024-09-15T03:47:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001781803 2024-09-15T03:47:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.127982+0000", 2024-09-15T03:47:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7129999999999999e-05 2024-09-15T03:47:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.137154+0000", 2024-09-15T03:47:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0091719469999999997 2024-09-15T03:47:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:243 3.3s0 3:c7c3ef16:::smithi00132172-25:head [create,setxattr iZnWfebab`ptjURR\\f`psW]tpS`qpUaeWkY[Oqep (1330) in=40b,setxattr f[mZ_qQmcZejQ`fbdeic_tkPUaTYcnm[WprjVmc] (12488) in=40b,setxattr tkZgQ]S_pdlhZnqrVaZRa`oQQr_USmUfhado^dXT (1121) in=40b,setxattr srWjTqXhcth`hfZmQY_fg]UqNjWofa^dfgOV`sYP (455) in=40b,setxattr ^_mdiRroj_TtQ^gcNj`XtkW[ooqY`PWXqYmcOk_n (138) in=40b,setxattr jQlajdtcalhkcfYaakZfNb[^_TjkfcS`Vrk]q_d[ (806) in=40b,setxattr ZfYgkVcrsXcnpchgfcbldNrUTasnQNmfZW]Nbh\\o (561) in=40b,setxattr _hkYOqjPNhmhUlSNktfeQpsqSWjO\\fbf[fpTlZiO (1227) in=40b,setxattr ]mdV^Xdc\\sViXdkXNU[js_Qhh[b[sbfnZVXkdids (11788) in=40b,setxattr tb^mbOrNhfrhtce^g (455) in=17b,setxattr koTkVgrfU\\m`StnZ`Rirjhbhg`PhVqm^a^Xgaaqn (35) in=40b,setxattr aUtZPUPXPnQmT]\\\\n\\lZpnXakdNkTldVQoQhPUf] (559) in=40b,setxattr mrhR]^U`Tteo[Topmql`SNma\\pqfk`if]OppnYt` (770) in=40b,setxattr Oetk]RrnWeXbVctVtVrhXf_RfTdelqX^lWp[sTkp (1226) in=40b,setxattr WtedXoOd`VqnbksOqQkatk`gdTnW^lXbUtrlYgkl (3576) in=40b,setxattr dddOWN\\j\\XXg^fr[rZcskeVX]jOQ^ZmoWepk[]\\a (17824) in=40b,setxattr TeRO^djPgNZiemQ_aRXgP[]jgUZa\\^\\iUgesaqaN (208) in=40b,setxattr teqm]mUobZbWjcdQ_jTa[f[k[Sd\\lh]n\\osigqkN (7436) in=40b,setxattr dphaol[XkeitbkomWYaQctWfataUhbSiNntkWo`P (1893) in=40b,setxattr oQsnhO]ZklSVYQ[WUpaelY[p[Qie^UepOt`Z[q (455) in=38b] snapc 7=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:47:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.398312916000002, 2024-09-15T03:47:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012365956000000001, 2024-09-15T03:47:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 243 2024-09-15T03:47:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168917+0000", 2024-09-15T03:47:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.725e-06 2024-09-15T03:47:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169030+0000", 2024-09-15T03:47:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113035 2024-09-15T03:47:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169047+0000", 2024-09-15T03:47:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6701000000000002e-05 2024-09-15T03:47:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169054+0000", 2024-09-15T03:47:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1369999999999996e-06 2024-09-15T03:47:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169140+0000", 2024-09-15T03:47:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6069999999999994e-05 2024-09-15T03:47:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169270+0000", 2024-09-15T03:47:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130071 2024-09-15T03:47:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.170661+0000", 2024-09-15T03:47:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001390666 2024-09-15T03:47:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171751+0000", 2024-09-15T03:47:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010899079999999999 2024-09-15T03:47:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171812+0000", 2024-09-15T03:47:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0797000000000001e-05 2024-09-15T03:47:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172279+0000", 2024-09-15T03:47:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046704500000000003 2024-09-15T03:47:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172299+0000", 2024-09-15T03:47:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0387999999999999e-05 2024-09-15T03:47:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.176110+0000", 2024-09-15T03:47:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038112340000000001 2024-09-15T03:47:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.181282+0000", 2024-09-15T03:47:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051711789999999997 2024-09-15T03:47:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:258 3.3s0 3:c9b663a2:::smithi00132172-7:head [delete] snapc 7=[6,5,3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e40)", 2024-09-15T03:47:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:47.146116+0000", 2024-09-15T03:47:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.421112098000002, 2024-09-15T03:47:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096428380000000008, 2024-09-15T03:47:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 258 2024-09-15T03:47:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146116+0000", 2024-09-15T03:47:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146116+0000", 2024-09-15T03:47:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146119+0000", 2024-09-15T03:47:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.227e-06 2024-09-15T03:47:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146127+0000", 2024-09-15T03:47:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1529999999999997e-06 2024-09-15T03:47:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146146+0000", 2024-09-15T03:47:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9318999999999999e-05 2024-09-15T03:47:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146154+0000", 2024-09-15T03:47:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7200000000000006e-06 2024-09-15T03:47:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146233+0000", 2024-09-15T03:47:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8725000000000003e-05 2024-09-15T03:47:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146244+0000", 2024-09-15T03:47:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1725e-05 2024-09-15T03:47:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.149207+0000", 2024-09-15T03:47:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029627999999999998 2024-09-15T03:47:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.149326+0000", 2024-09-15T03:47:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011869199999999999 2024-09-15T03:47:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.150266+0000", 2024-09-15T03:47:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00094048500000000002 2024-09-15T03:47:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.150626+0000", 2024-09-15T03:47:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035946699999999998 2024-09-15T03:47:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.150751+0000", 2024-09-15T03:47:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012544800000000001 2024-09-15T03:47:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.151656+0000", 2024-09-15T03:47:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00090447499999999996 2024-09-15T03:47:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.153489+0000", 2024-09-15T03:47:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001833115 2024-09-15T03:47:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.155171+0000", 2024-09-15T03:47:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001682455 2024-09-15T03:47:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.155759+0000", 2024-09-15T03:47:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00058803200000000005 2024-09-15T03:47:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/28 ECSubWrite(tid=15, reqid=client.4521.0:337, at_version=49'10, trim_to=0'0, roll_forward_to=43'8))", 2024-09-15T03:47:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 34.709323007000002, 2024-09-15T03:47:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010500893000000001, 2024-09-15T03:47:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857906+0000", 2024-09-15T03:47:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.63e-07 2024-09-15T03:47:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857928+0000", 2024-09-15T03:47:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2155999999999999e-05 2024-09-15T03:47:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857937+0000", 2024-09-15T03:47:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2359999999999993e-06 2024-09-15T03:47:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857941+0000", 2024-09-15T03:47:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0960000000000003e-06 2024-09-15T03:47:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857999+0000", 2024-09-15T03:47:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7407000000000001e-05 2024-09-15T03:47:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858011+0000", 2024-09-15T03:47:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1759999999999999e-05 2024-09-15T03:47:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858240+0000", 2024-09-15T03:47:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022949299999999999 2024-09-15T03:47:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858251+0000", 2024-09-15T03:47:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1168999999999999e-05 2024-09-15T03:47:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858332+0000", 2024-09-15T03:47:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1204000000000004e-05 2024-09-15T03:47:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858360+0000", 2024-09-15T03:47:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7767999999999999e-05 2024-09-15T03:47:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.868406+0000", 2024-09-15T03:47:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010046141 2024-09-15T03:47:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/28 ECSubWrite(tid=16, reqid=client.4521.0:337, at_version=0'0, trim_to=0'0, roll_forward_to=49'10))", 2024-09-15T03:47:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:47:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 34.707966978999998, 2024-09-15T03:47:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093024639999999999, 2024-09-15T03:47:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:47:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859261+0000", 2024-09-15T03:47:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859263+0000", 2024-09-15T03:47:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1969999999999999e-06 2024-09-15T03:47:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859275+0000", 2024-09-15T03:47:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2153999999999999e-05 2024-09-15T03:47:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859294+0000", 2024-09-15T03:47:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8875000000000001e-05 2024-09-15T03:47:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859301+0000", 2024-09-15T03:47:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9360000000000002e-06 2024-09-15T03:47:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859397+0000", 2024-09-15T03:47:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6831999999999997e-05 2024-09-15T03:47:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859407+0000", 2024-09-15T03:47:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0939999999999997e-06 2024-09-15T03:47:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859500+0000", 2024-09-15T03:47:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3338000000000007e-05 2024-09-15T03:47:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859528+0000", 2024-09-15T03:47:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8039999999999999e-05 2024-09-15T03:47:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859568+0000", 2024-09-15T03:47:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0221e-05 2024-09-15T03:47:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859583+0000", 2024-09-15T03:47:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5217e-05 2024-09-15T03:47:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.868564+0000", 2024-09-15T03:47:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089805600000000003 2024-09-15T03:47:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:349 3.3s0 3:c7c3ef16:::smithi00132172-25:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.728746981, 2024-09-15T03:47:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005145406, 2024-09-15T03:47:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 349 2024-09-15T03:47:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838485+0000", 2024-09-15T03:47:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0620000000000001e-06 2024-09-15T03:47:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838494+0000", 2024-09-15T03:47:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3179999999999996e-06 2024-09-15T03:47:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3689999999999998e-05 2024-09-15T03:47:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838812+0000", 2024-09-15T03:47:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029437300000000001 2024-09-15T03:47:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838905+0000", 2024-09-15T03:47:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3196999999999996e-05 2024-09-15T03:47:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838914+0000", 2024-09-15T03:47:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0890000000000006e-06 2024-09-15T03:47:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840773+0000", 2024-09-15T03:47:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018584820000000001 2024-09-15T03:47:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840889+0000", 2024-09-15T03:47:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115986 2024-09-15T03:47:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842016+0000", 2024-09-15T03:47:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001126878 2024-09-15T03:47:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842320+0000", 2024-09-15T03:47:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030409799999999999 2024-09-15T03:47:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842360+0000", 2024-09-15T03:47:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0750000000000001e-05 2024-09-15T03:47:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842448+0000", 2024-09-15T03:47:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7842000000000004e-05 2024-09-15T03:47:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844580+0000", 2024-09-15T03:47:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021313719999999999 2024-09-15T03:47:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843574+0000", 2024-09-15T03:47:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99899468199999997 2024-09-15T03:47:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843627+0000", 2024-09-15T03:47:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2587000000000003e-05 2024-09-15T03:47:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:350 3.3s0 3:c7c3ef16:::smithi00132172-25:head [rollback 14] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.728395319999997, 2024-09-15T03:47:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0050383839999999, 2024-09-15T03:47:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 350 2024-09-15T03:47:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838834+0000", 2024-09-15T03:47:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0200000000000001e-07 2024-09-15T03:47:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838839+0000", 2024-09-15T03:47:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9790000000000001e-06 2024-09-15T03:47:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838852+0000", 2024-09-15T03:47:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3138e-05 2024-09-15T03:47:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838858+0000", 2024-09-15T03:47:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8320000000000002e-06 2024-09-15T03:47:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839003+0000", 2024-09-15T03:47:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014513500000000001 2024-09-15T03:47:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839011+0000", 2024-09-15T03:47:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7719999999999994e-06 2024-09-15T03:47:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842395+0000", 2024-09-15T03:47:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033840110000000001 2024-09-15T03:47:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842580+0000", 2024-09-15T03:47:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000185067 2024-09-15T03:47:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843394+0000", 2024-09-15T03:47:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081377299999999995 2024-09-15T03:47:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843510+0000", 2024-09-15T03:47:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0001165 2024-09-15T03:47:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843524+0000", 2024-09-15T03:47:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4022999999999999e-05 2024-09-15T03:47:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843579+0000", 2024-09-15T03:47:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4654999999999998e-05 2024-09-15T03:47:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844583+0000", 2024-09-15T03:47:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010042429999999999 2024-09-15T03:47:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:30.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843772+0000", 2024-09-15T03:47:30.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99918911200000005 2024-09-15T03:47:30.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843872+0000", 2024-09-15T03:47:30.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9341999999999999e-05 2024-09-15T03:47:30.661 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.661 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.661 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.662 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:351 3.3s0 3:c7c3ef16:::smithi00132172-25:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:30.662 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:30.662 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.728337687, 2024-09-15T03:47:30.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0051167679999999, 2024-09-15T03:47:30.662 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.662 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:30.662 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:30.662 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:30.662 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:30.662 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 351 2024-09-15T03:47:30.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0800000000000004e-07 2024-09-15T03:47:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838895+0000", 2024-09-15T03:47:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.241e-06 2024-09-15T03:47:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838904+0000", 2024-09-15T03:47:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9549999999999998e-06 2024-09-15T03:47:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838909+0000", 2024-09-15T03:47:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7869999999999998e-06 2024-09-15T03:47:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839045+0000", 2024-09-15T03:47:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013556399999999999 2024-09-15T03:47:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839052+0000", 2024-09-15T03:47:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8199999999999999e-06 2024-09-15T03:47:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843587+0000", 2024-09-15T03:47:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045349739999999998 2024-09-15T03:47:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843708+0000", 2024-09-15T03:47:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120776 2024-09-15T03:47:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844246+0000", 2024-09-15T03:47:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053838200000000001 2024-09-15T03:47:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844335+0000", 2024-09-15T03:47:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9240999999999995e-05 2024-09-15T03:47:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844347+0000", 2024-09-15T03:47:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2187e-05 2024-09-15T03:47:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844460+0000", 2024-09-15T03:47:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112571 2024-09-15T03:47:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844584+0000", 2024-09-15T03:47:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012403800000000001 2024-09-15T03:47:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843972+0000", 2024-09-15T03:47:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99938787299999998 2024-09-15T03:47:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844008+0000", 2024-09-15T03:47:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5750999999999997e-05 2024-09-15T03:47:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.721203736, 2024-09-15T03:47:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013010173, 2024-09-15T03:47:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846026+0000", 2024-09-15T03:47:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1230000000000001e-06 2024-09-15T03:47:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846047+0000", 2024-09-15T03:47:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0774000000000001e-05 2024-09-15T03:47:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846064+0000", 2024-09-15T03:47:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7560000000000001e-05 2024-09-15T03:47:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846072+0000", 2024-09-15T03:47:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5020000000000003e-06 2024-09-15T03:47:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846187+0000", 2024-09-15T03:47:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115717 2024-09-15T03:47:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846201+0000", 2024-09-15T03:47:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3329999999999999e-05 2024-09-15T03:47:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846303+0000", 2024-09-15T03:47:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101807 2024-09-15T03:47:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846312+0000", 2024-09-15T03:47:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5529999999999998e-06 2024-09-15T03:47:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846382+0000", 2024-09-15T03:47:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0146000000000002e-05 2024-09-15T03:47:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846520+0000", 2024-09-15T03:47:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013814099999999999 2024-09-15T03:47:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.859035+0000", 2024-09-15T03:47:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012514519999999999 2024-09-15T03:47:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.721106988999999, 2024-09-15T03:47:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013044694000000001, 2024-09-15T03:47:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846122+0000", 2024-09-15T03:47:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0399999999999996e-07 2024-09-15T03:47:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846174+0000", 2024-09-15T03:47:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1672999999999998e-05 2024-09-15T03:47:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846184+0000", 2024-09-15T03:47:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0514e-05 2024-09-15T03:47:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846190+0000", 2024-09-15T03:47:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2820000000000002e-06 2024-09-15T03:47:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846341+0000", 2024-09-15T03:47:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150803 2024-09-15T03:47:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846351+0000", 2024-09-15T03:47:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3899999999999999e-06 2024-09-15T03:47:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846450+0000", 2024-09-15T03:47:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9839000000000004e-05 2024-09-15T03:47:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846482+0000", 2024-09-15T03:47:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1089e-05 2024-09-15T03:47:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846615+0000", 2024-09-15T03:47:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013305899999999999 2024-09-15T03:47:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846634+0000", 2024-09-15T03:47:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8899999999999999e-05 2024-09-15T03:47:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.859166+0000", 2024-09-15T03:47:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012532641000000001 2024-09-15T03:47:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 60/31 ECSubWrite(tid=70, reqid=client.4521.0:421, at_version=60'22, trim_to=0'0, roll_forward_to=47'20))", 2024-09-15T03:47:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:06.837638+0000", 2024-09-15T03:47:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.729590914999999, 2024-09-15T03:47:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092785220000000009, 2024-09-15T03:47:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837638+0000", 2024-09-15T03:47:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837638+0000", 2024-09-15T03:47:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837638+0000", 2024-09-15T03:47:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6399999999999998e-07 2024-09-15T03:47:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837664+0000", 2024-09-15T03:47:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5788000000000001e-05 2024-09-15T03:47:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837675+0000", 2024-09-15T03:47:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1269e-05 2024-09-15T03:47:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837682+0000", 2024-09-15T03:47:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5120000000000002e-06 2024-09-15T03:47:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837771+0000", 2024-09-15T03:47:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8802999999999997e-05 2024-09-15T03:47:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.837793+0000", 2024-09-15T03:47:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2141e-05 2024-09-15T03:47:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.838114+0000", 2024-09-15T03:47:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000320453 2024-09-15T03:47:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.838243+0000", 2024-09-15T03:47:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012918599999999999 2024-09-15T03:47:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.838298+0000", 2024-09-15T03:47:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5164999999999998e-05 2024-09-15T03:47:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.838665+0000", 2024-09-15T03:47:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036705199999999998 2024-09-15T03:47:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.846916+0000", 2024-09-15T03:47:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082508890000000008 2024-09-15T03:47:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 60/28 ECSubWrite(tid=73, reqid=client.4521.0:424, at_version=60'41, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.666798142000001, 2024-09-15T03:47:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011124821, 2024-09-15T03:47:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900431+0000", 2024-09-15T03:47:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2900000000000003e-07 2024-09-15T03:47:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900451+0000", 2024-09-15T03:47:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0066999999999999e-05 2024-09-15T03:47:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900466+0000", 2024-09-15T03:47:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5062000000000001e-05 2024-09-15T03:47:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900474+0000", 2024-09-15T03:47:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6310000000000003e-06 2024-09-15T03:47:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900524+0000", 2024-09-15T03:47:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0037000000000002e-05 2024-09-15T03:47:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900535+0000", 2024-09-15T03:47:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1209000000000001e-05 2024-09-15T03:47:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900659+0000", 2024-09-15T03:47:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012441000000000001 2024-09-15T03:47:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900971+0000", 2024-09-15T03:47:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031183799999999999 2024-09-15T03:47:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901144+0000", 2024-09-15T03:47:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172261 2024-09-15T03:47:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901734+0000", 2024-09-15T03:47:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059069700000000005 2024-09-15T03:47:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.911555+0000", 2024-09-15T03:47:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0098208800000000006 2024-09-15T03:47:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 60/28 ECSubWrite(tid=74, reqid=client.4521.0:425, at_version=60'42, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:06.901240+0000", 2024-09-15T03:47:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.665988562999999, 2024-09-15T03:47:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010429678, 2024-09-15T03:47:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901240+0000", 2024-09-15T03:47:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901240+0000", 2024-09-15T03:47:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901241+0000", 2024-09-15T03:47:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7999999999999993e-07 2024-09-15T03:47:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901273+0000", 2024-09-15T03:47:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2087e-05 2024-09-15T03:47:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901302+0000", 2024-09-15T03:47:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8705e-05 2024-09-15T03:47:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901311+0000", 2024-09-15T03:47:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9190000000000005e-06 2024-09-15T03:47:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901797+0000", 2024-09-15T03:47:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048606499999999999 2024-09-15T03:47:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901808+0000", 2024-09-15T03:47:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1075e-05 2024-09-15T03:47:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901934+0000", 2024-09-15T03:47:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012674299999999999 2024-09-15T03:47:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901964+0000", 2024-09-15T03:47:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9833999999999999e-05 2024-09-15T03:47:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.902006+0000", 2024-09-15T03:47:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1823999999999998e-05 2024-09-15T03:47:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:30.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.902020+0000", 2024-09-15T03:47:30.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.363e-05 2024-09-15T03:47:30.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:30.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:30.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:30.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.911670+0000", 2024-09-15T03:47:30.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096498160000000003 2024-09-15T03:47:30.697 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.697 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.697 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.697 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:30.697 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:30.697 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:30.699 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:47:30.700 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:30.698+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:30.800 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:30.799+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:30.801 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:30.801 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:30.801 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:30.801 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:30.809 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:47:30.901 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:30.900+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:30.910 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:30.911 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:30.911 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:30.911 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:30.911 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:30.919 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:47:30.982 INFO:tasks.rados.rados.0.smithi001.stdout:279: done (10 left) 2024-09-15T03:47:30.983 INFO:tasks.rados.rados.0.smithi001.stdout:280: rollback oid 20 current snap is 32 2024-09-15T03:47:30.983 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 20 to 30 2024-09-15T03:47:30.983 INFO:tasks.rados.rados.0.smithi001.stdout:272: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-15T03:47:31.003 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:31.003+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:31.006 INFO:tasks.rados.rados.0.smithi001.stdout:277: finishing rollback tid 0 to smithi00132172-19 2024-09-15T03:47:31.006 INFO:tasks.rados.rados.0.smithi001.stdout:273: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-15T03:47:31.016 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 25 v 95 (ObjNum 66 snap 6 seq_num 66) dirty exists 2024-09-15T03:47:31.016 INFO:tasks.rados.rados.0.smithi001.stdout:277: finishing rollback tid 1 to smithi00132172-19 2024-09-15T03:47:31.016 INFO:tasks.rados.rados.0.smithi001.stdout:277: finishing rollback tid 2 to smithi00132172-19 2024-09-15T03:47:31.016 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 19 v 100 (ObjNum 66 snap 6 seq_num 66) dirty exists 2024-09-15T03:47:31.017 INFO:tasks.rados.rados.0.smithi001.stdout:271: expect (ObjNum 53 snap 0 seq_num 53) 2024-09-15T03:47:31.066 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:197 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [delete] snapc 3=[3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e34)", 2024-09-15T03:47:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 49.951467174999998, 2024-09-15T03:47:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021934704999999999, 2024-09-15T03:47:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197 2024-09-15T03:47:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112266+0000", 2024-09-15T03:47:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5000000000000002e-07 2024-09-15T03:47:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112269+0000", 2024-09-15T03:47:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.066e-06 2024-09-15T03:47:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112276+0000", 2024-09-15T03:47:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-09-15T03:47:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112281+0000", 2024-09-15T03:47:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3970000000000004e-06 2024-09-15T03:47:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123281+0000", 2024-09-15T03:47:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011000575 2024-09-15T03:47:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123395+0000", 2024-09-15T03:47:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113398 2024-09-15T03:47:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123963+0000", 2024-09-15T03:47:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000568101 2024-09-15T03:47:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124220+0000", 2024-09-15T03:47:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025704900000000003 2024-09-15T03:47:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124229+0000", 2024-09-15T03:47:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4860000000000003e-06 2024-09-15T03:47:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124277+0000", 2024-09-15T03:47:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7574000000000002e-05 2024-09-15T03:47:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124312+0000", 2024-09-15T03:47:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5256000000000003e-05 2024-09-15T03:47:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125716+0000", 2024-09-15T03:47:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014043720000000001 2024-09-15T03:47:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.134200+0000", 2024-09-15T03:47:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084835120000000003 2024-09-15T03:47:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es4 34/28 ECSubWrite(tid=33, reqid=client.4521.0:195, at_version=34'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-15T03:47:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:47:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 49.945639816000003, 2024-09-15T03:47:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015907482000000001, 2024-09-15T03:47:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:47:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:47:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118093+0000", 2024-09-15T03:47:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9400000000000004e-07 2024-09-15T03:47:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118299+0000", 2024-09-15T03:47:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000205677 2024-09-15T03:47:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118315+0000", 2024-09-15T03:47:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5429000000000001e-05 2024-09-15T03:47:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118322+0000", 2024-09-15T03:47:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8290000000000008e-06 2024-09-15T03:47:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118416+0000", 2024-09-15T03:47:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3140000000000006e-05 2024-09-15T03:47:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118431+0000", 2024-09-15T03:47:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5648e-05 2024-09-15T03:47:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118678+0000", 2024-09-15T03:47:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024701200000000001 2024-09-15T03:47:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.118691+0000", 2024-09-15T03:47:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2653000000000001e-05 2024-09-15T03:47:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.119431+0000", 2024-09-15T03:47:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00074060399999999996 2024-09-15T03:47:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124280+0000", 2024-09-15T03:47:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0048484410000000002 2024-09-15T03:47:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.134000+0000", 2024-09-15T03:47:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0097201549999999994 2024-09-15T03:47:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:220 3.bs0 3:d99ddedc:::smithi00132172-44:head [copy-from ver 22] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.936087505000003, 2024-09-15T03:47:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024273429999999999, 2024-09-15T03:47:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 220 2024-09-15T03:47:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127647+0000", 2024-09-15T03:47:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1799999999999999e-06 2024-09-15T03:47:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127656+0000", 2024-09-15T03:47:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3759999999999997e-06 2024-09-15T03:47:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127674+0000", 2024-09-15T03:47:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7374e-05 2024-09-15T03:47:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127680+0000", 2024-09-15T03:47:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6949999999999997e-06 2024-09-15T03:47:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127736+0000", 2024-09-15T03:47:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6011999999999998e-05 2024-09-15T03:47:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127744+0000", 2024-09-15T03:47:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2870000000000004e-06 2024-09-15T03:47:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130338+0000", 2024-09-15T03:47:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025940249999999998 2024-09-15T03:47:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130423+0000", 2024-09-15T03:47:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5303000000000006e-05 2024-09-15T03:47:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.147840+0000", 2024-09-15T03:47:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017416580000000001 2024-09-15T03:47:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148318+0000", 2024-09-15T03:47:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047822899999999999 2024-09-15T03:47:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148329+0000", 2024-09-15T03:47:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0611000000000001e-05 2024-09-15T03:47:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148488+0000", 2024-09-15T03:47:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015938399999999999 2024-09-15T03:47:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148870+0000", 2024-09-15T03:47:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000382405 2024-09-15T03:47:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.151092+0000", 2024-09-15T03:47:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022212529999999999 2024-09-15T03:47:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.151918+0000", 2024-09-15T03:47:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082671600000000002 2024-09-15T03:47:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.087 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:221 3.bs0 3:d99ddedc:::smithi00132172-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:31.087 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:31.087 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.936043349999998, 2024-09-15T03:47:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021194474000000001, 2024-09-15T03:47:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 221 2024-09-15T03:47:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127690+0000", 2024-09-15T03:47:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:47:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127692+0000", 2024-09-15T03:47:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6589999999999999e-06 2024-09-15T03:47:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127700+0000", 2024-09-15T03:47:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4769999999999997e-06 2024-09-15T03:47:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127704+0000", 2024-09-15T03:47:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3819999999999997e-06 2024-09-15T03:47:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127805+0000", 2024-09-15T03:47:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100639 2024-09-15T03:47:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127809+0000", 2024-09-15T03:47:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8870000000000002e-06 2024-09-15T03:47:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130527+0000", 2024-09-15T03:47:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002718579 2024-09-15T03:47:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130558+0000", 2024-09-15T03:47:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0827999999999997e-05 2024-09-15T03:47:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148546+0000", 2024-09-15T03:47:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017988356 2024-09-15T03:47:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148683+0000", 2024-09-15T03:47:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013650899999999999 2024-09-15T03:47:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148884+0000", 2024-09-15T03:47:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020061499999999999 2024-09-15T03:47:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.094 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:31.094 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.094 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:252 3.bs0 3:d6f79b7b:::smithi00132172-29:head [copy-from ver 6] snapc 7=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:47:31.094 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.169146+0000", 2024-09-15T03:47:31.094 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.894586081, 2024-09-15T03:47:31.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016731068000000002, 2024-09-15T03:47:31.094 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:31.094 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 252 2024-09-15T03:47:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169146+0000", 2024-09-15T03:47:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169146+0000", 2024-09-15T03:47:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169147+0000", 2024-09-15T03:47:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1400000000000001e-07 2024-09-15T03:47:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169149+0000", 2024-09-15T03:47:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0279999999999999e-06 2024-09-15T03:47:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169152+0000", 2024-09-15T03:47:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0800000000000002e-06 2024-09-15T03:47:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169154+0000", 2024-09-15T03:47:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4679999999999999e-06 2024-09-15T03:47:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.174427+0000", 2024-09-15T03:47:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052724349999999998 2024-09-15T03:47:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.174543+0000", 2024-09-15T03:47:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116803 2024-09-15T03:47:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.178297+0000", 2024-09-15T03:47:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037531539999999999 2024-09-15T03:47:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179112+0000", 2024-09-15T03:47:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081548600000000003 2024-09-15T03:47:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179151+0000", 2024-09-15T03:47:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9158e-05 2024-09-15T03:47:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179272+0000", 2024-09-15T03:47:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120802 2024-09-15T03:47:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179531+0000", 2024-09-15T03:47:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000259351 2024-09-15T03:47:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.179723+0000", 2024-09-15T03:47:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000191447 2024-09-15T03:47:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.185877+0000", 2024-09-15T03:47:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061546420000000001 2024-09-15T03:47:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:357 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.214691815999998, 2024-09-15T03:47:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.02017917, 2024-09-15T03:47:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 357 2024-09-15T03:47:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849043+0000", 2024-09-15T03:47:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.295e-06 2024-09-15T03:47:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849051+0000", 2024-09-15T03:47:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4079999999999998e-06 2024-09-15T03:47:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849069+0000", 2024-09-15T03:47:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7462999999999999e-05 2024-09-15T03:47:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849077+0000", 2024-09-15T03:47:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8869999999999992e-06 2024-09-15T03:47:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849194+0000", 2024-09-15T03:47:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011723899999999999 2024-09-15T03:47:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849296+0000", 2024-09-15T03:47:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102011 2024-09-15T03:47:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861106+0000", 2024-09-15T03:47:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0118097610000001 2024-09-15T03:47:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861222+0000", 2024-09-15T03:47:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116561 2024-09-15T03:47:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.869220+0000", 2024-09-15T03:47:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007997545 2024-09-15T03:47:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:358 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.21463687, 2024-09-15T03:47:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0228638050000001, 2024-09-15T03:47:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 358 2024-09-15T03:47:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4099999999999997e-07 2024-09-15T03:47:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849101+0000", 2024-09-15T03:47:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8180000000000003e-06 2024-09-15T03:47:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849110+0000", 2024-09-15T03:47:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1409999999999994e-06 2024-09-15T03:47:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849116+0000", 2024-09-15T03:47:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2390000000000002e-06 2024-09-15T03:47:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849326+0000", 2024-09-15T03:47:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000210029 2024-09-15T03:47:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849421+0000", 2024-09-15T03:47:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5775999999999999e-05 2024-09-15T03:47:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861647+0000", 2024-09-15T03:47:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0122257139999999 2024-09-15T03:47:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861777+0000", 2024-09-15T03:47:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130087 2024-09-15T03:47:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.871959+0000", 2024-09-15T03:47:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010182160000000001 2024-09-15T03:47:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:359 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.214611718, 2024-09-15T03:47:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0240468149999999, 2024-09-15T03:47:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 359 2024-09-15T03:47:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:47:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849124+0000", 2024-09-15T03:47:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2679999999999999e-06 2024-09-15T03:47:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849132+0000", 2024-09-15T03:47:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4030000000000001e-06 2024-09-15T03:47:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849136+0000", 2024-09-15T03:47:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2289999999999997e-06 2024-09-15T03:47:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849446+0000", 2024-09-15T03:47:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031043299999999999 2024-09-15T03:47:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849553+0000", 2024-09-15T03:47:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106415 2024-09-15T03:47:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.863404+0000", 2024-09-15T03:47:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0138511029999999 2024-09-15T03:47:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.863475+0000", 2024-09-15T03:47:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0814000000000003e-05 2024-09-15T03:47:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.873168+0000", 2024-09-15T03:47:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00969274 2024-09-15T03:47:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:474 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~786432] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.188687043000002, 2024-09-15T03:47:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98604943700000003, 2024-09-15T03:47:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 474 2024-09-15T03:47:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875048+0000", 2024-09-15T03:47:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1600000000000001e-06 2024-09-15T03:47:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875844+0000", 2024-09-15T03:47:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079686299999999998 2024-09-15T03:47:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875857+0000", 2024-09-15T03:47:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2889e-05 2024-09-15T03:47:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875864+0000", 2024-09-15T03:47:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6139999999999997e-06 2024-09-15T03:47:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875916+0000", 2024-09-15T03:47:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1992e-05 2024-09-15T03:47:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875918+0000", 2024-09-15T03:47:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3300000000000001e-06 2024-09-15T03:47:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.845918+0000", 2024-09-15T03:47:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96999959199999997 2024-09-15T03:47:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.846229+0000", 2024-09-15T03:47:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000311601 2024-09-15T03:47:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848445+0000", 2024-09-15T03:47:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022160499999999998 2024-09-15T03:47:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848880+0000", 2024-09-15T03:47:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043432600000000002 2024-09-15T03:47:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848952+0000", 2024-09-15T03:47:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2509000000000007e-05 2024-09-15T03:47:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.849311+0000", 2024-09-15T03:47:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000358611 2024-09-15T03:47:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851131+0000", 2024-09-15T03:47:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001820115 2024-09-15T03:47:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861048+0000", 2024-09-15T03:47:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099164830000000002 2024-09-15T03:47:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861095+0000", 2024-09-15T03:47:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7302000000000002e-05 2024-09-15T03:47:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:475 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~491520] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.185920572000001, 2024-09-15T03:47:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98407319000000004, 2024-09-15T03:47:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 475 2024-09-15T03:47:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877813+0000", 2024-09-15T03:47:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8800000000000002e-07 2024-09-15T03:47:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878260+0000", 2024-09-15T03:47:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044732099999999998 2024-09-15T03:47:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878270+0000", 2024-09-15T03:47:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9799999999999993e-06 2024-09-15T03:47:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878276+0000", 2024-09-15T03:47:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2269999999999998e-06 2024-09-15T03:47:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878313+0000", 2024-09-15T03:47:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6570999999999999e-05 2024-09-15T03:47:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878315+0000", 2024-09-15T03:47:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8729999999999999e-06 2024-09-15T03:47:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851316+0000", 2024-09-15T03:47:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97300154800000005 2024-09-15T03:47:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851440+0000", 2024-09-15T03:47:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124287 2024-09-15T03:47:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852732+0000", 2024-09-15T03:47:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001291174 2024-09-15T03:47:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852991+0000", 2024-09-15T03:47:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025905299999999997 2024-09-15T03:47:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853003+0000", 2024-09-15T03:47:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2908000000000001e-05 2024-09-15T03:47:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853104+0000", 2024-09-15T03:47:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010011800000000001 2024-09-15T03:47:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854342+0000", 2024-09-15T03:47:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012380189999999999 2024-09-15T03:47:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861206+0000", 2024-09-15T03:47:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068648499999999996 2024-09-15T03:47:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861885+0000", 2024-09-15T03:47:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00067857300000000002 2024-09-15T03:47:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:476 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~278528] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.184293559, 2024-09-15T03:47:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98268067299999995, 2024-09-15T03:47:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 476 2024-09-15T03:47:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:31.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:31.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:31.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:31.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.13e-07 2024-09-15T03:47:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879706+0000", 2024-09-15T03:47:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026656100000000001 2024-09-15T03:47:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879715+0000", 2024-09-15T03:47:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9330000000000007e-06 2024-09-15T03:47:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879720+0000", 2024-09-15T03:47:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0880000000000002e-06 2024-09-15T03:47:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879743+0000", 2024-09-15T03:47:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2654e-05 2024-09-15T03:47:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879743+0000", 2024-09-15T03:47:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.99e-07 2024-09-15T03:47:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853170+0000", 2024-09-15T03:47:31.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97342687900000002 2024-09-15T03:47:31.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853297+0000", 2024-09-15T03:47:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127157 2024-09-15T03:47:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854150+0000", 2024-09-15T03:47:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085288900000000001 2024-09-15T03:47:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854292+0000", 2024-09-15T03:47:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014199300000000001 2024-09-15T03:47:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854559+0000", 2024-09-15T03:47:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026668999999999998 2024-09-15T03:47:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855998+0000", 2024-09-15T03:47:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001438714 2024-09-15T03:47:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:31.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856547+0000", 2024-09-15T03:47:31.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054880399999999996 2024-09-15T03:47:31.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:31.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861858+0000", 2024-09-15T03:47:31.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053110320000000003 2024-09-15T03:47:31.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:31.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.862120+0000", 2024-09-15T03:47:31.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026196699999999999 2024-09-15T03:47:31.144 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.144 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:31.144 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.144 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:477 3.bs0 3:d52faf55:::smithi00132172-40:head [setxattr _header (57) in=7b] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:31.144 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:31.144 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.184005622000001, 2024-09-15T03:47:31.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98562198000000001, 2024-09-15T03:47:31.144 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:31.145 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:31.145 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:31.145 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:31.145 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.145 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 477 2024-09-15T03:47:31.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.145 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:31.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:31.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:31.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:31.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:31.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:31.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:31.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2800000000000002e-07 2024-09-15T03:47:31.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:31.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879730+0000", 2024-09-15T03:47:31.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0589999999999998e-06 2024-09-15T03:47:31.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:31.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879737+0000", 2024-09-15T03:47:31.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0419999999999997e-06 2024-09-15T03:47:31.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879741+0000", 2024-09-15T03:47:31.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9960000000000004e-06 2024-09-15T03:47:31.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879756+0000", 2024-09-15T03:47:31.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4681999999999999e-05 2024-09-15T03:47:31.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:31.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879756+0000", 2024-09-15T03:47:31.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.22e-07 2024-09-15T03:47:31.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854590+0000", 2024-09-15T03:47:31.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97483319000000002 2024-09-15T03:47:31.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:31.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854718+0000", 2024-09-15T03:47:31.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012842100000000001 2024-09-15T03:47:31.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:31.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855416+0000", 2024-09-15T03:47:31.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069759200000000003 2024-09-15T03:47:31.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:31.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855551+0000", 2024-09-15T03:47:31.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013517600000000001 2024-09-15T03:47:31.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:31.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856002+0000", 2024-09-15T03:47:31.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045169399999999999 2024-09-15T03:47:31.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:31.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856065+0000", 2024-09-15T03:47:31.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3028999999999996e-05 2024-09-15T03:47:31.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:31.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856554+0000", 2024-09-15T03:47:31.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000488486 2024-09-15T03:47:31.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:31.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865278+0000", 2024-09-15T03:47:31.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087241929999999999 2024-09-15T03:47:31.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:31.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865349+0000", 2024-09-15T03:47:31.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0669999999999999e-05 2024-09-15T03:47:31.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.152 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:31.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.152 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:478 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:31.152 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:31.152 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.183986826999998, 2024-09-15T03:47:31.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99471090500000003, 2024-09-15T03:47:31.153 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:31.153 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:31.153 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:31.153 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:31.153 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.153 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 478 2024-09-15T03:47:31.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.153 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:31.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:31.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:31.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:31.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:31.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:31.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:31.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2599999999999998e-07 2024-09-15T03:47:31.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:31.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879748+0000", 2024-09-15T03:47:31.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9609999999999999e-06 2024-09-15T03:47:31.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:31.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879755+0000", 2024-09-15T03:47:31.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1879999999999996e-06 2024-09-15T03:47:31.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879760+0000", 2024-09-15T03:47:31.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2560000000000002e-06 2024-09-15T03:47:31.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879797+0000", 2024-09-15T03:47:31.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6452999999999998e-05 2024-09-15T03:47:31.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:31.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879799+0000", 2024-09-15T03:47:31.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6619999999999999e-06 2024-09-15T03:47:31.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855637+0000", 2024-09-15T03:47:31.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97583850999999999 2024-09-15T03:47:31.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:31.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855763+0000", 2024-09-15T03:47:31.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012612699999999999 2024-09-15T03:47:31.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865707+0000", 2024-09-15T03:47:31.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099439660000000003 2024-09-15T03:47:31.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:31.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865803+0000", 2024-09-15T03:47:31.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5946999999999995e-05 2024-09-15T03:47:31.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:31.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.874456+0000", 2024-09-15T03:47:31.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0086534090000000008 2024-09-15T03:47:31.158 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.158 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:31.158 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.158 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:486 3.bs0 3:df8b18a3:::smithi00132172-17:head [delete] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e64)", 2024-09-15T03:47:31.158 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:31.158 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.181174347999999, 2024-09-15T03:47:31.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98362617100000005, 2024-09-15T03:47:31.159 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:31.159 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:31.159 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:31.159 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:31.159 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.159 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 486 2024-09-15T03:47:31.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.159 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:31.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:31.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:31.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:31.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:31.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:31.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882559+0000", 2024-09-15T03:47:31.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0359999999999999e-06 2024-09-15T03:47:31.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:31.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882564+0000", 2024-09-15T03:47:31.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6530000000000002e-06 2024-09-15T03:47:31.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:31.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882574+0000", 2024-09-15T03:47:31.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0464e-05 2024-09-15T03:47:31.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882578+0000", 2024-09-15T03:47:31.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2459999999999997e-06 2024-09-15T03:47:31.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882624+0000", 2024-09-15T03:47:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5698000000000002e-05 2024-09-15T03:47:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882627+0000", 2024-09-15T03:47:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3350000000000001e-06 2024-09-15T03:47:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856574+0000", 2024-09-15T03:47:31.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97394739399999997 2024-09-15T03:47:31.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:31.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856853+0000", 2024-09-15T03:47:31.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027918399999999998 2024-09-15T03:47:31.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:31.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.866184+0000", 2024-09-15T03:47:31.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093311609999999993 2024-09-15T03:47:31.163 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.163 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:31.164 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.164 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:495 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [copy-from ver 51] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-15T03:47:31.164 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:31.164 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.175273299000001, 2024-09-15T03:47:31.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023555178, 2024-09-15T03:47:31.164 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:31.164 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:31.164 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:31.164 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:31.164 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.164 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 495 2024-09-15T03:47:31.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.165 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:31.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:31.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:31.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:31.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:31.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:31.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888461+0000", 2024-09-15T03:47:31.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.719e-06 2024-09-15T03:47:31.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:31.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888471+0000", 2024-09-15T03:47:31.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0161e-05 2024-09-15T03:47:31.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:31.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888500+0000", 2024-09-15T03:47:31.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8989e-05 2024-09-15T03:47:31.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888508+0000", 2024-09-15T03:47:31.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5299999999999999e-06 2024-09-15T03:47:31.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888554+0000", 2024-09-15T03:47:31.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6489000000000002e-05 2024-09-15T03:47:31.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:31.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888809+0000", 2024-09-15T03:47:31.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000255467 2024-09-15T03:47:31.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:31.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.905724+0000", 2024-09-15T03:47:31.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016914348999999999 2024-09-15T03:47:31.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:31.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906244+0000", 2024-09-15T03:47:31.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052054900000000003 2024-09-15T03:47:31.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:31.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906262+0000", 2024-09-15T03:47:31.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7499999999999998e-05 2024-09-15T03:47:31.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:31.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.907367+0000", 2024-09-15T03:47:31.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011049460000000001 2024-09-15T03:47:31.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:31.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.907384+0000", 2024-09-15T03:47:31.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7000999999999999e-05 2024-09-15T03:47:31.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:31.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.911965+0000", 2024-09-15T03:47:31.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045809470000000001 2024-09-15T03:47:31.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:31.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.912014+0000", 2024-09-15T03:47:31.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9530999999999997e-05 2024-09-15T03:47:31.170 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.171 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:31.171 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.171 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:496 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-15T03:47:31.171 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.888524+0000", 2024-09-15T03:47:31.171 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.175208207000001, 2024-09-15T03:47:31.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01823953, 2024-09-15T03:47:31.171 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:31.171 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:31.171 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:31.171 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:31.171 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.171 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 496 2024-09-15T03:47:31.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.172 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:31.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:31.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888524+0000", 2024-09-15T03:47:31.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:31.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888524+0000", 2024-09-15T03:47:31.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:31.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888525+0000", 2024-09-15T03:47:31.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2699999999999999e-07 2024-09-15T03:47:31.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:31.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888529+0000", 2024-09-15T03:47:31.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7840000000000001e-06 2024-09-15T03:47:31.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:31.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888540+0000", 2024-09-15T03:47:31.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1477e-05 2024-09-15T03:47:31.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888546+0000", 2024-09-15T03:47:31.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4990000000000002e-06 2024-09-15T03:47:31.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888971+0000", 2024-09-15T03:47:31.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042491300000000002 2024-09-15T03:47:31.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:31.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.889024+0000", 2024-09-15T03:47:31.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3046e-05 2024-09-15T03:47:31.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906517+0000", 2024-09-15T03:47:31.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017492909000000001 2024-09-15T03:47:31.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:31.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906638+0000", 2024-09-15T03:47:31.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012138000000000001 2024-09-15T03:47:31.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:31.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906764+0000", 2024-09-15T03:47:31.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125795 2024-09-15T03:47:31.176 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.176 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:31.176 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.176 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:689 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~5144576,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:31.176 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.088846+0000", 2024-09-15T03:47:31.176 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.97488687900000004, 2024-09-15T03:47:31.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020311780000000002, 2024-09-15T03:47:31.177 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:31.177 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:31.177 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:31.177 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:31.177 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.177 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 689 2024-09-15T03:47:31.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.177 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:31.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:31.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088846+0000", 2024-09-15T03:47:31.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:31.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088846+0000", 2024-09-15T03:47:31.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:31.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088847+0000", 2024-09-15T03:47:31.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9300000000000006e-07 2024-09-15T03:47:31.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:31.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088852+0000", 2024-09-15T03:47:31.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1850000000000003e-06 2024-09-15T03:47:31.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:31.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088863+0000", 2024-09-15T03:47:31.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1418e-05 2024-09-15T03:47:31.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088873+0000", 2024-09-15T03:47:31.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4020000000000008e-06 2024-09-15T03:47:31.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.089560+0000", 2024-09-15T03:47:31.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068774899999999998 2024-09-15T03:47:31.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:31.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090073+0000", 2024-09-15T03:47:31.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051222799999999999 2024-09-15T03:47:31.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:31.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.109157+0000", 2024-09-15T03:47:31.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019084805 2024-09-15T03:47:31.181 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.181 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:31.181 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.181 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:690 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~5144576,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:31.181 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:31.181 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.974853741, 2024-09-15T03:47:31.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025340499999999998, 2024-09-15T03:47:31.182 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:31.182 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:31.182 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:31.182 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:31.182 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.182 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 690 2024-09-15T03:47:31.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.182 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:31.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:31.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:31.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:31.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:31.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:31.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:31.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7999999999999996e-07 2024-09-15T03:47:31.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:31.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088882+0000", 2024-09-15T03:47:31.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1839999999999999e-06 2024-09-15T03:47:31.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:31.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088892+0000", 2024-09-15T03:47:31.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3540000000000006e-06 2024-09-15T03:47:31.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088896+0000", 2024-09-15T03:47:31.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5210000000000004e-06 2024-09-15T03:47:31.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.093968+0000", 2024-09-15T03:47:31.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050715170000000002 2024-09-15T03:47:31.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:31.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.094098+0000", 2024-09-15T03:47:31.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013070000000000001 2024-09-15T03:47:31.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:31.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.114219+0000", 2024-09-15T03:47:31.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020120743999999999 2024-09-15T03:47:31.186 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.186 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:31.186 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.186 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:691 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~5144576,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:31.186 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.088900+0000", 2024-09-15T03:47:31.186 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.97483199799999998, 2024-09-15T03:47:31.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031407274999999998, 2024-09-15T03:47:31.186 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:31.186 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:31.186 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:31.186 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:31.187 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.187 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 691 2024-09-15T03:47:31.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.187 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:31.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:31.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088900+0000", 2024-09-15T03:47:31.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:31.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088900+0000", 2024-09-15T03:47:31.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:31.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088901+0000", 2024-09-15T03:47:31.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9099999999999999e-07 2024-09-15T03:47:31.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:31.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088904+0000", 2024-09-15T03:47:31.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0299999999999998e-06 2024-09-15T03:47:31.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:31.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088911+0000", 2024-09-15T03:47:31.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.708e-06 2024-09-15T03:47:31.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088915+0000", 2024-09-15T03:47:31.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0579999999999999e-06 2024-09-15T03:47:31.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.096143+0000", 2024-09-15T03:47:31.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007228099 2024-09-15T03:47:31.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:31.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.096202+0000", 2024-09-15T03:47:31.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9667999999999999e-05 2024-09-15T03:47:31.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:31.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.120308+0000", 2024-09-15T03:47:31.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024105320999999999 2024-09-15T03:47:31.190 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.191 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:31.191 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.191 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(osd.0.17:2 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [assert-version v51,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:31.191 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.090507+0000", 2024-09-15T03:47:31.191 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.97322553099999998, 2024-09-15T03:47:31.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025010186, 2024-09-15T03:47:31.191 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:31.191 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:31.191 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:31.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:31.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090507+0000", 2024-09-15T03:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090507+0000", 2024-09-15T03:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090508+0000", 2024-09-15T03:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.176e-06 2024-09-15T03:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:31.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090512+0000", 2024-09-15T03:47:31.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2989999999999998e-06 2024-09-15T03:47:31.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:31.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090522+0000", 2024-09-15T03:47:31.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6549999999999999e-06 2024-09-15T03:47:31.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090526+0000", 2024-09-15T03:47:31.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8500000000000004e-06 2024-09-15T03:47:31.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:31.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.094385+0000", 2024-09-15T03:47:31.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038591670000000002 2024-09-15T03:47:31.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:31.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.094541+0000", 2024-09-15T03:47:31.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015590999999999999 2024-09-15T03:47:31.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:31.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:31.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:31.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.115517+0000", 2024-09-15T03:47:31.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020976129 2024-09-15T03:47:31.195 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.195 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:31.195 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.195 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:31.195 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:31.195 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:31.197 INFO:tasks.rados.rados.0.smithi001.stdout:276: finishing copy_from racing read to smithi00132172-18 2024-09-15T03:47:31.197 INFO:tasks.rados.rados.0.smithi001.stdout:276: finishing copy_from to smithi00132172-18 2024-09-15T03:47:31.198 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 18 v 4 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-15T03:47:31.198 INFO:tasks.rados.rados.0.smithi001.stdout:270: finishing copy_from racing read to smithi00132172-10 2024-09-15T03:47:31.198 INFO:tasks.rados.rados.0.smithi001.stdout:270: finishing copy_from to smithi00132172-10 2024-09-15T03:47:31.198 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 10 v 31 (ObjNum 102 snap 28 seq_num 102) dirty exists 2024-09-15T03:47:31.198 INFO:tasks.rados.rados.0.smithi001.stdout:278: finishing write tid 1 to smithi00132172-37 2024-09-15T03:47:31.198 INFO:tasks.rados.rados.0.smithi001.stdout:278: finishing write tid 2 to smithi00132172-37 2024-09-15T03:47:31.198 INFO:tasks.rados.rados.0.smithi001.stdout:278: finishing write tid 3 to smithi00132172-37 2024-09-15T03:47:31.198 INFO:tasks.rados.rados.0.smithi001.stdout:270: done (10 left) 2024-09-15T03:47:31.198 INFO:tasks.rados.rados.0.smithi001.stdout:271: done (9 left) 2024-09-15T03:47:31.198 INFO:tasks.rados.rados.0.smithi001.stdout:272: done (8 left) 2024-09-15T03:47:31.198 INFO:tasks.rados.rados.0.smithi001.stdout:273: done (7 left) 2024-09-15T03:47:31.198 INFO:tasks.rados.rados.0.smithi001.stdout:274: done (6 left) 2024-09-15T03:47:31.198 INFO:tasks.rados.rados.0.smithi001.stdout:276: done (5 left) 2024-09-15T03:47:31.199 INFO:tasks.rados.rados.0.smithi001.stdout:277: done (4 left) 2024-09-15T03:47:31.199 INFO:tasks.rados.rados.0.smithi001.stdout:281: setattr oid 36 current snap is 32 2024-09-15T03:47:31.199 INFO:tasks.rados.rados.0.smithi001.stdout:278: finishing write tid 4 to smithi00132172-37 2024-09-15T03:47:31.199 INFO:tasks.rados.rados.0.smithi001.stdout:278: finishing write tid 5 to smithi00132172-37 2024-09-15T03:47:31.199 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 37 v 47 (ObjNum 115 snap 32 seq_num 115) dirty exists 2024-09-15T03:47:31.199 INFO:tasks.rados.rados.0.smithi001.stdout:278: left oid 37 (ObjNum 115 snap 32 seq_num 115) 2024-09-15T03:47:31.199 INFO:tasks.rados.rados.0.smithi001.stdout:269: finishing copy_from racing read to smithi00132172-27 2024-09-15T03:47:31.199 INFO:tasks.rados.rados.0.smithi001.stdout:275: expect (ObjNum 106 snap 29 seq_num 106) 2024-09-15T03:47:31.199 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:47:31.200 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:31.196+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:31.298 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:31.297+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:31.308 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:31.308 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:31.309 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:31.309 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:31.314 INFO:tasks.rados.rados.0.smithi001.stdout:269: finishing copy_from to smithi00132172-27 2024-09-15T03:47:31.315 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 27 v 51 (ObjNum 81 snap 16 seq_num 81) dirty exists 2024-09-15T03:47:31.315 INFO:tasks.rados.rados.0.smithi001.stdout:280: finishing rollback tid 0 to smithi00132172-20 2024-09-15T03:47:31.315 INFO:tasks.rados.rados.0.smithi001.stdout:280: finishing rollback tid 1 to smithi00132172-20 2024-09-15T03:47:31.315 INFO:tasks.rados.rados.0.smithi001.stdout:280: finishing rollback tid 2 to smithi00132172-20 2024-09-15T03:47:31.315 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 20 v 28 (ObjNum 99 snap 24 seq_num 99) dirty exists 2024-09-15T03:47:31.315 INFO:tasks.rados.rados.0.smithi001.stdout:269: done (4 left) 2024-09-15T03:47:31.315 INFO:tasks.rados.rados.0.smithi001.stdout:275: done (3 left) 2024-09-15T03:47:31.316 INFO:tasks.rados.rados.0.smithi001.stdout:278: done (2 left) 2024-09-15T03:47:31.316 INFO:tasks.rados.rados.0.smithi001.stdout:280: done (1 left) 2024-09-15T03:47:31.316 INFO:tasks.rados.rados.0.smithi001.stdout:282: snap_create 2024-09-15T03:47:31.316 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 36 v 30 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-15T03:47:31.318 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:47:31.401 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:31.401+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:31.455 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:31.455 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:31.455 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:31.455 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:31.456 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:31.463 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:47:31.500 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:31.500+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:31.601 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:31.600+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:31.602 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:31.602 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:31.602 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:31.603 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:47:31.603 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.603 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:47:31.603 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:31.603 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 73.698049101999999, 2024-09-15T03:47:31.603 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10988437500000001, 2024-09-15T03:47:31.603 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:31.603 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:31.603 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:31.603 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:47:31.603 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:47:31.603 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1 2024-09-15T03:47:31.604 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.604 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:31.604 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.604 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:31.604 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:31.604 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.604 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.604 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.604 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:31.604 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:31.604 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.604 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.604 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.605 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:31.605 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902355+0000", 2024-09-15T03:47:31.605 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.429e-06 2024-09-15T03:47:31.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.605 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.605 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:31.605 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902362+0000", 2024-09-15T03:47:31.605 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7290000000000003e-06 2024-09-15T03:47:31.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.605 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.605 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:31.605 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902373+0000", 2024-09-15T03:47:31.606 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0533999999999999e-05 2024-09-15T03:47:31.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.606 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.606 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.606 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902379+0000", 2024-09-15T03:47:31.606 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4350000000000002e-06 2024-09-15T03:47:31.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.606 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.606 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.606 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902428+0000", 2024-09-15T03:47:31.606 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8571e-05 2024-09-15T03:47:31.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.606 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.607 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:47:31.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902434+0000", 2024-09-15T03:47:31.607 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5610000000000004e-06 2024-09-15T03:47:31.607 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.607 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.607 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011307+0000", 2024-09-15T03:47:31.607 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.108872306 2024-09-15T03:47:31.607 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.607 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.607 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:31.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011448+0000", 2024-09-15T03:47:31.607 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014137199999999999 2024-09-15T03:47:31.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.608 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.608 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:47:31.608 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011569+0000", 2024-09-15T03:47:31.608 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120925 2024-09-15T03:47:31.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.608 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.608 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:31.608 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011695+0000", 2024-09-15T03:47:31.608 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012586800000000001 2024-09-15T03:47:31.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.608 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.608 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:31.609 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011703+0000", 2024-09-15T03:47:31.609 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4190000000000002e-06 2024-09-15T03:47:31.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.609 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.609 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:31.609 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011739+0000", 2024-09-15T03:47:31.609 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5474e-05 2024-09-15T03:47:31.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.609 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.609 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:47:31.609 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011940+0000", 2024-09-15T03:47:31.609 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020108699999999999 2024-09-15T03:47:31.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.610 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.610 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:47:31.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012179+0000", 2024-09-15T03:47:31.610 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023950299999999999 2024-09-15T03:47:31.610 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.610 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.610 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:31.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012215+0000", 2024-09-15T03:47:31.610 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5332e-05 2024-09-15T03:47:31.610 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.610 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.610 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:31.611 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012238+0000", 2024-09-15T03:47:31.611 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3830000000000001e-05 2024-09-15T03:47:31.611 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.611 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:31.611 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.611 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.611 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4263.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2024-09-15T03:47:31.611 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:31.611 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 67.556652416000006, 2024-09-15T03:47:31.611 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019495788999999999, 2024-09-15T03:47:31.611 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:31.612 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:31.612 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:31.612 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4263", 2024-09-15T03:47:31.612 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/646873808", 2024-09-15T03:47:31.612 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2 2024-09-15T03:47:31.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.612 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:31.612 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.612 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:31.612 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:31.612 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.613 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.613 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:31.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:31.613 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.613 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.613 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:31.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043752+0000", 2024-09-15T03:47:31.613 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.815e-06 2024-09-15T03:47:31.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.613 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.613 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:31.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043757+0000", 2024-09-15T03:47:31.614 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4240000000000003e-06 2024-09-15T03:47:31.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.614 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.614 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:31.614 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043766+0000", 2024-09-15T03:47:31.614 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8629999999999997e-06 2024-09-15T03:47:31.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.614 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.614 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.614 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043773+0000", 2024-09-15T03:47:31.614 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0099999999999998e-06 2024-09-15T03:47:31.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.614 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.614 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.615 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043830+0000", 2024-09-15T03:47:31.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7089999999999997e-05 2024-09-15T03:47:31.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.615 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.615 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:47:31.615 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043837+0000", 2024-09-15T03:47:31.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.08e-06 2024-09-15T03:47:31.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.615 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.615 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.615 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063103+0000", 2024-09-15T03:47:31.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019265619000000001 2024-09-15T03:47:31.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:31.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063246+0000", 2024-09-15T03:47:31.616 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000143888 2024-09-15T03:47:31.616 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.616 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:31.616 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.616 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:47:31.616 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:31.616 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 45.429579027000003, 2024-09-15T03:47:31.616 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011878227, 2024-09-15T03:47:31.616 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:31.617 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:31.617 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:31.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:31.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:31.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:31.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:31.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:31.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170825+0000", 2024-09-15T03:47:31.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9599999999999999e-07 2024-09-15T03:47:31.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.618 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:31.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171100+0000", 2024-09-15T03:47:31.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027548500000000001 2024-09-15T03:47:31.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.618 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:31.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171112+0000", 2024-09-15T03:47:31.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2152e-05 2024-09-15T03:47:31.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.619 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.619 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171119+0000", 2024-09-15T03:47:31.619 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5050000000000004e-06 2024-09-15T03:47:31.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.619 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.619 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171162+0000", 2024-09-15T03:47:31.619 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2908000000000003e-05 2024-09-15T03:47:31.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.619 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:31.619 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171169+0000", 2024-09-15T03:47:31.619 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9739999999999996e-06 2024-09-15T03:47:31.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:31.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171398+0000", 2024-09-15T03:47:31.620 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022954799999999999 2024-09-15T03:47:31.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:31.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173640+0000", 2024-09-15T03:47:31.620 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022417000000000001 2024-09-15T03:47:31.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:31.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173705+0000", 2024-09-15T03:47:31.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4907999999999996e-05 2024-09-15T03:47:31.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.621 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:31.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173742+0000", 2024-09-15T03:47:31.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6940999999999997e-05 2024-09-15T03:47:31.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.621 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:31.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.182702+0000", 2024-09-15T03:47:31.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0089604100000000002 2024-09-15T03:47:31.622 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.622 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:31.622 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.622 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 39/31 ECSubWrite(tid=43, reqid=client.4521.0:252, at_version=39'27, trim_to=0'0, roll_forward_to=39'25))", 2024-09-15T03:47:31.622 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:31.622 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 45.421861708999998, 2024-09-15T03:47:31.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011286757, 2024-09-15T03:47:31.622 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:31.622 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:31.623 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:31.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:31.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:31.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:31.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:31.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:31.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178542+0000", 2024-09-15T03:47:31.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9400000000000004e-07 2024-09-15T03:47:31.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:31.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178754+0000", 2024-09-15T03:47:31.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021228800000000001 2024-09-15T03:47:31.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:31.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178765+0000", 2024-09-15T03:47:31.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0176e-05 2024-09-15T03:47:31.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178769+0000", 2024-09-15T03:47:31.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1359999999999999e-06 2024-09-15T03:47:31.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178810+0000", 2024-09-15T03:47:31.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0825999999999998e-05 2024-09-15T03:47:31.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:31.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178820+0000", 2024-09-15T03:47:31.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0237999999999999e-05 2024-09-15T03:47:31.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:31.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179033+0000", 2024-09-15T03:47:31.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021361500000000001 2024-09-15T03:47:31.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:31.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179040+0000", 2024-09-15T03:47:31.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0509999999999999e-06 2024-09-15T03:47:31.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:31.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179102+0000", 2024-09-15T03:47:31.627 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1977000000000006e-05 2024-09-15T03:47:31.627 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.627 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.627 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:31.627 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179114+0000", 2024-09-15T03:47:31.627 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1223000000000001e-05 2024-09-15T03:47:31.627 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.627 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.627 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:31.627 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.189828+0000", 2024-09-15T03:47:31.627 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010714433000000001 2024-09-15T03:47:31.627 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.627 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:31.627 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.628 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.628 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.628 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 51/31 ECSubWrite(tid=67, reqid=client.4521.0:350, at_version=51'33, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:31.628 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:31.628 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 33.756561668000003, 2024-09-15T03:47:31.628 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.015569239, 2024-09-15T03:47:31.628 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:31.628 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:31.628 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:31.628 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.628 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:31.628 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:31.628 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.629 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:31.629 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:31.629 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.629 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:31.629 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843843+0000", 2024-09-15T03:47:31.629 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.195e-06 2024-09-15T03:47:31.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.629 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:31.629 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843880+0000", 2024-09-15T03:47:31.630 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7527999999999997e-05 2024-09-15T03:47:31.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:31.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843898+0000", 2024-09-15T03:47:31.630 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7696000000000001e-05 2024-09-15T03:47:31.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843907+0000", 2024-09-15T03:47:31.630 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8820000000000007e-06 2024-09-15T03:47:31.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.631 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843978+0000", 2024-09-15T03:47:31.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1537000000000003e-05 2024-09-15T03:47:31.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.631 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.631 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:31.631 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843989+0000", 2024-09-15T03:47:31.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0720000000000001e-05 2024-09-15T03:47:31.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.631 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.631 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:31.631 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844102+0000", 2024-09-15T03:47:31.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000113406 2024-09-15T03:47:31.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.632 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:31.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844131+0000", 2024-09-15T03:47:31.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8762e-05 2024-09-15T03:47:31.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.632 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:31.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844180+0000", 2024-09-15T03:47:31.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9329999999999997e-05 2024-09-15T03:47:31.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.632 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:31.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844201+0000", 2024-09-15T03:47:31.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0772999999999999e-05 2024-09-15T03:47:31.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.633 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:31.636 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.859411+0000", 2024-09-15T03:47:31.636 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01520941 2024-09-15T03:47:31.636 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.636 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:31.636 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.636 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.637 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:31.637 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:31.637 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 33.755682757000002, 2024-09-15T03:47:31.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012156452, 2024-09-15T03:47:31.637 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:31.637 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:31.637 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:31.637 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.637 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:31.637 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:31.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.637 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.637 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:31.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:31.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.638 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:31.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844721+0000", 2024-09-15T03:47:31.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1100000000000005e-07 2024-09-15T03:47:31.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.638 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:31.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844743+0000", 2024-09-15T03:47:31.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1999000000000001e-05 2024-09-15T03:47:31.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:31.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844758+0000", 2024-09-15T03:47:31.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4718999999999999e-05 2024-09-15T03:47:31.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844772+0000", 2024-09-15T03:47:31.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4637000000000001e-05 2024-09-15T03:47:31.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844842+0000", 2024-09-15T03:47:31.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9596000000000002e-05 2024-09-15T03:47:31.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:31.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844853+0000", 2024-09-15T03:47:31.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0912e-05 2024-09-15T03:47:31.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:31.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844966+0000", 2024-09-15T03:47:31.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011276 2024-09-15T03:47:31.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:31.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844977+0000", 2024-09-15T03:47:31.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1676e-05 2024-09-15T03:47:31.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:31.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847414+0000", 2024-09-15T03:47:31.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024363760000000001 2024-09-15T03:47:31.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:31.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847438+0000", 2024-09-15T03:47:31.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3737e-05 2024-09-15T03:47:31.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:31.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856877+0000", 2024-09-15T03:47:31.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094392290000000004 2024-09-15T03:47:31.642 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.642 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:31.642 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.642 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:31.642 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:31.642 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 33.755613361999998, 2024-09-15T03:47:31.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012258211, 2024-09-15T03:47:31.642 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:31.643 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:31.643 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:31.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:31.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:31.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:31.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:31.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:31.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:31.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.58e-07 2024-09-15T03:47:31.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:31.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844818+0000", 2024-09-15T03:47:31.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.796e-05 2024-09-15T03:47:31.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:31.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844829+0000", 2024-09-15T03:47:31.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0895e-05 2024-09-15T03:47:31.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844836+0000", 2024-09-15T03:47:31.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.618e-06 2024-09-15T03:47:31.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845023+0000", 2024-09-15T03:47:31.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018743500000000001 2024-09-15T03:47:31.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:31.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845035+0000", 2024-09-15T03:47:31.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1958e-05 2024-09-15T03:47:31.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:31.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845147+0000", 2024-09-15T03:47:31.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112281 2024-09-15T03:47:31.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:31.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847417+0000", 2024-09-15T03:47:31.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022693969999999998 2024-09-15T03:47:31.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:31.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847462+0000", 2024-09-15T03:47:31.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5466999999999997e-05 2024-09-15T03:47:31.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.647 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:31.647 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847503+0000", 2024-09-15T03:47:31.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1022999999999997e-05 2024-09-15T03:47:31.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.647 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:31.647 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.857048+0000", 2024-09-15T03:47:31.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095447190000000001 2024-09-15T03:47:31.647 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.648 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:31.648 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.649 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 51/31 ECSubWrite(tid=68, reqid=client.4521.0:351, at_version=51'34, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:31.653 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:31.653 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 33.755195243999999, 2024-09-15T03:47:31.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.014544793, 2024-09-15T03:47:31.653 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:31.653 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:31.653 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:31.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:31.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:31.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:31.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:31.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:31.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845209+0000", 2024-09-15T03:47:31.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5300000000000003e-07 2024-09-15T03:47:31.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:31.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845227+0000", 2024-09-15T03:47:31.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8759000000000001e-05 2024-09-15T03:47:31.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:31.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845241+0000", 2024-09-15T03:47:31.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3740000000000001e-05 2024-09-15T03:47:31.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845247+0000", 2024-09-15T03:47:31.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2779999999999999e-06 2024-09-15T03:47:31.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845308+0000", 2024-09-15T03:47:31.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0702000000000003e-05 2024-09-15T03:47:31.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:31.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845317+0000", 2024-09-15T03:47:31.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3770000000000002e-06 2024-09-15T03:47:31.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:31.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845413+0000", 2024-09-15T03:47:31.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5356e-05 2024-09-15T03:47:31.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:31.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847421+0000", 2024-09-15T03:47:31.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020077200000000002 2024-09-15T03:47:31.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:31.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847466+0000", 2024-09-15T03:47:31.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5754999999999998e-05 2024-09-15T03:47:31.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:31.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847572+0000", 2024-09-15T03:47:31.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010594000000000001 2024-09-15T03:47:31.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.659 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:31.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.859753+0000", 2024-09-15T03:47:31.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012180412999999999 2024-09-15T03:47:31.659 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.659 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:31.659 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.659 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 60/28 ECSubWrite(tid=73, reqid=client.4521.0:424, at_version=60'41, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:31.659 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:31.659 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 24.699991990000001, 2024-09-15T03:47:31.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01137208, 2024-09-15T03:47:31.660 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:31.660 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:31.660 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:31.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.660 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:31.660 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:31.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:31.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:31.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:31.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900412+0000", 2024-09-15T03:47:31.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7599999999999996e-07 2024-09-15T03:47:31.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:31.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:31.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7986999999999999e-05 2024-09-15T03:47:31.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:31.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900445+0000", 2024-09-15T03:47:31.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.468e-05 2024-09-15T03:47:31.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900451+0000", 2024-09-15T03:47:31.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5799999999999997e-06 2024-09-15T03:47:31.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900507+0000", 2024-09-15T03:47:31.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5855e-05 2024-09-15T03:47:31.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:31.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900530+0000", 2024-09-15T03:47:31.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2929e-05 2024-09-15T03:47:31.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:31.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900640+0000", 2024-09-15T03:47:31.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000110536 2024-09-15T03:47:31.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.664 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:31.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900648+0000", 2024-09-15T03:47:31.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9470000000000003e-06 2024-09-15T03:47:31.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.664 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:31.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900764+0000", 2024-09-15T03:47:31.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011518099999999999 2024-09-15T03:47:31.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.665 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:31.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900803+0000", 2024-09-15T03:47:31.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9941999999999998e-05 2024-09-15T03:47:31.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.665 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:31.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.911783+0000", 2024-09-15T03:47:31.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010979667 2024-09-15T03:47:31.665 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.665 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:31.665 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.666 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:517 3.3s1 3:caaee79c:::smithi00132172-10:head [read 0~3145728,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:31.666 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.947050+0000", 2024-09-15T03:47:31.666 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.653353057, 2024-09-15T03:47:31.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012110134, 2024-09-15T03:47:31.666 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:31.666 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:31.666 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:31.666 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:31.666 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.666 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 517 2024-09-15T03:47:31.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.666 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:31.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:31.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947050+0000", 2024-09-15T03:47:31.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:31.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947050+0000", 2024-09-15T03:47:31.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:31.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947052+0000", 2024-09-15T03:47:31.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9099999999999999e-06 2024-09-15T03:47:31.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:31.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947060+0000", 2024-09-15T03:47:31.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8529999999999992e-06 2024-09-15T03:47:31.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:31.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947074+0000", 2024-09-15T03:47:31.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4344e-05 2024-09-15T03:47:31.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947082+0000", 2024-09-15T03:47:31.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9249999999999995e-06 2024-09-15T03:47:31.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947138+0000", 2024-09-15T03:47:31.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6054000000000002e-05 2024-09-15T03:47:31.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:31.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947339+0000", 2024-09-15T03:47:31.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000200829 2024-09-15T03:47:31.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.670 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:31.670 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.959160+0000", 2024-09-15T03:47:31.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011821219000000001 2024-09-15T03:47:31.670 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.670 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:31.670 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.670 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.670 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:518 3.3s1 3:caaee79c:::smithi00132172-10:head [read 0~3145728,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:31.670 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.947091+0000", 2024-09-15T03:47:31.670 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.653311915, 2024-09-15T03:47:31.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017608233000000001, 2024-09-15T03:47:31.670 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:31.671 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:31.671 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:31.671 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:31.671 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.671 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 518 2024-09-15T03:47:31.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.671 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:31.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.671 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:31.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947091+0000", 2024-09-15T03:47:31.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.671 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:31.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947091+0000", 2024-09-15T03:47:31.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:31.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947092+0000", 2024-09-15T03:47:31.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7800000000000002e-07 2024-09-15T03:47:31.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:31.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947095+0000", 2024-09-15T03:47:31.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2959999999999999e-06 2024-09-15T03:47:31.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.673 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:31.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947102+0000", 2024-09-15T03:47:31.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7469999999999999e-06 2024-09-15T03:47:31.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.673 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947106+0000", 2024-09-15T03:47:31.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6550000000000003e-06 2024-09-15T03:47:31.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.673 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947622+0000", 2024-09-15T03:47:31.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000515727 2024-09-15T03:47:31.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:31.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947751+0000", 2024-09-15T03:47:31.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012920600000000001 2024-09-15T03:47:31.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:31.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.964699+0000", 2024-09-15T03:47:31.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.016948123999999998 2024-09-15T03:47:31.674 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.674 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:31.674 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.675 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:519 3.3s1 3:caaee79c:::smithi00132172-10:head [read 0~3145728,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:31.675 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:31.675 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.653278209, 2024-09-15T03:47:31.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.021743243999999998, 2024-09-15T03:47:31.675 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:31.675 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:31.675 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:31.675 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:31.675 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.675 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 519 2024-09-15T03:47:31.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.675 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:31.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.676 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:31.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:31.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.676 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:31.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:31.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.676 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:31.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:31.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5999999999999999e-07 2024-09-15T03:47:31.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:31.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947130+0000", 2024-09-15T03:47:31.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7720000000000002e-06 2024-09-15T03:47:31.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:31.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947137+0000", 2024-09-15T03:47:31.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2370000000000003e-06 2024-09-15T03:47:31.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947141+0000", 2024-09-15T03:47:31.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8809999999999998e-06 2024-09-15T03:47:31.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.948008+0000", 2024-09-15T03:47:31.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00086656700000000005 2024-09-15T03:47:31.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:31.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.948113+0000", 2024-09-15T03:47:31.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104904 2024-09-15T03:47:31.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:31.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.968868+0000", 2024-09-15T03:47:31.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.020755422999999999 2024-09-15T03:47:31.679 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.679 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:31.679 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.679 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.3s1 69/68 ECSubReadReply(tid=163, attrs_read=0))", 2024-09-15T03:47:31.679 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:31.679 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.646702465000001, 2024-09-15T03:47:31.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011102266, 2024-09-15T03:47:31.679 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:31.680 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:31.680 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:31.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:31.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:31.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:31.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:31.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.681 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:31.681 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:31.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1199999999999998e-07 2024-09-15T03:47:31.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.681 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:31.681 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956300+0000", 2024-09-15T03:47:31.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025987459999999999 2024-09-15T03:47:31.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.681 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:31.681 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956314+0000", 2024-09-15T03:47:31.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4469e-05 2024-09-15T03:47:31.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.682 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.682 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.682 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956319+0000", 2024-09-15T03:47:31.682 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.746e-06 2024-09-15T03:47:31.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.682 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.682 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.682 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.959591+0000", 2024-09-15T03:47:31.682 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032721510000000001 2024-09-15T03:47:31.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.682 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.682 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:31.682 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.964803+0000", 2024-09-15T03:47:31.682 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0052117420000000001 2024-09-15T03:47:31.683 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.683 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:31.683 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.683 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.3s1 69/68 ECSubReadReply(tid=164, attrs_read=0))", 2024-09-15T03:47:31.683 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:31.683 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.644078031999999, 2024-09-15T03:47:31.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012633276000000001, 2024-09-15T03:47:31.683 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:31.683 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:31.683 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:31.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.684 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:31.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:31.684 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.684 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.684 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:31.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:31.684 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.684 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.684 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:31.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:31.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5900000000000003e-07 2024-09-15T03:47:31.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.685 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:31.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957786+0000", 2024-09-15T03:47:31.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001460755 2024-09-15T03:47:31.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.685 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:31.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957807+0000", 2024-09-15T03:47:31.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0937e-05 2024-09-15T03:47:31.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.685 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.686 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957814+0000", 2024-09-15T03:47:31.686 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5989999999999998e-06 2024-09-15T03:47:31.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.686 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.686 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.964911+0000", 2024-09-15T03:47:31.686 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0070976370000000004 2024-09-15T03:47:31.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.686 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:31.686 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.968958+0000", 2024-09-15T03:47:31.686 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.004046989 2024-09-15T03:47:31.686 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.687 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:31.687 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.687 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.687 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.687 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:567 3.ds0 3:b166213b:::smithi00132172-21:head [rollback 3] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:31.687 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.947576+0000", 2024-09-15T03:47:31.687 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.652827104, 2024-09-15T03:47:31.687 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013871319999999999, 2024-09-15T03:47:31.687 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:31.687 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:31.687 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:31.687 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:31.687 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.688 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 567 2024-09-15T03:47:31.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.688 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:31.688 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.688 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:31.688 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947576+0000", 2024-09-15T03:47:31.688 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.688 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.688 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:31.688 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947576+0000", 2024-09-15T03:47:31.688 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.689 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:31.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947577+0000", 2024-09-15T03:47:31.689 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.37e-07 2024-09-15T03:47:31.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.689 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:31.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947580+0000", 2024-09-15T03:47:31.689 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5710000000000002e-06 2024-09-15T03:47:31.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.689 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:31.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947588+0000", 2024-09-15T03:47:31.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8939999999999993e-06 2024-09-15T03:47:31.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.690 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.690 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.690 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947593+0000", 2024-09-15T03:47:31.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7049999999999998e-06 2024-09-15T03:47:31.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.690 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.690 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.690 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948070+0000", 2024-09-15T03:47:31.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047685099999999998 2024-09-15T03:47:31.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.690 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.691 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:31.691 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948177+0000", 2024-09-15T03:47:31.691 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010765299999999999 2024-09-15T03:47:31.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.691 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.691 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:31.691 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948986+0000", 2024-09-15T03:47:31.691 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000808977 2024-09-15T03:47:31.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.691 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.691 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:31.691 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949117+0000", 2024-09-15T03:47:31.691 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000131032 2024-09-15T03:47:31.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:31.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949133+0000", 2024-09-15T03:47:31.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5798999999999999e-05 2024-09-15T03:47:31.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:31.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950936+0000", 2024-09-15T03:47:31.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00180316 2024-09-15T03:47:31.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:31.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950994+0000", 2024-09-15T03:47:31.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7884999999999999e-05 2024-09-15T03:47:31.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:31.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951532+0000", 2024-09-15T03:47:31.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00053787699999999997 2024-09-15T03:47:31.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:31.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961447+0000", 2024-09-15T03:47:31.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0099152790000000008 2024-09-15T03:47:31.693 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.693 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:31.694 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.694 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:568 3.ds0 3:b166213b:::smithi00132172-21:head [append 0~0] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:31.694 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.948048+0000", 2024-09-15T03:47:31.694 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.652355179000001, 2024-09-15T03:47:31.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013579269, 2024-09-15T03:47:31.694 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:31.694 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:31.694 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:31.694 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:31.694 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.694 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 568 2024-09-15T03:47:31.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.695 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:31.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:31.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948048+0000", 2024-09-15T03:47:31.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:31.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948048+0000", 2024-09-15T03:47:31.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:31.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948049+0000", 2024-09-15T03:47:31.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7199999999999998e-07 2024-09-15T03:47:31.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:31.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948056+0000", 2024-09-15T03:47:31.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0280000000000003e-06 2024-09-15T03:47:31.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:31.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948066+0000", 2024-09-15T03:47:31.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0141e-05 2024-09-15T03:47:31.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948072+0000", 2024-09-15T03:47:31.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1519999999999997e-06 2024-09-15T03:47:31.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949221+0000", 2024-09-15T03:47:31.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001149125 2024-09-15T03:47:31.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:31.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949313+0000", 2024-09-15T03:47:31.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1469000000000001e-05 2024-09-15T03:47:31.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:31.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949818+0000", 2024-09-15T03:47:31.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00050549700000000004 2024-09-15T03:47:31.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:31.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950002+0000", 2024-09-15T03:47:31.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018414700000000001 2024-09-15T03:47:31.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:31.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950940+0000", 2024-09-15T03:47:31.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00093786100000000001 2024-09-15T03:47:31.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:31.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951004+0000", 2024-09-15T03:47:31.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3859e-05 2024-09-15T03:47:31.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:31.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951020+0000", 2024-09-15T03:47:31.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.628e-05 2024-09-15T03:47:31.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:31.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951753+0000", 2024-09-15T03:47:31.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00073276999999999999 2024-09-15T03:47:31.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:31.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961627+0000", 2024-09-15T03:47:31.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098741679999999991 2024-09-15T03:47:31.700 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.700 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:31.700 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.701 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:677 3.3s1 3:caaee79c:::smithi00132172-10:head [copy-from ver 31] snapc 2f=[2d,2b,29,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:31.701 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:30.087209+0000", 2024-09-15T03:47:31.701 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.513193988, 2024-09-15T03:47:31.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.025257667000000001, 2024-09-15T03:47:31.701 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:31.701 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:31.701 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:31.701 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:31.701 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.701 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 677 2024-09-15T03:47:31.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.701 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:31.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:31.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087209+0000", 2024-09-15T03:47:31.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:31.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087209+0000", 2024-09-15T03:47:31.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:31.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087211+0000", 2024-09-15T03:47:31.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2500000000000001e-06 2024-09-15T03:47:31.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:31.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087222+0000", 2024-09-15T03:47:31.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.113e-05 2024-09-15T03:47:31.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:31.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087252+0000", 2024-09-15T03:47:31.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9488999999999998e-05 2024-09-15T03:47:31.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087264+0000", 2024-09-15T03:47:31.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1875999999999999e-05 2024-09-15T03:47:31.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087368+0000", 2024-09-15T03:47:31.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104035 2024-09-15T03:47:31.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:31.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087560+0000", 2024-09-15T03:47:31.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019211400000000001 2024-09-15T03:47:31.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:31.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.104984+0000", 2024-09-15T03:47:31.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017424489000000001 2024-09-15T03:47:31.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:31.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105388+0000", 2024-09-15T03:47:31.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000403686 2024-09-15T03:47:31.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:31.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105404+0000", 2024-09-15T03:47:31.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6268000000000001e-05 2024-09-15T03:47:31.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:31.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105641+0000", 2024-09-15T03:47:31.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023648399999999999 2024-09-15T03:47:31.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:31.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105662+0000", 2024-09-15T03:47:31.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0602999999999999e-05 2024-09-15T03:47:31.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:31.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.110231+0000", 2024-09-15T03:47:31.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0045697089999999999 2024-09-15T03:47:31.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:31.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.112467+0000", 2024-09-15T03:47:31.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002235534 2024-09-15T03:47:31.707 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.707 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:31.707 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.707 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:678 3.3s1 3:caaee79c:::smithi00132172-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:31.708 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:30.087279+0000", 2024-09-15T03:47:31.708 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.513124533, 2024-09-15T03:47:31.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.018991965, 2024-09-15T03:47:31.708 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:31.708 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:31.708 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:31.708 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:31.708 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.708 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 678 2024-09-15T03:47:31.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.708 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:31.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:31.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087279+0000", 2024-09-15T03:47:31.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:31.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087279+0000", 2024-09-15T03:47:31.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:31.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087280+0000", 2024-09-15T03:47:31.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.147e-06 2024-09-15T03:47:31.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:31.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087285+0000", 2024-09-15T03:47:31.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2059999999999998e-06 2024-09-15T03:47:31.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:31.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087303+0000", 2024-09-15T03:47:31.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7646e-05 2024-09-15T03:47:31.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087313+0000", 2024-09-15T03:47:31.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0862999999999999e-05 2024-09-15T03:47:31.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087765+0000", 2024-09-15T03:47:31.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045120400000000003 2024-09-15T03:47:31.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:31.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087962+0000", 2024-09-15T03:47:31.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000197095 2024-09-15T03:47:31.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.712 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105788+0000", 2024-09-15T03:47:31.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017826143999999999 2024-09-15T03:47:31.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.712 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:31.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.106034+0000", 2024-09-15T03:47:31.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024586999999999998 2024-09-15T03:47:31.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.712 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:31.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.106270+0000", 2024-09-15T03:47:31.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023678999999999999 2024-09-15T03:47:31.713 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.713 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:31.713 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.713 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:688 3.3s1 3:c7c3ef16:::smithi00132172-25:head [create,setxattr og^`]cfbTbdPrNph_pkYgeiTrgatpl] (805) in=31b,setxattr `Trb[\\aaaksnX\\dQOYP^TPXpYtmQ_maX\\[hbePkT (981) in=40b,setxattr UkQ^W[[sWh]fSrWg]^\\oZpWaamVkVUnZYbR[Vql\\ (663) in=40b,setxattr ofr[Os^]tjr^jicnNpPaUeXQTfldVQPdSPjcgWcT (7228) in=40b,setxattr Vl`NPsTVpnj_gO_srrbrZ\\eq (805) in=24b,setxattr ^_mcpljcT[qX_UeeYZY\\_ZTSsSsojP\\esUkVphT] (1963) in=40b,setxattr [oY^bSn[ealaPaOhprWW_SNaa]VhaXN[jmo`YYgt (138) in=40b,setxattr tT[SSseaPa\\UPS`\\g\\pRqsd`^lo__arabQlQmob` (663) in=40b,setxattr o^\\]jXsesZhoknr\\dagnjidgsPOhkfg[ffVe[PQW (1259) in=40b,setxattr c^VX_^n^gsh^P`aVbXpaS\\cgkpZO]OmXmchTtWZd (1612) in=40b,setxattr _^aeiTb[Uhh`rNNUOZOPt_Z_Qmeps`XXQ^qidetf (629) in=40b,setxattr ms[glaQ`N\\pqbicVTmgp^\\Rg]h`fefatkfg\\js_[ (139) in=40b,setxattr O\\gsPtYmrReq]if[pkOgPtqS`]Nrq_eYpsefkSgl (700) in=40b,setxattr RgefRtSsOT`jTQb^h\\igU`_lbscW`fgkXroNdmea (1751) in=40b,setxattr sgaV_cSaelbWQYill^k`RsVajcXpUSfW]eNd_nat (807) in=40b,setxattr P\\lm\\r`\\jpssZmkPqcQSbje_]]`YOSZTeRUUSSY[ (1365) in=40b,setxattr aa^ePena\\opOdeWXTj[XU]N^jQS]bens\\Tc_g]ea (16140) in=40b,setxattr R[qckQQO`n[[egbgcZsdtUbdPdnPNlOmsbP_OXOQ (1227) in=40b,setxattr SpRUpSaQZ]nlQ]kpXZnOWSUS]tsmf`qZiPTkq]Yi (209) in=40b,setxattr Y[Y[X`O_XaVOb]NZ_PgmhRX]Vds_dfbp]h[SPkYb (417) in=40b] snapc 2f=[2d,2b,29,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:31.713 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:30.088531+0000", 2024-09-15T03:47:31.713 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.511871798, 2024-09-15T03:47:31.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013293503999999999, 2024-09-15T03:47:31.713 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:31.713 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:31.713 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:31.713 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:31.714 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:31.714 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 688 2024-09-15T03:47:31.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.714 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:31.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.714 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:31.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088531+0000", 2024-09-15T03:47:31.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.714 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:31.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088531+0000", 2024-09-15T03:47:31.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:31.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088533+0000", 2024-09-15T03:47:31.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5379999999999999e-06 2024-09-15T03:47:31.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:31.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088662+0000", 2024-09-15T03:47:31.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000129002 2024-09-15T03:47:31.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:31.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088688+0000", 2024-09-15T03:47:31.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6084999999999998e-05 2024-09-15T03:47:31.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088696+0000", 2024-09-15T03:47:31.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0490000000000004e-06 2024-09-15T03:47:31.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088746+0000", 2024-09-15T03:47:31.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0074000000000002e-05 2024-09-15T03:47:31.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:31.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.089084+0000", 2024-09-15T03:47:31.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033810200000000001 2024-09-15T03:47:31.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:31.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090498+0000", 2024-09-15T03:47:31.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001414205 2024-09-15T03:47:31.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:31.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090936+0000", 2024-09-15T03:47:31.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043802799999999997 2024-09-15T03:47:31.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:31.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090951+0000", 2024-09-15T03:47:31.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4953999999999999e-05 2024-09-15T03:47:31.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:31.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.091069+0000", 2024-09-15T03:47:31.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000117236 2024-09-15T03:47:31.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:31.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.091128+0000", 2024-09-15T03:47:31.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9580999999999998e-05 2024-09-15T03:47:31.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.719 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.719 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:31.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.097312+0000", 2024-09-15T03:47:31.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0061838129999999998 2024-09-15T03:47:31.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.719 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.719 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:31.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.101825+0000", 2024-09-15T03:47:31.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0045128369999999996 2024-09-15T03:47:31.719 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.720 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:31.720 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.720 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s4 83/80 ECSubWrite(tid=24, reqid=client.4521.0:675, at_version=83'37, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:31.720 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:30.126290+0000", 2024-09-15T03:47:31.720 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.4741129550000001, 2024-09-15T03:47:31.720 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.014228881000000001, 2024-09-15T03:47:31.720 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:31.720 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:31.720 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:31.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.720 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:31.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.126290+0000", 2024-09-15T03:47:31.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:31.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.126290+0000", 2024-09-15T03:47:31.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:31.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:31.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.126291+0000", 2024-09-15T03:47:31.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6499999999999998e-07 2024-09-15T03:47:31.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:31.722 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128867+0000", 2024-09-15T03:47:31.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025759749999999999 2024-09-15T03:47:31.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:31.722 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128881+0000", 2024-09-15T03:47:31.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3823e-05 2024-09-15T03:47:31.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:31.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128888+0000", 2024-09-15T03:47:31.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.605e-06 2024-09-15T03:47:31.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:31.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128949+0000", 2024-09-15T03:47:31.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0949999999999998e-05 2024-09-15T03:47:31.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:31.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128957+0000", 2024-09-15T03:47:31.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4760000000000001e-06 2024-09-15T03:47:31.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:31.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.129323+0000", 2024-09-15T03:47:31.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036638900000000001 2024-09-15T03:47:31.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:31.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.129331+0000", 2024-09-15T03:47:31.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1319999999999994e-06 2024-09-15T03:47:31.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:31.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.129567+0000", 2024-09-15T03:47:31.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000235393 2024-09-15T03:47:31.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:31.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.129586+0000", 2024-09-15T03:47:31.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9517999999999998e-05 2024-09-15T03:47:31.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:31.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:31.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:31.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.140519+0000", 2024-09-15T03:47:31.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010932855 2024-09-15T03:47:31.725 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.726 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:31.726 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.726 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:31.726 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:31.726 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:31.727 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:47:31.728 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:31.726+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:31.828 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:31.827+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:31.832 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:47:31.865 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:31.865 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:31.865 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:31.865 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:31.874 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:47:31.928 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:31.927+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:31.956 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:31.956 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:31.970 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:31.970 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:31.971 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:31.971 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:31.971 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:31.971 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:31.971 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:31.979 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:47:31.983 INFO:tasks.rados.rados.0.smithi001.stdout:281: done (1 left) 2024-09-15T03:47:31.983 INFO:tasks.rados.rados.0.smithi001.stdout:282: done (0 left) 2024-09-15T03:47:31.983 INFO:tasks.rados.rados.0.smithi001.stdout:283: copy_from oid 45 from oid 20 current snap is 33 2024-09-15T03:47:31.983 INFO:tasks.rados.rados.0.smithi001.stdout:284: read oid 37 snap -1 2024-09-15T03:47:31.983 INFO:tasks.rados.rados.0.smithi001.stdout:284: expect (ObjNum 115 snap 32 seq_num 115) 2024-09-15T03:47:31.984 INFO:tasks.rados.rados.0.smithi001.stdout:285: rollback oid 27 current snap is 33 2024-09-15T03:47:31.984 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 27 to 25 2024-09-15T03:47:31.984 INFO:tasks.rados.rados.0.smithi001.stdout:286: snap_remove snap 7 2024-09-15T03:47:31.989 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:31.989 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:32.002 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T03:47:32.010 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:32.010 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:32.029 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:32.028+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:32.029 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:32.029 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:32.044 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:32.045 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:32.058 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:32.058 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:32.072 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:32.073 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:32.087 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:32.087 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:32.089 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:32.089 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:32.089 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:32.089 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:47:32.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.090 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:47:32.090 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:32.090 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 45.916693940999998, 2024-09-15T03:47:32.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012183955999999999, 2024-09-15T03:47:32.090 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:32.090 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:32.090 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:32.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.090 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:32.090 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:32.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.090 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.091 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.091 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:32.091 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:32.091 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.091 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.091 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.091 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:32.092 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171102+0000", 2024-09-15T03:47:32.092 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.17e-07 2024-09-15T03:47:32.092 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.092 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:32.092 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171454+0000", 2024-09-15T03:47:32.092 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00035226700000000002 2024-09-15T03:47:32.092 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.092 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:32.092 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171465+0000", 2024-09-15T03:47:32.092 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0943e-05 2024-09-15T03:47:32.092 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:32.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171471+0000", 2024-09-15T03:47:32.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7400000000000001e-06 2024-09-15T03:47:32.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:32.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171508+0000", 2024-09-15T03:47:32.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7147999999999997e-05 2024-09-15T03:47:32.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:32.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171517+0000", 2024-09-15T03:47:32.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2560000000000008e-06 2024-09-15T03:47:32.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.094 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:32.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171716+0000", 2024-09-15T03:47:32.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019899699999999999 2024-09-15T03:47:32.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.094 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:32.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173686+0000", 2024-09-15T03:47:32.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019696829999999999 2024-09-15T03:47:32.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.095 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.095 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:32.095 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173777+0000", 2024-09-15T03:47:32.095 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0637999999999995e-05 2024-09-15T03:47:32.095 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.096 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:32.096 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173814+0000", 2024-09-15T03:47:32.096 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7499000000000001e-05 2024-09-15T03:47:32.096 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.096 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:32.096 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.183286+0000", 2024-09-15T03:47:32.096 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094716680000000008 2024-09-15T03:47:32.097 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.097 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:32.097 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.097 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 51/31 ECSubWrite(tid=67, reqid=client.4521.0:350, at_version=51'33, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:32.097 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.843852+0000", 2024-09-15T03:47:32.097 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 34.243943979999997, 2024-09-15T03:47:32.097 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010895471, 2024-09-15T03:47:32.097 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:32.097 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:32.097 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:32.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.098 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:32.098 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843852+0000", 2024-09-15T03:47:32.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.098 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.098 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.098 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:32.098 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843852+0000", 2024-09-15T03:47:32.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.098 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.098 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.098 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:32.098 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843853+0000", 2024-09-15T03:47:32.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.359e-06 2024-09-15T03:47:32.099 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:32.099 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843882+0000", 2024-09-15T03:47:32.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8711999999999999e-05 2024-09-15T03:47:32.099 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:32.099 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843901+0000", 2024-09-15T03:47:32.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9477e-05 2024-09-15T03:47:32.099 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.100 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:32.100 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843911+0000", 2024-09-15T03:47:32.100 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2590000000000006e-06 2024-09-15T03:47:32.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.100 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:32.100 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843973+0000", 2024-09-15T03:47:32.100 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2310999999999993e-05 2024-09-15T03:47:32.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:32.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843984+0000", 2024-09-15T03:47:32.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.132e-05 2024-09-15T03:47:32.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.101 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:32.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844108+0000", 2024-09-15T03:47:32.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012384399999999999 2024-09-15T03:47:32.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.101 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:32.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844139+0000", 2024-09-15T03:47:32.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1148000000000001e-05 2024-09-15T03:47:32.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:32.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844185+0000", 2024-09-15T03:47:32.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5388e-05 2024-09-15T03:47:32.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:32.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844202+0000", 2024-09-15T03:47:32.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7708999999999999e-05 2024-09-15T03:47:32.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:32.103 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.854747+0000", 2024-09-15T03:47:32.103 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010544944000000001 2024-09-15T03:47:32.103 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.103 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:32.103 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.103 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.103 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.103 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 51/31 ECSubWrite(tid=68, reqid=client.4521.0:351, at_version=51'34, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:32.103 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:32.103 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 34.242580158999999, 2024-09-15T03:47:32.103 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098406970000000007, 2024-09-15T03:47:32.103 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:32.103 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:32.104 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:32.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.104 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:32.104 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:32.104 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.104 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.104 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:32.104 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:32.104 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.104 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.104 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:32.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845216+0000", 2024-09-15T03:47:32.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1299999999999999e-07 2024-09-15T03:47:32.105 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:32.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845232+0000", 2024-09-15T03:47:32.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5366000000000002e-05 2024-09-15T03:47:32.105 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:32.106 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845242+0000", 2024-09-15T03:47:32.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0403e-05 2024-09-15T03:47:32.106 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.106 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.106 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:32.106 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845247+0000", 2024-09-15T03:47:32.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4870000000000004e-06 2024-09-15T03:47:32.106 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.106 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.106 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:32.106 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845292+0000", 2024-09-15T03:47:32.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5762e-05 2024-09-15T03:47:32.106 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:32.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845302+0000", 2024-09-15T03:47:32.107 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5969999999999998e-06 2024-09-15T03:47:32.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:32.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845378+0000", 2024-09-15T03:47:32.107 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5628000000000002e-05 2024-09-15T03:47:32.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:32.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847479+0000", 2024-09-15T03:47:32.107 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021009380000000001 2024-09-15T03:47:32.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.108 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:32.108 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847530+0000", 2024-09-15T03:47:32.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1548000000000002e-05 2024-09-15T03:47:32.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.108 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:32.108 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847544+0000", 2024-09-15T03:47:32.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4355000000000001e-05 2024-09-15T03:47:32.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.108 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:32.109 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.855056+0000", 2024-09-15T03:47:32.109 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075119000000000002 2024-09-15T03:47:32.109 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.109 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:32.109 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.109 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.109 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.109 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 51/31 ECSubWrite(tid=93, reqid=client.4521.0:352, at_version=51'24, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:47:32.109 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:32.109 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 34.240516225, 2024-09-15T03:47:32.109 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0096136880000000004, 2024-09-15T03:47:32.109 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:32.109 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:32.109 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:32.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.110 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:32.110 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:32.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.110 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.110 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:32.110 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:32.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.110 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.110 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:32.110 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847280+0000", 2024-09-15T03:47:32.111 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9100000000000003e-07 2024-09-15T03:47:32.111 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:32.111 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847669+0000", 2024-09-15T03:47:32.111 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00038855999999999998 2024-09-15T03:47:32.111 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:32.111 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847682+0000", 2024-09-15T03:47:32.112 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3112000000000001e-05 2024-09-15T03:47:32.112 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.112 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.112 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:32.112 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847687+0000", 2024-09-15T03:47:32.112 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0880000000000002e-06 2024-09-15T03:47:32.112 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.112 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.112 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:32.112 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847745+0000", 2024-09-15T03:47:32.112 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8400999999999999e-05 2024-09-15T03:47:32.112 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:32.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847756+0000", 2024-09-15T03:47:32.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0325e-05 2024-09-15T03:47:32.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:32.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.848092+0000", 2024-09-15T03:47:32.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033636099999999997 2024-09-15T03:47:32.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:32.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.848101+0000", 2024-09-15T03:47:32.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6240000000000008e-06 2024-09-15T03:47:32.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:32.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850132+0000", 2024-09-15T03:47:32.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020309239999999999 2024-09-15T03:47:32.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:32.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.850174+0000", 2024-09-15T03:47:32.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2630999999999999e-05 2024-09-15T03:47:32.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:32.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856893+0000", 2024-09-15T03:47:32.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0067188710000000004 2024-09-15T03:47:32.115 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.115 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:32.115 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.115 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 60/28 ECSubWrite(tid=73, reqid=client.4521.0:424, at_version=60'41, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:32.115 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.900505+0000", 2024-09-15T03:47:32.115 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 25.187290562000001, 2024-09-15T03:47:32.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011912449, 2024-09-15T03:47:32.115 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:32.116 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:32.116 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:32.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.116 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:32.116 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900505+0000", 2024-09-15T03:47:32.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.116 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:32.116 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900505+0000", 2024-09-15T03:47:32.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.117 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:32.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900506+0000", 2024-09-15T03:47:32.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0800000000000003e-07 2024-09-15T03:47:32.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.117 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:32.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900527+0000", 2024-09-15T03:47:32.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0682999999999998e-05 2024-09-15T03:47:32.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.117 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:32.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900541+0000", 2024-09-15T03:47:32.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4217e-05 2024-09-15T03:47:32.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.118 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.118 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:32.118 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900547+0000", 2024-09-15T03:47:32.118 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3010000000000003e-06 2024-09-15T03:47:32.118 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.118 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.118 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:32.118 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900596+0000", 2024-09-15T03:47:32.118 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8612000000000001e-05 2024-09-15T03:47:32.118 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.118 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.118 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:32.118 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900607+0000", 2024-09-15T03:47:32.118 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.063e-05 2024-09-15T03:47:32.119 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.119 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:32.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900714+0000", 2024-09-15T03:47:32.119 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000107321 2024-09-15T03:47:32.119 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.119 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:32.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900721+0000", 2024-09-15T03:47:32.119 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1709999999999996e-06 2024-09-15T03:47:32.119 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.119 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:32.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900753+0000", 2024-09-15T03:47:32.119 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1565e-05 2024-09-15T03:47:32.120 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.120 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.120 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:32.120 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900766+0000", 2024-09-15T03:47:32.120 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3406e-05 2024-09-15T03:47:32.120 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.120 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.120 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:32.120 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.912418+0000", 2024-09-15T03:47:32.120 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011651635 2024-09-15T03:47:32.120 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.120 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:32.120 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.121 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.121 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.121 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as3 60/28 ECSubWrite(tid=74, reqid=client.4521.0:425, at_version=60'42, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:32.121 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.901156+0000", 2024-09-15T03:47:32.121 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 25.186639809999999, 2024-09-15T03:47:32.121 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011338225, 2024-09-15T03:47:32.121 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:32.121 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:32.121 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:32.121 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.121 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:32.121 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901156+0000", 2024-09-15T03:47:32.122 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.122 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.122 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.122 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:32.122 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901156+0000", 2024-09-15T03:47:32.122 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.122 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.122 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.122 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:32.122 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901157+0000", 2024-09-15T03:47:32.122 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9400000000000005e-07 2024-09-15T03:47:32.122 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.122 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.123 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:32.123 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901168+0000", 2024-09-15T03:47:32.123 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1924e-05 2024-09-15T03:47:32.123 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.123 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.123 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:32.123 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901178+0000", 2024-09-15T03:47:32.123 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7040000000000006e-06 2024-09-15T03:47:32.123 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.123 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.123 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:32.123 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901182+0000", 2024-09-15T03:47:32.123 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3080000000000001e-06 2024-09-15T03:47:32.123 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.124 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.124 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:32.124 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901226+0000", 2024-09-15T03:47:32.124 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3436999999999998e-05 2024-09-15T03:47:32.124 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.124 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.124 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:32.124 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901235+0000", 2024-09-15T03:47:32.124 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9400000000000008e-06 2024-09-15T03:47:32.124 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.124 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.124 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:32.124 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901329+0000", 2024-09-15T03:47:32.125 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3686999999999999e-05 2024-09-15T03:47:32.125 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.125 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.125 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:32.125 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901337+0000", 2024-09-15T03:47:32.125 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0129999999999993e-06 2024-09-15T03:47:32.125 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.125 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.126 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:32.126 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903268+0000", 2024-09-15T03:47:32.126 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001931545 2024-09-15T03:47:32.126 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.126 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.126 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:32.126 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.903279+0000", 2024-09-15T03:47:32.126 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0626e-05 2024-09-15T03:47:32.126 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.126 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.126 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:32.126 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.912494+0000", 2024-09-15T03:47:32.127 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092154469999999999 2024-09-15T03:47:32.127 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.127 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:32.127 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.127 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.127 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.127 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 60/31 ECSubWrite(tid=129, reqid=client.4521.0:422, at_version=60'41, trim_to=0'0, roll_forward_to=60'39))", 2024-09-15T03:47:32.127 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.901373+0000", 2024-09-15T03:47:32.127 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 25.186422632999999, 2024-09-15T03:47:32.127 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013085485000000001, 2024-09-15T03:47:32.127 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:32.127 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:32.128 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:32.128 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.128 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:32.128 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901373+0000", 2024-09-15T03:47:32.128 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.128 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.128 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.128 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:32.128 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901373+0000", 2024-09-15T03:47:32.128 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.128 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.128 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.128 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:32.129 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901374+0000", 2024-09-15T03:47:32.129 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5600000000000005e-07 2024-09-15T03:47:32.129 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.129 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.129 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:32.129 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901713+0000", 2024-09-15T03:47:32.129 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033933299999999998 2024-09-15T03:47:32.129 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.129 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.129 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:32.129 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901726+0000", 2024-09-15T03:47:32.129 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2843999999999999e-05 2024-09-15T03:47:32.129 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.129 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.130 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:32.130 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901730+0000", 2024-09-15T03:47:32.130 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4660000000000001e-06 2024-09-15T03:47:32.130 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.130 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.130 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:32.130 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901773+0000", 2024-09-15T03:47:32.130 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2542e-05 2024-09-15T03:47:32.130 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.130 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.130 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:32.130 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.901783+0000", 2024-09-15T03:47:32.130 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.041e-05 2024-09-15T03:47:32.130 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.131 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.131 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:32.131 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906331+0000", 2024-09-15T03:47:32.131 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0045479049999999997 2024-09-15T03:47:32.131 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.131 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.131 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:32.131 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906340+0000", 2024-09-15T03:47:32.131 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8920000000000006e-06 2024-09-15T03:47:32.131 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.131 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.131 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:32.131 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906403+0000", 2024-09-15T03:47:32.131 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2793999999999995e-05 2024-09-15T03:47:32.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.132 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:32.132 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.906413+0000", 2024-09-15T03:47:32.132 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0370999999999999e-05 2024-09-15T03:47:32.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.132 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:32.132 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.914459+0000", 2024-09-15T03:47:32.132 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0080452719999999991 2024-09-15T03:47:32.132 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.132 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:32.132 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.133 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.133 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.fs7 65/64 ECSubWrite(tid=108, reqid=client.4521.0:466, at_version=65'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:32.133 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:11.853993+0000", 2024-09-15T03:47:32.133 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 20.233802444999998, 2024-09-15T03:47:32.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013335351, 2024-09-15T03:47:32.133 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:32.133 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:32.133 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:32.133 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.133 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:32.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.853993+0000", 2024-09-15T03:47:32.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:32.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.853993+0000", 2024-09-15T03:47:32.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.134 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:32.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.853994+0000", 2024-09-15T03:47:32.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9999999999999997e-07 2024-09-15T03:47:32.134 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:32.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854199+0000", 2024-09-15T03:47:32.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020490999999999999 2024-09-15T03:47:32.135 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.135 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:32.135 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854210+0000", 2024-09-15T03:47:32.135 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0813e-05 2024-09-15T03:47:32.135 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.135 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:32.135 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854215+0000", 2024-09-15T03:47:32.135 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8270000000000003e-06 2024-09-15T03:47:32.135 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.135 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:32.136 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854264+0000", 2024-09-15T03:47:32.136 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8477999999999999e-05 2024-09-15T03:47:32.136 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.136 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.136 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:32.136 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854274+0000", 2024-09-15T03:47:32.136 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5729999999999997e-06 2024-09-15T03:47:32.136 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.136 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.136 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:32.136 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.854563+0000", 2024-09-15T03:47:32.136 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000289208 2024-09-15T03:47:32.136 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.136 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:32.137 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.855280+0000", 2024-09-15T03:47:32.137 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00071722500000000005 2024-09-15T03:47:32.137 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:32.137 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.855441+0000", 2024-09-15T03:47:32.137 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016084199999999999 2024-09-15T03:47:32.137 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:32.137 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.855456+0000", 2024-09-15T03:47:32.138 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5123000000000001e-05 2024-09-15T03:47:32.138 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.138 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.138 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:32.138 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:11.867329+0000", 2024-09-15T03:47:32.138 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011872752 2024-09-15T03:47:32.138 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.138 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:32.138 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.138 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.138 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.139 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.ds6 74/70 ECSubWrite(tid=198, reqid=client.4521.0:566, at_version=74'32, trim_to=0'0, roll_forward_to=73'30))", 2024-09-15T03:47:32.139 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.944959+0000", 2024-09-15T03:47:32.139 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.142837242000001, 2024-09-15T03:47:32.139 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010023647, 2024-09-15T03:47:32.139 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:32.139 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:32.139 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:32.139 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.139 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:32.139 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.944959+0000", 2024-09-15T03:47:32.139 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.139 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.139 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.139 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:32.140 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.944959+0000", 2024-09-15T03:47:32.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.140 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.140 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:32.140 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.944959+0000", 2024-09-15T03:47:32.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3099999999999998e-07 2024-09-15T03:47:32.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.140 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.140 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:32.140 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.944986+0000", 2024-09-15T03:47:32.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7297999999999999e-05 2024-09-15T03:47:32.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.141 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.141 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:32.141 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.944997+0000", 2024-09-15T03:47:32.141 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0441999999999999e-05 2024-09-15T03:47:32.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.141 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.141 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:32.141 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.945008+0000", 2024-09-15T03:47:32.141 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1687e-05 2024-09-15T03:47:32.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.141 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.141 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:32.141 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.945058+0000", 2024-09-15T03:47:32.142 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9410999999999998e-05 2024-09-15T03:47:32.142 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.142 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.142 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:32.142 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.945069+0000", 2024-09-15T03:47:32.142 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0812000000000001e-05 2024-09-15T03:47:32.142 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.142 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.142 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:32.142 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.945348+0000", 2024-09-15T03:47:32.142 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027976800000000001 2024-09-15T03:47:32.142 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.142 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.143 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:32.143 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.945362+0000", 2024-09-15T03:47:32.143 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3834e-05 2024-09-15T03:47:32.143 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.143 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.143 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:32.143 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.945658+0000", 2024-09-15T03:47:32.143 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029610200000000002 2024-09-15T03:47:32.143 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.143 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.143 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:32.143 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.945707+0000", 2024-09-15T03:47:32.143 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9029000000000001e-05 2024-09-15T03:47:32.144 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.144 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.144 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:32.144 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.954982+0000", 2024-09-15T03:47:32.144 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0092748329999999997 2024-09-15T03:47:32.144 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.144 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:32.144 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.144 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.144 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.144 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.ds6 74/70 ECSubWrite(tid=200, reqid=client.4521.0:567, at_version=74'33, trim_to=0'0, roll_forward_to=74'32))", 2024-09-15T03:47:32.144 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.948998+0000", 2024-09-15T03:47:32.144 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.138797859, 2024-09-15T03:47:32.145 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011947362, 2024-09-15T03:47:32.145 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:32.145 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:32.145 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:32.145 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.145 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:32.145 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948998+0000", 2024-09-15T03:47:32.145 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.145 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.145 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.145 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:32.145 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948998+0000", 2024-09-15T03:47:32.145 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.145 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.145 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.146 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:32.146 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948999+0000", 2024-09-15T03:47:32.146 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.71e-07 2024-09-15T03:47:32.146 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.146 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.146 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:32.146 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949020+0000", 2024-09-15T03:47:32.146 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1488000000000002e-05 2024-09-15T03:47:32.146 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.146 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.146 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:32.146 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949047+0000", 2024-09-15T03:47:32.146 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6409000000000001e-05 2024-09-15T03:47:32.146 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.147 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.147 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:32.147 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949054+0000", 2024-09-15T03:47:32.147 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3830000000000003e-06 2024-09-15T03:47:32.147 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.147 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.147 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:32.147 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949111+0000", 2024-09-15T03:47:32.147 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7074999999999998e-05 2024-09-15T03:47:32.147 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.147 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.147 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:32.147 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949120+0000", 2024-09-15T03:47:32.148 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1549999999999996e-06 2024-09-15T03:47:32.148 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.148 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.148 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:32.148 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949214+0000", 2024-09-15T03:47:32.148 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4046e-05 2024-09-15T03:47:32.148 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.148 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.148 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:32.148 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949225+0000", 2024-09-15T03:47:32.148 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0567999999999999e-05 2024-09-15T03:47:32.148 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.148 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.148 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:32.149 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949303+0000", 2024-09-15T03:47:32.149 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8065999999999994e-05 2024-09-15T03:47:32.149 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.149 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.149 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:32.149 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949317+0000", 2024-09-15T03:47:32.149 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4035e-05 2024-09-15T03:47:32.149 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.149 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.149 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:32.149 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.960945+0000", 2024-09-15T03:47:32.149 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011628266 2024-09-15T03:47:32.149 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.150 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:32.150 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.150 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.150 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.150 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.9s4 74/70 ECSubWrite(tid=108, reqid=client.4521.0:569, at_version=74'33, trim_to=0'0, roll_forward_to=74'28))", 2024-09-15T03:47:32.150 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.949550+0000", 2024-09-15T03:47:32.150 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.138245961000001, 2024-09-15T03:47:32.150 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012286695, 2024-09-15T03:47:32.150 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:32.150 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:32.150 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:32.150 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.150 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:32.151 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949550+0000", 2024-09-15T03:47:32.151 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.151 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.151 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.151 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:32.151 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949550+0000", 2024-09-15T03:47:32.151 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.151 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.151 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.151 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:32.151 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949551+0000", 2024-09-15T03:47:32.151 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9999999999999996e-07 2024-09-15T03:47:32.151 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.151 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.152 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:32.152 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949590+0000", 2024-09-15T03:47:32.152 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9560000000000001e-05 2024-09-15T03:47:32.152 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.152 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.152 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:32.152 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949611+0000", 2024-09-15T03:47:32.152 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1229e-05 2024-09-15T03:47:32.152 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.152 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.152 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:32.153 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949620+0000", 2024-09-15T03:47:32.153 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5140000000000001e-06 2024-09-15T03:47:32.153 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.153 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.153 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:32.153 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949670+0000", 2024-09-15T03:47:32.153 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9545000000000001e-05 2024-09-15T03:47:32.153 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.153 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.153 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:32.153 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949678+0000", 2024-09-15T03:47:32.153 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9260000000000006e-06 2024-09-15T03:47:32.153 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.154 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.154 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:32.154 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949987+0000", 2024-09-15T03:47:32.154 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030801899999999998 2024-09-15T03:47:32.154 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.154 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.154 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:32.154 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949997+0000", 2024-09-15T03:47:32.155 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0930999999999999e-05 2024-09-15T03:47:32.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.155 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.155 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:32.155 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950049+0000", 2024-09-15T03:47:32.155 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1384000000000001e-05 2024-09-15T03:47:32.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.155 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.155 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:32.155 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950063+0000", 2024-09-15T03:47:32.155 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4664000000000001e-05 2024-09-15T03:47:32.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.155 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.155 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:32.156 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961837+0000", 2024-09-15T03:47:32.156 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011773023000000001 2024-09-15T03:47:32.156 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.156 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:32.156 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.156 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.ds6 74/70 ECSubWrite(tid=201, reqid=client.4521.0:568, at_version=74'34, trim_to=0'0, roll_forward_to=74'32))", 2024-09-15T03:47:32.156 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.950074+0000", 2024-09-15T03:47:32.156 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.137721490000001, 2024-09-15T03:47:32.156 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011003285, 2024-09-15T03:47:32.156 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:32.156 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:32.156 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:32.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:32.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950074+0000", 2024-09-15T03:47:32.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:32.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950074+0000", 2024-09-15T03:47:32.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:32.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950075+0000", 2024-09-15T03:47:32.158 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0999999999999997e-07 2024-09-15T03:47:32.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.158 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.158 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:32.158 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950095+0000", 2024-09-15T03:47:32.158 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0256000000000001e-05 2024-09-15T03:47:32.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.158 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.158 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:32.158 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950109+0000", 2024-09-15T03:47:32.158 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3325e-05 2024-09-15T03:47:32.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.158 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.158 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:32.158 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950123+0000", 2024-09-15T03:47:32.159 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3893000000000001e-05 2024-09-15T03:47:32.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.159 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.159 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:32.159 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950170+0000", 2024-09-15T03:47:32.159 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7496000000000001e-05 2024-09-15T03:47:32.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.159 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.159 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:32.159 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950178+0000", 2024-09-15T03:47:32.159 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4190000000000002e-06 2024-09-15T03:47:32.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.160 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.160 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:32.160 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950271+0000", 2024-09-15T03:47:32.160 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2961999999999995e-05 2024-09-15T03:47:32.160 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.160 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.160 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:32.160 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950281+0000", 2024-09-15T03:47:32.160 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0960000000000007e-06 2024-09-15T03:47:32.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.161 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.161 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:32.161 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950317+0000", 2024-09-15T03:47:32.161 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6529999999999998e-05 2024-09-15T03:47:32.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.161 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.161 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:32.161 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950326+0000", 2024-09-15T03:47:32.161 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1419999999999999e-06 2024-09-15T03:47:32.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.161 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.161 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:32.162 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961078+0000", 2024-09-15T03:47:32.162 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010751356 2024-09-15T03:47:32.162 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.162 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:32.162 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.162 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.162 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 75/70 ECSubWrite(tid=198, reqid=client.4521.0:593, at_version=75'53, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:32.162 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:21.946647+0000", 2024-09-15T03:47:32.162 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.14114829, 2024-09-15T03:47:32.162 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012484423, 2024-09-15T03:47:32.162 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:32.162 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:32.162 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:32.163 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:32.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946647+0000", 2024-09-15T03:47:32.163 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.163 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:32.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946647+0000", 2024-09-15T03:47:32.163 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.163 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:32.164 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946648+0000", 2024-09-15T03:47:32.164 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6000000000000003e-07 2024-09-15T03:47:32.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.164 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:32.164 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946690+0000", 2024-09-15T03:47:32.164 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1430000000000001e-05 2024-09-15T03:47:32.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.164 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:32.164 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946705+0000", 2024-09-15T03:47:32.164 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4851000000000001e-05 2024-09-15T03:47:32.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.165 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:32.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946711+0000", 2024-09-15T03:47:32.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1380000000000004e-06 2024-09-15T03:47:32.165 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.165 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.165 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:32.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946755+0000", 2024-09-15T03:47:32.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4512999999999999e-05 2024-09-15T03:47:32.165 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.165 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.165 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:32.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946764+0000", 2024-09-15T03:47:32.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.337e-06 2024-09-15T03:47:32.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.166 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:32.166 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946900+0000", 2024-09-15T03:47:32.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013671500000000001 2024-09-15T03:47:32.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.166 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:32.166 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946922+0000", 2024-09-15T03:47:32.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1848000000000002e-05 2024-09-15T03:47:32.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.166 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:32.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.946994+0000", 2024-09-15T03:47:32.167 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1790000000000002e-05 2024-09-15T03:47:32.167 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.167 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.167 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:32.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947046+0000", 2024-09-15T03:47:32.167 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2565000000000002e-05 2024-09-15T03:47:32.167 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.167 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.167 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:32.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.959132+0000", 2024-09-15T03:47:32.167 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012085476 2024-09-15T03:47:32.167 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.167 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:32.168 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.168 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 75/70 ECSubWrite(tid=200, reqid=client.4521.0:594, at_version=75'54, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:32.168 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:21.947454+0000", 2024-09-15T03:47:32.168 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.14034146, 2024-09-15T03:47:32.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012079137, 2024-09-15T03:47:32.168 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:32.168 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:32.168 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:32.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.168 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:32.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947454+0000", 2024-09-15T03:47:32.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:32.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947454+0000", 2024-09-15T03:47:32.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:32.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947455+0000", 2024-09-15T03:47:32.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9800000000000003e-07 2024-09-15T03:47:32.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:32.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947472+0000", 2024-09-15T03:47:32.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7084000000000001e-05 2024-09-15T03:47:32.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.170 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.170 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:32.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947486+0000", 2024-09-15T03:47:32.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3684e-05 2024-09-15T03:47:32.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.170 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.170 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:32.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947492+0000", 2024-09-15T03:47:32.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5830000000000003e-06 2024-09-15T03:47:32.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.170 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.170 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:32.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947534+0000", 2024-09-15T03:47:32.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2107000000000002e-05 2024-09-15T03:47:32.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.171 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:32.171 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947543+0000", 2024-09-15T03:47:32.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0850000000000003e-06 2024-09-15T03:47:32.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.171 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:32.171 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947657+0000", 2024-09-15T03:47:32.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000113465 2024-09-15T03:47:32.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.172 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:32.172 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.947664+0000", 2024-09-15T03:47:32.172 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.525e-06 2024-09-15T03:47:32.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.172 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.172 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:32.172 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948013+0000", 2024-09-15T03:47:32.172 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034869399999999998 2024-09-15T03:47:32.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.172 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.172 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:32.172 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948031+0000", 2024-09-15T03:47:32.172 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7571000000000002e-05 2024-09-15T03:47:32.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.173 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.173 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:32.173 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.959533+0000", 2024-09-15T03:47:32.173 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011502741 2024-09-15T03:47:32.173 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.173 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:32.173 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.173 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.173 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s5 75/70 ECSubWrite(tid=201, reqid=client.4521.0:585, at_version=0'0, trim_to=0'0, roll_forward_to=75'33))", 2024-09-15T03:47:32.173 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:21.948682+0000", 2024-09-15T03:47:32.173 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.139114134, 2024-09-15T03:47:32.174 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01048613, 2024-09-15T03:47:32.174 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:32.174 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:32.174 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:32.174 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.174 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:32.174 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948682+0000", 2024-09-15T03:47:32.174 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.174 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.174 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:32.174 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948682+0000", 2024-09-15T03:47:32.174 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:32.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948682+0000", 2024-09-15T03:47:32.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2200000000000004e-07 2024-09-15T03:47:32.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:32.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948690+0000", 2024-09-15T03:47:32.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9170000000000006e-06 2024-09-15T03:47:32.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:32.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948701+0000", 2024-09-15T03:47:32.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0414999999999999e-05 2024-09-15T03:47:32.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:32.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948708+0000", 2024-09-15T03:47:32.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2169999999999997e-06 2024-09-15T03:47:32.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:32.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948755+0000", 2024-09-15T03:47:32.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6968999999999997e-05 2024-09-15T03:47:32.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.177 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:32.177 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948760+0000", 2024-09-15T03:47:32.177 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.78e-06 2024-09-15T03:47:32.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.177 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:32.177 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948838+0000", 2024-09-15T03:47:32.177 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8143e-05 2024-09-15T03:47:32.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.177 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:32.177 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948853+0000", 2024-09-15T03:47:32.177 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5449999999999999e-05 2024-09-15T03:47:32.178 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.178 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.178 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:32.178 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948905+0000", 2024-09-15T03:47:32.178 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1422000000000003e-05 2024-09-15T03:47:32.178 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.178 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.178 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:32.178 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.948919+0000", 2024-09-15T03:47:32.178 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4448e-05 2024-09-15T03:47:32.178 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.178 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.178 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:32.178 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.959168+0000", 2024-09-15T03:47:32.179 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010248747000000001 2024-09-15T03:47:32.179 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.179 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:32.179 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.179 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.179 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.179 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s2 75/70 ECSubWrite(tid=203, reqid=client.4521.0:594, at_version=0'0, trim_to=0'0, roll_forward_to=75'54))", 2024-09-15T03:47:32.179 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:21.949416+0000", 2024-09-15T03:47:32.179 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.138380123999999, 2024-09-15T03:47:32.179 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010405688999999999, 2024-09-15T03:47:32.179 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:32.179 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:32.179 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:32.179 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.180 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:32.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949416+0000", 2024-09-15T03:47:32.180 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.180 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.180 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:32.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949416+0000", 2024-09-15T03:47:32.180 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.180 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.180 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:32.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949416+0000", 2024-09-15T03:47:32.181 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9200000000000001e-07 2024-09-15T03:47:32.181 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.181 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:32.181 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949426+0000", 2024-09-15T03:47:32.181 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0169999999999999e-05 2024-09-15T03:47:32.181 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.181 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:32.181 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949441+0000", 2024-09-15T03:47:32.181 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4168000000000001e-05 2024-09-15T03:47:32.181 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.181 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:32.182 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949451+0000", 2024-09-15T03:47:32.182 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0003000000000001e-05 2024-09-15T03:47:32.182 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.182 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.182 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:32.182 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949485+0000", 2024-09-15T03:47:32.182 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4489999999999997e-05 2024-09-15T03:47:32.182 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.182 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.182 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:32.182 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949492+0000", 2024-09-15T03:47:32.182 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4139999999999997e-06 2024-09-15T03:47:32.182 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.182 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.183 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:32.183 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949600+0000", 2024-09-15T03:47:32.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000107219 2024-09-15T03:47:32.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.183 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.183 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:32.183 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949614+0000", 2024-09-15T03:47:32.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4586999999999999e-05 2024-09-15T03:47:32.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.183 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.183 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:32.183 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949645+0000", 2024-09-15T03:47:32.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1000999999999997e-05 2024-09-15T03:47:32.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.184 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:32.184 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.949660+0000", 2024-09-15T03:47:32.184 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4843e-05 2024-09-15T03:47:32.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.184 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:32.184 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.959821+0000", 2024-09-15T03:47:32.184 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010161201999999999 2024-09-15T03:47:32.184 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.184 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:32.185 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.185 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.185 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.0s2 75/70 ECSubWrite(tid=109, reqid=client.4521.0:587, at_version=75'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:32.185 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:21.950286+0000", 2024-09-15T03:47:32.185 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.13750933, 2024-09-15T03:47:32.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013522467999999999, 2024-09-15T03:47:32.185 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:32.185 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:32.185 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:32.185 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.185 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:32.186 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950286+0000", 2024-09-15T03:47:32.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.186 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.186 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:32.186 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950286+0000", 2024-09-15T03:47:32.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.186 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.186 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:32.186 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950287+0000", 2024-09-15T03:47:32.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.54e-07 2024-09-15T03:47:32.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:32.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950503+0000", 2024-09-15T03:47:32.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000216725 2024-09-15T03:47:32.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:32.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950517+0000", 2024-09-15T03:47:32.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3988999999999999e-05 2024-09-15T03:47:32.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:32.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950527+0000", 2024-09-15T03:47:32.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.302e-06 2024-09-15T03:47:32.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:32.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950577+0000", 2024-09-15T03:47:32.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0395000000000001e-05 2024-09-15T03:47:32.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:32.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950586+0000", 2024-09-15T03:47:32.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5080000000000005e-06 2024-09-15T03:47:32.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.189 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:32.189 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950919+0000", 2024-09-15T03:47:32.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033313599999999999 2024-09-15T03:47:32.189 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.189 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.189 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:32.189 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.950927+0000", 2024-09-15T03:47:32.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2279999999999998e-06 2024-09-15T03:47:32.189 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.189 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.189 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:32.189 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.951002+0000", 2024-09-15T03:47:32.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4975000000000006e-05 2024-09-15T03:47:32.190 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.190 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:32.190 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.951026+0000", 2024-09-15T03:47:32.190 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4258000000000001e-05 2024-09-15T03:47:32.190 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.190 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:32.190 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.963809+0000", 2024-09-15T03:47:32.190 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012782798 2024-09-15T03:47:32.190 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.190 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:32.190 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.191 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.0s2 75/70 ECSubWrite(tid=110, reqid=client.4521.0:587, at_version=0'0, trim_to=0'0, roll_forward_to=75'28))", 2024-09-15T03:47:32.191 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:21.953245+0000", 2024-09-15T03:47:32.191 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.134550672, 2024-09-15T03:47:32.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010716737, 2024-09-15T03:47:32.191 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:32.191 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:32.191 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:32.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:32.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953245+0000", 2024-09-15T03:47:32.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:32.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953245+0000", 2024-09-15T03:47:32.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:32.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953246+0000", 2024-09-15T03:47:32.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.096e-06 2024-09-15T03:47:32.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:32.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953256+0000", 2024-09-15T03:47:32.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.026e-05 2024-09-15T03:47:32.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:32.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953270+0000", 2024-09-15T03:47:32.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3293e-05 2024-09-15T03:47:32.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:32.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953275+0000", 2024-09-15T03:47:32.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4770000000000002e-06 2024-09-15T03:47:32.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:32.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953322+0000", 2024-09-15T03:47:32.194 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7108999999999999e-05 2024-09-15T03:47:32.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.194 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:32.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953332+0000", 2024-09-15T03:47:32.194 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9559999999999992e-06 2024-09-15T03:47:32.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.194 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:32.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953403+0000", 2024-09-15T03:47:32.194 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0653000000000003e-05 2024-09-15T03:47:32.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.195 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.195 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:32.195 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953410+0000", 2024-09-15T03:47:32.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1969999999999999e-06 2024-09-15T03:47:32.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.195 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.195 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:32.195 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953456+0000", 2024-09-15T03:47:32.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6220000000000001e-05 2024-09-15T03:47:32.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.195 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.195 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:32.195 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.953471+0000", 2024-09-15T03:47:32.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5057e-05 2024-09-15T03:47:32.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.196 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:32.196 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.963962+0000", 2024-09-15T03:47:32.196 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010490419000000001 2024-09-15T03:47:32.196 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.196 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:32.196 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.196 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs7 75/70 ECSubWrite(tid=160, reqid=client.4521.0:595, at_version=75'87, trim_to=0'0, roll_forward_to=73'85))", 2024-09-15T03:47:32.196 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:21.968440+0000", 2024-09-15T03:47:32.196 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.119355531, 2024-09-15T03:47:32.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010079088999999999, 2024-09-15T03:47:32.197 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:32.197 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:32.197 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:32.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:32.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.968440+0000", 2024-09-15T03:47:32.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:32.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.968440+0000", 2024-09-15T03:47:32.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:32.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.968440+0000", 2024-09-15T03:47:32.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1500000000000001e-07 2024-09-15T03:47:32.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:32.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969096+0000", 2024-09-15T03:47:32.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00065533300000000002 2024-09-15T03:47:32.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.199 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:32.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969112+0000", 2024-09-15T03:47:32.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5943999999999998e-05 2024-09-15T03:47:32.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.199 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:32.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969118+0000", 2024-09-15T03:47:32.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3659999999999997e-06 2024-09-15T03:47:32.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.199 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:32.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969171+0000", 2024-09-15T03:47:32.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2951000000000002e-05 2024-09-15T03:47:32.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.200 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:32.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969184+0000", 2024-09-15T03:47:32.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2536e-05 2024-09-15T03:47:32.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.200 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:32.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969454+0000", 2024-09-15T03:47:32.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027025600000000002 2024-09-15T03:47:32.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.200 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:32.201 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969463+0000", 2024-09-15T03:47:32.201 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6569999999999992e-06 2024-09-15T03:47:32.201 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.201 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.201 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:32.201 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969584+0000", 2024-09-15T03:47:32.201 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012004 2024-09-15T03:47:32.201 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.201 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.201 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:32.201 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.969604+0000", 2024-09-15T03:47:32.201 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0678000000000001e-05 2024-09-15T03:47:32.201 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:21.978519+0000", 2024-09-15T03:47:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0089151130000000006 2024-09-15T03:47:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s6 83/80 ECSubWrite(tid=226, reqid=client.4521.0:688, at_version=83'95, trim_to=0'0, roll_forward_to=83'93))", 2024-09-15T03:47:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:30.090752+0000", 2024-09-15T03:47:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.9970438700000002, 2024-09-15T03:47:32.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013202034, 2024-09-15T03:47:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090752+0000", 2024-09-15T03:47:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090752+0000", 2024-09-15T03:47:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:32.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090753+0000", 2024-09-15T03:47:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1999999999999998e-07 2024-09-15T03:47:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090834+0000", 2024-09-15T03:47:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1649000000000004e-05 2024-09-15T03:47:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090852+0000", 2024-09-15T03:47:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7044000000000001e-05 2024-09-15T03:47:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090862+0000", 2024-09-15T03:47:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0209e-05 2024-09-15T03:47:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090914+0000", 2024-09-15T03:47:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2624000000000003e-05 2024-09-15T03:47:32.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090925+0000", 2024-09-15T03:47:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0669000000000001e-05 2024-09-15T03:47:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.091336+0000", 2024-09-15T03:47:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00041115200000000002 2024-09-15T03:47:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.091372+0000", 2024-09-15T03:47:32.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5345999999999997e-05 2024-09-15T03:47:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.091474+0000", 2024-09-15T03:47:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000102216 2024-09-15T03:47:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.091496+0000", 2024-09-15T03:47:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2340999999999998e-05 2024-09-15T03:47:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:32.207 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:32.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.103954+0000", 2024-09-15T03:47:32.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012457864000000001 2024-09-15T03:47:32.208 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.208 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:32.208 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.208 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:32.208 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:32.208 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:32.208 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10] out_osds: [6] dead_osds: [10] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] 2024-09-15T03:47:32.209 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T03:47:32.209 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-09-15T03:47:32.209 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 6 2024-09-15T03:47:32.210 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:32.211 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:32.211 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-15T03:47:32.211 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:32.212 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:47:32.212 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T03:47:32.213 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:32.213 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T03:47:32.213 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:32.208+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:32.310 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:32.309+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:32.349 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:32.349 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:32.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:47:32.410 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:32.410+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:32.454 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:32.456 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:32.462 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:47:32.512 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:32.511+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:32.532 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-09-15T03:47:32.531+0000 7ff94fdf1640 -1 mon.a@0(leader).osd e85 definitely_dead 0 2024-09-15T03:47:32.597 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:32.597 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:32.605 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:47:32.612 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:32.612+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:32.707 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:32.707 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:32.712 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:32.712+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:32.715 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:47:32.813 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:32.813+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:32.857 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:32.857 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:32.865 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:47:32.914 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:32.914+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:32.967 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:32.968 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:32.976 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:47:32.987 INFO:tasks.rados.rados.0.smithi001.stdout:286: done (3 left) 2024-09-15T03:47:32.987 INFO:tasks.rados.rados.0.smithi001.stdout:287: read oid 32 snap 16 2024-09-15T03:47:32.987 INFO:tasks.rados.rados.0.smithi001.stdout:287: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-15T03:47:32.987 INFO:tasks.rados.rados.0.smithi001.stdout:288: rollback oid 25 current snap is 33 2024-09-15T03:47:32.987 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 25 to 20 2024-09-15T03:47:32.987 INFO:tasks.rados.rados.0.smithi001.stdout:285: finishing rollback tid 0 to smithi00132172-27 2024-09-15T03:47:32.987 INFO:tasks.rados.rados.0.smithi001.stdout:285: finishing rollback tid 1 to smithi00132172-27 2024-09-15T03:47:32.987 INFO:tasks.rados.rados.0.smithi001.stdout:285: finishing rollback tid 2 to smithi00132172-27 2024-09-15T03:47:32.988 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 27 v 54 (ObjNum 85 snap 18 seq_num 85) dirty exists 2024-09-15T03:47:32.988 INFO:tasks.rados.rados.0.smithi001.stdout:283: finishing copy_from racing read to smithi00132172-45 2024-09-15T03:47:32.988 INFO:tasks.rados.rados.0.smithi001.stdout:283: finishing copy_from to smithi00132172-45 2024-09-15T03:47:32.988 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 45 v 28 (ObjNum 99 snap 24 seq_num 99) dirty exists 2024-09-15T03:47:32.988 INFO:tasks.rados.rados.0.smithi001.stdout:284: expect (ObjNum 115 snap 32 seq_num 115) 2024-09-15T03:47:32.988 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-15T03:47:32.989 INFO:teuthology.orchestra.run.smithi001.stderr:marked in osd.6. 2024-09-15T03:47:32.997 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:32.996+0000 7f6f815b7640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:32.998 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:32.997+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:32.998 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:32.998+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:32.999 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:32.998+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:33.000 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:32.997+0000 7f0a725b8640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:33.015 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-09-15T03:47:33.016 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:33.014+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:33.100 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:33.100 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:33.109 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:47:33.116 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:33.115+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:33.131 INFO:tasks.rados.rados.0.smithi001.stdout:288: finishing rollback tid 0 to smithi00132172-25 2024-09-15T03:47:33.131 INFO:tasks.rados.rados.0.smithi001.stdout:288: finishing rollback tid 1 to smithi00132172-25 2024-09-15T03:47:33.131 INFO:tasks.rados.rados.0.smithi001.stdout:288: finishing rollback tid 2 to smithi00132172-25 2024-09-15T03:47:33.131 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 25 v 100 (ObjNum 66 snap 6 seq_num 66) dirty exists 2024-09-15T03:47:33.131 INFO:tasks.rados.rados.0.smithi001.stdout:283: done (4 left) 2024-09-15T03:47:33.131 INFO:tasks.rados.rados.0.smithi001.stdout:284: done (3 left) 2024-09-15T03:47:33.131 INFO:tasks.rados.rados.0.smithi001.stdout:285: done (2 left) 2024-09-15T03:47:33.131 INFO:tasks.rados.rados.0.smithi001.stdout:288: done (1 left) 2024-09-15T03:47:33.131 INFO:tasks.rados.rados.0.smithi001.stdout:289: rollback oid 48 current snap is 33 2024-09-15T03:47:33.132 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 48 to 24 2024-09-15T03:47:33.132 INFO:tasks.rados.rados.0.smithi001.stdout:290: rollback oid 39 current snap is 33 2024-09-15T03:47:33.132 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 39 to 24 2024-09-15T03:47:33.132 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 14 current snap is 33 2024-09-15T03:47:33.132 INFO:tasks.rados.rados.0.smithi001.stdout:291: seq_num 116 ranges {2621440=786432,3407872=458752,3866624=442368,4308992=262144} 2024-09-15T03:47:33.150 INFO:tasks.rados.rados.0.smithi001.stdout:291: writing smithi00132172-14 from 2621440 to 3407872 tid 1 2024-09-15T03:47:33.153 INFO:tasks.rados.rados.0.smithi001.stdout:291: writing smithi00132172-14 from 3407872 to 3866624 tid 2 2024-09-15T03:47:33.156 INFO:tasks.rados.rados.0.smithi001.stdout:291: writing smithi00132172-14 from 3866624 to 4308992 tid 3 2024-09-15T03:47:33.158 INFO:tasks.rados.rados.0.smithi001.stdout:291: writing smithi00132172-14 from 4308992 to 4571136 tid 4 2024-09-15T03:47:33.158 INFO:tasks.rados.rados.0.smithi001.stdout:292: copy_from oid 6 from oid 24 current snap is 33 2024-09-15T03:47:33.158 INFO:tasks.rados.rados.0.smithi001.stdout:289: finishing rollback tid 0 to smithi00132172-48 2024-09-15T03:47:33.158 INFO:tasks.rados.rados.0.smithi001.stdout:290: finishing rollback tid 0 to smithi00132172-39 2024-09-15T03:47:33.159 INFO:tasks.rados.rados.0.smithi001.stdout:289: finishing rollback tid 1 to smithi00132172-48 2024-09-15T03:47:33.159 INFO:tasks.rados.rados.0.smithi001.stdout:289: finishing rollback tid 2 to smithi00132172-48 2024-09-15T03:47:33.159 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 48 v 67 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-15T03:47:33.159 INFO:tasks.rados.rados.0.smithi001.stdout:290: finishing rollback tid 1 to smithi00132172-39 2024-09-15T03:47:33.159 INFO:tasks.rados.rados.0.smithi001.stdout:290: finishing rollback tid 2 to smithi00132172-39 2024-09-15T03:47:33.159 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 39 v 116 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-15T03:47:33.159 INFO:tasks.rados.rados.0.smithi001.stdout:291: finishing write tid 1 to smithi00132172-14 2024-09-15T03:47:33.159 INFO:tasks.rados.rados.0.smithi001.stdout:289: done (4 left) 2024-09-15T03:47:33.159 INFO:tasks.rados.rados.0.smithi001.stdout:290: done (3 left) 2024-09-15T03:47:33.159 INFO:tasks.rados.rados.0.smithi001.stdout:293: rollback oid 29 current snap is 33 2024-09-15T03:47:33.159 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 29 to 16 2024-09-15T03:47:33.159 INFO:tasks.rados.rados.0.smithi001.stdout:294: delete oid 40 current snap is 33 2024-09-15T03:47:33.159 INFO:tasks.rados.rados.0.smithi001.stdout:291: finishing write tid 2 to smithi00132172-14 2024-09-15T03:47:33.160 INFO:tasks.rados.rados.0.smithi001.stdout:291: finishing write tid 3 to smithi00132172-14 2024-09-15T03:47:33.160 INFO:tasks.rados.rados.0.smithi001.stdout:291: finishing write tid 4 to smithi00132172-14 2024-09-15T03:47:33.162 INFO:tasks.rados.rados.0.smithi001.stdout:291: finishing write tid 5 to smithi00132172-14 2024-09-15T03:47:33.163 INFO:tasks.rados.rados.0.smithi001.stdout:291: finishing write tid 6 to smithi00132172-14 2024-09-15T03:47:33.163 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 14 v 122 (ObjNum 116 snap 33 seq_num 116) dirty exists 2024-09-15T03:47:33.163 INFO:tasks.rados.rados.0.smithi001.stdout:291: left oid 14 (ObjNum 116 snap 33 seq_num 116) 2024-09-15T03:47:33.217 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:33.217 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:33.217 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:33.216+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:33.226 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:47:33.318 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:33.317+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:33.367 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:33.367 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:33.376 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:47:33.419 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:33.418+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:33.481 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:33.481 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:33.486 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:47:33.520 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:33.519+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:33.586 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:33.586 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:33.595 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:47:33.639 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:33.638+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:33.696 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:33.697 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:33.705 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:47:33.747 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:33.746+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:33.823 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:33.822+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:33.846 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:33.847 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:33.855 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:47:33.924 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:33.923+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:33.956 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:33.956 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:33.964 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:47:33.996 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:33.995+0000 7f6f855bf640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:33.996 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:33.995+0000 7f6f855bf640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:33.996 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:33.995+0000 7f6f855bf640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:33.996 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:33.995+0000 7f6f855bf640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:33.996 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:33.995+0000 7f6f855bf640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:33.999 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.995+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:33.999 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.995+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:33.999 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.995+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:33.999 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.995+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:33.999 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.995+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:33.999 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.995+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.000 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.995+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.000 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.996+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.000 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.996+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.000 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.996+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.000 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.996+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.000 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.996+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.000 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.996+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.000 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.996+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.001 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.998+0000 7f0a725b8640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.001 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.998+0000 7f0a725b8640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.001 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.998+0000 7f0a725b8640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.002 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.998+0000 7f0a725b8640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.002 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.998+0000 7f0a725b8640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.002 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.998+0000 7f0a725b8640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.002 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.998+0000 7f0a725b8640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.002 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.999+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.002 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.999+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.002 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.999+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.002 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.999+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.002 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.999+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.002 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.999+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.002 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.999+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.002 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.999+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.002 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.999+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.002 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.999+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.003 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.999+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.003 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.999+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.003 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.999+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.003 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:33.999+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.003 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:34.000+0000 7f0a725b8640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.003 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:34.000+0000 7f0a725b8640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.003 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:34.000+0000 7f0a725b8640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.003 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:34.000+0000 7f0a725b8640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.003 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:34.000+0000 7f0a725b8640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.003 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:34.000+0000 7f0a725b8640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.003 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:34.000+0000 7f0a725b8640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.004 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:34.000+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.004 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:34.000+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.004 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:34.000+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.004 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:34.000+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.005 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:34.000+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.005 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:34.000+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.005 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:34.000+0000 7f0a765c0640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.005 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:34.004+0000 7f0a725b8640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.005 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:34.004+0000 7f0a725b8640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.006 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.004+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.007 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:34.006+0000 7f0a725b8640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.007 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:34.006+0000 7f0a725b8640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.008 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.008+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.008 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.008+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.019 INFO:tasks.rados.rados.0.smithi001.stdout:292: finishing copy_from to smithi00132172-6 2024-09-15T03:47:34.019 INFO:tasks.rados.rados.0.smithi001.stdout:292: got expected ENOENT (src dne) 2024-09-15T03:47:34.020 INFO:tasks.rados.rados.0.smithi001.stdout:292: finishing copy_from racing read to smithi00132172-6 2024-09-15T03:47:34.021 INFO:tasks.rados.rados.0.smithi001.stdout:287: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-15T03:47:34.025 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:34.024+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:34.106 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:34.107 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:34.115 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:47:34.126 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:34.125+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:34.213 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:47:34.224 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:34.224 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:34.226 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:34.226+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:34.233 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:47:34.327 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:34.326+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:34.330 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:34.330 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:34.334 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:34.334 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:34.342 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:47:34.344 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:34.345 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:34.359 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:34.359 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:34.373 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:34.374 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:34.388 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:34.388 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:34.403 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:34.403 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:34.418 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:34.418 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:34.428 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:34.427+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:34.434 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:34.435 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:34.446 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:34.447 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:34.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:34.447 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:34.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.447 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs6 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:34.447 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108711+0000", 2024-09-15T03:47:34.447 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.335878770999997, 2024-09-15T03:47:34.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01472483, 2024-09-15T03:47:34.447 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:34.447 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:34.447 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:34.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:34.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108711+0000", 2024-09-15T03:47:34.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:34.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108711+0000", 2024-09-15T03:47:34.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:34.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108712+0000", 2024-09-15T03:47:34.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.85e-07 2024-09-15T03:47:34.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:34.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108745+0000", 2024-09-15T03:47:34.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2762000000000002e-05 2024-09-15T03:47:34.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:34.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108760+0000", 2024-09-15T03:47:34.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4567e-05 2024-09-15T03:47:34.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:34.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108780+0000", 2024-09-15T03:47:34.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0122000000000001e-05 2024-09-15T03:47:34.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108842+0000", 2024-09-15T03:47:34.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2297000000000003e-05 2024-09-15T03:47:34.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:34.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108851+0000", 2024-09-15T03:47:34.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0129999999999999e-06 2024-09-15T03:47:34.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:34.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108942+0000", 2024-09-15T03:47:34.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0713999999999999e-05 2024-09-15T03:47:34.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:34.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108949+0000", 2024-09-15T03:47:34.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1600000000000001e-06 2024-09-15T03:47:34.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:34.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109975+0000", 2024-09-15T03:47:34.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010264429999999999 2024-09-15T03:47:34.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:34.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109994+0000", 2024-09-15T03:47:34.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8374000000000001e-05 2024-09-15T03:47:34.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:34.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123436+0000", 2024-09-15T03:47:34.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013442592999999999 2024-09-15T03:47:34.452 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.453 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:34.453 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.453 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs6 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:34.453 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108809+0000", 2024-09-15T03:47:34.453 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.335781042999997, 2024-09-15T03:47:34.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014757497, 2024-09-15T03:47:34.453 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:34.453 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:34.453 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:34.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:34.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108809+0000", 2024-09-15T03:47:34.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:34.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108809+0000", 2024-09-15T03:47:34.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:34.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108810+0000", 2024-09-15T03:47:34.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.15e-07 2024-09-15T03:47:34.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:34.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108825+0000", 2024-09-15T03:47:34.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5092999999999999e-05 2024-09-15T03:47:34.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:34.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108835+0000", 2024-09-15T03:47:34.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0251000000000001e-05 2024-09-15T03:47:34.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:34.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108841+0000", 2024-09-15T03:47:34.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0510000000000002e-06 2024-09-15T03:47:34.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108971+0000", 2024-09-15T03:47:34.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012976599999999999 2024-09-15T03:47:34.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:34.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108976+0000", 2024-09-15T03:47:34.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8450000000000002e-06 2024-09-15T03:47:34.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:34.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109032+0000", 2024-09-15T03:47:34.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6119999999999998e-05 2024-09-15T03:47:34.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:34.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109978+0000", 2024-09-15T03:47:34.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00094603899999999997 2024-09-15T03:47:34.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:34.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110007+0000", 2024-09-15T03:47:34.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9062999999999999e-05 2024-09-15T03:47:34.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:34.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110025+0000", 2024-09-15T03:47:34.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7717999999999999e-05 2024-09-15T03:47:34.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:34.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123567+0000", 2024-09-15T03:47:34.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013542136 2024-09-15T03:47:34.458 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.458 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:34.458 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.458 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:324 3.7s0 3:e40d1694:::smithi00132172-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:34.458 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:34.458 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.600467608999999, 2024-09-15T03:47:34.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0031668960000002, 2024-09-15T03:47:34.459 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:34.459 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:34.459 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:34.459 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:34.459 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:34.459 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 324 2024-09-15T03:47:34.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.459 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:34.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:34.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:34.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:34.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:34.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:34.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:34.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:47:34.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:34.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844125+0000", 2024-09-15T03:47:34.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.142e-06 2024-09-15T03:47:34.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:34.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844132+0000", 2024-09-15T03:47:34.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0439999999999998e-06 2024-09-15T03:47:34.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:34.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844136+0000", 2024-09-15T03:47:34.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.126e-06 2024-09-15T03:47:34.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.847618+0000", 2024-09-15T03:47:34.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0034813790000000002 2024-09-15T03:47:34.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:34.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.847709+0000", 2024-09-15T03:47:34.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0835999999999996e-05 2024-09-15T03:47:34.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.849483+0000", 2024-09-15T03:47:34.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001774929 2024-09-15T03:47:34.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.849574+0000", 2024-09-15T03:47:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0847999999999995e-05 2024-09-15T03:47:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.847290+0000", 2024-09-15T03:47:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9977151819999999 2024-09-15T03:47:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:334 3.7s0 3:ece1472c:::smithi00132172-43:head [append 0~0] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.587740521000001, 2024-09-15T03:47:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.997592021, 2024-09-15T03:47:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 334 2024-09-15T03:47:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856851+0000", 2024-09-15T03:47:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.238e-06 2024-09-15T03:47:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856855+0000", 2024-09-15T03:47:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3640000000000001e-06 2024-09-15T03:47:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856868+0000", 2024-09-15T03:47:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2588999999999999e-05 2024-09-15T03:47:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856873+0000", 2024-09-15T03:47:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5350000000000004e-06 2024-09-15T03:47:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856916+0000", 2024-09-15T03:47:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2889000000000002e-05 2024-09-15T03:47:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856983+0000", 2024-09-15T03:47:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6746000000000001e-05 2024-09-15T03:47:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857448+0000", 2024-09-15T03:47:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046498500000000001 2024-09-15T03:47:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857720+0000", 2024-09-15T03:47:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027227600000000001 2024-09-15T03:47:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857741+0000", 2024-09-15T03:47:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1149999999999999e-05 2024-09-15T03:47:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857795+0000", 2024-09-15T03:47:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3919999999999999e-05 2024-09-15T03:47:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859582+0000", 2024-09-15T03:47:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017862640000000001 2024-09-15T03:47:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854347+0000", 2024-09-15T03:47:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9947654780000001 2024-09-15T03:47:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854442+0000", 2024-09-15T03:47:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4586999999999994e-05 2024-09-15T03:47:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:335 3.7s0 3:ece1472c:::smithi00132172-43:head [rollback 12] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.587710504, 2024-09-15T03:47:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9977859200000001, 2024-09-15T03:47:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 335 2024-09-15T03:47:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8099999999999999e-07 2024-09-15T03:47:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856882+0000", 2024-09-15T03:47:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.649e-06 2024-09-15T03:47:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856885+0000", 2024-09-15T03:47:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.664e-06 2024-09-15T03:47:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:34.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856888+0000", 2024-09-15T03:47:34.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3910000000000001e-06 2024-09-15T03:47:34.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857826+0000", 2024-09-15T03:47:34.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00093817900000000003 2024-09-15T03:47:34.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857940+0000", 2024-09-15T03:47:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114424 2024-09-15T03:47:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858504+0000", 2024-09-15T03:47:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056400899999999997 2024-09-15T03:47:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858593+0000", 2024-09-15T03:47:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8463999999999999e-05 2024-09-15T03:47:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:34.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858604+0000", 2024-09-15T03:47:34.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1567e-05 2024-09-15T03:47:34.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:34.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858642+0000", 2024-09-15T03:47:34.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7369000000000001e-05 2024-09-15T03:47:34.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:34.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859588+0000", 2024-09-15T03:47:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00094661799999999998 2024-09-15T03:47:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854594+0000", 2024-09-15T03:47:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9950055550000001 2024-09-15T03:47:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854666+0000", 2024-09-15T03:47:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1749999999999996e-05 2024-09-15T03:47:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:336 3.7s0 3:ece1472c:::smithi00132172-43:head [append 0~0] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.587679454000003, 2024-09-15T03:47:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9980017820000002, 2024-09-15T03:47:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 336 2024-09-15T03:47:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8000000000000002e-07 2024-09-15T03:47:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856913+0000", 2024-09-15T03:47:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5400000000000001e-06 2024-09-15T03:47:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856918+0000", 2024-09-15T03:47:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1540000000000001e-06 2024-09-15T03:47:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856921+0000", 2024-09-15T03:47:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2550000000000002e-06 2024-09-15T03:47:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858655+0000", 2024-09-15T03:47:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017336070000000001 2024-09-15T03:47:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858741+0000", 2024-09-15T03:47:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6305999999999997e-05 2024-09-15T03:47:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859193+0000", 2024-09-15T03:47:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045221200000000001 2024-09-15T03:47:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859319+0000", 2024-09-15T03:47:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012542900000000001 2024-09-15T03:47:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859331+0000", 2024-09-15T03:47:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2028000000000001e-05 2024-09-15T03:47:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859369+0000", 2024-09-15T03:47:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8522999999999997e-05 2024-09-15T03:47:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859649+0000", 2024-09-15T03:47:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028008200000000002 2024-09-15T03:47:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854800+0000", 2024-09-15T03:47:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9951506540000001 2024-09-15T03:47:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854913+0000", 2024-09-15T03:47:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112712 2024-09-15T03:47:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:346 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:47:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.606183084000001, 2024-09-15T03:47:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005631948, 2024-09-15T03:47:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 346 2024-09-15T03:47:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:47:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:47:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838410+0000", 2024-09-15T03:47:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4430000000000002e-06 2024-09-15T03:47:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838419+0000", 2024-09-15T03:47:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1730000000000001e-06 2024-09-15T03:47:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838444+0000", 2024-09-15T03:47:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5111000000000001e-05 2024-09-15T03:47:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838455+0000", 2024-09-15T03:47:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0839000000000001e-05 2024-09-15T03:47:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3415000000000002e-05 2024-09-15T03:47:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838527+0000", 2024-09-15T03:47:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0550000000000005e-06 2024-09-15T03:47:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840638+0000", 2024-09-15T03:47:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021105379999999999 2024-09-15T03:47:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840729+0000", 2024-09-15T03:47:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0977999999999995e-05 2024-09-15T03:47:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.841834+0000", 2024-09-15T03:47:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011055640000000001 2024-09-15T03:47:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842223+0000", 2024-09-15T03:47:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038899900000000001 2024-09-15T03:47:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842241+0000", 2024-09-15T03:47:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7317e-05 2024-09-15T03:47:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842298+0000", 2024-09-15T03:47:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7825000000000003e-05 2024-09-15T03:47:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844866+0000", 2024-09-15T03:47:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025675289999999998 2024-09-15T03:47:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843973+0000", 2024-09-15T03:47:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99910710599999997 2024-09-15T03:47:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844039+0000", 2024-09-15T03:47:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6056000000000006e-05 2024-09-15T03:47:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:347 3.2s0 3:48ceb02f:::smithi00132172-5:head [rollback c] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:47:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.606098287999998, 2024-09-15T03:47:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005723841, 2024-09-15T03:47:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 347 2024-09-15T03:47:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:47:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:47:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838493+0000", 2024-09-15T03:47:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7300000000000005e-07 2024-09-15T03:47:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838498+0000", 2024-09-15T03:47:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9300000000000002e-06 2024-09-15T03:47:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838508+0000", 2024-09-15T03:47:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9790000000000005e-06 2024-09-15T03:47:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838513+0000", 2024-09-15T03:47:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8890000000000001e-06 2024-09-15T03:47:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838583+0000", 2024-09-15T03:47:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0696000000000002e-05 2024-09-15T03:47:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838590+0000", 2024-09-15T03:47:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2990000000000002e-06 2024-09-15T03:47:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842313+0000", 2024-09-15T03:47:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037238810000000001 2024-09-15T03:47:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842432+0000", 2024-09-15T03:47:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118474 2024-09-15T03:47:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843108+0000", 2024-09-15T03:47:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00067639299999999996 2024-09-15T03:47:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843227+0000", 2024-09-15T03:47:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119092 2024-09-15T03:47:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843242+0000", 2024-09-15T03:47:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4216000000000001e-05 2024-09-15T03:47:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843292+0000", 2024-09-15T03:47:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0800999999999997e-05 2024-09-15T03:47:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844878+0000", 2024-09-15T03:47:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015852749999999999 2024-09-15T03:47:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844147+0000", 2024-09-15T03:47:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99926949200000004 2024-09-15T03:47:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844216+0000", 2024-09-15T03:47:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8551000000000001e-05 2024-09-15T03:47:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:348 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.606072345000001, 2024-09-15T03:47:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.006019768, 2024-09-15T03:47:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 348 2024-09-15T03:47:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:34.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:34.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:34.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:34.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:34.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8700000000000001e-07 2024-09-15T03:47:34.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838521+0000", 2024-09-15T03:47:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3740000000000001e-06 2024-09-15T03:47:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838527+0000", 2024-09-15T03:47:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1999999999999999e-06 2024-09-15T03:47:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838532+0000", 2024-09-15T03:47:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2059999999999998e-06 2024-09-15T03:47:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838609+0000", 2024-09-15T03:47:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7193000000000004e-05 2024-09-15T03:47:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838615+0000", 2024-09-15T03:47:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7980000000000002e-06 2024-09-15T03:47:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843986+0000", 2024-09-15T03:47:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053706450000000003 2024-09-15T03:47:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844106+0000", 2024-09-15T03:47:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120079 2024-09-15T03:47:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844669+0000", 2024-09-15T03:47:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056290300000000001 2024-09-15T03:47:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844804+0000", 2024-09-15T03:47:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013575600000000001 2024-09-15T03:47:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844820+0000", 2024-09-15T03:47:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5145e-05 2024-09-15T03:47:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844893+0000", 2024-09-15T03:47:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3270999999999997e-05 2024-09-15T03:47:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845043+0000", 2024-09-15T03:47:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150144 2024-09-15T03:47:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844473+0000", 2024-09-15T03:47:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99943048700000003 2024-09-15T03:47:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844538+0000", 2024-09-15T03:47:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4179999999999999e-05 2024-09-15T03:47:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:352 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~557056] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:47:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.599874700999997, 2024-09-15T03:47:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.016265065, 2024-09-15T03:47:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 352 2024-09-15T03:47:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:47:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:47:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844718+0000", 2024-09-15T03:47:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2340000000000001e-06 2024-09-15T03:47:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845237+0000", 2024-09-15T03:47:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051953399999999999 2024-09-15T03:47:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845273+0000", 2024-09-15T03:47:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5683999999999999e-05 2024-09-15T03:47:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845281+0000", 2024-09-15T03:47:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8439999999999998e-06 2024-09-15T03:47:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845365+0000", 2024-09-15T03:47:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4651e-05 2024-09-15T03:47:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845486+0000", 2024-09-15T03:47:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012009499999999999 2024-09-15T03:47:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847101+0000", 2024-09-15T03:47:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016157350000000001 2024-09-15T03:47:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847602+0000", 2024-09-15T03:47:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050066200000000003 2024-09-15T03:47:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847622+0000", 2024-09-15T03:47:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9749999999999999e-05 2024-09-15T03:47:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848616+0000", 2024-09-15T03:47:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099390799999999994 2024-09-15T03:47:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848709+0000", 2024-09-15T03:47:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2874000000000006e-05 2024-09-15T03:47:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.860888+0000", 2024-09-15T03:47:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.012179707 2024-09-15T03:47:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.860981+0000", 2024-09-15T03:47:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2386999999999995e-05 2024-09-15T03:47:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:353 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~671744] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:47:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.596312507999997, 2024-09-15T03:47:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.012912206, 2024-09-15T03:47:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 353 2024-09-15T03:47:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:47:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:47:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848279+0000", 2024-09-15T03:47:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.362e-06 2024-09-15T03:47:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849024+0000", 2024-09-15T03:47:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00074495500000000005 2024-09-15T03:47:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849039+0000", 2024-09-15T03:47:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5119e-05 2024-09-15T03:47:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849046+0000", 2024-09-15T03:47:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6359999999999997e-06 2024-09-15T03:47:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849115+0000", 2024-09-15T03:47:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8998999999999994e-05 2024-09-15T03:47:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849251+0000", 2024-09-15T03:47:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013589799999999999 2024-09-15T03:47:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850605+0000", 2024-09-15T03:47:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013544659999999999 2024-09-15T03:47:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850703+0000", 2024-09-15T03:47:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7576000000000002e-05 2024-09-15T03:47:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850719+0000", 2024-09-15T03:47:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5767e-05 2024-09-15T03:47:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851562+0000", 2024-09-15T03:47:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084361899999999999 2024-09-15T03:47:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851705+0000", 2024-09-15T03:47:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000142893 2024-09-15T03:47:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861112+0000", 2024-09-15T03:47:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.009407349 2024-09-15T03:47:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861190+0000", 2024-09-15T03:47:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7566999999999998e-05 2024-09-15T03:47:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:354 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~98304] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:47:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.595510072000003, 2024-09-15T03:47:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0177595669999999, 2024-09-15T03:47:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 354 2024-09-15T03:47:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:47:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:47:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849081+0000", 2024-09-15T03:47:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0610000000000001e-06 2024-09-15T03:47:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849157+0000", 2024-09-15T03:47:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5804999999999997e-05 2024-09-15T03:47:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849168+0000", 2024-09-15T03:47:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0730000000000001e-05 2024-09-15T03:47:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849174+0000", 2024-09-15T03:47:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3609999999999997e-06 2024-09-15T03:47:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850814+0000", 2024-09-15T03:47:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001640124 2024-09-15T03:47:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850894+0000", 2024-09-15T03:47:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0167999999999995e-05 2024-09-15T03:47:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851406+0000", 2024-09-15T03:47:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051203200000000005 2024-09-15T03:47:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851506+0000", 2024-09-15T03:47:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9457999999999995e-05 2024-09-15T03:47:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851574+0000", 2024-09-15T03:47:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8145000000000006e-05 2024-09-15T03:47:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851632+0000", 2024-09-15T03:47:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8108e-05 2024-09-15T03:47:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851712+0000", 2024-09-15T03:47:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9380000000000002e-05 2024-09-15T03:47:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.866753+0000", 2024-09-15T03:47:34.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0150417979999999 2024-09-15T03:47:34.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:34.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.866840+0000", 2024-09-15T03:47:34.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6397000000000006e-05 2024-09-15T03:47:34.529 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.529 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:34.529 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.529 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:355 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [setxattr _header (57) in=7b] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:34.529 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:47:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.595380759000001, 2024-09-15T03:47:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.018024966, 2024-09-15T03:47:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 355 2024-09-15T03:47:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:47:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:47:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849210+0000", 2024-09-15T03:47:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.33e-07 2024-09-15T03:47:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849215+0000", 2024-09-15T03:47:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4379999999999997e-06 2024-09-15T03:47:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849234+0000", 2024-09-15T03:47:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9409999999999999e-05 2024-09-15T03:47:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849241+0000", 2024-09-15T03:47:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6370000000000002e-06 2024-09-15T03:47:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851823+0000", 2024-09-15T03:47:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025817689999999998 2024-09-15T03:47:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851934+0000", 2024-09-15T03:47:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111357 2024-09-15T03:47:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852466+0000", 2024-09-15T03:47:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053248100000000001 2024-09-15T03:47:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852580+0000", 2024-09-15T03:47:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011379699999999999 2024-09-15T03:47:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852607+0000", 2024-09-15T03:47:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279e-05 2024-09-15T03:47:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852693+0000", 2024-09-15T03:47:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6792999999999994e-05 2024-09-15T03:47:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852869+0000", 2024-09-15T03:47:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017537499999999999 2024-09-15T03:47:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867006+0000", 2024-09-15T03:47:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.014136996 2024-09-15T03:47:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867234+0000", 2024-09-15T03:47:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022870099999999999 2024-09-15T03:47:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:356 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:47:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.595343995999997, 2024-09-15T03:47:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.020403521, 2024-09-15T03:47:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 356 2024-09-15T03:47:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:47:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:47:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849247+0000", 2024-09-15T03:47:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2699999999999999e-07 2024-09-15T03:47:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849250+0000", 2024-09-15T03:47:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2610000000000002e-06 2024-09-15T03:47:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849257+0000", 2024-09-15T03:47:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.703e-06 2024-09-15T03:47:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849261+0000", 2024-09-15T03:47:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7910000000000002e-06 2024-09-15T03:47:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852672+0000", 2024-09-15T03:47:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003411627 2024-09-15T03:47:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852801+0000", 2024-09-15T03:47:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000128918 2024-09-15T03:47:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867347+0000", 2024-09-15T03:47:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0145458380000001 2024-09-15T03:47:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867476+0000", 2024-09-15T03:47:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012915100000000001 2024-09-15T03:47:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.869650+0000", 2024-09-15T03:47:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021737050000000002 2024-09-15T03:47:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs3 65/64 ECSubWrite(tid=110, reqid=client.4521.0:474, at_version=65'72, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.850059+0000", 2024-09-15T03:47:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.594531074999999, 2024-09-15T03:47:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013621852, 2024-09-15T03:47:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.850059+0000", 2024-09-15T03:47:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.850059+0000", 2024-09-15T03:47:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.850060+0000", 2024-09-15T03:47:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8100000000000003e-07 2024-09-15T03:47:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.850200+0000", 2024-09-15T03:47:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014009 2024-09-15T03:47:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851295+0000", 2024-09-15T03:47:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010954579999999999 2024-09-15T03:47:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851302+0000", 2024-09-15T03:47:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5370000000000003e-06 2024-09-15T03:47:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851797+0000", 2024-09-15T03:47:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049567000000000003 2024-09-15T03:47:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851809+0000", 2024-09-15T03:47:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1270000000000001e-05 2024-09-15T03:47:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852166+0000", 2024-09-15T03:47:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035684699999999998 2024-09-15T03:47:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852186+0000", 2024-09-15T03:47:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0737000000000002e-05 2024-09-15T03:47:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852323+0000", 2024-09-15T03:47:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013634699999999999 2024-09-15T03:47:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852351+0000", 2024-09-15T03:47:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8515000000000001e-05 2024-09-15T03:47:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.863681+0000", 2024-09-15T03:47:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011329799999999999 2024-09-15T03:47:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/64 ECSubWrite(tid=108, reqid=client.4521.0:466, at_version=65'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.856519+0000", 2024-09-15T03:47:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.588071649, 2024-09-15T03:47:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015639281000000001, 2024-09-15T03:47:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856519+0000", 2024-09-15T03:47:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856519+0000", 2024-09-15T03:47:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856519+0000", 2024-09-15T03:47:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4200000000000005e-07 2024-09-15T03:47:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857060+0000", 2024-09-15T03:47:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054053200000000004 2024-09-15T03:47:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857069+0000", 2024-09-15T03:47:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2979999999999997e-06 2024-09-15T03:47:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857074+0000", 2024-09-15T03:47:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7570000000000004e-06 2024-09-15T03:47:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857122+0000", 2024-09-15T03:47:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7828999999999999e-05 2024-09-15T03:47:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857128+0000", 2024-09-15T03:47:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6409999999999996e-06 2024-09-15T03:47:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857491+0000", 2024-09-15T03:47:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036239300000000003 2024-09-15T03:47:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857503+0000", 2024-09-15T03:47:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2468e-05 2024-09-15T03:47:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857718+0000", 2024-09-15T03:47:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021492200000000001 2024-09-15T03:47:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857745+0000", 2024-09-15T03:47:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7189e-05 2024-09-15T03:47:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.872158+0000", 2024-09-15T03:47:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01441241 2024-09-15T03:47:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/64 ECSubWrite(tid=111, reqid=client.4521.0:467, at_version=65'29, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.858407+0000", 2024-09-15T03:47:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.586182893, 2024-09-15T03:47:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013931720999999999, 2024-09-15T03:47:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858407+0000", 2024-09-15T03:47:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858407+0000", 2024-09-15T03:47:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858408+0000", 2024-09-15T03:47:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5199999999999996e-07 2024-09-15T03:47:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858647+0000", 2024-09-15T03:47:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000238743 2024-09-15T03:47:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858659+0000", 2024-09-15T03:47:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2536e-05 2024-09-15T03:47:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858665+0000", 2024-09-15T03:47:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3340000000000001e-06 2024-09-15T03:47:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858724+0000", 2024-09-15T03:47:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8887e-05 2024-09-15T03:47:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858733+0000", 2024-09-15T03:47:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.781e-06 2024-09-15T03:47:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858903+0000", 2024-09-15T03:47:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016952199999999999 2024-09-15T03:47:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858918+0000", 2024-09-15T03:47:34.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.472e-05 2024-09-15T03:47:34.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:34.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.860148+0000", 2024-09-15T03:47:34.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012305980000000001 2024-09-15T03:47:34.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:34.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.860176+0000", 2024-09-15T03:47:34.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7393e-05 2024-09-15T03:47:34.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.872339+0000", 2024-09-15T03:47:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012163455 2024-09-15T03:47:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/64 ECSubWrite(tid=112, reqid=client.4521.0:468, at_version=65'30, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.858922+0000", 2024-09-15T03:47:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.585668396999999, 2024-09-15T03:47:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013555977, 2024-09-15T03:47:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858922+0000", 2024-09-15T03:47:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858922+0000", 2024-09-15T03:47:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858922+0000", 2024-09-15T03:47:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6600000000000002e-07 2024-09-15T03:47:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.859065+0000", 2024-09-15T03:47:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014284399999999999 2024-09-15T03:47:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.859091+0000", 2024-09-15T03:47:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6151000000000001e-05 2024-09-15T03:47:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.859098+0000", 2024-09-15T03:47:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6669999999999999e-06 2024-09-15T03:47:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.859148+0000", 2024-09-15T03:47:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9771000000000002e-05 2024-09-15T03:47:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.859157+0000", 2024-09-15T03:47:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4390000000000006e-06 2024-09-15T03:47:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.859295+0000", 2024-09-15T03:47:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013837300000000001 2024-09-15T03:47:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.860152+0000", 2024-09-15T03:47:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085642699999999999 2024-09-15T03:47:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.860226+0000", 2024-09-15T03:47:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4301000000000006e-05 2024-09-15T03:47:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.860245+0000", 2024-09-15T03:47:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8516e-05 2024-09-15T03:47:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.872478+0000", 2024-09-15T03:47:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012233122000000001 2024-09-15T03:47:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:527 3.7s0 3:e4b8e19d:::smithi00132172-41:head [copy-from ver 51] snapc 23=[23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:47:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.495775034000001, 2024-09-15T03:47:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030362388000000001, 2024-09-15T03:47:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 527 2024-09-15T03:47:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:47:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:47:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948817+0000", 2024-09-15T03:47:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3230000000000001e-06 2024-09-15T03:47:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948825+0000", 2024-09-15T03:47:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5900000000000008e-06 2024-09-15T03:47:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948838+0000", 2024-09-15T03:47:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2712e-05 2024-09-15T03:47:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948842+0000", 2024-09-15T03:47:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4170000000000002e-06 2024-09-15T03:47:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948923+0000", 2024-09-15T03:47:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0686000000000006e-05 2024-09-15T03:47:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:47:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949020+0000", 2024-09-15T03:47:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6984999999999994e-05 2024-09-15T03:47:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954390+0000", 2024-09-15T03:47:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053704790000000001 2024-09-15T03:47:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954558+0000", 2024-09-15T03:47:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000167202 2024-09-15T03:47:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.973998+0000", 2024-09-15T03:47:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019440316999999999 2024-09-15T03:47:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974527+0000", 2024-09-15T03:47:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052943800000000002 2024-09-15T03:47:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974541+0000", 2024-09-15T03:47:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3774999999999999e-05 2024-09-15T03:47:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974841+0000", 2024-09-15T03:47:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029942899999999998 2024-09-15T03:47:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.976067+0000", 2024-09-15T03:47:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012265399999999999 2024-09-15T03:47:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.978952+0000", 2024-09-15T03:47:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028848839999999999 2024-09-15T03:47:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.979178+0000", 2024-09-15T03:47:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022561099999999999 2024-09-15T03:47:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:528 3.7s0 3:e4b8e19d:::smithi00132172-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.495740625, 2024-09-15T03:47:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026927036000000001, 2024-09-15T03:47:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 528 2024-09-15T03:47:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9700000000000003e-07 2024-09-15T03:47:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948852+0000", 2024-09-15T03:47:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.604e-06 2024-09-15T03:47:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948855+0000", 2024-09-15T03:47:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9369999999999997e-06 2024-09-15T03:47:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948858+0000", 2024-09-15T03:47:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2709999999999999e-06 2024-09-15T03:47:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949080+0000", 2024-09-15T03:47:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000222148 2024-09-15T03:47:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:47:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949156+0000", 2024-09-15T03:47:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6215000000000001e-05 2024-09-15T03:47:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954754+0000", 2024-09-15T03:47:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055978060000000003 2024-09-15T03:47:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954858+0000", 2024-09-15T03:47:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010419199999999999 2024-09-15T03:47:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974861+0000", 2024-09-15T03:47:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020002671999999999 2024-09-15T03:47:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974987+0000", 2024-09-15T03:47:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012638699999999999 2024-09-15T03:47:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.975777+0000", 2024-09-15T03:47:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00078950699999999995 2024-09-15T03:47:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:34.578 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.578 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:34.578 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:34.578 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:34.579 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:34.579 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:34.580 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:34.580 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:47:34.581 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:34.581 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-15T03:47:34.581 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:34.582 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T03:47:34.582 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:34.582 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T03:47:34.582 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:34.578+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:34.680 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:34.679+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:34.683 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:34.683 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:34.683 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:34.683 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:34.691 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:47:34.781 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:34.780+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:34.791 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:34.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:34.792 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:34.792 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:34.792 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:34.799 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:47:34.846 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.846 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.846 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.846 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.846 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.846 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.846 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.846 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.846 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.847 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.847 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.847 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.847 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.847 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.847 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.847 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.847 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.847 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.847 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.847 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.847 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.847 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.848 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.848 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.848 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.848 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.848 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.848 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.848 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.848 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.848 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.848 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.849 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.849 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.849 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.849 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.849 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.849 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.849 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.849 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.849 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.849 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.849 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.849 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.849 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.850 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.850 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.850 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.850 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.850 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.850 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.850 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.850 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.850 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.850 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.850 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.850 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.850 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.851 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.851 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.851 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.851 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.851 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.851 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.851 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.851 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.851 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.851 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.851 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.851 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.851 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.852 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.852 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.852 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.852 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.852 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.852 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.852 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.852 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.852 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.852 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.852 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.852 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.852 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.853 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.853 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.839+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.853 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.853 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.853 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.853 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.853 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.853 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.853 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.853 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.853 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.853 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.853 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.853 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.854 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.854 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.854 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.854 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.854 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.854 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.854 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.854 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.854 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.854 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.854 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.854 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.854 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.855 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.855 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.855 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.855 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.855 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.855 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.855 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.855 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.855 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.855 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.855 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.855 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.855 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.856 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.856 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.856 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.856 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.856 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.856 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.856 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.856 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.856 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.856 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.856 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.856 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.856 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.856 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.857 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.857 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.857 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.857 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.857 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.857 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.857 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.857 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.857 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.857 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.857 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.857 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.857 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.858 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.858 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.858 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.858 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.858 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.858 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.858 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.858 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.858 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.858 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.858 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.858 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.858 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.858 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.859 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.859 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.859 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.859 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.859 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.859 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.859 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.859 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.859 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.859 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.859 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.859 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.859 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.859 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.860 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.860 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.861 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.861 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.840+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.861 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.861 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.861 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.861 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.861 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.861 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.861 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.861 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.861 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.861 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.862 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.862 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.862 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.862 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.862 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.862 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.862 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.862 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.862 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.862 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.862 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.863 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.863 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.863 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.863 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.863 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.863 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.863 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.863 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.863 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.863 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.863 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.863 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.863 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.864 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.864 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.864 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.864 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.864 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.864 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.864 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.864 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.864 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.864 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.864 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.864 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.864 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.865 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.865 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.865 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.865 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.865 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.865 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.865 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.865 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.865 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.865 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.865 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.865 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.865 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.866 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.866 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.866 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.866 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.866 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.866 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.866 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.866 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.866 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.866 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.866 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.866 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.866 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.867 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.867 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.867 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.867 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.867 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.867 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.867 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.867 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.867 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.867 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.867 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.867 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.867 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.867 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.868 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.868 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.868 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.868 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.868 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.868 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.868 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.868 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.868 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.868 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.868 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.868 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.868 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.869 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.869 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.869 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.869 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.869 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.869 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.869 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.869 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.869 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.869 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.869 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.869 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.869 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.869 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.870 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.870 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.870 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.870 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.870 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.870 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.870 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.870 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.870 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.870 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.870 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.870 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.870 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.870 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.871 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.871 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.871 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.871 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.871 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.871 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.871 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.871 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.871 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.871 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.871 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.871 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.871 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.871 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.872 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.872 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.872 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.872 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.872 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.872 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.872 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.872 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.873 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.873 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.873 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.873 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.873 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.873 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.873 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.873 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.873 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.873 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.873 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.873 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.873 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.873 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.874 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.874 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.874 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.874 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.874 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.874 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.841+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.875 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.875 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.875 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.875 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.875 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.875 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.875 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.876 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.876 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.876 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.876 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.876 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.876 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.876 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.876 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.876 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.876 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.876 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.876 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.876 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.877 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.877 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.877 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.877 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.877 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.877 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.877 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.877 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.877 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.877 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.877 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.877 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.877 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.877 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.878 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.878 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.878 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.878 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.878 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.878 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.878 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.878 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.878 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.878 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.878 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.878 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.878 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.878 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.879 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.879 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.879 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.879 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.879 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.842+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.879 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.879 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.879 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.879 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.879 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.879 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.879 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.879 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.879 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.880 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.880 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.880 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.880 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.880 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.880 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.880 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.880 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.880 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.880 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.880 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.880 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.880 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.881 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.881 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.881 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.881 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.881 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.881 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.881 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.881 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.881 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.881 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.882 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.882 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.882 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.882 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.882 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.882 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.882 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.882 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.882 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.882 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.882 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.882 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.883 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.883 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.883 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.883 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.883 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.883 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.883 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.883 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.883 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.884 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.884 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.884 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.884 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.884 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.884 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.884 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.884 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.884 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.884 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.884 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.884 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.884 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.885 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.885 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.885 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.885 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.885 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.885 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.885 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.885 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.885 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.885 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.885 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.885 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.887 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.887 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.887 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.887 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.887 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.887 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.887 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.887 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.887 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.887 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.887 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.887 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.888 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.888 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.888 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.888 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.888 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.888 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.888 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.888 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.888 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.888 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.888 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.888 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.888 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.888 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.889 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.889 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.889 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.889 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.889 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.889 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.889 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.889 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.889 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.889 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.889 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.889 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.889 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.890 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.890 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.890 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.890 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.890 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.890 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.890 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.890 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.890 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.892 INFO:tasks.rados.rados.0.smithi001.stdout:293: finishing rollback tid 0 to smithi00132172-29 2024-09-15T03:47:34.892 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:34.891+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:34.892 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.892 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.892 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.893 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.893 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.893 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.893 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.893 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.893 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.893 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.893 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.893 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.893 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.893 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.893 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.893 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.894 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.894 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.894 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.894 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.894 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.894 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.894 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.897 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.897 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.897 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.898 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.898 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.898 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.898 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.898 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.898 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.898 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.898 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.898 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.899 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.899 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.899 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.899 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.899 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.899 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.899 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.899 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.899 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.899 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.899 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.899 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.899 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.899 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.900 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.900 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.900 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.900 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.900 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.900 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.900 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.900 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.900 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.900 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.900 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.900 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.900 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.901 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.901 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.843+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.901 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.901 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.901 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.901 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.901 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.901 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.901 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.901 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.901 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.901 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.901 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.901 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.902 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.902 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.902 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.902 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.902 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.902 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.902 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.903 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.903 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.903 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.903 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.903 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.903 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.904 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.904 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.904 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.904 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.904 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.904 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.904 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.904 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.904 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.905 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.905 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.905 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.905 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.909 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.909 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.909 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.909 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.909 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.910 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.910 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.910 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.910 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.910 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.910 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.910 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.910 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.911 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.911 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.911 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.911 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.911 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.911 INFO:tasks.rados.rados.0.smithi001.stdout:293: finishing rollback tid 1 to smithi00132172-29 2024-09-15T03:47:34.911 INFO:tasks.rados.rados.0.smithi001.stdout:293: finishing rollback tid 2 to smithi00132172-29 2024-09-15T03:47:34.911 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 29 v 111 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-15T03:47:34.912 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.912 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.912 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.912 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.912 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.912 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.912 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.912 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.912 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.912 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.912 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.915 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.915 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.915 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.915 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.916 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.916 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.916 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.916 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.916 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.916 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.916 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.916 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.916 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.916 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.916 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.916 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.916 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.917 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.917 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.917 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.917 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.917 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.917 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.917 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.917 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.917 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.917 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.917 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.917 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.917 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.918 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.918 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.918 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.918 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.918 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.918 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.918 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.918 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.918 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.918 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.918 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.918 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.918 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.918 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.919 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.919 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.919 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.919 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.919 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.919 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.919 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.919 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.919 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.919 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.919 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.919 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.919 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.919 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.920 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.920 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.920 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.920 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.920 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.920 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.920 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.920 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.920 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.920 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.920 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.920 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.920 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.921 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.844+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.921 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.921 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.921 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.921 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.921 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.921 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.922 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.922 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.922 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.922 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.922 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.922 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.922 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.922 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.922 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.922 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.922 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.922 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.922 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.923 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.923 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.923 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.923 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.923 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.923 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.923 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.923 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.924 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.924 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.924 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.924 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.924 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.924 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.924 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.925 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.926 INFO:tasks.rados.rados.0.smithi001.stdout:287: done (4 left) 2024-09-15T03:47:34.926 INFO:tasks.rados.rados.0.smithi001.stdout:291: done (3 left) 2024-09-15T03:47:34.926 INFO:tasks.rados.rados.0.smithi001.stdout:292: done (2 left) 2024-09-15T03:47:34.926 INFO:tasks.rados.rados.0.smithi001.stdout:293: done (1 left) 2024-09-15T03:47:34.926 INFO:tasks.rados.rados.0.smithi001.stdout:294: done (0 left) 2024-09-15T03:47:34.926 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 43 current snap is 33 2024-09-15T03:47:34.926 INFO:tasks.rados.rados.0.smithi001.stdout:295: seq_num 118 ranges {720896=442368,1163264=753664,1916928=655360,2572288=147456} 2024-09-15T03:47:34.926 INFO:tasks.rados.rados.0.smithi001.stdout:295: writing smithi00132172-43 from 720896 to 1163264 tid 1 2024-09-15T03:47:34.928 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.928 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.928 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.929 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.929 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.929 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.929 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.929 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.929 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.929 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.929 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.929 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.929 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.929 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.929 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.929 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.929 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.930 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.930 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.930 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.930 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.930 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.930 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.930 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.930 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.930 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.930 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.930 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.930 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.931 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.931 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.931 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.931 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.931 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.931 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.931 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.931 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.931 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.931 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.931 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.931 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.931 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.931 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.932 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.932 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.932 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.932 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.932 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.932 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.932 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.932 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.932 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.932 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.932 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.932 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.932 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.933 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.933 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.933 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.933 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.933 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.933 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.933 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.933 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.933 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.933 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.933 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.933 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.933 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.934 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.934 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.934 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.934 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.934 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.934 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.934 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.934 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.934 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.934 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.935 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.935 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.935 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.935 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.935 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.935 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.935 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.936 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.936 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.936 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.936 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.936 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.936 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.936 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.937 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.937 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.937 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.937 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.937 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.937 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.937 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.937 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.938 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.938 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.939 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.939 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.939 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.939 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.939 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.939 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.939 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.939 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.939 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.939 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.940 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.940 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.940 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.940 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.940 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.849+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.942 INFO:tasks.rados.rados.0.smithi001.stdout:295: writing smithi00132172-43 from 1163264 to 1916928 tid 2 2024-09-15T03:47:34.943 INFO:tasks.rados.rados.0.smithi001.stdout:295: writing smithi00132172-43 from 1916928 to 2572288 tid 3 2024-09-15T03:47:34.943 INFO:tasks.rados.rados.0.smithi001.stdout:295: writing smithi00132172-43 from 2572288 to 2719744 tid 4 2024-09-15T03:47:34.943 INFO:tasks.rados.rados.0.smithi001.stdout:296: read oid 44 snap -1 2024-09-15T03:47:34.943 INFO:tasks.rados.rados.0.smithi001.stdout:296: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-15T03:47:34.943 INFO:tasks.rados.rados.0.smithi001.stdout:297: setattr oid 3 current snap is 33 2024-09-15T03:47:34.943 INFO:tasks.rados.rados.0.smithi001.stdout:295: finishing write tid 1 to smithi00132172-43 2024-09-15T03:47:34.943 INFO:tasks.rados.rados.0.smithi001.stdout:298: rollback oid 2 current snap is 33 2024-09-15T03:47:34.943 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 2 to 10 2024-09-15T03:47:34.943 INFO:tasks.rados.rados.0.smithi001.stdout:299: read oid 34 snap 24 2024-09-15T03:47:34.943 INFO:tasks.rados.rados.0.smithi001.stdout:299: expect (ObjNum 70 snap 8 seq_num 70) 2024-09-15T03:47:34.943 INFO:tasks.rados.rados.0.smithi001.stdout:300: read oid 37 snap -1 2024-09-15T03:47:34.943 INFO:tasks.rados.rados.0.smithi001.stdout:300: expect (ObjNum 115 snap 32 seq_num 115) 2024-09-15T03:47:34.943 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 16 current snap is 33 2024-09-15T03:47:34.944 INFO:tasks.rados.rados.0.smithi001.stdout:301: seq_num 119 ranges {1720320=458752,2179072=507904,2686976=507904,3194880=458752,3653632=81920} 2024-09-15T03:47:34.944 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.944 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.944 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.945 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.945 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.845+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.945 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.945 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.945 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.945 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.945 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.945 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.945 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.945 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.945 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.945 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.945 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.945 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.946 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.946 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.946 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.946 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.946 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.946 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.946 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.946 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.946 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.947 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.947 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.947 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.947 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.947 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.947 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.947 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.948 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.948 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.948 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.948 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.948 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.948 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.948 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.949 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.949 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.949 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.949 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.949 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.949 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.949 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.949 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.949 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.949 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.949 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.950 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.950 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.950 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.950 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.950 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.950 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.950 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.950 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.951 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.951 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.951 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.951 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.951 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.951 INFO:tasks.rados.rados.0.smithi001.stdout:301: writing smithi00132172-16 from 1720320 to 2179072 tid 1 2024-09-15T03:47:34.952 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.952 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.952 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.952 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.952 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.953 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.953 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.953 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.953 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.953 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.953 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.953 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.953 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.953 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.953 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.953 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.953 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.954 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.954 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.954 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.954 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.954 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.954 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.954 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.955 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.955 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.955 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.955 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.955 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.955 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.956 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.956 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.956 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.956 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.956 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.956 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.956 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.957 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.957 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.957 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.957 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.957 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.957 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.957 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.957 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.958 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.958 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.958 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.958 INFO:tasks.rados.rados.0.smithi001.stdout:301: writing smithi00132172-16 from 2179072 to 2686976 tid 2 2024-09-15T03:47:34.958 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.958 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.958 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.959 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.959 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.959 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.959 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.959 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.960 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.960 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.966 INFO:tasks.rados.rados.0.smithi001.stdout:301: writing smithi00132172-16 from 2686976 to 3194880 tid 3 2024-09-15T03:47:34.966 INFO:tasks.rados.rados.0.smithi001.stdout:301: writing smithi00132172-16 from 3194880 to 3653632 tid 4 2024-09-15T03:47:34.966 INFO:tasks.rados.rados.0.smithi001.stdout:301: writing smithi00132172-16 from 3653632 to 3735552 tid 5 2024-09-15T03:47:34.966 INFO:tasks.rados.rados.0.smithi001.stdout:302: snap_create 2024-09-15T03:47:34.966 INFO:tasks.rados.rados.0.smithi001.stdout:298: finishing rollback tid 0 to smithi00132172-2 2024-09-15T03:47:34.966 INFO:tasks.rados.rados.0.smithi001.stdout:298: finishing rollback tid 1 to smithi00132172-2 2024-09-15T03:47:34.966 INFO:tasks.rados.rados.0.smithi001.stdout:298: finishing rollback tid 2 to smithi00132172-2 2024-09-15T03:47:34.966 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 2 v 104 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-15T03:47:34.966 INFO:tasks.rados.rados.0.smithi001.stdout:295: finishing write tid 2 to smithi00132172-43 2024-09-15T03:47:34.966 INFO:tasks.rados.rados.0.smithi001.stdout:295: finishing write tid 3 to smithi00132172-43 2024-09-15T03:47:34.966 INFO:tasks.rados.rados.0.smithi001.stdout:295: finishing write tid 4 to smithi00132172-43 2024-09-15T03:47:34.966 INFO:tasks.rados.rados.0.smithi001.stdout:295: finishing write tid 5 to smithi00132172-43 2024-09-15T03:47:34.966 INFO:tasks.rados.rados.0.smithi001.stdout:295: finishing write tid 6 to smithi00132172-43 2024-09-15T03:47:34.967 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 43 v 73 (ObjNum 118 snap 33 seq_num 118) dirty exists 2024-09-15T03:47:34.967 INFO:tasks.rados.rados.0.smithi001.stdout:295: left oid 43 (ObjNum 118 snap 33 seq_num 118) 2024-09-15T03:47:34.967 INFO:tasks.rados.rados.0.smithi001.stdout:299: expect (ObjNum 70 snap 8 seq_num 70) 2024-09-15T03:47:34.972 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.972 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.972 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.972 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.972 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.972 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.972 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.972 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.972 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.973 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.973 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.973 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.973 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.973 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.973 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.973 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.973 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.973 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.973 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.973 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.973 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.973 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.974 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.974 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.974 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.974 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.974 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.974 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.974 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.974 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.974 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.974 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.974 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.974 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.846+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.974 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.847+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.974 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.847+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.975 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.847+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.975 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.847+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.975 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.847+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.975 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.847+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.975 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.847+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.975 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.975 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.975 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.975 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.975 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.975 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.975 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.975 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.975 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.976 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.976 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.976 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.976 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.976 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.976 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.976 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.976 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.976 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.976 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.976 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.976 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.976 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.976 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.977 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.977 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.977 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.977 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.977 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.848+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.977 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.977 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.977 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.977 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.977 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.977 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.977 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.977 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.978 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.978 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.978 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.978 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.978 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.978 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.978 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.978 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.978 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.978 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.978 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.978 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.978 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.978 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.979 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.979 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.979 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.979 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.979 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.979 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.979 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.979 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.979 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.979 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.979 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.979 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.979 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.979 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.979 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.980 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.980 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.980 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.980 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.980 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.980 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.980 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.980 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.980 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.980 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.980 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.980 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.980 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.981 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.981 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.981 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.981 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.981 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.981 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.981 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.981 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.981 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.981 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.981 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.981 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.981 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.981 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.982 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.982 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.982 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.982 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.982 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.982 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.982 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.982 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.982 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.982 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.982 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.982 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.982 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.983 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.983 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.983 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.983 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.983 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.983 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.983 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.983 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.983 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.983 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.983 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.983 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.983 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.850+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.983 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.984 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.984 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.984 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.984 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.984 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.984 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.984 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.984 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.984 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.984 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.984 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.984 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.984 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.985 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.985 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.985 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.985 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.985 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.985 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.985 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.985 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.985 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.985 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.985 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.985 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.985 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.985 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.986 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.986 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.986 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.986 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.986 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.986 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.986 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.986 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.986 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.986 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.986 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.986 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.986 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.987 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.987 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.987 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.987 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.987 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.987 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.987 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.987 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.987 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.987 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.987 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.987 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.987 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.987 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.987 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.988 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.988 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.988 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.988 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.988 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.988 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.988 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.988 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.988 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.988 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.988 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.988 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.988 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.989 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.989 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.989 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.989 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.989 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.989 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.989 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.989 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.989 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.989 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.989 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.989 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.989 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.989 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.990 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.990 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.990 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.990 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.990 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.990 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.990 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.990 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.990 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.990 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.990 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.991 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.991 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.991 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.992 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.992 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.992 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.992 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.992 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.992 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.992 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.992 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.992 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.992 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.992 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.992 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.992 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.993 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.993 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:34.993 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.003 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.003 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.003 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.003 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.004 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.004 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.004 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.004 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.004 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.004 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.004 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.851+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.004 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.004 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.004 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.004 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.004 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.004 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.005 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.005 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.005 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.005 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.005 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.005 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.005 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.005 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.005 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.005 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.005 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.005 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.005 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.006 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.006 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.006 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.006 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.006 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.006 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.006 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.006 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.006 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.006 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.006 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.006 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.006 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.006 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.007 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.007 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.007 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.007 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.007 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.007 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.007 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.007 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.007 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.007 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.007 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.007 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.007 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.007 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.008 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.008 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.008 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.008 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.008 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.008 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.008 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.008 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.008 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.008 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.008 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.008 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.008 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.008 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.009 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.009 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.009 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.009 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.009 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.009 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.009 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.009 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.009 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.009 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.009 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.009 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.009 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.010 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.010 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.010 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.010 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.010 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.010 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.010 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.010 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.010 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.010 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.010 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.010 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.010 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.010 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.011 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.011 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.011 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.011 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.011 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.011 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.011 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.011 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.011 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.011 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.011 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.011 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.011 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.012 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.012 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.012 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.012 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.012 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.012 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.012 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.852+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.012 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.012 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.012 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.012 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.012 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.012 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.013 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.013 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.013 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.013 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.013 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.013 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.013 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.013 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.013 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.013 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.013 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.013 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.014 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.014 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.014 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.014 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.014 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.014 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.014 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.014 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.014 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.014 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.014 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.014 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.014 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.015 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.015 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.015 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.015 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.015 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.015 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.015 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.015 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.015 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.015 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.015 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.015 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.015 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.015 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.016 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.016 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.016 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.016 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.016 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.016 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.016 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.016 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.016 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.016 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.016 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.016 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.016 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.017 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.017 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.017 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.017 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.017 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.017 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.017 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.017 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.017 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.017 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.017 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.017 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.017 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.018 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.018 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.018 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.018 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.018 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.018 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.018 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.018 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.018 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.018 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.018 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.018 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.018 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.019 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.019 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.019 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.019 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.019 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.019 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.019 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.019 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.019 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.019 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.019 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.019 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.019 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.019 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.020 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.020 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.020 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.020 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.020 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.020 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.020 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.020 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.020 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.020 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.020 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.020 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.020 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.021 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.021 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.021 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.021 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.021 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.021 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.021 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.021 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.021 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.021 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.853+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.021 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.021 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.021 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.021 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.022 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.022 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.022 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.022 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.022 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.022 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.022 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.022 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.022 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.022 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.022 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.022 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.022 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.022 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.023 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.023 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.023 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.023 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.023 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.023 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.023 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.023 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.023 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.023 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.023 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.023 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.023 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.023 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.024 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.024 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.024 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.024 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.024 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.024 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.024 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.024 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.024 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.024 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.024 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.024 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.024 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.025 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.025 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.025 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.025 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.025 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.025 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.025 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.025 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.025 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.025 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.025 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.025 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.025 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.025 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.026 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.026 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.026 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.026 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.026 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.026 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.026 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.026 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.026 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.026 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.026 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.026 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.026 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.026 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.027 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.027 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.027 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.027 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.027 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.027 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.027 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.027 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.027 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.027 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.027 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.028 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.028 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.028 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.028 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.028 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.028 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.028 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.028 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.029 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.029 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T03:47:35.031 INFO:tasks.rados.rados.0.smithi001.stdout:301: finishing write tid 1 to smithi00132172-16 2024-09-15T03:47:35.031 INFO:tasks.rados.rados.0.smithi001.stdout:296: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-15T03:47:35.031 INFO:tasks.rados.rados.0.smithi001.stdout:301: finishing write tid 2 to smithi00132172-16 2024-09-15T03:47:35.031 INFO:tasks.rados.rados.0.smithi001.stdout:301: finishing write tid 3 to smithi00132172-16 2024-09-15T03:47:35.031 INFO:tasks.rados.rados.0.smithi001.stdout:300: expect (ObjNum 115 snap 32 seq_num 115) 2024-09-15T03:47:35.033 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.033 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.033 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.033 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.033 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.033 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.033 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.033 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.034 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.034 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.034 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.034 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.034 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.034 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.034 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.034 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.034 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.854+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.034 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.034 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.034 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.034 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.035 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.035 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.035 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.035 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.035 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.035 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.035 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.035 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.035 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.035 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.035 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.035 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.035 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.035 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.036 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.036 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.036 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.036 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.036 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.036 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.036 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.036 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.036 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.036 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.036 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.036 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.036 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.036 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.037 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.037 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.037 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.037 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.037 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.037 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.037 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.037 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.037 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.037 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.037 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.037 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.037 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.037 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.038 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.038 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.038 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.038 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.038 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.038 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.038 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.038 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.038 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.038 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.038 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.038 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.038 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.038 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.039 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.039 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.039 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.039 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.039 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.039 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.085 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.085 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.085 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.085 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.085 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.085 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.086 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.086 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.086 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.086 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.086 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.086 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.086 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.086 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.086 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.086 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.086 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.086 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.086 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.087 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.087 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.087 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.087 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.087 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.087 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.087 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.087 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.087 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.087 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.087 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.087 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.087 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.088 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.088 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.088 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.088 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.088 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.088 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.088 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.855+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.088 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.088 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.088 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.088 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.088 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.088 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.089 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.089 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.089 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.089 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.089 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.089 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.089 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.089 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.089 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.089 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.089 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.089 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.089 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.089 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.090 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.090 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.090 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.090 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.090 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.090 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.090 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.090 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.090 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.090 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.090 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.090 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.090 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.091 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.091 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.091 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.091 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.091 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.091 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.091 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.091 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.091 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.091 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.091 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.091 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.091 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.091 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.092 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.092 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.092 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.092 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.092 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.092 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.092 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.092 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.092 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.092 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.092 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.092 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.092 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.093 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.093 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.093 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.093 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.093 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.093 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.093 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.093 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.093 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.093 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.093 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.093 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.093 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.094 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.094 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.094 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.094 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.094 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.094 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.094 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.094 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.094 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.094 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.094 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.094 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.094 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.095 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.095 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.095 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.095 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.095 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.095 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.095 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.095 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.095 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.095 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.095 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.095 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.096 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.096 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.096 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.096 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.096 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.096 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.096 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.096 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.856+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.096 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.096 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.096 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.096 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.096 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.097 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.097 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.097 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.097 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.097 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.097 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.097 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.097 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.097 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.097 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.097 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.097 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.097 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.097 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.098 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.098 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.098 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.098 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.098 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.098 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.098 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.098 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.098 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.098 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.098 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.099 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.099 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.099 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.099 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.099 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.099 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.099 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.099 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.099 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.099 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.099 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.100 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.100 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.100 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.100 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.100 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.100 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.100 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.100 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.100 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.100 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.100 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.100 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.101 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.101 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.101 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.101 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.101 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.101 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.101 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.101 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.101 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.101 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.101 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.101 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.101 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.101 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.102 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.102 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.102 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.102 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.102 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.102 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.102 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.102 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.102 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.102 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.102 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.102 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.102 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.103 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.103 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.103 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.103 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.103 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.103 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.103 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.103 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.103 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.103 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.104 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.104 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.104 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.104 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.104 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.104 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.104 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.104 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.104 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.104 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.104 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.104 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.104 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.105 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.105 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.105 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.105 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.105 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.105 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.105 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.105 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.105 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.857+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.105 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.105 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.105 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.105 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.106 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.106 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.106 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.106 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.106 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.106 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.106 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.106 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.106 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.106 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.106 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.106 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.106 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.107 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.107 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.107 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.107 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.107 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.107 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.107 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.107 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.107 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.107 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.107 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.107 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.108 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.108 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.108 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.108 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.108 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.108 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.108 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.108 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.108 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.108 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.108 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.108 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.108 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.109 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.109 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.109 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.109 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.109 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.109 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.109 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.109 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.109 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.109 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.109 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.109 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.110 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.110 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.110 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.110 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.110 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.110 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.110 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.110 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.110 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.110 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.110 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.110 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.110 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.111 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.111 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.111 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.111 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.111 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.111 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.111 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.111 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.111 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.111 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.111 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.111 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.111 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.112 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.112 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.112 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.112 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.112 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.112 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.112 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.112 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.112 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.112 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.112 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.112 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.112 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.113 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.113 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.113 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.113 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.113 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.113 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.113 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.113 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.113 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.113 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.113 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.113 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.113 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.113 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.113 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.114 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.114 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.114 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.114 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.114 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.114 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.116 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.116 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.116 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.116 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.116 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.858+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.116 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.116 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.116 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.116 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.116 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.116 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.117 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.117 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.117 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.117 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.117 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.117 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.117 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.117 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.117 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.117 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.117 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.117 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.117 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.118 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.118 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.118 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.118 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.118 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.118 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.118 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.118 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.118 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.118 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.118 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.118 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.118 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.118 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.119 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.119 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.119 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.119 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.119 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.119 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.119 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.119 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.119 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.119 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.119 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.119 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.119 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.119 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.120 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.120 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.120 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.120 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.120 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.120 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.120 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.120 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.121 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.121 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.121 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.121 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.121 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.121 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.121 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.121 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.121 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.121 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.121 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.121 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.122 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.122 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.122 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.122 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.122 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.122 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.122 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.122 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.122 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.122 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.122 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.122 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.122 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.123 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.123 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.123 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.123 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.123 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.123 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.123 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.123 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.123 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.123 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.123 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.123 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.124 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.124 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.124 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.124 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.124 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.124 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.124 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.124 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.124 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.124 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.124 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.124 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.125 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.859+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.125 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.125 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.125 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.125 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.125 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.125 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.125 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.125 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.125 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.125 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.125 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.126 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.126 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.126 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.126 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.126 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.126 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.126 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.126 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.126 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.126 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.126 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.126 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.127 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.127 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.127 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.127 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.127 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.127 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.127 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.127 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.127 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.127 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.127 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.127 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.127 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.128 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.128 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.128 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.128 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.128 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.128 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.128 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.128 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.128 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.128 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.128 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.128 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.129 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.129 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.129 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.129 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.129 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.129 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.129 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.129 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.129 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.129 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.129 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.129 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.129 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.129 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.130 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.130 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.130 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.130 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.130 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.130 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.130 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.130 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.130 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.130 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.130 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.130 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.130 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.131 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.131 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.131 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.131 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.131 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.131 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.131 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.131 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.131 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.131 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.131 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.131 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.132 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.132 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.132 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.132 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.132 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.132 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.132 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.132 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.132 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.132 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.132 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.132 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.133 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.133 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.133 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.133 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.133 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.133 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.133 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.133 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.133 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.133 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.133 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.133 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.133 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.134 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.134 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.134 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.134 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.134 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.134 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.134 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.134 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.134 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.134 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.134 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.134 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.135 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.135 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.860+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.135 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.135 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.135 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.135 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.135 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.135 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.135 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.135 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.135 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.135 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.135 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.136 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.136 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.136 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.136 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.136 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.136 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.136 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.136 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.136 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.136 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.136 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.136 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.136 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.137 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.137 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.137 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.137 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.137 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.137 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.137 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.137 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.137 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.137 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.137 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.137 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.138 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.138 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.138 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.138 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.138 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.138 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.138 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.138 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.138 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.138 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.138 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.138 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.138 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.139 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.139 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.139 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.139 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.139 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.139 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.139 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.139 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.139 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.139 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.139 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.139 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.140 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.140 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.140 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.140 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.140 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.140 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.140 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.140 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.140 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.140 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.140 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.140 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.141 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.141 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.141 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.141 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.141 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.141 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.141 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.141 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.141 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.141 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.141 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.141 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.142 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.142 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.142 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.142 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.142 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.142 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.142 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.142 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.142 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.142 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.142 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.142 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.142 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.143 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.143 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.143 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.143 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.143 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.143 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.143 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.143 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.143 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.143 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.143 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.143 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.143 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.143 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.144 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.144 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.144 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.144 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.144 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.144 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.144 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.861+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.144 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.144 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.144 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.144 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.144 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.144 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.145 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.145 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.145 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.145 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.145 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.145 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.145 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.145 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.145 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.145 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.145 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.145 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.145 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.146 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.146 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.146 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.146 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.146 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.146 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.146 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.146 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.146 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.146 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.146 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.146 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.146 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.147 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.147 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.147 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.147 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.147 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.147 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.147 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.147 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.147 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.147 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.147 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.147 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.147 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.148 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.148 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.148 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.148 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.148 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.148 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.148 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.148 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.148 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.148 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.148 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.148 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.148 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.149 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.149 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.149 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.149 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.149 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.149 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.149 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.149 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.149 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.149 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.149 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.149 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.149 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.150 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.150 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.150 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.150 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.150 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.150 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.150 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.150 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.150 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.150 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.150 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.150 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.150 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.150 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.151 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.151 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.151 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.151 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.151 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.151 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.151 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.151 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.151 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.151 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.151 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.151 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.151 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.152 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.152 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.152 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.152 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.152 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.152 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.152 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.862+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.152 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.152 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.152 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.152 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.152 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.152 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.153 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.153 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.153 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.153 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.153 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.153 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.153 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.153 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.153 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.153 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.153 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.153 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.153 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.153 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.154 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.154 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.154 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.154 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.154 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.154 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.154 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.154 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.154 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.154 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.154 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.154 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.154 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.155 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.155 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.155 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.155 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.155 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.155 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.155 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.155 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.155 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.155 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.155 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.155 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.155 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.156 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.156 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.156 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.156 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.156 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.156 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.156 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.156 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.156 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.156 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.156 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.156 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.156 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.157 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.157 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.157 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.157 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.157 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.157 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.157 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.157 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.157 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.157 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.157 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.157 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.157 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.158 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.158 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.158 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.158 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.863+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.158 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.158 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.158 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.158 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.158 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.158 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.158 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.158 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.158 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.158 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.159 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.159 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.159 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.159 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.159 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.159 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.159 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.159 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.159 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.159 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.159 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.159 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.159 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.160 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.160 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.160 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.160 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.160 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.160 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.160 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.160 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.160 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.160 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.160 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.160 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.160 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.161 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.161 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.161 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.161 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.161 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.161 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.161 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.161 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.161 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.161 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.161 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.161 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.162 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.162 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.162 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.162 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.162 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.162 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.162 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.864+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.162 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.162 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.162 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.162 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.162 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.162 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.162 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.163 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.163 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.163 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.163 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.163 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.163 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.163 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.163 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.163 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.163 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.163 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.163 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.163 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.163 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.164 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.164 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.164 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.164 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.164 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.164 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.164 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.164 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.164 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.164 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.164 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.164 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.165 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.165 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.165 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.165 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.165 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.165 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.165 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.165 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.165 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.165 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.165 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.165 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.165 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.166 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.166 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.166 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.166 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.166 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.166 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.166 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.166 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.166 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.166 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.166 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.166 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.166 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.166 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.167 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.167 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.167 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.167 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.167 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.167 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.167 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.167 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.167 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.865+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.167 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.167 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.167 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.167 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.168 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.168 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.168 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.168 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.168 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.168 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.168 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.168 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.168 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.168 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.168 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.168 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.168 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.169 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.169 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.169 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.169 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.169 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.169 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.169 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.169 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.169 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.169 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.169 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.169 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.169 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.170 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.170 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.170 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.170 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.170 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.170 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.170 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.170 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.170 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.170 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.170 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.170 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.170 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.171 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.171 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.171 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.171 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.171 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.171 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.171 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.171 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.171 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.171 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.171 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.171 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.171 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.171 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.172 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.172 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.172 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.172 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.172 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.172 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.172 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.172 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.172 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.866+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.172 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.172 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.172 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.172 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.173 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.173 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.173 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.173 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.173 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.173 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.173 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.173 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.173 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.173 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.173 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.173 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.173 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.174 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.174 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.174 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.174 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.174 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.174 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.174 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.174 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.174 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.174 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.174 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.174 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.174 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.175 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.175 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.175 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.175 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.175 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.175 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.175 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.175 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.175 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.175 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.175 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.175 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.175 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.176 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.176 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.176 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.176 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.176 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.176 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.176 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.176 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.176 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.176 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.176 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.176 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.176 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.177 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.177 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.177 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.177 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.177 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.177 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.177 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.177 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.177 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.177 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.177 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.177 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.177 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.178 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.178 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.178 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.178 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.178 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.178 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.178 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.178 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.178 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.178 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.178 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.178 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.178 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.179 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.179 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.179 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.179 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.179 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.179 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.179 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.179 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.179 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.179 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.179 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.179 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.179 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.179 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.180 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.180 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.180 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.180 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.180 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.180 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.180 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.180 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.180 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.180 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.180 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.180 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.180 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.181 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.181 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.181 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.181 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.181 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.867+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.181 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.181 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.181 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.181 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.181 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.181 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.181 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.181 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.182 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.182 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.182 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.182 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.182 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.182 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.182 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.182 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.182 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.182 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.182 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.182 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.182 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.183 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.183 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.183 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.183 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.183 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.183 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.183 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.183 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.183 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.183 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.183 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.183 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.183 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.184 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.184 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.184 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.184 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.184 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.184 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.184 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.184 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.184 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.184 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.184 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.184 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.185 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.185 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.185 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.185 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.185 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.185 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.185 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.185 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.185 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.185 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.185 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.185 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.186 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.186 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.186 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.186 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.186 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.186 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.186 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.186 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.186 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.186 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.186 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.186 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.186 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.187 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.187 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.187 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.187 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.187 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.187 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.187 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.187 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.187 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.187 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.187 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.187 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.188 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.188 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.188 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.188 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.188 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.188 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.188 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.188 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.188 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.188 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.188 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.188 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.188 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.188 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.189 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.868+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.189 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.189 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.189 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.189 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.189 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.189 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.189 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.189 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.189 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.189 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.189 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.189 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.190 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.190 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.190 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.190 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.190 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.190 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.190 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.190 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.190 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.190 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.190 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.190 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.190 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.190 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.191 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.191 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.191 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.191 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.191 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.869+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.191 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.906+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.191 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.927+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.192 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.934+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.192 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.934+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.192 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.935+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.192 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.935+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.192 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.935+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.192 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.935+0000 7f38af7b5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.192 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.936+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.192 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:34.936+0000 7f38b37bd640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,3,4,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.193 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.949+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.194 INFO:tasks.rados.rados.0.smithi001.stdout:301: finishing write tid 4 to smithi00132172-16 2024-09-15T03:47:35.194 INFO:tasks.rados.rados.0.smithi001.stdout:301: finishing write tid 5 to smithi00132172-16 2024-09-15T03:47:35.194 INFO:tasks.rados.rados.0.smithi001.stdout:301: finishing write tid 6 to smithi00132172-16 2024-09-15T03:47:35.195 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 3 v 115 (ObjNum 81 snap 16 seq_num 81) dirty exists 2024-09-15T03:47:35.195 INFO:tasks.rados.rados.0.smithi001.stdout:301: finishing write tid 7 to smithi00132172-16 2024-09-15T03:47:35.195 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 16 v 46 (ObjNum 119 snap 33 seq_num 119) dirty exists 2024-09-15T03:47:35.195 INFO:tasks.rados.rados.0.smithi001.stdout:301: left oid 16 (ObjNum 119 snap 33 seq_num 119) 2024-09-15T03:47:35.195 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:35.194+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:35.196 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:35.196 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:35.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:35.196 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:35.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.196 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 38/28 ECSubWrite(tid=33, reqid=client.4521.0:222, at_version=38'14, trim_to=0'0, roll_forward_to=35'12))", 2024-09-15T03:47:35.196 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:47:35.196 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 49.822812431999999, 2024-09-15T03:47:35.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9949187450000001, 2024-09-15T03:47:35.197 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.197 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.197 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:47:35.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:47:35.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130996+0000", 2024-09-15T03:47:35.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8809999999999998e-06 2024-09-15T03:47:35.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131020+0000", 2024-09-15T03:47:35.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4190999999999999e-05 2024-09-15T03:47:35.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131039+0000", 2024-09-15T03:47:35.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9228999999999998e-05 2024-09-15T03:47:35.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131048+0000", 2024-09-15T03:47:35.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2380000000000003e-06 2024-09-15T03:47:35.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131122+0000", 2024-09-15T03:47:35.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3146999999999996e-05 2024-09-15T03:47:35.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131132+0000", 2024-09-15T03:47:35.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.08e-05 2024-09-15T03:47:35.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131381+0000", 2024-09-15T03:47:35.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000248936 2024-09-15T03:47:35.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131392+0000", 2024-09-15T03:47:35.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0987e-05 2024-09-15T03:47:35.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131444+0000", 2024-09-15T03:47:35.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1585000000000002e-05 2024-09-15T03:47:35.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131465+0000", 2024-09-15T03:47:35.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1365999999999998e-05 2024-09-15T03:47:35.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.125911+0000", 2024-09-15T03:47:35.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9944453850000001 2024-09-15T03:47:35.202 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.202 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.202 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.202 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 38/28 ECSubWrite(tid=34, reqid=client.4521.0:222, at_version=0'0, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T03:47:35.202 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:47:35.202 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 49.821413333000002, 2024-09-15T03:47:35.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.002752697, 2024-09-15T03:47:35.203 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.203 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.203 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:47:35.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:47:35.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132392+0000", 2024-09-15T03:47:35.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5899999999999995e-07 2024-09-15T03:47:35.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132415+0000", 2024-09-15T03:47:35.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3561e-05 2024-09-15T03:47:35.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132436+0000", 2024-09-15T03:47:35.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0879000000000001e-05 2024-09-15T03:47:35.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132444+0000", 2024-09-15T03:47:35.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9200000000000004e-06 2024-09-15T03:47:35.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132497+0000", 2024-09-15T03:47:35.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2299000000000002e-05 2024-09-15T03:47:35.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132508+0000", 2024-09-15T03:47:35.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1100999999999999e-05 2024-09-15T03:47:35.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132615+0000", 2024-09-15T03:47:35.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107546 2024-09-15T03:47:35.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132635+0000", 2024-09-15T03:47:35.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9374000000000001e-05 2024-09-15T03:47:35.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132677+0000", 2024-09-15T03:47:35.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2456999999999997e-05 2024-09-15T03:47:35.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132711+0000", 2024-09-15T03:47:35.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4118000000000001e-05 2024-09-15T03:47:35.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135144+0000", 2024-09-15T03:47:35.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.002432583 2024-09-15T03:47:35.208 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.208 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.208 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.208 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=45, reqid=client.4521.0:227, at_version=38'24, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:47:35.208 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:35.208 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 49.798486791000002, 2024-09-15T03:47:35.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.974765339, 2024-09-15T03:47:35.208 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.208 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.209 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:35.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:35.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:35.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.43e-07 2024-09-15T03:47:35.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156110+0000", 2024-09-15T03:47:35.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000792409 2024-09-15T03:47:35.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156139+0000", 2024-09-15T03:47:35.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9207e-05 2024-09-15T03:47:35.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156148+0000", 2024-09-15T03:47:35.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1269999999999992e-06 2024-09-15T03:47:35.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156248+0000", 2024-09-15T03:47:35.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9421999999999998e-05 2024-09-15T03:47:35.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156258+0000", 2024-09-15T03:47:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6989999999999998e-06 2024-09-15T03:47:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156674+0000", 2024-09-15T03:47:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041682600000000002 2024-09-15T03:47:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156705+0000", 2024-09-15T03:47:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0525000000000003e-05 2024-09-15T03:47:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157742+0000", 2024-09-15T03:47:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010374 2024-09-15T03:47:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157777+0000", 2024-09-15T03:47:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4878e-05 2024-09-15T03:47:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.130083+0000", 2024-09-15T03:47:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972305703 2024-09-15T03:47:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=37, reqid=client.4521.0:236, at_version=38'5, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 49.796873335000001, 2024-09-15T03:47:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9740836640000001, 2024-09-15T03:47:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1300000000000001e-07 2024-09-15T03:47:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156953+0000", 2024-09-15T03:47:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1585000000000001e-05 2024-09-15T03:47:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156974+0000", 2024-09-15T03:47:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0764e-05 2024-09-15T03:47:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156981+0000", 2024-09-15T03:47:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2760000000000003e-06 2024-09-15T03:47:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157061+0000", 2024-09-15T03:47:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9583e-05 2024-09-15T03:47:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157075+0000", 2024-09-15T03:47:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4019000000000001e-05 2024-09-15T03:47:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157351+0000", 2024-09-15T03:47:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027641100000000001 2024-09-15T03:47:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157746+0000", 2024-09-15T03:47:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039553199999999998 2024-09-15T03:47:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157838+0000", 2024-09-15T03:47:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1632000000000006e-05 2024-09-15T03:47:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157856+0000", 2024-09-15T03:47:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7873999999999999e-05 2024-09-15T03:47:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.131015+0000", 2024-09-15T03:47:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9731586750000001 2024-09-15T03:47:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=46, reqid=client.4521.0:228, at_version=38'25, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:47:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 49.796814552999997, 2024-09-15T03:47:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9732838510000001, 2024-09-15T03:47:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0999999999999998e-07 2024-09-15T03:47:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157149+0000", 2024-09-15T03:47:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000158763 2024-09-15T03:47:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157165+0000", 2024-09-15T03:47:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5883999999999999e-05 2024-09-15T03:47:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157175+0000", 2024-09-15T03:47:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4949999999999997e-06 2024-09-15T03:47:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157219+0000", 2024-09-15T03:47:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4069000000000001e-05 2024-09-15T03:47:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157228+0000", 2024-09-15T03:47:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7829999999999993e-06 2024-09-15T03:47:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157345+0000", 2024-09-15T03:47:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116655 2024-09-15T03:47:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157746+0000", 2024-09-15T03:47:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040045100000000002 2024-09-15T03:47:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157838+0000", 2024-09-15T03:47:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1955999999999998e-05 2024-09-15T03:47:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157888+0000", 2024-09-15T03:47:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0943000000000003e-05 2024-09-15T03:47:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.130274+0000", 2024-09-15T03:47:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972385142 2024-09-15T03:47:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=38, reqid=client.4521.0:237, at_version=38'6, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:47:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 49.795471982999999, 2024-09-15T03:47:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9764011529999999, 2024-09-15T03:47:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:47:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:47:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158333+0000", 2024-09-15T03:47:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8599999999999997e-07 2024-09-15T03:47:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158353+0000", 2024-09-15T03:47:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9504000000000002e-05 2024-09-15T03:47:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158368+0000", 2024-09-15T03:47:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4966999999999999e-05 2024-09-15T03:47:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158374+0000", 2024-09-15T03:47:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.798e-06 2024-09-15T03:47:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158430+0000", 2024-09-15T03:47:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5744e-05 2024-09-15T03:47:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158440+0000", 2024-09-15T03:47:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4560000000000006e-06 2024-09-15T03:47:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158529+0000", 2024-09-15T03:47:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9172999999999995e-05 2024-09-15T03:47:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158543+0000", 2024-09-15T03:47:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.395e-05 2024-09-15T03:47:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158654+0000", 2024-09-15T03:47:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011110700000000001 2024-09-15T03:47:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158671+0000", 2024-09-15T03:47:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7068999999999999e-05 2024-09-15T03:47:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134733+0000", 2024-09-15T03:47:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.976062599 2024-09-15T03:47:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=47, reqid=client.4521.0:229, at_version=38'26, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:47:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:47:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 49.795468020999998, 2024-09-15T03:47:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9727810969999999, 2024-09-15T03:47:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:47:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:47:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158337+0000", 2024-09-15T03:47:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.15e-07 2024-09-15T03:47:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158351+0000", 2024-09-15T03:47:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3871e-05 2024-09-15T03:47:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158372+0000", 2024-09-15T03:47:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1058999999999999e-05 2024-09-15T03:47:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158376+0000", 2024-09-15T03:47:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5249999999999999e-06 2024-09-15T03:47:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158420+0000", 2024-09-15T03:47:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3918000000000002e-05 2024-09-15T03:47:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158429+0000", 2024-09-15T03:47:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5150000000000006e-06 2024-09-15T03:47:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158514+0000", 2024-09-15T03:47:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5252000000000003e-05 2024-09-15T03:47:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158542+0000", 2024-09-15T03:47:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.798e-05 2024-09-15T03:47:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158653+0000", 2024-09-15T03:47:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111315 2024-09-15T03:47:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158701+0000", 2024-09-15T03:47:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8241000000000001e-05 2024-09-15T03:47:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.131117+0000", 2024-09-15T03:47:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972416006 2024-09-15T03:47:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=39, reqid=client.4521.0:238, at_version=38'7, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:47:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 49.794780865, 2024-09-15T03:47:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9758214509999998, 2024-09-15T03:47:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:47:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:47:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159024+0000", 2024-09-15T03:47:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6700000000000003e-07 2024-09-15T03:47:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159096+0000", 2024-09-15T03:47:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1386999999999999e-05 2024-09-15T03:47:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159113+0000", 2024-09-15T03:47:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7020999999999998e-05 2024-09-15T03:47:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159122+0000", 2024-09-15T03:47:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4720000000000001e-06 2024-09-15T03:47:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159168+0000", 2024-09-15T03:47:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5766000000000002e-05 2024-09-15T03:47:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159176+0000", 2024-09-15T03:47:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3070000000000003e-06 2024-09-15T03:47:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159258+0000", 2024-09-15T03:47:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1398999999999998e-05 2024-09-15T03:47:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159292+0000", 2024-09-15T03:47:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4480000000000002e-05 2024-09-15T03:47:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159342+0000", 2024-09-15T03:47:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9450999999999998e-05 2024-09-15T03:47:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159362+0000", 2024-09-15T03:47:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0633e-05 2024-09-15T03:47:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134845+0000", 2024-09-15T03:47:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.975482768 2024-09-15T03:47:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=48, reqid=client.4521.0:229, at_version=0'0, trim_to=0'0, roll_forward_to=38'26))", 2024-09-15T03:47:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:47:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 49.793414069000001, 2024-09-15T03:47:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.974485472, 2024-09-15T03:47:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:47:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:47:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160391+0000", 2024-09-15T03:47:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.136e-06 2024-09-15T03:47:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160405+0000", 2024-09-15T03:47:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3613e-05 2024-09-15T03:47:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160426+0000", 2024-09-15T03:47:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1234999999999999e-05 2024-09-15T03:47:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160434+0000", 2024-09-15T03:47:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.61e-06 2024-09-15T03:47:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160496+0000", 2024-09-15T03:47:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2384000000000004e-05 2024-09-15T03:47:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160507+0000", 2024-09-15T03:47:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1005000000000001e-05 2024-09-15T03:47:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160632+0000", 2024-09-15T03:47:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124566 2024-09-15T03:47:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160653+0000", 2024-09-15T03:47:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0707999999999999e-05 2024-09-15T03:47:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160701+0000", 2024-09-15T03:47:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8751999999999997e-05 2024-09-15T03:47:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160739+0000", 2024-09-15T03:47:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7486e-05 2024-09-15T03:47:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134876+0000", 2024-09-15T03:47:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9741369770000001 2024-09-15T03:47:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=40, reqid=client.4521.0:238, at_version=0'0, trim_to=0'0, roll_forward_to=38'7))", 2024-09-15T03:47:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:47:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 49.791567647999997, 2024-09-15T03:47:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9727862730000001, 2024-09-15T03:47:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:47:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:47:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162238+0000", 2024-09-15T03:47:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.672e-06 2024-09-15T03:47:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162253+0000", 2024-09-15T03:47:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4932e-05 2024-09-15T03:47:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162284+0000", 2024-09-15T03:47:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1211000000000003e-05 2024-09-15T03:47:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162296+0000", 2024-09-15T03:47:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1229999999999999e-05 2024-09-15T03:47:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162373+0000", 2024-09-15T03:47:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7146999999999998e-05 2024-09-15T03:47:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162387+0000", 2024-09-15T03:47:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4290999999999999e-05 2024-09-15T03:47:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162502+0000", 2024-09-15T03:47:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115054 2024-09-15T03:47:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162516+0000", 2024-09-15T03:47:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3597999999999999e-05 2024-09-15T03:47:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162558+0000", 2024-09-15T03:47:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2562999999999999e-05 2024-09-15T03:47:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162584+0000", 2024-09-15T03:47:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5738e-05 2024-09-15T03:47:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135023+0000", 2024-09-15T03:47:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9724388369999999 2024-09-15T03:47:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 39/28 ECSubWrite(tid=38, reqid=osd.11.0:38, at_version=39'16, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T03:47:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 48.810185347999997, 2024-09-15T03:47:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99210653800000004, 2024-09-15T03:47:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1800000000000002e-07 2024-09-15T03:47:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143648+0000", 2024-09-15T03:47:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.849e-05 2024-09-15T03:47:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143670+0000", 2024-09-15T03:47:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2036e-05 2024-09-15T03:47:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143680+0000", 2024-09-15T03:47:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8320000000000001e-06 2024-09-15T03:47:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143783+0000", 2024-09-15T03:47:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103569 2024-09-15T03:47:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143801+0000", 2024-09-15T03:47:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8077999999999999e-05 2024-09-15T03:47:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144077+0000", 2024-09-15T03:47:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027569500000000001 2024-09-15T03:47:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144459+0000", 2024-09-15T03:47:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038227299999999998 2024-09-15T03:47:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144519+0000", 2024-09-15T03:47:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9907000000000001e-05 2024-09-15T03:47:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144628+0000", 2024-09-15T03:47:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108958 2024-09-15T03:47:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135726+0000", 2024-09-15T03:47:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99109738199999997 2024-09-15T03:47:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 39/31 ECSubWrite(tid=41, reqid=osd.6.0:41, at_version=39'9, trim_to=0'0, roll_forward_to=38'7))", 2024-09-15T03:47:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 48.809535093000001, 2024-09-15T03:47:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99106653099999997, 2024-09-15T03:47:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.97e-07 2024-09-15T03:47:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144297+0000", 2024-09-15T03:47:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7815000000000001e-05 2024-09-15T03:47:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144314+0000", 2024-09-15T03:47:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6269e-05 2024-09-15T03:47:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144322+0000", 2024-09-15T03:47:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1880000000000002e-06 2024-09-15T03:47:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144767+0000", 2024-09-15T03:47:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044555300000000001 2024-09-15T03:47:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144777+0000", 2024-09-15T03:47:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5950000000000005e-06 2024-09-15T03:47:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144992+0000", 2024-09-15T03:47:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000215479 2024-09-15T03:47:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145142+0000", 2024-09-15T03:47:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149302 2024-09-15T03:47:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145249+0000", 2024-09-15T03:47:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106971 2024-09-15T03:47:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145299+0000", 2024-09-15T03:47:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0413e-05 2024-09-15T03:47:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135336+0000", 2024-09-15T03:47:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99003674900000005 2024-09-15T03:47:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 39/28 ECSubWrite(tid=50, reqid=osd.2.0:50, at_version=39'28, trim_to=0'0, roll_forward_to=38'26))", 2024-09-15T03:47:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 48.809237611, 2024-09-15T03:47:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99095576299999999, 2024-09-15T03:47:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.03e-07 2024-09-15T03:47:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144593+0000", 2024-09-15T03:47:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6520999999999999e-05 2024-09-15T03:47:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144611+0000", 2024-09-15T03:47:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7963000000000001e-05 2024-09-15T03:47:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144619+0000", 2024-09-15T03:47:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7409999999999992e-06 2024-09-15T03:47:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144821+0000", 2024-09-15T03:47:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000201652 2024-09-15T03:47:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144832+0000", 2024-09-15T03:47:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0827e-05 2024-09-15T03:47:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145099+0000", 2024-09-15T03:47:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026688299999999999 2024-09-15T03:47:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145143+0000", 2024-09-15T03:47:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.481e-05 2024-09-15T03:47:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145249+0000", 2024-09-15T03:47:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010618799999999999 2024-09-15T03:47:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145340+0000", 2024-09-15T03:47:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0340000000000006e-05 2024-09-15T03:47:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135522+0000", 2024-09-15T03:47:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99018263500000003 2024-09-15T03:47:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=43, reqid=client.4521.0:240, at_version=39'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:47:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 48.805896891000003, 2024-09-15T03:47:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99104975500000003, 2024-09-15T03:47:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:47:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:47:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147908+0000", 2024-09-15T03:47:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.05e-07 2024-09-15T03:47:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147931+0000", 2024-09-15T03:47:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3621e-05 2024-09-15T03:47:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147945+0000", 2024-09-15T03:47:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3648999999999999e-05 2024-09-15T03:47:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147951+0000", 2024-09-15T03:47:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0510000000000002e-06 2024-09-15T03:47:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148055+0000", 2024-09-15T03:47:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104421 2024-09-15T03:47:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148068+0000", 2024-09-15T03:47:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3019e-05 2024-09-15T03:47:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148375+0000", 2024-09-15T03:47:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030683400000000002 2024-09-15T03:47:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148399+0000", 2024-09-15T03:47:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3760999999999998e-05 2024-09-15T03:47:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148453+0000", 2024-09-15T03:47:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3851999999999999e-05 2024-09-15T03:47:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148481+0000", 2024-09-15T03:47:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8637999999999999e-05 2024-09-15T03:47:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.138957+0000", 2024-09-15T03:47:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99047580400000002 2024-09-15T03:47:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 39/28 ECSubWrite(tid=39, reqid=osd.11.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:47:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:47:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 48.805764402000001, 2024-09-15T03:47:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99452509, 2024-09-15T03:47:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:47:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:47:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148041+0000", 2024-09-15T03:47:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0999999999999998e-07 2024-09-15T03:47:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148052+0000", 2024-09-15T03:47:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.129e-05 2024-09-15T03:47:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148064+0000", 2024-09-15T03:47:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173e-05 2024-09-15T03:47:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148068+0000", 2024-09-15T03:47:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5680000000000001e-06 2024-09-15T03:47:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148136+0000", 2024-09-15T03:47:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7508999999999993e-05 2024-09-15T03:47:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148148+0000", 2024-09-15T03:47:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2002e-05 2024-09-15T03:47:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148265+0000", 2024-09-15T03:47:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117669 2024-09-15T03:47:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148289+0000", 2024-09-15T03:47:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3672e-05 2024-09-15T03:47:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148348+0000", 2024-09-15T03:47:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8838000000000001e-05 2024-09-15T03:47:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148368+0000", 2024-09-15T03:47:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9942999999999999e-05 2024-09-15T03:47:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142565+0000", 2024-09-15T03:47:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99419715900000005 2024-09-15T03:47:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=45, reqid=client.4521.0:241, at_version=39'17, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:47:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 48.802761379000003, 2024-09-15T03:47:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99118202200000005, 2024-09-15T03:47:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:47:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:47:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151044+0000", 2024-09-15T03:47:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1340000000000001e-06 2024-09-15T03:47:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151073+0000", 2024-09-15T03:47:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8894000000000002e-05 2024-09-15T03:47:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151095+0000", 2024-09-15T03:47:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1889e-05 2024-09-15T03:47:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151107+0000", 2024-09-15T03:47:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2136999999999999e-05 2024-09-15T03:47:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151189+0000", 2024-09-15T03:47:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1883000000000002e-05 2024-09-15T03:47:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151203+0000", 2024-09-15T03:47:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3825999999999999e-05 2024-09-15T03:47:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151305+0000", 2024-09-15T03:47:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101816 2024-09-15T03:47:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151337+0000", 2024-09-15T03:47:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2703000000000002e-05 2024-09-15T03:47:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151387+0000", 2024-09-15T03:47:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9366000000000001e-05 2024-09-15T03:47:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151405+0000", 2024-09-15T03:47:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8748000000000001e-05 2024-09-15T03:47:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142225+0000", 2024-09-15T03:47:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99081962599999995 2024-09-15T03:47:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 39/31 ECSubWrite(tid=42, reqid=osd.6.0:41, at_version=0'0, trim_to=0'0, roll_forward_to=39'9))", 2024-09-15T03:47:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:47:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 48.802465337999998, 2024-09-15T03:47:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99092639999999999, 2024-09-15T03:47:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:47:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:47:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151340+0000", 2024-09-15T03:47:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8299999999999996e-07 2024-09-15T03:47:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151350+0000", 2024-09-15T03:47:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0675e-05 2024-09-15T03:47:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151362+0000", 2024-09-15T03:47:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2048999999999999e-05 2024-09-15T03:47:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151369+0000", 2024-09-15T03:47:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3230000000000003e-06 2024-09-15T03:47:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151425+0000", 2024-09-15T03:47:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5964000000000002e-05 2024-09-15T03:47:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151435+0000", 2024-09-15T03:47:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9240000000000002e-06 2024-09-15T03:47:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151530+0000", 2024-09-15T03:47:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5649000000000005e-05 2024-09-15T03:47:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151558+0000", 2024-09-15T03:47:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7976000000000001e-05 2024-09-15T03:47:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151599+0000", 2024-09-15T03:47:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1152000000000002e-05 2024-09-15T03:47:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151617+0000", 2024-09-15T03:47:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7676000000000002e-05 2024-09-15T03:47:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142265+0000", 2024-09-15T03:47:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99064822900000005 2024-09-15T03:47:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=46, reqid=client.4521.0:242, at_version=39'18, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 48.802439702000001, 2024-09-15T03:47:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.991000204, 2024-09-15T03:47:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3399999999999999e-07 2024-09-15T03:47:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151393+0000", 2024-09-15T03:47:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8149000000000001e-05 2024-09-15T03:47:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151405+0000", 2024-09-15T03:47:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2063e-05 2024-09-15T03:47:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151415+0000", 2024-09-15T03:47:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4579999999999999e-06 2024-09-15T03:47:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151472+0000", 2024-09-15T03:47:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7018999999999997e-05 2024-09-15T03:47:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151480+0000", 2024-09-15T03:47:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1410000000000005e-06 2024-09-15T03:47:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151571+0000", 2024-09-15T03:47:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1243999999999996e-05 2024-09-15T03:47:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151602+0000", 2024-09-15T03:47:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0834000000000003e-05 2024-09-15T03:47:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151632+0000", 2024-09-15T03:47:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9527e-05 2024-09-15T03:47:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151685+0000", 2024-09-15T03:47:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3599e-05 2024-09-15T03:47:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142365+0000", 2024-09-15T03:47:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99067953600000003 2024-09-15T03:47:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 39/28 ECSubWrite(tid=51, reqid=osd.2.0:50, at_version=0'0, trim_to=0'0, roll_forward_to=39'28))", 2024-09-15T03:47:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:47:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 48.799076264, 2024-09-15T03:47:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98766307399999997, 2024-09-15T03:47:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:47:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:47:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154729+0000", 2024-09-15T03:47:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3799999999999996e-07 2024-09-15T03:47:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154742+0000", 2024-09-15T03:47:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2909e-05 2024-09-15T03:47:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154755+0000", 2024-09-15T03:47:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3251999999999999e-05 2024-09-15T03:47:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154761+0000", 2024-09-15T03:47:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6559999999999997e-06 2024-09-15T03:47:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154829+0000", 2024-09-15T03:47:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8549999999999999e-05 2024-09-15T03:47:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154840+0000", 2024-09-15T03:47:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0159999999999999e-05 2024-09-15T03:47:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154943+0000", 2024-09-15T03:47:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010347400000000001 2024-09-15T03:47:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154955+0000", 2024-09-15T03:47:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1935e-05 2024-09-15T03:47:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156442+0000", 2024-09-15T03:47:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014867859999999999 2024-09-15T03:47:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156466+0000", 2024-09-15T03:47:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4077999999999999e-05 2024-09-15T03:47:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142391+0000", 2024-09-15T03:47:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98592533599999999 2024-09-15T03:47:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=48, reqid=client.4521.0:242, at_version=0'0, trim_to=0'0, roll_forward_to=39'18))", 2024-09-15T03:47:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:47:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 48.797189244999998, 2024-09-15T03:47:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98592495099999999, 2024-09-15T03:47:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:47:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:47:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156616+0000", 2024-09-15T03:47:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1999999999999998e-07 2024-09-15T03:47:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156629+0000", 2024-09-15T03:47:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2870999999999999e-05 2024-09-15T03:47:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156649+0000", 2024-09-15T03:47:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9786e-05 2024-09-15T03:47:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156657+0000", 2024-09-15T03:47:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5110000000000003e-06 2024-09-15T03:47:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156723+0000", 2024-09-15T03:47:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6209000000000003e-05 2024-09-15T03:47:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156734+0000", 2024-09-15T03:47:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0475999999999999e-05 2024-09-15T03:47:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156868+0000", 2024-09-15T03:47:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013402999999999999 2024-09-15T03:47:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156889+0000", 2024-09-15T03:47:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1146e-05 2024-09-15T03:47:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156937+0000", 2024-09-15T03:47:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8031999999999997e-05 2024-09-15T03:47:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156970+0000", 2024-09-15T03:47:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2462999999999997e-05 2024-09-15T03:47:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142540+0000", 2024-09-15T03:47:35.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98557050700000004 2024-09-15T03:47:35.317 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.317 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.317 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.317 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.317 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.317 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:35.318 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:47:35.318 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.980+0000 7f63ad7be640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.318 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:34.991+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.319 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:34.997+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:35.319 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:35.010+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.319 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:35.024+0000 7f63a97b6640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,3,6 to read 0,1,2,3,4,5,6,7 2024-09-15T03:47:35.322 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:35.321+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:35.422 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:35.422 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:35.423 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:35.422+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:35.430 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:47:35.523 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:35.522+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:35.530 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:35.530 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:35.531 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:35.531 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:35.531 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:35.539 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:47:35.624 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:35.623+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:35.682 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:35.682 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:35.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:35.683 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:35.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.683 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=21, reqid=client.4521.0:184, at_version=33'6, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:35.683 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:35.683 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 55.573799317999999, 2024-09-15T03:47:35.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010901256999999999, 2024-09-15T03:47:35.683 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.683 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.683 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:35.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:35.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:35.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0200000000000001e-07 2024-09-15T03:47:35.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106529+0000", 2024-09-15T03:47:35.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9539e-05 2024-09-15T03:47:35.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106542+0000", 2024-09-15T03:47:35.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2731999999999999e-05 2024-09-15T03:47:35.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106549+0000", 2024-09-15T03:47:35.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3339999999999996e-06 2024-09-15T03:47:35.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106613+0000", 2024-09-15T03:47:35.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4527000000000001e-05 2024-09-15T03:47:35.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106622+0000", 2024-09-15T03:47:35.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.755e-06 2024-09-15T03:47:35.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106841+0000", 2024-09-15T03:47:35.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021906200000000001 2024-09-15T03:47:35.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106869+0000", 2024-09-15T03:47:35.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.739e-05 2024-09-15T03:47:35.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106948+0000", 2024-09-15T03:47:35.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9422e-05 2024-09-15T03:47:35.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106965+0000", 2024-09-15T03:47:35.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7010000000000001e-05 2024-09-15T03:47:35.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.117410+0000", 2024-09-15T03:47:35.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010445284000000001 2024-09-15T03:47:35.688 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.688 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.688 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.689 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:35.689 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:35.689 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 55.571885831000003, 2024-09-15T03:47:35.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015296944, 2024-09-15T03:47:35.689 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.689 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.689 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:35.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:35.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:35.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5499999999999998e-07 2024-09-15T03:47:35.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108447+0000", 2024-09-15T03:47:35.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3314000000000001e-05 2024-09-15T03:47:35.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108460+0000", 2024-09-15T03:47:35.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3444e-05 2024-09-15T03:47:35.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108464+0000", 2024-09-15T03:47:35.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3610000000000003e-06 2024-09-15T03:47:35.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108494+0000", 2024-09-15T03:47:35.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9688000000000001e-05 2024-09-15T03:47:35.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108502+0000", 2024-09-15T03:47:35.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.678e-06 2024-09-15T03:47:35.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108601+0000", 2024-09-15T03:47:35.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9748999999999997e-05 2024-09-15T03:47:35.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108610+0000", 2024-09-15T03:47:35.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2490000000000001e-06 2024-09-15T03:47:35.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109977+0000", 2024-09-15T03:47:35.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013669450000000001 2024-09-15T03:47:35.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109993+0000", 2024-09-15T03:47:35.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6688e-05 2024-09-15T03:47:35.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123720+0000", 2024-09-15T03:47:35.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013726172999999999 2024-09-15T03:47:35.695 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.695 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.695 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.695 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:35.695 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:35.695 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 55.571836169999997, 2024-09-15T03:47:35.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015105210000000001, 2024-09-15T03:47:35.695 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.696 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.696 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:35.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:35.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108473+0000", 2024-09-15T03:47:35.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.84e-07 2024-09-15T03:47:35.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108486+0000", 2024-09-15T03:47:35.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3845999999999999e-05 2024-09-15T03:47:35.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108491+0000", 2024-09-15T03:47:35.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4719999999999997e-06 2024-09-15T03:47:35.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108495+0000", 2024-09-15T03:47:35.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.822e-06 2024-09-15T03:47:35.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108546+0000", 2024-09-15T03:47:35.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0853000000000003e-05 2024-09-15T03:47:35.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108555+0000", 2024-09-15T03:47:35.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3319999999999998e-06 2024-09-15T03:47:35.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108638+0000", 2024-09-15T03:47:35.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3580999999999998e-05 2024-09-15T03:47:35.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109983+0000", 2024-09-15T03:47:35.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001344392 2024-09-15T03:47:35.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110023+0000", 2024-09-15T03:47:35.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9895999999999999e-05 2024-09-15T03:47:35.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110040+0000", 2024-09-15T03:47:35.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.747e-05 2024-09-15T03:47:35.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123577+0000", 2024-09-15T03:47:35.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013537262 2024-09-15T03:47:35.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.702 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.703 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:35.703 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:35.703 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 55.571802290999997, 2024-09-15T03:47:35.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015311313999999999, 2024-09-15T03:47:35.703 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.703 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.703 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:35.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:35.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:35.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8500000000000002e-07 2024-09-15T03:47:35.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108517+0000", 2024-09-15T03:47:35.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0543000000000001e-05 2024-09-15T03:47:35.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108524+0000", 2024-09-15T03:47:35.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7170000000000002e-06 2024-09-15T03:47:35.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108528+0000", 2024-09-15T03:47:35.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3970000000000004e-06 2024-09-15T03:47:35.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108632+0000", 2024-09-15T03:47:35.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103687 2024-09-15T03:47:35.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108637+0000", 2024-09-15T03:47:35.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4389999999999999e-06 2024-09-15T03:47:35.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108713+0000", 2024-09-15T03:47:35.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6223000000000005e-05 2024-09-15T03:47:35.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109984+0000", 2024-09-15T03:47:35.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012700719999999999 2024-09-15T03:47:35.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110023+0000", 2024-09-15T03:47:35.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9864999999999999e-05 2024-09-15T03:47:35.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110041+0000", 2024-09-15T03:47:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8040999999999999e-05 2024-09-15T03:47:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123817+0000", 2024-09-15T03:47:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013776045000000001 2024-09-15T03:47:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 55.571308682999998, 2024-09-15T03:47:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014864502, 2024-09-15T03:47:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:47:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109016+0000", 2024-09-15T03:47:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5889999999999999e-05 2024-09-15T03:47:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109029+0000", 2024-09-15T03:47:35.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2812e-05 2024-09-15T03:47:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109035+0000", 2024-09-15T03:47:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0109999999999997e-06 2024-09-15T03:47:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109073+0000", 2024-09-15T03:47:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8260000000000003e-05 2024-09-15T03:47:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109080+0000", 2024-09-15T03:47:35.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0190000000000001e-06 2024-09-15T03:47:35.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109165+0000", 2024-09-15T03:47:35.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4957000000000007e-05 2024-09-15T03:47:35.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109984+0000", 2024-09-15T03:47:35.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081866399999999998 2024-09-15T03:47:35.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110024+0000", 2024-09-15T03:47:35.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9644999999999997e-05 2024-09-15T03:47:35.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110072+0000", 2024-09-15T03:47:35.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8418999999999998e-05 2024-09-15T03:47:35.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123864+0000", 2024-09-15T03:47:35.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013792234 2024-09-15T03:47:35.715 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.715 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.715 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.715 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=34, reqid=client.4521.0:197, at_version=34'21, trim_to=0'0, roll_forward_to=34'18))", 2024-09-15T03:47:35.715 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:35.716 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.556296803000002, 2024-09-15T03:47:35.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012992399, 2024-09-15T03:47:35.716 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.716 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.716 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:35.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:35.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124013+0000", 2024-09-15T03:47:35.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6299999999999993e-07 2024-09-15T03:47:35.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124044+0000", 2024-09-15T03:47:35.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0995999999999999e-05 2024-09-15T03:47:35.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124062+0000", 2024-09-15T03:47:35.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8413000000000001e-05 2024-09-15T03:47:35.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124069+0000", 2024-09-15T03:47:35.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3930000000000002e-06 2024-09-15T03:47:35.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124150+0000", 2024-09-15T03:47:35.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1097e-05 2024-09-15T03:47:35.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124161+0000", 2024-09-15T03:47:35.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0720000000000001e-05 2024-09-15T03:47:35.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124402+0000", 2024-09-15T03:47:35.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000240442 2024-09-15T03:47:35.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125141+0000", 2024-09-15T03:47:35.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00073915500000000002 2024-09-15T03:47:35.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125191+0000", 2024-09-15T03:47:35.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9951999999999999e-05 2024-09-15T03:47:35.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125251+0000", 2024-09-15T03:47:35.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9938000000000001e-05 2024-09-15T03:47:35.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.137004+0000", 2024-09-15T03:47:35.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011753329999999999 2024-09-15T03:47:35.721 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.721 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.721 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.722 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=35, reqid=client.4521.0:197, at_version=0'0, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:47:35.722 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:35.722 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.554356759000001, 2024-09-15T03:47:35.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011202456, 2024-09-15T03:47:35.722 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.722 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.722 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:35.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:35.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125953+0000", 2024-09-15T03:47:35.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9100000000000002e-07 2024-09-15T03:47:35.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125964+0000", 2024-09-15T03:47:35.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1399999999999999e-05 2024-09-15T03:47:35.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125979+0000", 2024-09-15T03:47:35.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4810000000000001e-05 2024-09-15T03:47:35.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126004+0000", 2024-09-15T03:47:35.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4913e-05 2024-09-15T03:47:35.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126059+0000", 2024-09-15T03:47:35.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5668999999999998e-05 2024-09-15T03:47:35.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126071+0000", 2024-09-15T03:47:35.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1219999999999999e-05 2024-09-15T03:47:35.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126151+0000", 2024-09-15T03:47:35.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0572e-05 2024-09-15T03:47:35.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126163+0000", 2024-09-15T03:47:35.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2101e-05 2024-09-15T03:47:35.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.127945+0000", 2024-09-15T03:47:35.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001781803 2024-09-15T03:47:35.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.127982+0000", 2024-09-15T03:47:35.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7129999999999999e-05 2024-09-15T03:47:35.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.137154+0000", 2024-09-15T03:47:35.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0091719469999999997 2024-09-15T03:47:35.727 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.727 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.727 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.728 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:243 3.3s0 3:c7c3ef16:::smithi00132172-25:head [create,setxattr iZnWfebab`ptjURR\\f`psW]tpS`qpUaeWkY[Oqep (1330) in=40b,setxattr f[mZ_qQmcZejQ`fbdeic_tkPUaTYcnm[WprjVmc] (12488) in=40b,setxattr tkZgQ]S_pdlhZnqrVaZRa`oQQr_USmUfhado^dXT (1121) in=40b,setxattr srWjTqXhcth`hfZmQY_fg]UqNjWofa^dfgOV`sYP (455) in=40b,setxattr ^_mdiRroj_TtQ^gcNj`XtkW[ooqY`PWXqYmcOk_n (138) in=40b,setxattr jQlajdtcalhkcfYaakZfNb[^_TjkfcS`Vrk]q_d[ (806) in=40b,setxattr ZfYgkVcrsXcnpchgfcbldNrUTasnQNmfZW]Nbh\\o (561) in=40b,setxattr _hkYOqjPNhmhUlSNktfeQpsqSWjO\\fbf[fpTlZiO (1227) in=40b,setxattr ]mdV^Xdc\\sViXdkXNU[js_Qhh[b[sbfnZVXkdids (11788) in=40b,setxattr tb^mbOrNhfrhtce^g (455) in=17b,setxattr koTkVgrfU\\m`StnZ`Rirjhbhg`PhVqm^a^Xgaaqn (35) in=40b,setxattr aUtZPUPXPnQmT]\\\\n\\lZpnXakdNkTldVQoQhPUf] (559) in=40b,setxattr mrhR]^U`Tteo[Topmql`SNma\\pqfk`if]OppnYt` (770) in=40b,setxattr Oetk]RrnWeXbVctVtVrhXf_RfTdelqX^lWp[sTkp (1226) in=40b,setxattr WtedXoOd`VqnbksOqQkatk`gdTnW^lXbUtrlYgkl (3576) in=40b,setxattr dddOWN\\j\\XXg^fr[rZcskeVX]jOQ^ZmoWepk[]\\a (17824) in=40b,setxattr TeRO^djPgNZiemQ_aRXgP[]jgUZa\\^\\iUgesaqaN (208) in=40b,setxattr teqm]mUobZbWjcdQ_jTa[f[k[Sd\\lh]n\\osigqkN (7436) in=40b,setxattr dphaol[XkeitbkomWYaQctWfataUhbSiNntkWo`P (1893) in=40b,setxattr oQsnhO]ZklSVYQ[WUpaelY[p[Qie^UepOt`Z[q (455) in=38b] snapc 7=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:47:35.728 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:35.728 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 49.511392870999998, 2024-09-15T03:47:35.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012365956000000001, 2024-09-15T03:47:35.728 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.728 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:35.728 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:35.728 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:35.728 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:35.728 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 243 2024-09-15T03:47:35.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.729 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:35.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:35.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168917+0000", 2024-09-15T03:47:35.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.725e-06 2024-09-15T03:47:35.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169030+0000", 2024-09-15T03:47:35.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113035 2024-09-15T03:47:35.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169047+0000", 2024-09-15T03:47:35.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6701000000000002e-05 2024-09-15T03:47:35.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169054+0000", 2024-09-15T03:47:35.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1369999999999996e-06 2024-09-15T03:47:35.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169140+0000", 2024-09-15T03:47:35.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6069999999999994e-05 2024-09-15T03:47:35.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:35.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169270+0000", 2024-09-15T03:47:35.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130071 2024-09-15T03:47:35.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.170661+0000", 2024-09-15T03:47:35.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001390666 2024-09-15T03:47:35.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171751+0000", 2024-09-15T03:47:35.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010899079999999999 2024-09-15T03:47:35.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171812+0000", 2024-09-15T03:47:35.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0797000000000001e-05 2024-09-15T03:47:35.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172279+0000", 2024-09-15T03:47:35.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046704500000000003 2024-09-15T03:47:35.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172299+0000", 2024-09-15T03:47:35.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0387999999999999e-05 2024-09-15T03:47:35.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:35.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.176110+0000", 2024-09-15T03:47:35.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038112340000000001 2024-09-15T03:47:35.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.181282+0000", 2024-09-15T03:47:35.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051711789999999997 2024-09-15T03:47:35.735 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.735 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.736 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.736 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:258 3.3s0 3:c9b663a2:::smithi00132172-7:head [delete] snapc 7=[6,5,3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e40)", 2024-09-15T03:47:35.736 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:47.146116+0000", 2024-09-15T03:47:35.736 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 48.534192052999998, 2024-09-15T03:47:35.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096428380000000008, 2024-09-15T03:47:35.736 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.736 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:35.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:35.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:35.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:35.736 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 258 2024-09-15T03:47:35.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.737 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146116+0000", 2024-09-15T03:47:35.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146116+0000", 2024-09-15T03:47:35.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146119+0000", 2024-09-15T03:47:35.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.227e-06 2024-09-15T03:47:35.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146127+0000", 2024-09-15T03:47:35.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1529999999999997e-06 2024-09-15T03:47:35.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146146+0000", 2024-09-15T03:47:35.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9318999999999999e-05 2024-09-15T03:47:35.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146154+0000", 2024-09-15T03:47:35.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7200000000000006e-06 2024-09-15T03:47:35.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146233+0000", 2024-09-15T03:47:35.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8725000000000003e-05 2024-09-15T03:47:35.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:35.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146244+0000", 2024-09-15T03:47:35.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1725e-05 2024-09-15T03:47:35.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.149207+0000", 2024-09-15T03:47:35.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029627999999999998 2024-09-15T03:47:35.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:35.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.149326+0000", 2024-09-15T03:47:35.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011869199999999999 2024-09-15T03:47:35.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.150266+0000", 2024-09-15T03:47:35.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00094048500000000002 2024-09-15T03:47:35.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.150626+0000", 2024-09-15T03:47:35.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035946699999999998 2024-09-15T03:47:35.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.150751+0000", 2024-09-15T03:47:35.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012544800000000001 2024-09-15T03:47:35.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.151656+0000", 2024-09-15T03:47:35.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00090447499999999996 2024-09-15T03:47:35.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.153489+0000", 2024-09-15T03:47:35.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001833115 2024-09-15T03:47:35.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:35.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.155171+0000", 2024-09-15T03:47:35.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001682455 2024-09-15T03:47:35.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.155759+0000", 2024-09-15T03:47:35.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00058803200000000005 2024-09-15T03:47:35.744 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.744 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.744 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.744 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/28 ECSubWrite(tid=15, reqid=client.4521.0:337, at_version=49'10, trim_to=0'0, roll_forward_to=43'8))", 2024-09-15T03:47:35.744 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:35.744 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.822402961999998, 2024-09-15T03:47:35.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010500893000000001, 2024-09-15T03:47:35.744 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.744 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.745 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:35.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:35.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857906+0000", 2024-09-15T03:47:35.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.63e-07 2024-09-15T03:47:35.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857928+0000", 2024-09-15T03:47:35.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2155999999999999e-05 2024-09-15T03:47:35.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857937+0000", 2024-09-15T03:47:35.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2359999999999993e-06 2024-09-15T03:47:35.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857941+0000", 2024-09-15T03:47:35.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0960000000000003e-06 2024-09-15T03:47:35.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857999+0000", 2024-09-15T03:47:35.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7407000000000001e-05 2024-09-15T03:47:35.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858011+0000", 2024-09-15T03:47:35.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1759999999999999e-05 2024-09-15T03:47:35.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858240+0000", 2024-09-15T03:47:35.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022949299999999999 2024-09-15T03:47:35.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858251+0000", 2024-09-15T03:47:35.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1168999999999999e-05 2024-09-15T03:47:35.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858332+0000", 2024-09-15T03:47:35.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1204000000000004e-05 2024-09-15T03:47:35.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858360+0000", 2024-09-15T03:47:35.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7767999999999999e-05 2024-09-15T03:47:35.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.868406+0000", 2024-09-15T03:47:35.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010046141 2024-09-15T03:47:35.753 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.753 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.753 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.754 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:349 3.3s0 3:c7c3ef16:::smithi00132172-25:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:35.754 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:35.754 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 37.841826935999997, 2024-09-15T03:47:35.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005145406, 2024-09-15T03:47:35.754 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.754 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:35.754 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:35.754 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:35.754 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:35.754 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 349 2024-09-15T03:47:35.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.755 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:35.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:35.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838485+0000", 2024-09-15T03:47:35.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0620000000000001e-06 2024-09-15T03:47:35.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838494+0000", 2024-09-15T03:47:35.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3179999999999996e-06 2024-09-15T03:47:35.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:35.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3689999999999998e-05 2024-09-15T03:47:35.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838812+0000", 2024-09-15T03:47:35.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029437300000000001 2024-09-15T03:47:35.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838905+0000", 2024-09-15T03:47:35.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3196999999999996e-05 2024-09-15T03:47:35.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:35.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838914+0000", 2024-09-15T03:47:35.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0890000000000006e-06 2024-09-15T03:47:35.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840773+0000", 2024-09-15T03:47:35.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018584820000000001 2024-09-15T03:47:35.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:35.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840889+0000", 2024-09-15T03:47:35.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115986 2024-09-15T03:47:35.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842016+0000", 2024-09-15T03:47:35.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001126878 2024-09-15T03:47:35.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842320+0000", 2024-09-15T03:47:35.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030409799999999999 2024-09-15T03:47:35.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842360+0000", 2024-09-15T03:47:35.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0750000000000001e-05 2024-09-15T03:47:35.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842448+0000", 2024-09-15T03:47:35.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7842000000000004e-05 2024-09-15T03:47:35.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844580+0000", 2024-09-15T03:47:35.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021313719999999999 2024-09-15T03:47:35.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:35.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843574+0000", 2024-09-15T03:47:35.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99899468199999997 2024-09-15T03:47:35.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843627+0000", 2024-09-15T03:47:35.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2587000000000003e-05 2024-09-15T03:47:35.761 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.761 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.762 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.762 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:350 3.3s0 3:c7c3ef16:::smithi00132172-25:head [rollback 14] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:35.762 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:35.762 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 37.841475275000001, 2024-09-15T03:47:35.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0050383839999999, 2024-09-15T03:47:35.762 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.762 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:35.762 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:35.762 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:35.762 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:35.763 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 350 2024-09-15T03:47:35.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.763 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:35.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:35.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838834+0000", 2024-09-15T03:47:35.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0200000000000001e-07 2024-09-15T03:47:35.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838839+0000", 2024-09-15T03:47:35.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9790000000000001e-06 2024-09-15T03:47:35.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838852+0000", 2024-09-15T03:47:35.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3138e-05 2024-09-15T03:47:35.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838858+0000", 2024-09-15T03:47:35.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8320000000000002e-06 2024-09-15T03:47:35.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839003+0000", 2024-09-15T03:47:35.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014513500000000001 2024-09-15T03:47:35.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:35.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839011+0000", 2024-09-15T03:47:35.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7719999999999994e-06 2024-09-15T03:47:35.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842395+0000", 2024-09-15T03:47:35.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033840110000000001 2024-09-15T03:47:35.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:35.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842580+0000", 2024-09-15T03:47:35.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000185067 2024-09-15T03:47:35.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843394+0000", 2024-09-15T03:47:35.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081377299999999995 2024-09-15T03:47:35.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843510+0000", 2024-09-15T03:47:35.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0001165 2024-09-15T03:47:35.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843524+0000", 2024-09-15T03:47:35.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4022999999999999e-05 2024-09-15T03:47:35.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843579+0000", 2024-09-15T03:47:35.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4654999999999998e-05 2024-09-15T03:47:35.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844583+0000", 2024-09-15T03:47:35.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010042429999999999 2024-09-15T03:47:35.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:35.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843772+0000", 2024-09-15T03:47:35.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99918911200000005 2024-09-15T03:47:35.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843872+0000", 2024-09-15T03:47:35.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9341999999999999e-05 2024-09-15T03:47:35.770 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.770 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.770 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.770 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:351 3.3s0 3:c7c3ef16:::smithi00132172-25:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:35.770 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:35.770 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 37.841417642000003, 2024-09-15T03:47:35.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0051167679999999, 2024-09-15T03:47:35.770 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.770 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:35.771 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:35.771 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:35.771 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:35.771 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 351 2024-09-15T03:47:35.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.771 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:35.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:35.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:35.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0800000000000004e-07 2024-09-15T03:47:35.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838895+0000", 2024-09-15T03:47:35.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.241e-06 2024-09-15T03:47:35.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838904+0000", 2024-09-15T03:47:35.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9549999999999998e-06 2024-09-15T03:47:35.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838909+0000", 2024-09-15T03:47:35.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7869999999999998e-06 2024-09-15T03:47:35.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839045+0000", 2024-09-15T03:47:35.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013556399999999999 2024-09-15T03:47:35.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:35.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839052+0000", 2024-09-15T03:47:35.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8199999999999999e-06 2024-09-15T03:47:35.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843587+0000", 2024-09-15T03:47:35.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045349739999999998 2024-09-15T03:47:35.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:35.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843708+0000", 2024-09-15T03:47:35.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120776 2024-09-15T03:47:35.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844246+0000", 2024-09-15T03:47:35.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053838200000000001 2024-09-15T03:47:35.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844335+0000", 2024-09-15T03:47:35.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9240999999999995e-05 2024-09-15T03:47:35.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844347+0000", 2024-09-15T03:47:35.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2187e-05 2024-09-15T03:47:35.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844460+0000", 2024-09-15T03:47:35.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112571 2024-09-15T03:47:35.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844584+0000", 2024-09-15T03:47:35.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012403800000000001 2024-09-15T03:47:35.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:35.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843972+0000", 2024-09-15T03:47:35.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99938787299999998 2024-09-15T03:47:35.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844008+0000", 2024-09-15T03:47:35.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5750999999999997e-05 2024-09-15T03:47:35.777 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.777 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.777 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.778 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:35.778 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:35.778 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 37.834283691000003, 2024-09-15T03:47:35.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013010173, 2024-09-15T03:47:35.778 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.778 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.778 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:35.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:35.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846026+0000", 2024-09-15T03:47:35.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1230000000000001e-06 2024-09-15T03:47:35.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846047+0000", 2024-09-15T03:47:35.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0774000000000001e-05 2024-09-15T03:47:35.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846064+0000", 2024-09-15T03:47:35.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7560000000000001e-05 2024-09-15T03:47:35.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846072+0000", 2024-09-15T03:47:35.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5020000000000003e-06 2024-09-15T03:47:35.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846187+0000", 2024-09-15T03:47:35.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115717 2024-09-15T03:47:35.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846201+0000", 2024-09-15T03:47:35.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3329999999999999e-05 2024-09-15T03:47:35.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846303+0000", 2024-09-15T03:47:35.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101807 2024-09-15T03:47:35.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846312+0000", 2024-09-15T03:47:35.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5529999999999998e-06 2024-09-15T03:47:35.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846382+0000", 2024-09-15T03:47:35.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0146000000000002e-05 2024-09-15T03:47:35.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846520+0000", 2024-09-15T03:47:35.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013814099999999999 2024-09-15T03:47:35.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.859035+0000", 2024-09-15T03:47:35.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012514519999999999 2024-09-15T03:47:35.783 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.783 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.783 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.784 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:35.784 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:35.784 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 37.834186944000002, 2024-09-15T03:47:35.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013044694000000001, 2024-09-15T03:47:35.784 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.784 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.784 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:35.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:35.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846122+0000", 2024-09-15T03:47:35.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0399999999999996e-07 2024-09-15T03:47:35.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846174+0000", 2024-09-15T03:47:35.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1672999999999998e-05 2024-09-15T03:47:35.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846184+0000", 2024-09-15T03:47:35.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0514e-05 2024-09-15T03:47:35.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846190+0000", 2024-09-15T03:47:35.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2820000000000002e-06 2024-09-15T03:47:35.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846341+0000", 2024-09-15T03:47:35.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150803 2024-09-15T03:47:35.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846351+0000", 2024-09-15T03:47:35.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3899999999999999e-06 2024-09-15T03:47:35.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846450+0000", 2024-09-15T03:47:35.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9839000000000004e-05 2024-09-15T03:47:35.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846482+0000", 2024-09-15T03:47:35.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1089e-05 2024-09-15T03:47:35.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846615+0000", 2024-09-15T03:47:35.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013305899999999999 2024-09-15T03:47:35.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846634+0000", 2024-09-15T03:47:35.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8899999999999999e-05 2024-09-15T03:47:35.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.859166+0000", 2024-09-15T03:47:35.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012532641000000001 2024-09-15T03:47:35.790 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.790 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.790 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.791 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 60/28 ECSubWrite(tid=73, reqid=client.4521.0:424, at_version=60'41, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:35.791 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:35.791 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.779878097000001, 2024-09-15T03:47:35.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011124821, 2024-09-15T03:47:35.791 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.791 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.791 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:35.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:35.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900431+0000", 2024-09-15T03:47:35.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2900000000000003e-07 2024-09-15T03:47:35.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900451+0000", 2024-09-15T03:47:35.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0066999999999999e-05 2024-09-15T03:47:35.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900466+0000", 2024-09-15T03:47:35.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5062000000000001e-05 2024-09-15T03:47:35.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900474+0000", 2024-09-15T03:47:35.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6310000000000003e-06 2024-09-15T03:47:35.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900524+0000", 2024-09-15T03:47:35.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0037000000000002e-05 2024-09-15T03:47:35.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900535+0000", 2024-09-15T03:47:35.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1209000000000001e-05 2024-09-15T03:47:35.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900659+0000", 2024-09-15T03:47:35.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012441000000000001 2024-09-15T03:47:35.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900971+0000", 2024-09-15T03:47:35.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031183799999999999 2024-09-15T03:47:35.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901144+0000", 2024-09-15T03:47:35.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172261 2024-09-15T03:47:35.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901734+0000", 2024-09-15T03:47:35.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059069700000000005 2024-09-15T03:47:35.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.911555+0000", 2024-09-15T03:47:35.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0098208800000000006 2024-09-15T03:47:35.797 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.797 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.797 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.798 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 60/28 ECSubWrite(tid=74, reqid=client.4521.0:425, at_version=60'42, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:35.798 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:06.901240+0000", 2024-09-15T03:47:35.798 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.779068517999999, 2024-09-15T03:47:35.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010429678, 2024-09-15T03:47:35.798 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.798 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.798 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901240+0000", 2024-09-15T03:47:35.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901240+0000", 2024-09-15T03:47:35.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901241+0000", 2024-09-15T03:47:35.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7999999999999993e-07 2024-09-15T03:47:35.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901273+0000", 2024-09-15T03:47:35.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2087e-05 2024-09-15T03:47:35.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901302+0000", 2024-09-15T03:47:35.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8705e-05 2024-09-15T03:47:35.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901311+0000", 2024-09-15T03:47:35.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9190000000000005e-06 2024-09-15T03:47:35.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901797+0000", 2024-09-15T03:47:35.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048606499999999999 2024-09-15T03:47:35.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901808+0000", 2024-09-15T03:47:35.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1075e-05 2024-09-15T03:47:35.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901934+0000", 2024-09-15T03:47:35.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012674299999999999 2024-09-15T03:47:35.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901964+0000", 2024-09-15T03:47:35.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9833999999999999e-05 2024-09-15T03:47:35.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.902006+0000", 2024-09-15T03:47:35.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1823999999999998e-05 2024-09-15T03:47:35.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.902020+0000", 2024-09-15T03:47:35.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.363e-05 2024-09-15T03:47:35.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.911670+0000", 2024-09-15T03:47:35.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096498160000000003 2024-09-15T03:47:35.804 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.804 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.804 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.804 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPush(2.2 88/87 [PushOp(2:5cea7035:::rbd_info:head, version: 24'2, data_included: [0~19], data_size: 19, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:5cea7035:::rbd_info:head@24'2, size: 19, copy_subset: [0~19], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:19, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-15T03:47:35.804 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.846903+0000", 2024-09-15T03:47:35.804 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.83340497099999999, 2024-09-15T03:47:35.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013354107, 2024-09-15T03:47:35.805 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.805 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:35.805 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846903+0000", 2024-09-15T03:47:35.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846903+0000", 2024-09-15T03:47:35.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846904+0000", 2024-09-15T03:47:35.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4099999999999998e-07 2024-09-15T03:47:35.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846920+0000", 2024-09-15T03:47:35.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6061000000000001e-05 2024-09-15T03:47:35.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846945+0000", 2024-09-15T03:47:35.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4519e-05 2024-09-15T03:47:35.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846958+0000", 2024-09-15T03:47:35.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3546999999999999e-05 2024-09-15T03:47:35.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.847077+0000", 2024-09-15T03:47:35.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011864 2024-09-15T03:47:35.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for flush", 2024-09-15T03:47:35.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.847094+0000", 2024-09-15T03:47:35.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7017e-05 2024-09-15T03:47:35.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.859991+0000", 2024-09-15T03:47:35.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012897136 2024-09-15T03:47:35.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:35.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.860001+0000", 2024-09-15T03:47:35.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5089999999999999e-06 2024-09-15T03:47:35.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.860258+0000", 2024-09-15T03:47:35.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000256937 2024-09-15T03:47:35.809 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.809 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.809 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.809 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 88/87 ECSubWrite(tid=200, reqid=client.4521.0:737, at_version=88'109, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:35.809 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.883279+0000", 2024-09-15T03:47:35.809 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.79702904799999996, 2024-09-15T03:47:35.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010263239, 2024-09-15T03:47:35.810 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.810 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.810 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883279+0000", 2024-09-15T03:47:35.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883279+0000", 2024-09-15T03:47:35.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883280+0000", 2024-09-15T03:47:35.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5799999999999999e-07 2024-09-15T03:47:35.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883328+0000", 2024-09-15T03:47:35.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8198999999999997e-05 2024-09-15T03:47:35.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883341+0000", 2024-09-15T03:47:35.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2557e-05 2024-09-15T03:47:35.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883347+0000", 2024-09-15T03:47:35.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5359999999999998e-06 2024-09-15T03:47:35.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883406+0000", 2024-09-15T03:47:35.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8971999999999997e-05 2024-09-15T03:47:35.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883417+0000", 2024-09-15T03:47:35.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0346000000000001e-05 2024-09-15T03:47:35.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883682+0000", 2024-09-15T03:47:35.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026550500000000003 2024-09-15T03:47:35.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883705+0000", 2024-09-15T03:47:35.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2779e-05 2024-09-15T03:47:35.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883736+0000", 2024-09-15T03:47:35.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0626999999999997e-05 2024-09-15T03:47:35.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883752+0000", 2024-09-15T03:47:35.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6180000000000001e-05 2024-09-15T03:47:35.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.893543+0000", 2024-09-15T03:47:35.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0097908800000000001 2024-09-15T03:47:35.815 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.815 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.815 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.815 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 88/87 ECSubWrite(tid=210, reqid=client.4521.0:740, at_version=88'113, trim_to=0'0, roll_forward_to=88'111))", 2024-09-15T03:47:35.815 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.915188+0000", 2024-09-15T03:47:35.815 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.76512084599999997, 2024-09-15T03:47:35.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012691618999999999, 2024-09-15T03:47:35.816 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:35.816 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:35.816 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:35.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:35.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915188+0000", 2024-09-15T03:47:35.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:35.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915188+0000", 2024-09-15T03:47:35.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:35.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:35.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915188+0000", 2024-09-15T03:47:35.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3399999999999999e-07 2024-09-15T03:47:35.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:35.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915216+0000", 2024-09-15T03:47:35.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8067000000000001e-05 2024-09-15T03:47:35.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:35.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915233+0000", 2024-09-15T03:47:35.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.683e-05 2024-09-15T03:47:35.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:35.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915239+0000", 2024-09-15T03:47:35.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5720000000000002e-06 2024-09-15T03:47:35.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:35.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915313+0000", 2024-09-15T03:47:35.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4066000000000005e-05 2024-09-15T03:47:35.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:35.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915325+0000", 2024-09-15T03:47:35.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.252e-05 2024-09-15T03:47:35.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:35.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915603+0000", 2024-09-15T03:47:35.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027800699999999998 2024-09-15T03:47:35.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:35.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915645+0000", 2024-09-15T03:47:35.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1777000000000003e-05 2024-09-15T03:47:35.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:35.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915687+0000", 2024-09-15T03:47:35.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2181000000000002e-05 2024-09-15T03:47:35.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:35.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.917623+0000", 2024-09-15T03:47:35.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019357210000000001 2024-09-15T03:47:35.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:35.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:35.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:35.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.927879+0000", 2024-09-15T03:47:35.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010256243999999999 2024-09-15T03:47:35.821 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.821 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.821 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.821 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:35.821 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:35.821 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:35.823 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:47:35.823 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:35.822+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:35.837 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-15T03:47:35.838 INFO:tasks.rados.rados.0.smithi001.stdout:295: done (7 left) 2024-09-15T03:47:35.838 INFO:tasks.rados.rados.0.smithi001.stdout:296: done (6 left) 2024-09-15T03:47:35.838 INFO:tasks.rados.rados.0.smithi001.stdout:297: done (5 left) 2024-09-15T03:47:35.839 INFO:tasks.rados.rados.0.smithi001.stdout:298: done (4 left) 2024-09-15T03:47:35.839 INFO:tasks.rados.rados.0.smithi001.stdout:299: done (3 left) 2024-09-15T03:47:35.839 INFO:tasks.rados.rados.0.smithi001.stdout:300: done (2 left) 2024-09-15T03:47:35.839 INFO:tasks.rados.rados.0.smithi001.stdout:301: done (1 left) 2024-09-15T03:47:35.839 INFO:tasks.rados.rados.0.smithi001.stdout:302: done (0 left) 2024-09-15T03:47:35.839 INFO:tasks.rados.rados.0.smithi001.stdout:303: rmattr oid 42 current snap is 34 2024-09-15T03:47:35.839 INFO:tasks.rados.rados.0.smithi001.stdout:303: done (0 left) 2024-09-15T03:47:35.839 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 18 current snap is 34 2024-09-15T03:47:35.839 INFO:tasks.rados.rados.0.smithi001.stdout:304: seq_num 120 ranges {196608=475136,671744=655360,1327104=622592,1949696=278528} 2024-09-15T03:47:35.842 INFO:tasks.rados.rados.0.smithi001.stdout:304: writing smithi00132172-18 from 196608 to 671744 tid 1 2024-09-15T03:47:35.847 INFO:tasks.rados.rados.0.smithi001.stdout:304: writing smithi00132172-18 from 671744 to 1327104 tid 2 2024-09-15T03:47:35.851 INFO:tasks.rados.rados.0.smithi001.stdout:304: writing smithi00132172-18 from 1327104 to 1949696 tid 3 2024-09-15T03:47:35.852 INFO:tasks.rados.rados.0.smithi001.stdout:304: writing smithi00132172-18 from 1949696 to 2228224 tid 4 2024-09-15T03:47:35.852 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 11 current snap is 34 2024-09-15T03:47:35.852 INFO:tasks.rados.rados.0.smithi001.stdout:305: seq_num 121 ranges {1507328=491520,1998848=802816,2801664=720896,3522560=32768} 2024-09-15T03:47:35.863 INFO:tasks.rados.rados.0.smithi001.stdout:305: writing smithi00132172-11 from 1507328 to 1998848 tid 1 2024-09-15T03:47:35.868 INFO:tasks.rados.rados.0.smithi001.stdout:305: writing smithi00132172-11 from 1998848 to 2801664 tid 2 2024-09-15T03:47:35.872 INFO:tasks.rados.rados.0.smithi001.stdout:305: writing smithi00132172-11 from 2801664 to 3522560 tid 3 2024-09-15T03:47:35.872 INFO:tasks.rados.rados.0.smithi001.stdout:305: writing smithi00132172-11 from 3522560 to 3555328 tid 4 2024-09-15T03:47:35.873 INFO:tasks.rados.rados.0.smithi001.stdout:306: snap_create 2024-09-15T03:47:35.873 INFO:tasks.rados.rados.0.smithi001.stdout:305: finishing write tid 1 to smithi00132172-11 2024-09-15T03:47:35.873 INFO:tasks.rados.rados.0.smithi001.stdout:305: finishing write tid 2 to smithi00132172-11 2024-09-15T03:47:35.876 INFO:tasks.rados.rados.0.smithi001.stdout:305: finishing write tid 3 to smithi00132172-11 2024-09-15T03:47:35.876 INFO:tasks.rados.rados.0.smithi001.stdout:305: finishing write tid 4 to smithi00132172-11 2024-09-15T03:47:35.877 INFO:tasks.rados.rados.0.smithi001.stdout:305: finishing write tid 5 to smithi00132172-11 2024-09-15T03:47:35.878 INFO:tasks.rados.rados.0.smithi001.stdout:305: finishing write tid 6 to smithi00132172-11 2024-09-15T03:47:35.878 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 11 v 79 (ObjNum 121 snap 34 seq_num 121) dirty exists 2024-09-15T03:47:35.878 INFO:tasks.rados.rados.0.smithi001.stdout:305: left oid 11 (ObjNum 121 snap 34 seq_num 121) 2024-09-15T03:47:35.924 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:35.923+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:35.966 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:35.966 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:35.966 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:35.966 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:35.974 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:47:36.024 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:36.024+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:36.075 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:36.076 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:36.076 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:36.076 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:36.076 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:36.084 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:47:36.126 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:36.125+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:36.187 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:197 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [delete] snapc 3=[3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e34)", 2024-09-15T03:47:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 55.072674706999997, 2024-09-15T03:47:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021934704999999999, 2024-09-15T03:47:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197 2024-09-15T03:47:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112266+0000", 2024-09-15T03:47:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5000000000000002e-07 2024-09-15T03:47:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112269+0000", 2024-09-15T03:47:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.066e-06 2024-09-15T03:47:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112276+0000", 2024-09-15T03:47:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-09-15T03:47:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112281+0000", 2024-09-15T03:47:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3970000000000004e-06 2024-09-15T03:47:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123281+0000", 2024-09-15T03:47:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011000575 2024-09-15T03:47:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123395+0000", 2024-09-15T03:47:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113398 2024-09-15T03:47:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123963+0000", 2024-09-15T03:47:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000568101 2024-09-15T03:47:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124220+0000", 2024-09-15T03:47:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025704900000000003 2024-09-15T03:47:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124229+0000", 2024-09-15T03:47:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4860000000000003e-06 2024-09-15T03:47:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124277+0000", 2024-09-15T03:47:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7574000000000002e-05 2024-09-15T03:47:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124312+0000", 2024-09-15T03:47:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5256000000000003e-05 2024-09-15T03:47:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125716+0000", 2024-09-15T03:47:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014043720000000001 2024-09-15T03:47:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.134200+0000", 2024-09-15T03:47:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084835120000000003 2024-09-15T03:47:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:220 3.bs0 3:d99ddedc:::smithi00132172-44:head [copy-from ver 22] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.057295037000003, 2024-09-15T03:47:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024273429999999999, 2024-09-15T03:47:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 220 2024-09-15T03:47:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127647+0000", 2024-09-15T03:47:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1799999999999999e-06 2024-09-15T03:47:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127656+0000", 2024-09-15T03:47:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3759999999999997e-06 2024-09-15T03:47:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127674+0000", 2024-09-15T03:47:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7374e-05 2024-09-15T03:47:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127680+0000", 2024-09-15T03:47:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6949999999999997e-06 2024-09-15T03:47:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127736+0000", 2024-09-15T03:47:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6011999999999998e-05 2024-09-15T03:47:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127744+0000", 2024-09-15T03:47:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2870000000000004e-06 2024-09-15T03:47:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130338+0000", 2024-09-15T03:47:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025940249999999998 2024-09-15T03:47:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130423+0000", 2024-09-15T03:47:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5303000000000006e-05 2024-09-15T03:47:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.147840+0000", 2024-09-15T03:47:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017416580000000001 2024-09-15T03:47:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148318+0000", 2024-09-15T03:47:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047822899999999999 2024-09-15T03:47:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148329+0000", 2024-09-15T03:47:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0611000000000001e-05 2024-09-15T03:47:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148488+0000", 2024-09-15T03:47:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015938399999999999 2024-09-15T03:47:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148870+0000", 2024-09-15T03:47:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000382405 2024-09-15T03:47:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.151092+0000", 2024-09-15T03:47:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022212529999999999 2024-09-15T03:47:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.151918+0000", 2024-09-15T03:47:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082671600000000002 2024-09-15T03:47:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:221 3.bs0 3:d99ddedc:::smithi00132172-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.057250881999998, 2024-09-15T03:47:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021194474000000001, 2024-09-15T03:47:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 221 2024-09-15T03:47:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127690+0000", 2024-09-15T03:47:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:47:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127692+0000", 2024-09-15T03:47:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6589999999999999e-06 2024-09-15T03:47:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127700+0000", 2024-09-15T03:47:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4769999999999997e-06 2024-09-15T03:47:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127704+0000", 2024-09-15T03:47:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3819999999999997e-06 2024-09-15T03:47:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127805+0000", 2024-09-15T03:47:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100639 2024-09-15T03:47:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127809+0000", 2024-09-15T03:47:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8870000000000002e-06 2024-09-15T03:47:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130527+0000", 2024-09-15T03:47:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002718579 2024-09-15T03:47:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130558+0000", 2024-09-15T03:47:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0827999999999997e-05 2024-09-15T03:47:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148546+0000", 2024-09-15T03:47:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017988356 2024-09-15T03:47:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148683+0000", 2024-09-15T03:47:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013650899999999999 2024-09-15T03:47:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148884+0000", 2024-09-15T03:47:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020061499999999999 2024-09-15T03:47:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:357 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.335899347999998, 2024-09-15T03:47:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.02017917, 2024-09-15T03:47:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 357 2024-09-15T03:47:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849043+0000", 2024-09-15T03:47:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.295e-06 2024-09-15T03:47:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849051+0000", 2024-09-15T03:47:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4079999999999998e-06 2024-09-15T03:47:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849069+0000", 2024-09-15T03:47:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7462999999999999e-05 2024-09-15T03:47:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849077+0000", 2024-09-15T03:47:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8869999999999992e-06 2024-09-15T03:47:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849194+0000", 2024-09-15T03:47:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011723899999999999 2024-09-15T03:47:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849296+0000", 2024-09-15T03:47:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102011 2024-09-15T03:47:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861106+0000", 2024-09-15T03:47:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0118097610000001 2024-09-15T03:47:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861222+0000", 2024-09-15T03:47:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116561 2024-09-15T03:47:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.869220+0000", 2024-09-15T03:47:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007997545 2024-09-15T03:47:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:358 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.335844401999999, 2024-09-15T03:47:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0228638050000001, 2024-09-15T03:47:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 358 2024-09-15T03:47:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4099999999999997e-07 2024-09-15T03:47:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849101+0000", 2024-09-15T03:47:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8180000000000003e-06 2024-09-15T03:47:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849110+0000", 2024-09-15T03:47:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1409999999999994e-06 2024-09-15T03:47:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849116+0000", 2024-09-15T03:47:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2390000000000002e-06 2024-09-15T03:47:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849326+0000", 2024-09-15T03:47:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000210029 2024-09-15T03:47:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849421+0000", 2024-09-15T03:47:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5775999999999999e-05 2024-09-15T03:47:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861647+0000", 2024-09-15T03:47:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0122257139999999 2024-09-15T03:47:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861777+0000", 2024-09-15T03:47:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130087 2024-09-15T03:47:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.871959+0000", 2024-09-15T03:47:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010182160000000001 2024-09-15T03:47:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:359 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.33581925, 2024-09-15T03:47:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0240468149999999, 2024-09-15T03:47:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 359 2024-09-15T03:47:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:47:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849124+0000", 2024-09-15T03:47:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2679999999999999e-06 2024-09-15T03:47:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849132+0000", 2024-09-15T03:47:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4030000000000001e-06 2024-09-15T03:47:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849136+0000", 2024-09-15T03:47:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2289999999999997e-06 2024-09-15T03:47:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849446+0000", 2024-09-15T03:47:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031043299999999999 2024-09-15T03:47:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849553+0000", 2024-09-15T03:47:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106415 2024-09-15T03:47:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.863404+0000", 2024-09-15T03:47:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0138511029999999 2024-09-15T03:47:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.863475+0000", 2024-09-15T03:47:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0814000000000003e-05 2024-09-15T03:47:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.873168+0000", 2024-09-15T03:47:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00969274 2024-09-15T03:47:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:474 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~786432] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.309894575000001, 2024-09-15T03:47:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98604943700000003, 2024-09-15T03:47:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 474 2024-09-15T03:47:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875048+0000", 2024-09-15T03:47:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1600000000000001e-06 2024-09-15T03:47:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875844+0000", 2024-09-15T03:47:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079686299999999998 2024-09-15T03:47:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875857+0000", 2024-09-15T03:47:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2889e-05 2024-09-15T03:47:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875864+0000", 2024-09-15T03:47:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6139999999999997e-06 2024-09-15T03:47:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875916+0000", 2024-09-15T03:47:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1992e-05 2024-09-15T03:47:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875918+0000", 2024-09-15T03:47:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3300000000000001e-06 2024-09-15T03:47:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.845918+0000", 2024-09-15T03:47:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96999959199999997 2024-09-15T03:47:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.846229+0000", 2024-09-15T03:47:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000311601 2024-09-15T03:47:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848445+0000", 2024-09-15T03:47:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022160499999999998 2024-09-15T03:47:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848880+0000", 2024-09-15T03:47:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043432600000000002 2024-09-15T03:47:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848952+0000", 2024-09-15T03:47:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2509000000000007e-05 2024-09-15T03:47:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.849311+0000", 2024-09-15T03:47:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000358611 2024-09-15T03:47:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851131+0000", 2024-09-15T03:47:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001820115 2024-09-15T03:47:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861048+0000", 2024-09-15T03:47:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099164830000000002 2024-09-15T03:47:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861095+0000", 2024-09-15T03:47:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7302000000000002e-05 2024-09-15T03:47:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:475 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~491520] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.307128104, 2024-09-15T03:47:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98407319000000004, 2024-09-15T03:47:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 475 2024-09-15T03:47:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877813+0000", 2024-09-15T03:47:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8800000000000002e-07 2024-09-15T03:47:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878260+0000", 2024-09-15T03:47:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044732099999999998 2024-09-15T03:47:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878270+0000", 2024-09-15T03:47:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9799999999999993e-06 2024-09-15T03:47:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878276+0000", 2024-09-15T03:47:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2269999999999998e-06 2024-09-15T03:47:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878313+0000", 2024-09-15T03:47:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6570999999999999e-05 2024-09-15T03:47:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878315+0000", 2024-09-15T03:47:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8729999999999999e-06 2024-09-15T03:47:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851316+0000", 2024-09-15T03:47:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97300154800000005 2024-09-15T03:47:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851440+0000", 2024-09-15T03:47:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124287 2024-09-15T03:47:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852732+0000", 2024-09-15T03:47:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001291174 2024-09-15T03:47:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852991+0000", 2024-09-15T03:47:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025905299999999997 2024-09-15T03:47:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853003+0000", 2024-09-15T03:47:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2908000000000001e-05 2024-09-15T03:47:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853104+0000", 2024-09-15T03:47:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010011800000000001 2024-09-15T03:47:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854342+0000", 2024-09-15T03:47:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012380189999999999 2024-09-15T03:47:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861206+0000", 2024-09-15T03:47:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068648499999999996 2024-09-15T03:47:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861885+0000", 2024-09-15T03:47:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00067857300000000002 2024-09-15T03:47:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:476 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~278528] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.305501091, 2024-09-15T03:47:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98268067299999995, 2024-09-15T03:47:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 476 2024-09-15T03:47:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.13e-07 2024-09-15T03:47:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879706+0000", 2024-09-15T03:47:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026656100000000001 2024-09-15T03:47:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879715+0000", 2024-09-15T03:47:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9330000000000007e-06 2024-09-15T03:47:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879720+0000", 2024-09-15T03:47:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0880000000000002e-06 2024-09-15T03:47:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879743+0000", 2024-09-15T03:47:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2654e-05 2024-09-15T03:47:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879743+0000", 2024-09-15T03:47:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.99e-07 2024-09-15T03:47:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853170+0000", 2024-09-15T03:47:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97342687900000002 2024-09-15T03:47:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853297+0000", 2024-09-15T03:47:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127157 2024-09-15T03:47:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854150+0000", 2024-09-15T03:47:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085288900000000001 2024-09-15T03:47:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854292+0000", 2024-09-15T03:47:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014199300000000001 2024-09-15T03:47:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854559+0000", 2024-09-15T03:47:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026668999999999998 2024-09-15T03:47:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855998+0000", 2024-09-15T03:47:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001438714 2024-09-15T03:47:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856547+0000", 2024-09-15T03:47:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054880399999999996 2024-09-15T03:47:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861858+0000", 2024-09-15T03:47:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053110320000000003 2024-09-15T03:47:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.862120+0000", 2024-09-15T03:47:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026196699999999999 2024-09-15T03:47:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:477 3.bs0 3:d52faf55:::smithi00132172-40:head [setxattr _header (57) in=7b] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.305213154, 2024-09-15T03:47:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98562198000000001, 2024-09-15T03:47:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 477 2024-09-15T03:47:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2800000000000002e-07 2024-09-15T03:47:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879730+0000", 2024-09-15T03:47:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0589999999999998e-06 2024-09-15T03:47:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879737+0000", 2024-09-15T03:47:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0419999999999997e-06 2024-09-15T03:47:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879741+0000", 2024-09-15T03:47:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9960000000000004e-06 2024-09-15T03:47:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879756+0000", 2024-09-15T03:47:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4681999999999999e-05 2024-09-15T03:47:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879756+0000", 2024-09-15T03:47:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.22e-07 2024-09-15T03:47:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854590+0000", 2024-09-15T03:47:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97483319000000002 2024-09-15T03:47:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854718+0000", 2024-09-15T03:47:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012842100000000001 2024-09-15T03:47:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855416+0000", 2024-09-15T03:47:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069759200000000003 2024-09-15T03:47:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855551+0000", 2024-09-15T03:47:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013517600000000001 2024-09-15T03:47:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856002+0000", 2024-09-15T03:47:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045169399999999999 2024-09-15T03:47:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856065+0000", 2024-09-15T03:47:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3028999999999996e-05 2024-09-15T03:47:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856554+0000", 2024-09-15T03:47:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000488486 2024-09-15T03:47:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865278+0000", 2024-09-15T03:47:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087241929999999999 2024-09-15T03:47:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865349+0000", 2024-09-15T03:47:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0669999999999999e-05 2024-09-15T03:47:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:478 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.305194359000001, 2024-09-15T03:47:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99471090500000003, 2024-09-15T03:47:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 478 2024-09-15T03:47:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2599999999999998e-07 2024-09-15T03:47:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879748+0000", 2024-09-15T03:47:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9609999999999999e-06 2024-09-15T03:47:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879755+0000", 2024-09-15T03:47:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1879999999999996e-06 2024-09-15T03:47:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879760+0000", 2024-09-15T03:47:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2560000000000002e-06 2024-09-15T03:47:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879797+0000", 2024-09-15T03:47:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6452999999999998e-05 2024-09-15T03:47:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879799+0000", 2024-09-15T03:47:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6619999999999999e-06 2024-09-15T03:47:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855637+0000", 2024-09-15T03:47:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97583850999999999 2024-09-15T03:47:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855763+0000", 2024-09-15T03:47:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012612699999999999 2024-09-15T03:47:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865707+0000", 2024-09-15T03:47:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099439660000000003 2024-09-15T03:47:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865803+0000", 2024-09-15T03:47:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5946999999999995e-05 2024-09-15T03:47:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.874456+0000", 2024-09-15T03:47:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0086534090000000008 2024-09-15T03:47:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.264 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:36.264 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.264 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:486 3.bs0 3:df8b18a3:::smithi00132172-17:head [delete] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e64)", 2024-09-15T03:47:36.264 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:36.264 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.302381879999999, 2024-09-15T03:47:36.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98362617100000005, 2024-09-15T03:47:36.264 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:36.264 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:36.264 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:36.264 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:36.265 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.265 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 486 2024-09-15T03:47:36.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.265 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:36.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:36.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:36.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:36.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:36.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:36.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882559+0000", 2024-09-15T03:47:36.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0359999999999999e-06 2024-09-15T03:47:36.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:36.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882564+0000", 2024-09-15T03:47:36.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6530000000000002e-06 2024-09-15T03:47:36.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:36.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882574+0000", 2024-09-15T03:47:36.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0464e-05 2024-09-15T03:47:36.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882578+0000", 2024-09-15T03:47:36.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2459999999999997e-06 2024-09-15T03:47:36.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882624+0000", 2024-09-15T03:47:36.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5698000000000002e-05 2024-09-15T03:47:36.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:36.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882627+0000", 2024-09-15T03:47:36.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3350000000000001e-06 2024-09-15T03:47:36.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856574+0000", 2024-09-15T03:47:36.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97394739399999997 2024-09-15T03:47:36.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:36.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856853+0000", 2024-09-15T03:47:36.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027918399999999998 2024-09-15T03:47:36.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:36.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.866184+0000", 2024-09-15T03:47:36.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093311609999999993 2024-09-15T03:47:36.269 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.270 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:36.270 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.270 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:495 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [copy-from ver 51] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-15T03:47:36.270 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:36.270 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 24.296480831, 2024-09-15T03:47:36.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023555178, 2024-09-15T03:47:36.270 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:36.270 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:36.270 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:36.270 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:36.270 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.271 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 495 2024-09-15T03:47:36.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.271 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:36.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:36.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:36.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:36.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:36.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:36.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888461+0000", 2024-09-15T03:47:36.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.719e-06 2024-09-15T03:47:36.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:36.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888471+0000", 2024-09-15T03:47:36.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0161e-05 2024-09-15T03:47:36.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:36.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888500+0000", 2024-09-15T03:47:36.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8989e-05 2024-09-15T03:47:36.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888508+0000", 2024-09-15T03:47:36.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5299999999999999e-06 2024-09-15T03:47:36.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888554+0000", 2024-09-15T03:47:36.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6489000000000002e-05 2024-09-15T03:47:36.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:36.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888809+0000", 2024-09-15T03:47:36.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000255467 2024-09-15T03:47:36.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:36.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.905724+0000", 2024-09-15T03:47:36.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016914348999999999 2024-09-15T03:47:36.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:36.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906244+0000", 2024-09-15T03:47:36.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052054900000000003 2024-09-15T03:47:36.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:36.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906262+0000", 2024-09-15T03:47:36.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7499999999999998e-05 2024-09-15T03:47:36.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:36.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.907367+0000", 2024-09-15T03:47:36.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011049460000000001 2024-09-15T03:47:36.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:36.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.907384+0000", 2024-09-15T03:47:36.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7000999999999999e-05 2024-09-15T03:47:36.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:36.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.911965+0000", 2024-09-15T03:47:36.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045809470000000001 2024-09-15T03:47:36.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:36.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.912014+0000", 2024-09-15T03:47:36.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9530999999999997e-05 2024-09-15T03:47:36.277 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.277 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:36.277 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.277 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:496 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-15T03:47:36.277 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.888524+0000", 2024-09-15T03:47:36.277 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 24.296415739, 2024-09-15T03:47:36.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01823953, 2024-09-15T03:47:36.277 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:36.277 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:36.277 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:36.278 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:36.278 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.278 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 496 2024-09-15T03:47:36.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.278 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:36.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:36.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888524+0000", 2024-09-15T03:47:36.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:36.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888524+0000", 2024-09-15T03:47:36.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:36.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888525+0000", 2024-09-15T03:47:36.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2699999999999999e-07 2024-09-15T03:47:36.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:36.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888529+0000", 2024-09-15T03:47:36.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7840000000000001e-06 2024-09-15T03:47:36.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:36.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888540+0000", 2024-09-15T03:47:36.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1477e-05 2024-09-15T03:47:36.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888546+0000", 2024-09-15T03:47:36.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4990000000000002e-06 2024-09-15T03:47:36.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888971+0000", 2024-09-15T03:47:36.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042491300000000002 2024-09-15T03:47:36.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:36.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.889024+0000", 2024-09-15T03:47:36.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3046e-05 2024-09-15T03:47:36.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906517+0000", 2024-09-15T03:47:36.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017492909000000001 2024-09-15T03:47:36.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:36.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906638+0000", 2024-09-15T03:47:36.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012138000000000001 2024-09-15T03:47:36.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:36.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906764+0000", 2024-09-15T03:47:36.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125795 2024-09-15T03:47:36.282 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.282 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:36.282 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.283 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:689 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~5144576,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:36.283 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.088846+0000", 2024-09-15T03:47:36.283 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.0960944110000002, 2024-09-15T03:47:36.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020311780000000002, 2024-09-15T03:47:36.283 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:36.283 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:36.283 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:36.283 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:36.283 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.283 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 689 2024-09-15T03:47:36.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.284 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:36.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:36.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088846+0000", 2024-09-15T03:47:36.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:36.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088846+0000", 2024-09-15T03:47:36.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:36.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088847+0000", 2024-09-15T03:47:36.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9300000000000006e-07 2024-09-15T03:47:36.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:36.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088852+0000", 2024-09-15T03:47:36.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1850000000000003e-06 2024-09-15T03:47:36.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:36.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088863+0000", 2024-09-15T03:47:36.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1418e-05 2024-09-15T03:47:36.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088873+0000", 2024-09-15T03:47:36.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4020000000000008e-06 2024-09-15T03:47:36.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.089560+0000", 2024-09-15T03:47:36.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068774899999999998 2024-09-15T03:47:36.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:36.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090073+0000", 2024-09-15T03:47:36.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051222799999999999 2024-09-15T03:47:36.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:36.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.109157+0000", 2024-09-15T03:47:36.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019084805 2024-09-15T03:47:36.287 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.287 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:36.287 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.287 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:690 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~5144576,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:36.287 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:36.288 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.0960612730000001, 2024-09-15T03:47:36.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025340499999999998, 2024-09-15T03:47:36.288 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:36.288 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:36.288 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:36.288 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:36.288 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.288 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 690 2024-09-15T03:47:36.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.288 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:36.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:36.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:36.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:36.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:36.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:36.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:36.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7999999999999996e-07 2024-09-15T03:47:36.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:36.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088882+0000", 2024-09-15T03:47:36.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1839999999999999e-06 2024-09-15T03:47:36.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:36.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088892+0000", 2024-09-15T03:47:36.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3540000000000006e-06 2024-09-15T03:47:36.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088896+0000", 2024-09-15T03:47:36.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5210000000000004e-06 2024-09-15T03:47:36.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.093968+0000", 2024-09-15T03:47:36.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050715170000000002 2024-09-15T03:47:36.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:36.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.094098+0000", 2024-09-15T03:47:36.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013070000000000001 2024-09-15T03:47:36.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:36.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.114219+0000", 2024-09-15T03:47:36.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020120743999999999 2024-09-15T03:47:36.292 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.292 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:36.292 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.292 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:691 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~5144576,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:36.292 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.088900+0000", 2024-09-15T03:47:36.292 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.0960395299999997, 2024-09-15T03:47:36.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031407274999999998, 2024-09-15T03:47:36.292 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:36.292 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:36.293 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:36.293 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:36.293 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.293 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 691 2024-09-15T03:47:36.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.293 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:36.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:36.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088900+0000", 2024-09-15T03:47:36.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:36.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088900+0000", 2024-09-15T03:47:36.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:36.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088901+0000", 2024-09-15T03:47:36.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9099999999999999e-07 2024-09-15T03:47:36.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:36.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088904+0000", 2024-09-15T03:47:36.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0299999999999998e-06 2024-09-15T03:47:36.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:36.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088911+0000", 2024-09-15T03:47:36.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.708e-06 2024-09-15T03:47:36.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088915+0000", 2024-09-15T03:47:36.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0579999999999999e-06 2024-09-15T03:47:36.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.096143+0000", 2024-09-15T03:47:36.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007228099 2024-09-15T03:47:36.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:36.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.096202+0000", 2024-09-15T03:47:36.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9667999999999999e-05 2024-09-15T03:47:36.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:36.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.120308+0000", 2024-09-15T03:47:36.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024105320999999999 2024-09-15T03:47:36.297 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.297 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:36.297 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.297 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(osd.0.17:2 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [assert-version v51,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:36.297 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.090507+0000", 2024-09-15T03:47:36.297 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.0944330630000003, 2024-09-15T03:47:36.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025010186, 2024-09-15T03:47:36.297 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:36.297 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:36.298 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:36.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:36.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090507+0000", 2024-09-15T03:47:36.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:36.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090507+0000", 2024-09-15T03:47:36.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:36.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090508+0000", 2024-09-15T03:47:36.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.176e-06 2024-09-15T03:47:36.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:36.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090512+0000", 2024-09-15T03:47:36.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2989999999999998e-06 2024-09-15T03:47:36.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:36.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090522+0000", 2024-09-15T03:47:36.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6549999999999999e-06 2024-09-15T03:47:36.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090526+0000", 2024-09-15T03:47:36.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8500000000000004e-06 2024-09-15T03:47:36.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.094385+0000", 2024-09-15T03:47:36.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038591670000000002 2024-09-15T03:47:36.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:36.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.094541+0000", 2024-09-15T03:47:36.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015590999999999999 2024-09-15T03:47:36.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:36.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.115517+0000", 2024-09-15T03:47:36.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020976129 2024-09-15T03:47:36.301 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.301 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:36.301 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.301 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:749 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~737280,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-15T03:47:36.301 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.935286+0000", 2024-09-15T03:47:36.301 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.2496535230000001, 2024-09-15T03:47:36.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021703044000000001, 2024-09-15T03:47:36.301 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:36.302 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:36.302 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:36.302 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:36.302 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.302 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 749 2024-09-15T03:47:36.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.302 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:36.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:36.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935286+0000", 2024-09-15T03:47:36.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:36.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935286+0000", 2024-09-15T03:47:36.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:36.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935287+0000", 2024-09-15T03:47:36.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.51e-07 2024-09-15T03:47:36.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:36.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935290+0000", 2024-09-15T03:47:36.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8710000000000001e-06 2024-09-15T03:47:36.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:36.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935297+0000", 2024-09-15T03:47:36.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3390000000000004e-06 2024-09-15T03:47:36.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935302+0000", 2024-09-15T03:47:36.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5560000000000001e-06 2024-09-15T03:47:36.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936361+0000", 2024-09-15T03:47:36.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010593200000000001 2024-09-15T03:47:36.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:47:36.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936486+0000", 2024-09-15T03:47:36.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124657 2024-09-15T03:47:36.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.952979+0000", 2024-09-15T03:47:36.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016493154999999999 2024-09-15T03:47:36.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:36.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.953132+0000", 2024-09-15T03:47:36.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015346500000000001 2024-09-15T03:47:36.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:36.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.956989+0000", 2024-09-15T03:47:36.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038571299999999998 2024-09-15T03:47:36.307 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.307 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:36.307 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.307 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:750 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [create,setxattr \\^tkqj^qnPPfYPq\\WZoebWOkaebalTko_OhaYsrZ (1787) in=40b,setxattr cg]RoaNaVsOQTrQfp[VeRj_tmeegdoc`PtRYpskN (1998) in=40b,setxattr \\_N_s\\f^o_Ojh]gqWPkfsZ\\WTSbY]SOlX`mlV[^q (6944) in=40b,setxattr nsdiO]arkaqtYt (33) in=14b,setxattr ti`p[nirg_rWRRSe[oZhhn]b]ZsTimedg]]p[sQY (1049) in=40b,setxattr i[YNharOrm`bbP]UtYhjlqVWP\\XnccYlfhV_deTf (383) in=40b,setxattr ka_q[S]a]naRePWgcgUWmZSaW^RVV\\_rNXdZefal (875) in=40b,setxattr gaoaanUtdfqUrimS^_NfUtarRjNSVRf^_PbUpdtf (1645) in=40b,setxattr qNb^mqtq^Vsg]m\\aWdRTbbgOlZYptsW]O[_R`Vaa (1155) in=40b,setxattr gUPP_ttp`PdrngYsWrVdfZZVgff^Q`YRSjgOXTao (417) in=40b,setxattr mi`p\\f[PUQf^[[skf^\\Za_iXO_oTscOWNnQqcR[[ (4560) in=40b,setxattr N^_ka_abmi]seatXYWSQtibSdeOsfV\\OXsmmXlYm (1893) in=40b,setxattr aoqZmYaUpNW (33) in=11b,setxattr Z[bbsdRZTfkrP`]akaXifm`oV[YTOPrPaeeXV[fl (1611) in=40b,setxattr oZ`SalVqYt^T_[b_a]r[rlaaWOa_WZQ^^opf\\aN_ (243) in=40b,setxattr k\\V^enpkstj^VpoopX\\PcoTY]eogbs`pQQW]ofrm (347) in=40b,setxattr cfXfpnr`SogqjNa^cTVYtpgdciPp_eTUg]Ujn\\at (1926) in=40b,setxattr t`ojSZgRjbb[aaUXSepgQf_nc`kaOetle^n^`Vfd (1087) in=40b,setxattr ONYdltaQhe\\a\\da\\ehQRpsS^Ugdgmf\\WWigfUarV (631) in=40b,setxattr [Tj`Rbkt]qjatpSf_fPsanNr^ipYnOQlPnUZlmRU (4068) in=40b] snapc 32=[32,2d,2b,26,25,24,23,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T03:47:36.308 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.935912+0000", 2024-09-15T03:47:36.308 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.2490281990000001, 2024-09-15T03:47:36.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03915623, 2024-09-15T03:47:36.308 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:36.308 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:36.308 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:36.308 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:36.308 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.308 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 750 2024-09-15T03:47:36.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.308 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:36.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:36.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935912+0000", 2024-09-15T03:47:36.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:36.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935912+0000", 2024-09-15T03:47:36.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:36.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:36.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935913+0000", 2024-09-15T03:47:36.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7599999999999996e-07 2024-09-15T03:47:36.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:36.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935984+0000", 2024-09-15T03:47:36.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1136000000000005e-05 2024-09-15T03:47:36.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:36.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935997+0000", 2024-09-15T03:47:36.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3482000000000001e-05 2024-09-15T03:47:36.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936004+0000", 2024-09-15T03:47:36.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3260000000000001e-06 2024-09-15T03:47:36.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936878+0000", 2024-09-15T03:47:36.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087483999999999999 2024-09-15T03:47:36.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:47:36.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.937895+0000", 2024-09-15T03:47:36.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010163209999999999 2024-09-15T03:47:36.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:36.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.965388+0000", 2024-09-15T03:47:36.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027492986000000001 2024-09-15T03:47:36.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:36.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.965630+0000", 2024-09-15T03:47:36.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000242012 2024-09-15T03:47:36.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:36.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.966983+0000", 2024-09-15T03:47:36.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013535349999999999 2024-09-15T03:47:36.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:36.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.967360+0000", 2024-09-15T03:47:36.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037704399999999999 2024-09-15T03:47:36.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:36.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.967371+0000", 2024-09-15T03:47:36.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0988999999999999e-05 2024-09-15T03:47:36.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:36.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.967436+0000", 2024-09-15T03:47:36.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4679999999999997e-05 2024-09-15T03:47:36.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:36.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.969080+0000", 2024-09-15T03:47:36.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001644017 2024-09-15T03:47:36.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:36.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.969532+0000", 2024-09-15T03:47:36.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045157699999999999 2024-09-15T03:47:36.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:36.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:36.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:36.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.975068+0000", 2024-09-15T03:47:36.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055364089999999999 2024-09-15T03:47:36.315 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.315 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:36.315 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.315 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:36.315 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:36.315 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:36.317 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:47:36.317 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:36.316+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:36.417 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:36.417+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:36.425 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:36.425 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:36.425 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:36.426 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:36.434 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:47:36.519 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:36.518+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:36.531 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:36.531 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:36.531 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:36.531 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:36.531 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:36.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:47:36.583 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:47:36.628 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:36.627+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:36.638 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:36.638 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:36.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:36.638 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:47:36.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.638 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:47:36.638 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:36.639 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 78.733963040000006, 2024-09-15T03:47:36.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10988437500000001, 2024-09-15T03:47:36.639 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:36.639 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:36.639 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:36.639 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:47:36.639 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:47:36.639 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1 2024-09-15T03:47:36.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.640 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:36.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:36.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:36.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:36.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:36.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:36.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902355+0000", 2024-09-15T03:47:36.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.429e-06 2024-09-15T03:47:36.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:36.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902362+0000", 2024-09-15T03:47:36.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7290000000000003e-06 2024-09-15T03:47:36.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:36.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902373+0000", 2024-09-15T03:47:36.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0533999999999999e-05 2024-09-15T03:47:36.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902379+0000", 2024-09-15T03:47:36.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4350000000000002e-06 2024-09-15T03:47:36.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902428+0000", 2024-09-15T03:47:36.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8571e-05 2024-09-15T03:47:36.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:47:36.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902434+0000", 2024-09-15T03:47:36.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5610000000000004e-06 2024-09-15T03:47:36.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011307+0000", 2024-09-15T03:47:36.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.108872306 2024-09-15T03:47:36.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:36.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011448+0000", 2024-09-15T03:47:36.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014137199999999999 2024-09-15T03:47:36.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:47:36.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011569+0000", 2024-09-15T03:47:36.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120925 2024-09-15T03:47:36.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:36.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011695+0000", 2024-09-15T03:47:36.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012586800000000001 2024-09-15T03:47:36.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:36.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011703+0000", 2024-09-15T03:47:36.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4190000000000002e-06 2024-09-15T03:47:36.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:36.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011739+0000", 2024-09-15T03:47:36.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5474e-05 2024-09-15T03:47:36.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:47:36.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011940+0000", 2024-09-15T03:47:36.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020108699999999999 2024-09-15T03:47:36.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:47:36.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012179+0000", 2024-09-15T03:47:36.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023950299999999999 2024-09-15T03:47:36.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:36.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012215+0000", 2024-09-15T03:47:36.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5332e-05 2024-09-15T03:47:36.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:36.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012238+0000", 2024-09-15T03:47:36.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3830000000000001e-05 2024-09-15T03:47:36.646 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.646 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:36.646 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.647 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4263.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2024-09-15T03:47:36.647 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:36.647 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 72.592566353999999, 2024-09-15T03:47:36.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019495788999999999, 2024-09-15T03:47:36.647 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:36.647 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:36.647 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:36.647 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4263", 2024-09-15T03:47:36.647 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/646873808", 2024-09-15T03:47:36.648 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2 2024-09-15T03:47:36.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.648 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:36.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:36.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:36.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:36.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:36.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:36.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043752+0000", 2024-09-15T03:47:36.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.815e-06 2024-09-15T03:47:36.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:36.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043757+0000", 2024-09-15T03:47:36.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4240000000000003e-06 2024-09-15T03:47:36.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:36.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043766+0000", 2024-09-15T03:47:36.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8629999999999997e-06 2024-09-15T03:47:36.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043773+0000", 2024-09-15T03:47:36.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0099999999999998e-06 2024-09-15T03:47:36.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043830+0000", 2024-09-15T03:47:36.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7089999999999997e-05 2024-09-15T03:47:36.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.651 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:47:36.651 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043837+0000", 2024-09-15T03:47:36.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.08e-06 2024-09-15T03:47:36.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.651 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.651 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063103+0000", 2024-09-15T03:47:36.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019265619000000001 2024-09-15T03:47:36.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.651 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:36.651 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063246+0000", 2024-09-15T03:47:36.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000143888 2024-09-15T03:47:36.651 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.652 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:36.652 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.652 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:47:36.652 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:36.652 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 50.465492965000003, 2024-09-15T03:47:36.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011878227, 2024-09-15T03:47:36.652 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:36.652 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:36.652 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:36.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:36.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:36.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:36.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:36.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:36.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170825+0000", 2024-09-15T03:47:36.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9599999999999999e-07 2024-09-15T03:47:36.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:36.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171100+0000", 2024-09-15T03:47:36.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027548500000000001 2024-09-15T03:47:36.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:36.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171112+0000", 2024-09-15T03:47:36.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2152e-05 2024-09-15T03:47:36.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171119+0000", 2024-09-15T03:47:36.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5050000000000004e-06 2024-09-15T03:47:36.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171162+0000", 2024-09-15T03:47:36.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2908000000000003e-05 2024-09-15T03:47:36.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:36.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171169+0000", 2024-09-15T03:47:36.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9739999999999996e-06 2024-09-15T03:47:36.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:36.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171398+0000", 2024-09-15T03:47:36.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022954799999999999 2024-09-15T03:47:36.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:36.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173640+0000", 2024-09-15T03:47:36.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022417000000000001 2024-09-15T03:47:36.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:36.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173705+0000", 2024-09-15T03:47:36.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4907999999999996e-05 2024-09-15T03:47:36.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:36.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173742+0000", 2024-09-15T03:47:36.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6940999999999997e-05 2024-09-15T03:47:36.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:36.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.182702+0000", 2024-09-15T03:47:36.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0089604100000000002 2024-09-15T03:47:36.657 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.658 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:36.658 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.658 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 39/31 ECSubWrite(tid=43, reqid=client.4521.0:252, at_version=39'27, trim_to=0'0, roll_forward_to=39'25))", 2024-09-15T03:47:36.658 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:36.658 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 50.457775646999998, 2024-09-15T03:47:36.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011286757, 2024-09-15T03:47:36.658 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:36.658 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:36.658 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:36.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:36.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:36.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.659 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:36.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:36.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.659 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:36.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178542+0000", 2024-09-15T03:47:36.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9400000000000004e-07 2024-09-15T03:47:36.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.660 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:36.660 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178754+0000", 2024-09-15T03:47:36.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021228800000000001 2024-09-15T03:47:36.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.660 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:36.660 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178765+0000", 2024-09-15T03:47:36.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0176e-05 2024-09-15T03:47:36.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.660 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.660 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178769+0000", 2024-09-15T03:47:36.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1359999999999999e-06 2024-09-15T03:47:36.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178810+0000", 2024-09-15T03:47:36.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0825999999999998e-05 2024-09-15T03:47:36.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:36.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178820+0000", 2024-09-15T03:47:36.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0237999999999999e-05 2024-09-15T03:47:36.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:36.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179033+0000", 2024-09-15T03:47:36.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021361500000000001 2024-09-15T03:47:36.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:36.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179040+0000", 2024-09-15T03:47:36.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0509999999999999e-06 2024-09-15T03:47:36.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:36.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179102+0000", 2024-09-15T03:47:36.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1977000000000006e-05 2024-09-15T03:47:36.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:36.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179114+0000", 2024-09-15T03:47:36.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1223000000000001e-05 2024-09-15T03:47:36.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:36.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.189828+0000", 2024-09-15T03:47:36.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010714433000000001 2024-09-15T03:47:36.663 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.663 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:36.663 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.664 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 51/31 ECSubWrite(tid=67, reqid=client.4521.0:350, at_version=51'33, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:36.664 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:36.664 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 38.792475605999996, 2024-09-15T03:47:36.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.015569239, 2024-09-15T03:47:36.664 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:36.664 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:36.664 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:36.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.664 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:36.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:36.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.664 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:36.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:36.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.665 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:36.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843843+0000", 2024-09-15T03:47:36.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.195e-06 2024-09-15T03:47:36.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.665 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:36.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843880+0000", 2024-09-15T03:47:36.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7527999999999997e-05 2024-09-15T03:47:36.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:36.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843898+0000", 2024-09-15T03:47:36.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7696000000000001e-05 2024-09-15T03:47:36.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843907+0000", 2024-09-15T03:47:36.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8820000000000007e-06 2024-09-15T03:47:36.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843978+0000", 2024-09-15T03:47:36.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1537000000000003e-05 2024-09-15T03:47:36.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:36.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843989+0000", 2024-09-15T03:47:36.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0720000000000001e-05 2024-09-15T03:47:36.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:36.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844102+0000", 2024-09-15T03:47:36.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000113406 2024-09-15T03:47:36.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:36.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844131+0000", 2024-09-15T03:47:36.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8762e-05 2024-09-15T03:47:36.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:36.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844180+0000", 2024-09-15T03:47:36.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9329999999999997e-05 2024-09-15T03:47:36.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:36.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844201+0000", 2024-09-15T03:47:36.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0772999999999999e-05 2024-09-15T03:47:36.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:36.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.859411+0000", 2024-09-15T03:47:36.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01520941 2024-09-15T03:47:36.669 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.669 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:36.669 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.669 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:36.669 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:36.670 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 38.791596695000003, 2024-09-15T03:47:36.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012156452, 2024-09-15T03:47:36.670 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:36.670 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:36.670 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:36.670 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.670 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:36.670 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:36.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.670 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.670 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:36.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:36.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.671 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:36.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844721+0000", 2024-09-15T03:47:36.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1100000000000005e-07 2024-09-15T03:47:36.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.671 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:36.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844743+0000", 2024-09-15T03:47:36.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1999000000000001e-05 2024-09-15T03:47:36.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:36.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844758+0000", 2024-09-15T03:47:36.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4718999999999999e-05 2024-09-15T03:47:36.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844772+0000", 2024-09-15T03:47:36.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4637000000000001e-05 2024-09-15T03:47:36.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844842+0000", 2024-09-15T03:47:36.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9596000000000002e-05 2024-09-15T03:47:36.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.673 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:36.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844853+0000", 2024-09-15T03:47:36.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0912e-05 2024-09-15T03:47:36.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.673 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:36.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844966+0000", 2024-09-15T03:47:36.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011276 2024-09-15T03:47:36.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.673 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:36.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844977+0000", 2024-09-15T03:47:36.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1676e-05 2024-09-15T03:47:36.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:36.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847414+0000", 2024-09-15T03:47:36.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024363760000000001 2024-09-15T03:47:36.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:36.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847438+0000", 2024-09-15T03:47:36.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3737e-05 2024-09-15T03:47:36.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.675 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:36.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856877+0000", 2024-09-15T03:47:36.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094392290000000004 2024-09-15T03:47:36.675 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.675 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:36.675 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.675 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:36.675 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:36.675 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 38.791527299999998, 2024-09-15T03:47:36.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012258211, 2024-09-15T03:47:36.676 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:36.676 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:36.676 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:36.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.676 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:36.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:36.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:36.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:36.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:36.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:36.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.58e-07 2024-09-15T03:47:36.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:36.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844818+0000", 2024-09-15T03:47:36.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.796e-05 2024-09-15T03:47:36.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:36.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844829+0000", 2024-09-15T03:47:36.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0895e-05 2024-09-15T03:47:36.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844836+0000", 2024-09-15T03:47:36.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.618e-06 2024-09-15T03:47:36.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845023+0000", 2024-09-15T03:47:36.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018743500000000001 2024-09-15T03:47:36.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:36.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845035+0000", 2024-09-15T03:47:36.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1958e-05 2024-09-15T03:47:36.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:36.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845147+0000", 2024-09-15T03:47:36.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112281 2024-09-15T03:47:36.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:36.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847417+0000", 2024-09-15T03:47:36.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022693969999999998 2024-09-15T03:47:36.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:36.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847462+0000", 2024-09-15T03:47:36.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5466999999999997e-05 2024-09-15T03:47:36.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:36.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847503+0000", 2024-09-15T03:47:36.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1022999999999997e-05 2024-09-15T03:47:36.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.681 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:36.681 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.857048+0000", 2024-09-15T03:47:36.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095447190000000001 2024-09-15T03:47:36.681 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.681 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:36.681 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.681 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 51/31 ECSubWrite(tid=68, reqid=client.4521.0:351, at_version=51'34, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:36.681 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:36.681 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 38.791109182, 2024-09-15T03:47:36.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.014544793, 2024-09-15T03:47:36.682 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:36.682 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:36.682 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:36.682 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.682 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:36.682 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:36.682 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.682 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.682 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:36.682 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:36.682 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:36.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845209+0000", 2024-09-15T03:47:36.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5300000000000003e-07 2024-09-15T03:47:36.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:36.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845227+0000", 2024-09-15T03:47:36.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8759000000000001e-05 2024-09-15T03:47:36.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:36.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845241+0000", 2024-09-15T03:47:36.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3740000000000001e-05 2024-09-15T03:47:36.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.684 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.684 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845247+0000", 2024-09-15T03:47:36.684 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2779999999999999e-06 2024-09-15T03:47:36.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.684 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.684 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845308+0000", 2024-09-15T03:47:36.684 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0702000000000003e-05 2024-09-15T03:47:36.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.684 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.684 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:36.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845317+0000", 2024-09-15T03:47:36.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3770000000000002e-06 2024-09-15T03:47:36.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.685 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:36.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845413+0000", 2024-09-15T03:47:36.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5356e-05 2024-09-15T03:47:36.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.685 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:36.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847421+0000", 2024-09-15T03:47:36.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020077200000000002 2024-09-15T03:47:36.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.685 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:36.686 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847466+0000", 2024-09-15T03:47:36.686 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5754999999999998e-05 2024-09-15T03:47:36.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.686 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:36.686 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847572+0000", 2024-09-15T03:47:36.686 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010594000000000001 2024-09-15T03:47:36.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.686 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:36.686 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.859753+0000", 2024-09-15T03:47:36.686 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012180412999999999 2024-09-15T03:47:36.687 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.687 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:36.687 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.687 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.687 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.687 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 60/28 ECSubWrite(tid=73, reqid=client.4521.0:424, at_version=60'41, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:36.687 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:36.687 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 29.735905928000001, 2024-09-15T03:47:36.687 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01137208, 2024-09-15T03:47:36.687 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:36.687 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:36.687 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:36.687 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.687 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:36.688 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:36.688 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.688 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.688 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:36.688 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:36.688 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.688 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.688 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:36.688 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900412+0000", 2024-09-15T03:47:36.688 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7599999999999996e-07 2024-09-15T03:47:36.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.689 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:36.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:36.689 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7986999999999999e-05 2024-09-15T03:47:36.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.689 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:36.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900445+0000", 2024-09-15T03:47:36.689 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.468e-05 2024-09-15T03:47:36.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.689 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900451+0000", 2024-09-15T03:47:36.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5799999999999997e-06 2024-09-15T03:47:36.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.690 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.690 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.690 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900507+0000", 2024-09-15T03:47:36.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5855e-05 2024-09-15T03:47:36.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.690 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.690 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:36.690 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900530+0000", 2024-09-15T03:47:36.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2929e-05 2024-09-15T03:47:36.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.691 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.691 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:36.691 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900640+0000", 2024-09-15T03:47:36.691 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000110536 2024-09-15T03:47:36.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.691 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.691 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:36.691 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900648+0000", 2024-09-15T03:47:36.691 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9470000000000003e-06 2024-09-15T03:47:36.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.691 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.691 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:36.691 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900764+0000", 2024-09-15T03:47:36.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011518099999999999 2024-09-15T03:47:36.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:36.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900803+0000", 2024-09-15T03:47:36.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9941999999999998e-05 2024-09-15T03:47:36.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:36.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.911783+0000", 2024-09-15T03:47:36.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010979667 2024-09-15T03:47:36.692 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.693 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:36.693 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.693 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:517 3.3s1 3:caaee79c:::smithi00132172-10:head [read 0~3145728,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:36.693 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.947050+0000", 2024-09-15T03:47:36.693 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 20.689266995000001, 2024-09-15T03:47:36.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012110134, 2024-09-15T03:47:36.693 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:36.693 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:36.693 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:36.693 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:36.693 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.694 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 517 2024-09-15T03:47:36.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.694 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:36.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.694 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:36.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947050+0000", 2024-09-15T03:47:36.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.694 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:36.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947050+0000", 2024-09-15T03:47:36.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:36.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947052+0000", 2024-09-15T03:47:36.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9099999999999999e-06 2024-09-15T03:47:36.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:36.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947060+0000", 2024-09-15T03:47:36.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8529999999999992e-06 2024-09-15T03:47:36.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:36.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947074+0000", 2024-09-15T03:47:36.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4344e-05 2024-09-15T03:47:36.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947082+0000", 2024-09-15T03:47:36.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9249999999999995e-06 2024-09-15T03:47:36.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947138+0000", 2024-09-15T03:47:36.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6054000000000002e-05 2024-09-15T03:47:36.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:36.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947339+0000", 2024-09-15T03:47:36.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000200829 2024-09-15T03:47:36.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:36.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.959160+0000", 2024-09-15T03:47:36.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011821219000000001 2024-09-15T03:47:36.697 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.697 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:36.697 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.697 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:518 3.3s1 3:caaee79c:::smithi00132172-10:head [read 0~3145728,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:36.698 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.947091+0000", 2024-09-15T03:47:36.698 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 20.689225853, 2024-09-15T03:47:36.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017608233000000001, 2024-09-15T03:47:36.698 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:36.698 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:36.698 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:36.698 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:36.698 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.698 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 518 2024-09-15T03:47:36.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.698 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:36.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:36.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947091+0000", 2024-09-15T03:47:36.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:36.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947091+0000", 2024-09-15T03:47:36.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:36.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947092+0000", 2024-09-15T03:47:36.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7800000000000002e-07 2024-09-15T03:47:36.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:36.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947095+0000", 2024-09-15T03:47:36.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2959999999999999e-06 2024-09-15T03:47:36.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:36.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947102+0000", 2024-09-15T03:47:36.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7469999999999999e-06 2024-09-15T03:47:36.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947106+0000", 2024-09-15T03:47:36.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6550000000000003e-06 2024-09-15T03:47:36.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947622+0000", 2024-09-15T03:47:36.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000515727 2024-09-15T03:47:36.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:36.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947751+0000", 2024-09-15T03:47:36.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012920600000000001 2024-09-15T03:47:36.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:36.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.964699+0000", 2024-09-15T03:47:36.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.016948123999999998 2024-09-15T03:47:36.702 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.702 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:36.702 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.702 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:519 3.3s1 3:caaee79c:::smithi00132172-10:head [read 0~3145728,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:36.702 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:36.702 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 20.689192147, 2024-09-15T03:47:36.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.021743243999999998, 2024-09-15T03:47:36.702 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:36.703 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:36.703 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:36.703 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:36.703 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.703 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 519 2024-09-15T03:47:36.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.703 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:36.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:36.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:36.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:36.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:36.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:36.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:36.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5999999999999999e-07 2024-09-15T03:47:36.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:36.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947130+0000", 2024-09-15T03:47:36.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7720000000000002e-06 2024-09-15T03:47:36.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:36.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947137+0000", 2024-09-15T03:47:36.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2370000000000003e-06 2024-09-15T03:47:36.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947141+0000", 2024-09-15T03:47:36.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8809999999999998e-06 2024-09-15T03:47:36.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.948008+0000", 2024-09-15T03:47:36.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00086656700000000005 2024-09-15T03:47:36.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:36.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.948113+0000", 2024-09-15T03:47:36.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104904 2024-09-15T03:47:36.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:36.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.968868+0000", 2024-09-15T03:47:36.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.020755422999999999 2024-09-15T03:47:36.706 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.706 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:36.706 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.707 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.3s1 69/68 ECSubReadReply(tid=163, attrs_read=0))", 2024-09-15T03:47:36.707 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:36.707 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 20.682616403000001, 2024-09-15T03:47:36.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011102266, 2024-09-15T03:47:36.707 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:36.707 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:36.707 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:36.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:36.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:36.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.708 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:36.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:36.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.708 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:36.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:36.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1199999999999998e-07 2024-09-15T03:47:36.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.708 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:36.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956300+0000", 2024-09-15T03:47:36.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025987459999999999 2024-09-15T03:47:36.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:36.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956314+0000", 2024-09-15T03:47:36.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4469e-05 2024-09-15T03:47:36.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956319+0000", 2024-09-15T03:47:36.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.746e-06 2024-09-15T03:47:36.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.959591+0000", 2024-09-15T03:47:36.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032721510000000001 2024-09-15T03:47:36.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:36.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.964803+0000", 2024-09-15T03:47:36.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0052117420000000001 2024-09-15T03:47:36.710 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.710 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:36.710 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.710 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.3s1 69/68 ECSubReadReply(tid=164, attrs_read=0))", 2024-09-15T03:47:36.710 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:36.710 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 20.67999197, 2024-09-15T03:47:36.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012633276000000001, 2024-09-15T03:47:36.711 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:36.711 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:36.711 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:36.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:36.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:36.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:36.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:36.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.712 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:36.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:36.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5900000000000003e-07 2024-09-15T03:47:36.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.712 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:36.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957786+0000", 2024-09-15T03:47:36.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001460755 2024-09-15T03:47:36.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.713 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:36.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957807+0000", 2024-09-15T03:47:36.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0937e-05 2024-09-15T03:47:36.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.713 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957814+0000", 2024-09-15T03:47:36.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5989999999999998e-06 2024-09-15T03:47:36.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.713 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.964911+0000", 2024-09-15T03:47:36.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0070976370000000004 2024-09-15T03:47:36.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.714 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:36.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.968958+0000", 2024-09-15T03:47:36.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.004046989 2024-09-15T03:47:36.714 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.714 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:36.714 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.714 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:567 3.ds0 3:b166213b:::smithi00132172-21:head [rollback 3] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:36.714 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.947576+0000", 2024-09-15T03:47:36.715 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.688741042, 2024-09-15T03:47:36.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013871319999999999, 2024-09-15T03:47:36.715 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:36.715 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:36.715 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:36.715 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:36.715 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.715 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 567 2024-09-15T03:47:36.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.715 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:36.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:36.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947576+0000", 2024-09-15T03:47:36.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:36.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947576+0000", 2024-09-15T03:47:36.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:36.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947577+0000", 2024-09-15T03:47:36.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.37e-07 2024-09-15T03:47:36.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:36.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947580+0000", 2024-09-15T03:47:36.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5710000000000002e-06 2024-09-15T03:47:36.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:36.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947588+0000", 2024-09-15T03:47:36.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8939999999999993e-06 2024-09-15T03:47:36.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947593+0000", 2024-09-15T03:47:36.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7049999999999998e-06 2024-09-15T03:47:36.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948070+0000", 2024-09-15T03:47:36.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047685099999999998 2024-09-15T03:47:36.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:36.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948177+0000", 2024-09-15T03:47:36.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010765299999999999 2024-09-15T03:47:36.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.719 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:36.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948986+0000", 2024-09-15T03:47:36.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000808977 2024-09-15T03:47:36.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.719 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.719 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:36.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949117+0000", 2024-09-15T03:47:36.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000131032 2024-09-15T03:47:36.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.719 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.719 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:36.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949133+0000", 2024-09-15T03:47:36.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5798999999999999e-05 2024-09-15T03:47:36.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.720 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:36.720 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950936+0000", 2024-09-15T03:47:36.720 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00180316 2024-09-15T03:47:36.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.720 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:36.720 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950994+0000", 2024-09-15T03:47:36.720 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7884999999999999e-05 2024-09-15T03:47:36.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.720 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:36.720 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951532+0000", 2024-09-15T03:47:36.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00053787699999999997 2024-09-15T03:47:36.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:36.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961447+0000", 2024-09-15T03:47:36.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0099152790000000008 2024-09-15T03:47:36.721 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.721 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:36.721 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.721 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:568 3.ds0 3:b166213b:::smithi00132172-21:head [append 0~0] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:36.721 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.948048+0000", 2024-09-15T03:47:36.721 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.688269117000001, 2024-09-15T03:47:36.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013579269, 2024-09-15T03:47:36.722 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:36.722 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:36.722 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:36.722 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:36.722 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.722 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 568 2024-09-15T03:47:36.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.722 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:36.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:36.722 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948048+0000", 2024-09-15T03:47:36.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:36.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948048+0000", 2024-09-15T03:47:36.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:36.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948049+0000", 2024-09-15T03:47:36.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7199999999999998e-07 2024-09-15T03:47:36.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:36.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948056+0000", 2024-09-15T03:47:36.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0280000000000003e-06 2024-09-15T03:47:36.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:36.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948066+0000", 2024-09-15T03:47:36.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0141e-05 2024-09-15T03:47:36.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948072+0000", 2024-09-15T03:47:36.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1519999999999997e-06 2024-09-15T03:47:36.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949221+0000", 2024-09-15T03:47:36.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001149125 2024-09-15T03:47:36.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:36.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949313+0000", 2024-09-15T03:47:36.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1469000000000001e-05 2024-09-15T03:47:36.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:36.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949818+0000", 2024-09-15T03:47:36.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00050549700000000004 2024-09-15T03:47:36.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.726 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:36.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950002+0000", 2024-09-15T03:47:36.726 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018414700000000001 2024-09-15T03:47:36.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.726 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:36.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950940+0000", 2024-09-15T03:47:36.726 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00093786100000000001 2024-09-15T03:47:36.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.726 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:36.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951004+0000", 2024-09-15T03:47:36.727 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3859e-05 2024-09-15T03:47:36.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:36.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951020+0000", 2024-09-15T03:47:36.727 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.628e-05 2024-09-15T03:47:36.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:36.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951753+0000", 2024-09-15T03:47:36.727 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00073276999999999999 2024-09-15T03:47:36.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.728 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:36.728 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961627+0000", 2024-09-15T03:47:36.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098741679999999991 2024-09-15T03:47:36.728 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.728 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:36.728 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.728 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.728 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:677 3.3s1 3:caaee79c:::smithi00132172-10:head [copy-from ver 31] snapc 2f=[2d,2b,29,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:36.728 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:30.087209+0000", 2024-09-15T03:47:36.728 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.5491079259999996, 2024-09-15T03:47:36.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.025257667000000001, 2024-09-15T03:47:36.729 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:36.729 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:36.729 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:36.729 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:36.729 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.729 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 677 2024-09-15T03:47:36.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.729 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:36.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.729 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:36.729 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087209+0000", 2024-09-15T03:47:36.729 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.730 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:36.730 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087209+0000", 2024-09-15T03:47:36.730 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.730 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.730 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:36.730 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087211+0000", 2024-09-15T03:47:36.730 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2500000000000001e-06 2024-09-15T03:47:36.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.730 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.730 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:36.730 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087222+0000", 2024-09-15T03:47:36.730 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.113e-05 2024-09-15T03:47:36.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:36.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087252+0000", 2024-09-15T03:47:36.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9488999999999998e-05 2024-09-15T03:47:36.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087264+0000", 2024-09-15T03:47:36.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1875999999999999e-05 2024-09-15T03:47:36.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087368+0000", 2024-09-15T03:47:36.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104035 2024-09-15T03:47:36.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.732 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:36.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087560+0000", 2024-09-15T03:47:36.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019211400000000001 2024-09-15T03:47:36.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.732 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:36.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.104984+0000", 2024-09-15T03:47:36.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017424489000000001 2024-09-15T03:47:36.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:36.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105388+0000", 2024-09-15T03:47:36.733 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000403686 2024-09-15T03:47:36.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:36.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105404+0000", 2024-09-15T03:47:36.733 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6268000000000001e-05 2024-09-15T03:47:36.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:36.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105641+0000", 2024-09-15T03:47:36.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023648399999999999 2024-09-15T03:47:36.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.734 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.734 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:36.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105662+0000", 2024-09-15T03:47:36.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0602999999999999e-05 2024-09-15T03:47:36.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.734 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.734 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:36.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.110231+0000", 2024-09-15T03:47:36.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0045697089999999999 2024-09-15T03:47:36.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.735 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.735 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:36.735 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.112467+0000", 2024-09-15T03:47:36.735 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002235534 2024-09-15T03:47:36.735 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.735 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:36.735 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.735 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.735 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.735 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:678 3.3s1 3:caaee79c:::smithi00132172-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:36.735 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:30.087279+0000", 2024-09-15T03:47:36.735 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.5490384710000002, 2024-09-15T03:47:36.735 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.018991965, 2024-09-15T03:47:36.736 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:36.736 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:36.736 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:36.736 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:36.736 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.736 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 678 2024-09-15T03:47:36.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.736 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:36.736 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.736 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:36.736 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087279+0000", 2024-09-15T03:47:36.736 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.737 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.737 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:36.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087279+0000", 2024-09-15T03:47:36.737 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.737 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.737 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:36.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087280+0000", 2024-09-15T03:47:36.737 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.147e-06 2024-09-15T03:47:36.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.737 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.737 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:36.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087285+0000", 2024-09-15T03:47:36.737 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2059999999999998e-06 2024-09-15T03:47:36.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.738 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:36.738 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087303+0000", 2024-09-15T03:47:36.738 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7646e-05 2024-09-15T03:47:36.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.738 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.738 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087313+0000", 2024-09-15T03:47:36.738 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0862999999999999e-05 2024-09-15T03:47:36.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.738 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.738 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087765+0000", 2024-09-15T03:47:36.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045120400000000003 2024-09-15T03:47:36.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.739 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.739 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:36.739 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087962+0000", 2024-09-15T03:47:36.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000197095 2024-09-15T03:47:36.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.739 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.739 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.739 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105788+0000", 2024-09-15T03:47:36.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017826143999999999 2024-09-15T03:47:36.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.739 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.740 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:36.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.106034+0000", 2024-09-15T03:47:36.740 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024586999999999998 2024-09-15T03:47:36.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.740 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:36.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.106270+0000", 2024-09-15T03:47:36.740 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023678999999999999 2024-09-15T03:47:36.740 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.740 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:36.740 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.740 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:688 3.3s1 3:c7c3ef16:::smithi00132172-25:head [create,setxattr og^`]cfbTbdPrNph_pkYgeiTrgatpl] (805) in=31b,setxattr `Trb[\\aaaksnX\\dQOYP^TPXpYtmQ_maX\\[hbePkT (981) in=40b,setxattr UkQ^W[[sWh]fSrWg]^\\oZpWaamVkVUnZYbR[Vql\\ (663) in=40b,setxattr ofr[Os^]tjr^jicnNpPaUeXQTfldVQPdSPjcgWcT (7228) in=40b,setxattr Vl`NPsTVpnj_gO_srrbrZ\\eq (805) in=24b,setxattr ^_mcpljcT[qX_UeeYZY\\_ZTSsSsojP\\esUkVphT] (1963) in=40b,setxattr [oY^bSn[ealaPaOhprWW_SNaa]VhaXN[jmo`YYgt (138) in=40b,setxattr tT[SSseaPa\\UPS`\\g\\pRqsd`^lo__arabQlQmob` (663) in=40b,setxattr o^\\]jXsesZhoknr\\dagnjidgsPOhkfg[ffVe[PQW (1259) in=40b,setxattr c^VX_^n^gsh^P`aVbXpaS\\cgkpZO]OmXmchTtWZd (1612) in=40b,setxattr _^aeiTb[Uhh`rNNUOZOPt_Z_Qmeps`XXQ^qidetf (629) in=40b,setxattr ms[glaQ`N\\pqbicVTmgp^\\Rg]h`fefatkfg\\js_[ (139) in=40b,setxattr O\\gsPtYmrReq]if[pkOgPtqS`]Nrq_eYpsefkSgl (700) in=40b,setxattr RgefRtSsOT`jTQb^h\\igU`_lbscW`fgkXroNdmea (1751) in=40b,setxattr sgaV_cSaelbWQYill^k`RsVajcXpUSfW]eNd_nat (807) in=40b,setxattr P\\lm\\r`\\jpssZmkPqcQSbje_]]`YOSZTeRUUSSY[ (1365) in=40b,setxattr aa^ePena\\opOdeWXTj[XU]N^jQS]bens\\Tc_g]ea (16140) in=40b,setxattr R[qckQQO`n[[egbgcZsdtUbdPdnPNlOmsbP_OXOQ (1227) in=40b,setxattr SpRUpSaQZ]nlQ]kpXZnOWSUS]tsmf`qZiPTkq]Yi (209) in=40b,setxattr Y[Y[X`O_XaVOb]NZ_PgmhRX]Vds_dfbp]h[SPkYb (417) in=40b] snapc 2f=[2d,2b,29,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:36.741 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:30.088531+0000", 2024-09-15T03:47:36.741 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.5477857359999998, 2024-09-15T03:47:36.741 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013293503999999999, 2024-09-15T03:47:36.741 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:36.741 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:36.741 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:36.741 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:36.741 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:36.741 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 688 2024-09-15T03:47:36.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.741 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:36.741 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.741 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:36.742 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088531+0000", 2024-09-15T03:47:36.742 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.742 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.742 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:36.742 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088531+0000", 2024-09-15T03:47:36.742 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.742 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.742 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:36.742 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088533+0000", 2024-09-15T03:47:36.742 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5379999999999999e-06 2024-09-15T03:47:36.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.742 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:36.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088662+0000", 2024-09-15T03:47:36.743 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000129002 2024-09-15T03:47:36.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.743 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:36.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088688+0000", 2024-09-15T03:47:36.743 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6084999999999998e-05 2024-09-15T03:47:36.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.743 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088696+0000", 2024-09-15T03:47:36.743 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0490000000000004e-06 2024-09-15T03:47:36.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.744 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.744 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.744 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088746+0000", 2024-09-15T03:47:36.744 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0074000000000002e-05 2024-09-15T03:47:36.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.744 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.744 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:36.744 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.089084+0000", 2024-09-15T03:47:36.744 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033810200000000001 2024-09-15T03:47:36.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.744 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.744 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:36.745 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090498+0000", 2024-09-15T03:47:36.745 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001414205 2024-09-15T03:47:36.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.745 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.745 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:36.745 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090936+0000", 2024-09-15T03:47:36.745 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043802799999999997 2024-09-15T03:47:36.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.745 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.745 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:36.745 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090951+0000", 2024-09-15T03:47:36.745 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4953999999999999e-05 2024-09-15T03:47:36.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.746 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.746 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:36.746 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.091069+0000", 2024-09-15T03:47:36.746 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000117236 2024-09-15T03:47:36.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.746 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.746 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:36.746 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.091128+0000", 2024-09-15T03:47:36.746 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9580999999999998e-05 2024-09-15T03:47:36.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.746 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.746 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:36.747 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.097312+0000", 2024-09-15T03:47:36.747 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0061838129999999998 2024-09-15T03:47:36.747 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.747 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.747 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:36.747 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.101825+0000", 2024-09-15T03:47:36.747 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0045128369999999996 2024-09-15T03:47:36.747 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.747 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:36.747 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.747 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.747 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.747 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s4 83/80 ECSubWrite(tid=24, reqid=client.4521.0:675, at_version=83'37, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:36.747 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:30.126290+0000", 2024-09-15T03:47:36.748 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.510026893, 2024-09-15T03:47:36.748 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.014228881000000001, 2024-09-15T03:47:36.748 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:36.748 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:36.748 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:36.748 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.748 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:36.748 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.126290+0000", 2024-09-15T03:47:36.748 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.748 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.748 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.748 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:36.748 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.126290+0000", 2024-09-15T03:47:36.749 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:36.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.749 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.749 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:36.749 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.126291+0000", 2024-09-15T03:47:36.749 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6499999999999998e-07 2024-09-15T03:47:36.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.749 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.749 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:36.749 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128867+0000", 2024-09-15T03:47:36.749 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025759749999999999 2024-09-15T03:47:36.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.749 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.750 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:36.750 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128881+0000", 2024-09-15T03:47:36.750 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3823e-05 2024-09-15T03:47:36.750 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.750 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.750 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:36.750 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128888+0000", 2024-09-15T03:47:36.750 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.605e-06 2024-09-15T03:47:36.750 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.750 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.750 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:36.750 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128949+0000", 2024-09-15T03:47:36.750 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0949999999999998e-05 2024-09-15T03:47:36.751 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.751 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.751 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:36.751 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128957+0000", 2024-09-15T03:47:36.751 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4760000000000001e-06 2024-09-15T03:47:36.751 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.751 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.751 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:36.751 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.129323+0000", 2024-09-15T03:47:36.751 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036638900000000001 2024-09-15T03:47:36.751 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.751 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.751 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:36.752 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.129331+0000", 2024-09-15T03:47:36.752 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1319999999999994e-06 2024-09-15T03:47:36.752 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.752 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.752 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:36.752 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.129567+0000", 2024-09-15T03:47:36.752 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000235393 2024-09-15T03:47:36.752 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.752 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.752 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:36.752 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.129586+0000", 2024-09-15T03:47:36.752 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9517999999999998e-05 2024-09-15T03:47:36.753 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:36.753 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:36.753 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:36.753 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.140519+0000", 2024-09-15T03:47:36.753 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010932855 2024-09-15T03:47:36.753 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.753 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:36.753 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.753 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:36.753 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:36.753 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:36.754 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:36.755 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:36.755 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:36.756 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:36.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:47:36.756 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:36.757 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:36.757 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:36.757 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:36.758 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:36.754+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:36.771 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:36.772 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:36.786 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:36.787 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:36.801 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:36.801 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:36.815 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:36.816 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:36.830 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:36.830 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:36.839 INFO:tasks.rados.rados.0.smithi001.stdout:305: done (2 left) 2024-09-15T03:47:36.839 INFO:tasks.rados.rados.0.smithi001.stdout:306: done (1 left) 2024-09-15T03:47:36.839 INFO:tasks.rados.rados.0.smithi001.stdout:307: snap_remove snap 22 2024-09-15T03:47:36.852 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:36.853 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:36.853 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-15T03:47:36.856 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:36.855+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:36.864 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:36.864 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:36.871 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:36.871 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:36.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:47:36.874 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T03:47:36.874 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T03:47:36.957 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:36.957+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:36.970 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:36.970 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:36.978 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:47:37.058 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:37.057+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:37.074 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:37.074 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:37.082 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:47:37.161 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:37.160+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:37.178 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:37.178 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:37.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:47:37.262 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:37.262+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:37.282 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:37.282 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:37.291 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:47:37.365 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:37.364+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:37.388 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:37.388 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:37.396 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:47:37.466 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:37.466+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:37.502 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:37.502 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:37.511 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:47:37.596 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:37.595+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:37.651 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:37.651 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:37.660 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:47:37.670 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:37.669+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:37.771 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:37.770+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:37.801 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:37.801 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:37.809 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:47:37.841 INFO:tasks.rados.rados.0.smithi001.stdout:307: done (1 left) 2024-09-15T03:47:37.841 INFO:tasks.rados.rados.0.smithi001.stdout:308: read oid 13 snap -1 2024-09-15T03:47:37.841 INFO:tasks.rados.rados.0.smithi001.stdout:308: expect (ObjNum 68 snap 6 seq_num 68) 2024-09-15T03:47:37.841 INFO:tasks.rados.rados.0.smithi001.stdout:309: read oid 43 snap -1 2024-09-15T03:47:37.841 INFO:tasks.rados.rados.0.smithi001.stdout:309: expect (ObjNum 118 snap 33 seq_num 118) 2024-09-15T03:47:37.841 INFO:tasks.rados.rados.0.smithi001.stdout:310: copy_from oid 11 from oid 8 current snap is 35 2024-09-15T03:47:37.841 INFO:tasks.rados.rados.0.smithi001.stdout:304: finishing write tid 1 to smithi00132172-18 2024-09-15T03:47:37.841 INFO:tasks.rados.rados.0.smithi001.stdout:304: finishing write tid 2 to smithi00132172-18 2024-09-15T03:47:37.842 INFO:tasks.rados.rados.0.smithi001.stdout:304: finishing write tid 3 to smithi00132172-18 2024-09-15T03:47:37.842 INFO:tasks.rados.rados.0.smithi001.stdout:304: finishing write tid 4 to smithi00132172-18 2024-09-15T03:47:37.842 INFO:tasks.rados.rados.0.smithi001.stdout:311: snap_create 2024-09-15T03:47:37.842 INFO:tasks.rados.rados.0.smithi001.stdout:304: finishing write tid 5 to smithi00132172-18 2024-09-15T03:47:37.842 INFO:tasks.rados.rados.0.smithi001.stdout:304: finishing write tid 6 to smithi00132172-18 2024-09-15T03:47:37.842 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 18 v 52 (ObjNum 120 snap 34 seq_num 120) dirty exists 2024-09-15T03:47:37.842 INFO:tasks.rados.rados.0.smithi001.stdout:304: left oid 18 (ObjNum 120 snap 34 seq_num 120) 2024-09-15T03:47:37.857 INFO:tasks.rados.rados.0.smithi001.stdout:308: expect (ObjNum 68 snap 6 seq_num 68) 2024-09-15T03:47:37.860 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T03:47:37.871 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:37.870+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:37.890 INFO:tasks.rados.rados.0.smithi001.stdout:309: expect (ObjNum 118 snap 33 seq_num 118) 2024-09-15T03:47:37.919 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:37.919 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:37.928 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:47:37.973 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:37.971+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:37.975 INFO:tasks.rados.rados.0.smithi001.stdout:310: finishing copy_from racing read to smithi00132172-11 2024-09-15T03:47:37.975 INFO:tasks.rados.rados.0.smithi001.stdout:310: finishing copy_from to smithi00132172-11 2024-09-15T03:47:37.975 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 11 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-15T03:47:38.016 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [10] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] 2024-09-15T03:47:38.017 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T03:47:38.017 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-15T03:47:38.017 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-15T03:47:38.072 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:38.072 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:38.073 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:38.072+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:38.081 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:47:38.173 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:38.173+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:38.184 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:38.184 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:38.193 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:47:38.275 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:38.274+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:38.295 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:47:38.296 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:38.296 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 91, 2024-09-15T03:47:38.296 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "f27b540e-e5e3-4e95-afe8-c95ee66b0a62", 2024-09-15T03:47:38.296 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-09-15T03:46:09.843094+0000", 2024-09-15T03:47:38.297 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-09-15T03:47:37.837065+0000", 2024-09-15T03:47:38.297 INFO:teuthology.orchestra.run.smithi001.stdout: "last_up_change": "2024-09-15T03:47:26.955166+0000", 2024-09-15T03:47:38.297 INFO:teuthology.orchestra.run.smithi001.stdout: "last_in_change": "2024-09-15T03:47:32.982084+0000", 2024-09-15T03:47:38.297 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-15T03:47:38.298 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_num": 5806080, 2024-09-15T03:47:38.298 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_set": [ 2024-09-15T03:47:38.298 INFO:teuthology.orchestra.run.smithi001.stdout: "nodeep-scrub", 2024-09-15T03:47:38.298 INFO:teuthology.orchestra.run.smithi001.stdout: "noscrub", 2024-09-15T03:47:38.298 INFO:teuthology.orchestra.run.smithi001.stdout: "pglog_hardlimit", 2024-09-15T03:47:38.298 INFO:teuthology.orchestra.run.smithi001.stdout: "purged_snapdirs", 2024-09-15T03:47:38.298 INFO:teuthology.orchestra.run.smithi001.stdout: "recovery_deletes", 2024-09-15T03:47:38.298 INFO:teuthology.orchestra.run.smithi001.stdout: "sortbitwise" 2024-09-15T03:47:38.298 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:47:38.298 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_version": 5, 2024-09-15T03:47:38.298 INFO:teuthology.orchestra.run.smithi001.stdout: "full_ratio": 0.94999998807907104, 2024-09-15T03:47:38.299 INFO:teuthology.orchestra.run.smithi001.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-15T03:47:38.299 INFO:teuthology.orchestra.run.smithi001.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-15T03:47:38.299 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_snapshot": "", 2024-09-15T03:47:38.299 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_max": 3, 2024-09-15T03:47:38.299 INFO:teuthology.orchestra.run.smithi001.stdout: "max_osd": 12, 2024-09-15T03:47:38.299 INFO:teuthology.orchestra.run.smithi001.stdout: "require_min_compat_client": "luminous", 2024-09-15T03:47:38.300 INFO:teuthology.orchestra.run.smithi001.stdout: "min_compat_client": "luminous", 2024-09-15T03:47:38.300 INFO:teuthology.orchestra.run.smithi001.stdout: "require_osd_release": "quincy", 2024-09-15T03:47:38.300 INFO:teuthology.orchestra.run.smithi001.stdout: "pools": [ 2024-09-15T03:47:38.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.300 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 1, 2024-09-15T03:47:38.300 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": ".mgr", 2024-09-15T03:47:38.300 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-15T03:46:16.185530+0000", 2024-09-15T03:47:38.300 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-09-15T03:47:38.300 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool", 2024-09-15T03:47:38.300 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-15T03:47:38.301 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-15T03:47:38.301 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 1, 2024-09-15T03:47:38.301 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-15T03:47:38.301 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-15T03:47:38.301 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-15T03:47:38.301 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T03:47:38.301 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T03:47:38.301 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-15T03:47:38.301 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-15T03:47:38.301 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 1, 2024-09-15T03:47:38.301 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 1, 2024-09-15T03:47:38.301 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 1, 2024-09-15T03:47:38.302 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 1, 2024-09-15T03:47:38.302 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 1, 2024-09-15T03:47:38.302 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-15T03:47:38.302 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-15T03:47:38.302 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-15T03:47:38.302 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-15T03:47:38.302 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-15T03:47:38.302 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-15T03:47:38.302 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-15T03:47:38.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.303 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "20", 2024-09-15T03:47:38.303 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-15T03:47:38.303 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T03:47:38.303 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T03:47:38.303 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-15T03:47:38.303 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-15T03:47:38.303 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 0, 2024-09-15T03:47:38.303 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 0, 2024-09-15T03:47:38.304 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-15T03:47:38.304 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-15T03:47:38.304 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-15T03:47:38.304 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-15T03:47:38.304 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-15T03:47:38.304 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-15T03:47:38.304 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-15T03:47:38.304 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-15T03:47:38.305 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-15T03:47:38.305 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-15T03:47:38.305 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-15T03:47:38.305 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T03:47:38.305 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T03:47:38.305 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T03:47:38.305 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-15T03:47:38.305 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-15T03:47:38.305 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-15T03:47:38.305 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-15T03:47:38.305 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-15T03:47:38.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.306 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-15T03:47:38.306 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-15T03:47:38.306 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-15T03:47:38.306 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-15T03:47:38.306 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-15T03:47:38.306 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T03:47:38.306 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-15T03:47:38.306 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-15T03:47:38.306 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-15T03:47:38.306 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-15T03:47:38.306 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-15T03:47:38.306 INFO:teuthology.orchestra.run.smithi001.stdout: "options": { 2024-09-15T03:47:38.306 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_max": 32, 2024-09-15T03:47:38.307 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_min": 1 2024-09-15T03:47:38.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.307 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-15T03:47:38.307 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": {} 2024-09-15T03:47:38.307 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.307 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.307 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 2, 2024-09-15T03:47:38.307 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": "rbd", 2024-09-15T03:47:38.307 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-15T03:46:22.284406+0000", 2024-09-15T03:47:38.307 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 8193, 2024-09-15T03:47:38.307 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-15T03:47:38.307 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 1, 2024-09-15T03:47:38.308 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 2, 2024-09-15T03:47:38.308 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 1, 2024-09-15T03:47:38.308 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 0, 2024-09-15T03:47:38.308 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-15T03:47:38.308 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-15T03:47:38.308 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T03:47:38.308 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T03:47:38.308 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-15T03:47:38.308 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-15T03:47:38.308 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 8, 2024-09-15T03:47:38.308 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 8, 2024-09-15T03:47:38.309 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 8, 2024-09-15T03:47:38.309 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 8, 2024-09-15T03:47:38.309 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 8, 2024-09-15T03:47:38.309 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-15T03:47:38.309 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-15T03:47:38.309 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-15T03:47:38.309 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-15T03:47:38.309 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-15T03:47:38.309 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-15T03:47:38.309 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-15T03:47:38.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.310 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "24", 2024-09-15T03:47:38.310 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-15T03:47:38.310 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T03:47:38.310 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T03:47:38.310 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-15T03:47:38.310 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-15T03:47:38.310 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 3, 2024-09-15T03:47:38.310 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 24, 2024-09-15T03:47:38.311 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-15T03:47:38.311 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-15T03:47:38.311 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-15T03:47:38.311 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-15T03:47:38.311 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-15T03:47:38.311 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-15T03:47:38.311 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-15T03:47:38.311 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-15T03:47:38.311 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-15T03:47:38.311 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-15T03:47:38.311 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-15T03:47:38.311 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T03:47:38.312 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T03:47:38.312 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T03:47:38.312 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-15T03:47:38.312 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-15T03:47:38.312 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "", 2024-09-15T03:47:38.312 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-15T03:47:38.312 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-15T03:47:38.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.312 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-15T03:47:38.312 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-15T03:47:38.312 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-15T03:47:38.313 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-15T03:47:38.313 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-15T03:47:38.313 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T03:47:38.313 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-15T03:47:38.313 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-15T03:47:38.313 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 0, 2024-09-15T03:47:38.313 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-15T03:47:38.313 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-15T03:47:38.313 INFO:teuthology.orchestra.run.smithi001.stdout: "options": {}, 2024-09-15T03:47:38.313 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-15T03:47:38.313 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd": {} 2024-09-15T03:47:38.313 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.313 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.314 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 3, 2024-09-15T03:47:38.314 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": "unique_pool_0", 2024-09-15T03:47:38.314 INFO:teuthology.orchestra.run.smithi001.stdout: "create_time": "2024-09-15T03:46:34.105566+0000", 2024-09-15T03:47:38.314 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 8193, 2024-09-15T03:47:38.314 INFO:teuthology.orchestra.run.smithi001.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-15T03:47:38.314 INFO:teuthology.orchestra.run.smithi001.stdout: "type": 3, 2024-09-15T03:47:38.314 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 8, 2024-09-15T03:47:38.314 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": 5, 2024-09-15T03:47:38.314 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_rule": 1, 2024-09-15T03:47:38.314 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_count": 0, 2024-09-15T03:47:38.314 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_target": 0, 2024-09-15T03:47:38.314 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T03:47:38.315 INFO:teuthology.orchestra.run.smithi001.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T03:47:38.315 INFO:teuthology.orchestra.run.smithi001.stdout: "object_hash": 2, 2024-09-15T03:47:38.315 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscale_mode": "off", 2024-09-15T03:47:38.315 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": 16, 2024-09-15T03:47:38.315 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num": 16, 2024-09-15T03:47:38.315 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_placement_num_target": 16, 2024-09-15T03:47:38.315 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_target": 16, 2024-09-15T03:47:38.315 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num_pending": 16, 2024-09-15T03:47:38.315 INFO:teuthology.orchestra.run.smithi001.stdout: "last_pg_merge_meta": { 2024-09-15T03:47:38.315 INFO:teuthology.orchestra.run.smithi001.stdout: "source_pgid": "0.0", 2024-09-15T03:47:38.315 INFO:teuthology.orchestra.run.smithi001.stdout: "ready_epoch": 0, 2024-09-15T03:47:38.315 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_started": 0, 2024-09-15T03:47:38.316 INFO:teuthology.orchestra.run.smithi001.stdout: "last_epoch_clean": 0, 2024-09-15T03:47:38.316 INFO:teuthology.orchestra.run.smithi001.stdout: "source_version": "0'0", 2024-09-15T03:47:38.316 INFO:teuthology.orchestra.run.smithi001.stdout: "target_version": "0'0" 2024-09-15T03:47:38.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.316 INFO:teuthology.orchestra.run.smithi001.stdout: "last_change": "91", 2024-09-15T03:47:38.316 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend": "0", 2024-09-15T03:47:38.316 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T03:47:38.316 INFO:teuthology.orchestra.run.smithi001.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T03:47:38.316 INFO:teuthology.orchestra.run.smithi001.stdout: "auid": 0, 2024-09-15T03:47:38.316 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_mode": "selfmanaged", 2024-09-15T03:47:38.316 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_seq": 54, 2024-09-15T03:47:38.316 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_epoch": 91, 2024-09-15T03:47:38.317 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_snaps": [], 2024-09-15T03:47:38.317 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps": "[]", 2024-09-15T03:47:38.317 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_bytes": 0, 2024-09-15T03:47:38.317 INFO:teuthology.orchestra.run.smithi001.stdout: "quota_max_objects": 0, 2024-09-15T03:47:38.317 INFO:teuthology.orchestra.run.smithi001.stdout: "tiers": [], 2024-09-15T03:47:38.317 INFO:teuthology.orchestra.run.smithi001.stdout: "tier_of": -1, 2024-09-15T03:47:38.317 INFO:teuthology.orchestra.run.smithi001.stdout: "read_tier": -1, 2024-09-15T03:47:38.317 INFO:teuthology.orchestra.run.smithi001.stdout: "write_tier": -1, 2024-09-15T03:47:38.317 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_mode": "none", 2024-09-15T03:47:38.317 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_bytes": 0, 2024-09-15T03:47:38.317 INFO:teuthology.orchestra.run.smithi001.stdout: "target_max_objects": 0, 2024-09-15T03:47:38.317 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T03:47:38.318 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T03:47:38.318 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T03:47:38.318 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_flush_age": 0, 2024-09-15T03:47:38.318 INFO:teuthology.orchestra.run.smithi001.stdout: "cache_min_evict_age": 0, 2024-09-15T03:47:38.318 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profile": "lrcprofile", 2024-09-15T03:47:38.318 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_params": { 2024-09-15T03:47:38.318 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "none" 2024-09-15T03:47:38.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.318 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_period": 0, 2024-09-15T03:47:38.318 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_count": 0, 2024-09-15T03:47:38.318 INFO:teuthology.orchestra.run.smithi001.stdout: "use_gmt_hitset": true, 2024-09-15T03:47:38.318 INFO:teuthology.orchestra.run.smithi001.stdout: "min_read_recency_for_promote": 0, 2024-09-15T03:47:38.318 INFO:teuthology.orchestra.run.smithi001.stdout: "min_write_recency_for_promote": 0, 2024-09-15T03:47:38.319 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T03:47:38.319 INFO:teuthology.orchestra.run.smithi001.stdout: "hit_set_search_last_n": 0, 2024-09-15T03:47:38.319 INFO:teuthology.orchestra.run.smithi001.stdout: "grade_table": [], 2024-09-15T03:47:38.319 INFO:teuthology.orchestra.run.smithi001.stdout: "stripe_width": 16384, 2024-09-15T03:47:38.319 INFO:teuthology.orchestra.run.smithi001.stdout: "expected_num_objects": 0, 2024-09-15T03:47:38.319 INFO:teuthology.orchestra.run.smithi001.stdout: "fast_read": false, 2024-09-15T03:47:38.319 INFO:teuthology.orchestra.run.smithi001.stdout: "options": {}, 2024-09-15T03:47:38.319 INFO:teuthology.orchestra.run.smithi001.stdout: "application_metadata": { 2024-09-15T03:47:38.319 INFO:teuthology.orchestra.run.smithi001.stdout: "rados": {} 2024-09-15T03:47:38.319 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.319 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.319 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:47:38.319 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-15T03:47:38.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.320 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 0, 2024-09-15T03:47:38.320 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "cd433a6a-c875-4e6e-b89c-938caecbae3a", 2024-09-15T03:47:38.320 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:38.320 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:38.320 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:38.320 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:38.320 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:38.320 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:38.320 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:38.320 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 80, 2024-09-15T03:47:38.320 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:38.320 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:38.321 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:38.321 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.321 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.321 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.321 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6800", 2024-09-15T03:47:38.321 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:38.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.321 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.322 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.322 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6801", 2024-09-15T03:47:38.322 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:38.322 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.322 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.322 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.322 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:38.322 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.322 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.322 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.322 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6802", 2024-09-15T03:47:38.322 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:38.323 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.323 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.323 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6803", 2024-09-15T03:47:38.323 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:38.323 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.323 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.323 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.323 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:38.323 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.323 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.323 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6806", 2024-09-15T03:47:38.323 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:38.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.324 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.324 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6807", 2024-09-15T03:47:38.324 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:38.324 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.324 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.324 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:38.324 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.324 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.325 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6804", 2024-09-15T03:47:38.325 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:38.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.325 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.325 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.325 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6805", 2024-09-15T03:47:38.325 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28908 2024-09-15T03:47:38.325 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.325 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.325 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6801/28908", 2024-09-15T03:47:38.325 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6803/28908", 2024-09-15T03:47:38.326 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6807/28908", 2024-09-15T03:47:38.326 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6805/28908", 2024-09-15T03:47:38.326 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:38.326 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:38.326 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:38.326 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.326 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.326 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.326 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 1, 2024-09-15T03:47:38.326 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "789930f9-42bf-476b-9f09-a21bffe33a79", 2024-09-15T03:47:38.326 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:38.326 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:38.327 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:38.327 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:38.327 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:38.327 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:38.327 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:38.327 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 86, 2024-09-15T03:47:38.327 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:38.327 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:38.327 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:38.327 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.327 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.328 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6808", 2024-09-15T03:47:38.328 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:38.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.328 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.328 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6809", 2024-09-15T03:47:38.328 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:38.328 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.328 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.329 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:38.329 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.329 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.329 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.329 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6810", 2024-09-15T03:47:38.329 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:38.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.329 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.329 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.329 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6811", 2024-09-15T03:47:38.329 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:38.330 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.330 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.330 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:38.330 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.330 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.330 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6814", 2024-09-15T03:47:38.330 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:38.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.331 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.331 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6815", 2024-09-15T03:47:38.331 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:38.331 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.331 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.331 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:38.331 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.331 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.331 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6812", 2024-09-15T03:47:38.331 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:38.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.332 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.332 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6813", 2024-09-15T03:47:38.332 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28913 2024-09-15T03:47:38.332 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.332 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.332 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.332 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6809/28913", 2024-09-15T03:47:38.332 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6811/28913", 2024-09-15T03:47:38.332 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6815/28913", 2024-09-15T03:47:38.332 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6813/28913", 2024-09-15T03:47:38.332 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:38.333 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:38.333 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:38.333 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.333 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 2, 2024-09-15T03:47:38.333 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "9a83fec5-0d63-461c-b516-289443b3882b", 2024-09-15T03:47:38.333 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:38.333 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:38.333 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:38.333 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:38.333 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:38.333 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:38.334 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:38.334 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 88, 2024-09-15T03:47:38.334 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:38.334 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:38.334 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:38.334 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.334 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.334 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.334 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6821", 2024-09-15T03:47:38.334 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:38.334 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.334 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.334 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.335 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6823", 2024-09-15T03:47:38.335 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:38.335 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.335 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.335 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.335 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:38.336 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.336 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.336 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6825", 2024-09-15T03:47:38.336 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:38.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.337 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.337 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6827", 2024-09-15T03:47:38.337 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:38.337 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.337 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.337 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.337 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:38.337 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.337 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.337 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.337 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6830", 2024-09-15T03:47:38.337 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:38.337 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.338 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.338 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.338 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6831", 2024-09-15T03:47:38.338 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:38.338 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.338 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.338 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.338 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:38.338 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.338 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.338 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.338 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6828", 2024-09-15T03:47:38.338 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:38.339 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.339 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.339 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.339 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6829", 2024-09-15T03:47:38.339 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28911 2024-09-15T03:47:38.339 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.339 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.339 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.339 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6823/28911", 2024-09-15T03:47:38.339 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6827/28911", 2024-09-15T03:47:38.339 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6831/28911", 2024-09-15T03:47:38.339 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6829/28911", 2024-09-15T03:47:38.340 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:38.340 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:38.340 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:38.340 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.340 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.340 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.340 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 3, 2024-09-15T03:47:38.340 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "d944bf89-e183-496a-9ca3-7949741b4be4", 2024-09-15T03:47:38.340 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:38.340 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:38.341 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:38.341 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:38.341 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:38.341 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:38.341 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:38.341 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 89, 2024-09-15T03:47:38.341 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:38.341 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:38.341 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:38.341 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.342 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.342 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.342 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6816", 2024-09-15T03:47:38.342 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:38.342 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.342 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.342 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.342 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6817", 2024-09-15T03:47:38.342 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:38.342 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.343 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.343 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.343 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:38.343 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.343 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.344 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.344 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6818", 2024-09-15T03:47:38.344 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:38.344 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.344 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.344 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.344 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6819", 2024-09-15T03:47:38.344 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:38.344 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.344 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.344 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.345 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:38.345 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.345 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.345 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.345 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6824", 2024-09-15T03:47:38.345 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:38.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.345 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.345 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.345 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6826", 2024-09-15T03:47:38.345 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:38.346 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.346 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.346 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.346 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:38.346 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.346 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.346 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.346 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6820", 2024-09-15T03:47:38.346 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:38.346 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.347 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.347 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.347 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6822", 2024-09-15T03:47:38.347 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28914 2024-09-15T03:47:38.347 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.347 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.347 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.347 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6817/28914", 2024-09-15T03:47:38.347 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.1:6819/28914", 2024-09-15T03:47:38.348 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.1:6826/28914", 2024-09-15T03:47:38.348 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.1:6822/28914", 2024-09-15T03:47:38.348 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:38.348 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:38.348 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:38.348 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.348 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.349 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.349 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 4, 2024-09-15T03:47:38.349 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "15ec2dbd-d099-4aff-94b2-1bf9dbb00b56", 2024-09-15T03:47:38.349 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:38.349 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:38.349 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:38.349 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:38.349 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:38.349 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:38.349 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:38.350 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 89, 2024-09-15T03:47:38.350 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:38.350 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:38.350 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:38.350 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.350 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.350 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.350 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6803", 2024-09-15T03:47:38.350 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:38.350 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.350 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.351 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.351 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6805", 2024-09-15T03:47:38.351 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:38.351 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.351 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.351 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.351 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:38.351 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.351 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.351 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.351 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6807", 2024-09-15T03:47:38.351 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:38.352 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.352 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.352 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6809", 2024-09-15T03:47:38.352 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:38.352 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.352 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.352 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.352 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:38.352 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.353 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.353 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6814", 2024-09-15T03:47:38.353 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:38.353 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.353 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.353 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.353 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6815", 2024-09-15T03:47:38.353 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:38.353 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.354 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.354 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.354 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:38.354 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.354 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.354 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6811", 2024-09-15T03:47:38.354 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:38.354 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.354 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.354 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6813", 2024-09-15T03:47:38.354 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28468 2024-09-15T03:47:38.355 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.355 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.355 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.67:6805/28468", 2024-09-15T03:47:38.355 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.67:6809/28468", 2024-09-15T03:47:38.355 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.67:6815/28468", 2024-09-15T03:47:38.355 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.67:6813/28468", 2024-09-15T03:47:38.355 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:38.355 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:38.355 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:38.355 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.356 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 5, 2024-09-15T03:47:38.356 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "1e45de07-746b-4612-9a52-d05509cae9a3", 2024-09-15T03:47:38.356 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:38.356 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:38.356 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:38.356 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:38.356 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:38.356 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:38.356 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:38.357 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 86, 2024-09-15T03:47:38.357 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:38.357 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:38.357 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:38.357 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.357 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.357 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.357 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6824", 2024-09-15T03:47:38.357 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:38.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.358 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.358 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6825", 2024-09-15T03:47:38.358 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:38.358 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.358 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.359 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:38.359 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.359 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.359 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6826", 2024-09-15T03:47:38.360 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:38.360 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.360 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.360 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.360 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6827", 2024-09-15T03:47:38.361 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:38.361 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.361 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.361 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:38.361 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.362 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.362 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6830", 2024-09-15T03:47:38.362 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:38.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.363 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.363 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6831", 2024-09-15T03:47:38.363 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:38.363 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.364 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.364 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:38.364 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.365 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.365 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6828", 2024-09-15T03:47:38.366 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:38.366 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.366 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.366 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.366 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6829", 2024-09-15T03:47:38.367 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28495 2024-09-15T03:47:38.367 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.367 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.367 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.67:6825/28495", 2024-09-15T03:47:38.367 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.67:6827/28495", 2024-09-15T03:47:38.368 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.67:6831/28495", 2024-09-15T03:47:38.368 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.67:6829/28495", 2024-09-15T03:47:38.368 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:38.368 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:38.368 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:38.368 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.369 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.369 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.369 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 6, 2024-09-15T03:47:38.369 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "ccf785f8-751d-473e-b17c-3df9346be9f8", 2024-09-15T03:47:38.369 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:38.369 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:38.370 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:38.370 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:38.370 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:38.370 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:38.370 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:38.370 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 89, 2024-09-15T03:47:38.371 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:38.371 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:38.371 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:38.371 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.372 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.372 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6800", 2024-09-15T03:47:38.372 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:38.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.373 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.373 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6801", 2024-09-15T03:47:38.373 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:38.373 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.373 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.374 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:38.374 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.374 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.375 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6802", 2024-09-15T03:47:38.375 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:38.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.375 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.376 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6804", 2024-09-15T03:47:38.376 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:38.376 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.376 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.377 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:38.377 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.377 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.377 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6810", 2024-09-15T03:47:38.377 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:38.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.378 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.378 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6812", 2024-09-15T03:47:38.378 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:38.379 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.379 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.379 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:38.379 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.380 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.380 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6806", 2024-09-15T03:47:38.380 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:38.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.380 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.381 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6808", 2024-09-15T03:47:38.381 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28469 2024-09-15T03:47:38.381 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.381 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.382 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.67:6801/28469", 2024-09-15T03:47:38.382 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.67:6804/28469", 2024-09-15T03:47:38.382 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.67:6812/28469", 2024-09-15T03:47:38.382 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.67:6808/28469", 2024-09-15T03:47:38.382 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:38.383 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:38.383 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:38.383 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.383 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 7, 2024-09-15T03:47:38.384 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "f9f59987-869d-42eb-9eb6-08c39e3afe6d", 2024-09-15T03:47:38.384 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:38.384 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:38.384 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:38.384 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:38.384 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:38.385 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:38.385 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:38.386 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 89, 2024-09-15T03:47:38.386 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:38.386 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:38.386 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:38.386 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.386 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.386 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.386 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6816", 2024-09-15T03:47:38.387 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:38.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.387 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6817", 2024-09-15T03:47:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6818", 2024-09-15T03:47:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:38.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.389 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.389 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6819", 2024-09-15T03:47:38.389 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:38.389 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.389 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.389 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:38.390 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.390 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.390 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6822", 2024-09-15T03:47:38.390 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:38.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.390 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.390 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6823", 2024-09-15T03:47:38.390 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:38.390 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.390 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.391 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:38.391 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.391 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.391 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6820", 2024-09-15T03:47:38.391 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:38.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.391 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.391 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.67:6821", 2024-09-15T03:47:38.391 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 28470 2024-09-15T03:47:38.391 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.391 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.392 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.67:6817/28470", 2024-09-15T03:47:38.392 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.67:6819/28470", 2024-09-15T03:47:38.392 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.67:6823/28470", 2024-09-15T03:47:38.392 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.67:6821/28470", 2024-09-15T03:47:38.392 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:38.392 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:38.392 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:38.392 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.392 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 8, 2024-09-15T03:47:38.393 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "69ddfe52-11df-4608-9a10-e8907c4cd87f", 2024-09-15T03:47:38.393 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:38.393 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:38.393 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:38.393 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:38.393 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:38.393 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:38.393 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:38.393 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 89, 2024-09-15T03:47:38.393 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:38.393 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:38.393 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:38.393 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.394 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.394 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6808", 2024-09-15T03:47:38.394 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:38.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.394 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.394 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6809", 2024-09-15T03:47:38.394 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:38.394 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.394 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.394 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6810", 2024-09-15T03:47:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6811", 2024-09-15T03:47:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.396 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:38.396 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.396 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.396 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6814", 2024-09-15T03:47:38.396 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:38.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.396 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.396 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6815", 2024-09-15T03:47:38.396 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:38.396 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.396 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.397 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.397 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:38.397 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.397 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.397 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.397 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6812", 2024-09-15T03:47:38.397 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:38.397 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.397 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.397 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6813", 2024-09-15T03:47:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29472 2024-09-15T03:47:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.190:6809/29472", 2024-09-15T03:47:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.190:6811/29472", 2024-09-15T03:47:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.190:6815/29472", 2024-09-15T03:47:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.190:6813/29472", 2024-09-15T03:47:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 9, 2024-09-15T03:47:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "f92e1609-2a1a-4814-9ac0-44a73373d229", 2024-09-15T03:47:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 89, 2024-09-15T03:47:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6800", 2024-09-15T03:47:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6801", 2024-09-15T03:47:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6802", 2024-09-15T03:47:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6803", 2024-09-15T03:47:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6806", 2024-09-15T03:47:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6807", 2024-09-15T03:47:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6804", 2024-09-15T03:47:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6805", 2024-09-15T03:47:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29473 2024-09-15T03:47:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.190:6801/29473", 2024-09-15T03:47:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.190:6803/29473", 2024-09-15T03:47:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.190:6807/29473", 2024-09-15T03:47:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.190:6805/29473", 2024-09-15T03:47:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 10, 2024-09-15T03:47:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "5f2e8ca3-8c65-4595-8e82-27dc3768ab99", 2024-09-15T03:47:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 0, 2024-09-15T03:47:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 70, 2024-09-15T03:47:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 80, 2024-09-15T03:47:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6816", 2024-09-15T03:47:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6817", 2024-09-15T03:47:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6818", 2024-09-15T03:47:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6819", 2024-09-15T03:47:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6822", 2024-09-15T03:47:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6823", 2024-09-15T03:47:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6820", 2024-09-15T03:47:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6821", 2024-09-15T03:47:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29474 2024-09-15T03:47:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.190:6817/29474", 2024-09-15T03:47:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.190:6819/29474", 2024-09-15T03:47:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.190:6823/29474", 2024-09-15T03:47:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.190:6821/29474", 2024-09-15T03:47:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: "exists" 2024-09-15T03:47:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 11, 2024-09-15T03:47:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: "uuid": "cedf7df4-f875-42f5-a8cf-13cb82f05cc2", 2024-09-15T03:47:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: "up": 1, 2024-09-15T03:47:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: "in": 1, 2024-09-15T03:47:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 1, 2024-09-15T03:47:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_affinity": 1, 2024-09-15T03:47:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_begin": 0, 2024-09-15T03:47:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: "last_clean_end": 0, 2024-09-15T03:47:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: "up_from": 17, 2024-09-15T03:47:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: "up_thru": 80, 2024-09-15T03:47:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: "down_at": 0, 2024-09-15T03:47:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: "lost_at": 0, 2024-09-15T03:47:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-09-15T03:47:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.412 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6824", 2024-09-15T03:47:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6825", 2024-09-15T03:47:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addrs": { 2024-09-15T03:47:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6826", 2024-09-15T03:47:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6827", 2024-09-15T03:47:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addrs": { 2024-09-15T03:47:38.414 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6830", 2024-09-15T03:47:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6831", 2024-09-15T03:47:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addrs": { 2024-09-15T03:47:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-09-15T03:47:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-09-15T03:47:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6828", 2024-09-15T03:47:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-09-15T03:47:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.190:6829", 2024-09-15T03:47:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 29475 2024-09-15T03:47:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.416 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.190:6825/29475", 2024-09-15T03:47:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_addr": "172.21.15.190:6827/29475", 2024-09-15T03:47:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_back_addr": "172.21.15.190:6831/29475", 2024-09-15T03:47:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: "heartbeat_front_addr": "172.21.15.190:6829/29475", 2024-09-15T03:47:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: "state": [ 2024-09-15T03:47:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: "exists", 2024-09-15T03:47:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: "up" 2024-09-15T03:47:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:47:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_xinfo": [ 2024-09-15T03:47:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.417 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 0, 2024-09-15T03:47:38.418 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:38.418 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:38.418 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:38.418 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:38.418 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:38.418 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:47:37.565633+0000", 2024-09-15T03:47:38.418 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:38.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.418 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 1, 2024-09-15T03:47:38.418 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:38.418 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:38.418 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:47:18.975241+0000", 2024-09-15T03:47:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 2, 2024-09-15T03:47:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:38.419 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:47:25.983954+0000", 2024-09-15T03:47:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 3, 2024-09-15T03:47:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:47:26.751379+0000", 2024-09-15T03:47:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 4, 2024-09-15T03:47:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:47:25.908552+0000", 2024-09-15T03:47:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 5, 2024-09-15T03:47:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:38.421 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:47:34.158155+0000", 2024-09-15T03:47:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 6, 2024-09-15T03:47:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:38.422 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:47:37.031163+0000", 2024-09-15T03:47:38.423 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:38.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.423 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 7, 2024-09-15T03:47:38.423 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:38.423 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:38.423 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:38.423 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:38.423 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:38.423 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:47:18.853134+0000", 2024-09-15T03:47:38.423 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:38.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.424 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 8, 2024-09-15T03:47:38.424 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:38.424 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:38.424 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:38.424 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:38.424 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:38.424 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:47:29.597174+0000", 2024-09-15T03:47:38.424 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:38.424 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.424 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 9, 2024-09-15T03:47:38.424 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:38.425 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:38.425 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:38.425 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:38.425 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:38.425 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:47:37.409413+0000", 2024-09-15T03:47:38.425 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:38.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.425 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 10, 2024-09-15T03:47:38.425 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:38.425 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:38.425 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:38.425 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:38.425 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:38.426 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:13.993934+0000", 2024-09-15T03:47:38.426 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 80 2024-09-15T03:47:38.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.426 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": 11, 2024-09-15T03:47:38.426 INFO:teuthology.orchestra.run.smithi001.stdout: "down_stamp": "0.000000", 2024-09-15T03:47:38.426 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_probability": 0, 2024-09-15T03:47:38.426 INFO:teuthology.orchestra.run.smithi001.stdout: "laggy_interval": 0, 2024-09-15T03:47:38.426 INFO:teuthology.orchestra.run.smithi001.stdout: "features": 4540138320759226367, 2024-09-15T03:47:38.426 INFO:teuthology.orchestra.run.smithi001.stdout: "old_weight": 0, 2024-09-15T03:47:38.426 INFO:teuthology.orchestra.run.smithi001.stdout: "last_purged_snaps_scrub": "2024-09-15T03:47:27.962448+0000", 2024-09-15T03:47:38.426 INFO:teuthology.orchestra.run.smithi001.stdout: "dead_epoch": 0 2024-09-15T03:47:38.426 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.427 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:47:38.427 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap": [ 2024-09-15T03:47:38.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.427 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.a", 2024-09-15T03:47:38.427 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-15T03:47:38.427 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-09-15T03:47:38.427 INFO:teuthology.orchestra.run.smithi001.stdout: 10, 2024-09-15T03:47:38.427 INFO:teuthology.orchestra.run.smithi001.stdout: 8, 2024-09-15T03:47:38.427 INFO:teuthology.orchestra.run.smithi001.stdout: 9, 2024-09-15T03:47:38.427 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-09-15T03:47:38.427 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-09-15T03:47:38.427 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-09-15T03:47:38.427 INFO:teuthology.orchestra.run.smithi001.stdout: 0 2024-09-15T03:47:38.427 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.428 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.428 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:47:38.428 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_upmap_items": [], 2024-09-15T03:47:38.428 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_temp": [ 2024-09-15T03:47:38.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.428 INFO:teuthology.orchestra.run.smithi001.stdout: "pgid": "3.a", 2024-09-15T03:47:38.428 INFO:teuthology.orchestra.run.smithi001.stdout: "osds": [ 2024-09-15T03:47:38.428 INFO:teuthology.orchestra.run.smithi001.stdout: 4, 2024-09-15T03:47:38.428 INFO:teuthology.orchestra.run.smithi001.stdout: 8, 2024-09-15T03:47:38.428 INFO:teuthology.orchestra.run.smithi001.stdout: 8, 2024-09-15T03:47:38.428 INFO:teuthology.orchestra.run.smithi001.stdout: 9, 2024-09-15T03:47:38.428 INFO:teuthology.orchestra.run.smithi001.stdout: 5, 2024-09-15T03:47:38.428 INFO:teuthology.orchestra.run.smithi001.stdout: 6, 2024-09-15T03:47:38.429 INFO:teuthology.orchestra.run.smithi001.stdout: 2, 2024-09-15T03:47:38.429 INFO:teuthology.orchestra.run.smithi001.stdout: 0 2024-09-15T03:47:38.429 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.429 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.429 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:47:38.429 INFO:teuthology.orchestra.run.smithi001.stdout: "primary_temp": [], 2024-09-15T03:47:38.429 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklist": {}, 2024-09-15T03:47:38.429 INFO:teuthology.orchestra.run.smithi001.stdout: "range_blocklist": {}, 2024-09-15T03:47:38.429 INFO:teuthology.orchestra.run.smithi001.stdout: "erasure_code_profiles": { 2024-09-15T03:47:38.429 INFO:teuthology.orchestra.run.smithi001.stdout: "default": { 2024-09-15T03:47:38.429 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-failure-domain": "osd", 2024-09-15T03:47:38.429 INFO:teuthology.orchestra.run.smithi001.stdout: "k": "2", 2024-09-15T03:47:38.429 INFO:teuthology.orchestra.run.smithi001.stdout: "m": "1", 2024-09-15T03:47:38.430 INFO:teuthology.orchestra.run.smithi001.stdout: "plugin": "jerasure", 2024-09-15T03:47:38.430 INFO:teuthology.orchestra.run.smithi001.stdout: "technique": "reed_sol_van" 2024-09-15T03:47:38.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.430 INFO:teuthology.orchestra.run.smithi001.stdout: "lrcprofile": { 2024-09-15T03:47:38.430 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-device-class": "", 2024-09-15T03:47:38.430 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-failure-domain": "osd", 2024-09-15T03:47:38.430 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-root": "default", 2024-09-15T03:47:38.430 INFO:teuthology.orchestra.run.smithi001.stdout: "k": "4", 2024-09-15T03:47:38.430 INFO:teuthology.orchestra.run.smithi001.stdout: "l": "3", 2024-09-15T03:47:38.430 INFO:teuthology.orchestra.run.smithi001.stdout: "m": "2", 2024-09-15T03:47:38.430 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "lrcprofile", 2024-09-15T03:47:38.430 INFO:teuthology.orchestra.run.smithi001.stdout: "plugin": "lrc" 2024-09-15T03:47:38.430 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.431 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_snaps_queue": [ 2024-09-15T03:47:38.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.431 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 3, 2024-09-15T03:47:38.431 INFO:teuthology.orchestra.run.smithi001.stdout: "snaps": [ 2024-09-15T03:47:38.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.431 INFO:teuthology.orchestra.run.smithi001.stdout: "begin": 12, 2024-09-15T03:47:38.431 INFO:teuthology.orchestra.run.smithi001.stdout: "length": 1 2024-09-15T03:47:38.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.431 INFO:teuthology.orchestra.run.smithi001.stdout: "begin": 23, 2024-09-15T03:47:38.431 INFO:teuthology.orchestra.run.smithi001.stdout: "length": 1 2024-09-15T03:47:38.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.432 INFO:teuthology.orchestra.run.smithi001.stdout: "begin": 35, 2024-09-15T03:47:38.432 INFO:teuthology.orchestra.run.smithi001.stdout: "length": 1 2024-09-15T03:47:38.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.432 INFO:teuthology.orchestra.run.smithi001.stdout: "begin": 41, 2024-09-15T03:47:38.432 INFO:teuthology.orchestra.run.smithi001.stdout: "length": 1 2024-09-15T03:47:38.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:38.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.432 INFO:teuthology.orchestra.run.smithi001.stdout: "begin": 47, 2024-09-15T03:47:38.432 INFO:teuthology.orchestra.run.smithi001.stdout: "length": 1 2024-09-15T03:47:38.432 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.432 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.433 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.433 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:47:38.433 INFO:teuthology.orchestra.run.smithi001.stdout: "new_removed_snaps": [ 2024-09-15T03:47:38.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.433 INFO:teuthology.orchestra.run.smithi001.stdout: "pool": 3, 2024-09-15T03:47:38.433 INFO:teuthology.orchestra.run.smithi001.stdout: "snaps": [ 2024-09-15T03:47:38.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:38.433 INFO:teuthology.orchestra.run.smithi001.stdout: "begin": 35, 2024-09-15T03:47:38.433 INFO:teuthology.orchestra.run.smithi001.stdout: "length": 1 2024-09-15T03:47:38.433 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.433 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:38.433 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.433 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-09-15T03:47:38.434 INFO:teuthology.orchestra.run.smithi001.stdout: "new_purged_snaps": [], 2024-09-15T03:47:38.434 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_node_flags": {}, 2024-09-15T03:47:38.434 INFO:teuthology.orchestra.run.smithi001.stdout: "device_class_flags": {}, 2024-09-15T03:47:38.434 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": { 2024-09-15T03:47:38.434 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode_enabled": false, 2024-09-15T03:47:38.434 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_bucket_count": 0, 2024-09-15T03:47:38.434 INFO:teuthology.orchestra.run.smithi001.stdout: "degraded_stretch_mode": 0, 2024-09-15T03:47:38.434 INFO:teuthology.orchestra.run.smithi001.stdout: "recovering_stretch_mode": 0, 2024-09-15T03:47:38.434 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode_bucket": 0 2024-09-15T03:47:38.434 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:38.434 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:38.437 INFO:tasks.thrashosds.thrasher:j is {'epoch': 91, 'fsid': 'f27b540e-e5e3-4e95-afe8-c95ee66b0a62', 'created': '2024-09-15T03:46:09.843094+0000', 'modified': '2024-09-15T03:47:37.837065+0000', 'last_up_change': '2024-09-15T03:47:26.955166+0000', 'last_in_change': '2024-09-15T03:47:32.982084+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-15T03:46:16.185530+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-15T03:46:22.284406+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 24, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-15T03:46:34.105566+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 8, 'min_size': 5, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 54, 'snap_epoch': 91, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'lrcprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 16384, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'cd433a6a-c875-4e6e-b89c-938caecbae3a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 80, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6800', 'nonce': 28908}, {'type': 'v1', 'addr': '172.21.15.1:6801', 'nonce': 28908}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6802', 'nonce': 28908}, {'type': 'v1', 'addr': '172.21.15.1:6803', 'nonce': 28908}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6806', 'nonce': 28908}, {'type': 'v1', 'addr': '172.21.15.1:6807', 'nonce': 28908}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6804', 'nonce': 28908}, {'type': 'v1', 'addr': '172.21.15.1:6805', 'nonce': 28908}]}, 'public_addr': '172.21.15.1:6801/28908', 'cluster_addr': '172.21.15.1:6803/28908', 'heartbeat_back_addr': '172.21.15.1:6807/28908', 'heartbeat_front_addr': '172.21.15.1:6805/28908', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '789930f9-42bf-476b-9f09-a21bffe33a79', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 86, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6808', 'nonce': 28913}, {'type': 'v1', 'addr': '172.21.15.1:6809', 'nonce': 28913}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6810', 'nonce': 28913}, {'type': 'v1', 'addr': '172.21.15.1:6811', 'nonce': 28913}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6814', 'nonce': 28913}, {'type': 'v1', 'addr': '172.21.15.1:6815', 'nonce': 28913}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6812', 'nonce': 28913}, {'type': 'v1', 'addr': '172.21.15.1:6813', 'nonce': 28913}]}, 'public_addr': '172.21.15.1:6809/28913', 'cluster_addr': '172.21.15.1:6811/28913', 'heartbeat_back_addr': '172.21.15.1:6815/28913', 'heartbeat_front_addr': '172.21.15.1:6813/28913', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '9a83fec5-0d63-461c-b516-289443b3882b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 88, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6821', 'nonce': 28911}, {'type': 'v1', 'addr': '172.21.15.1:6823', 'nonce': 28911}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6825', 'nonce': 28911}, {'type': 'v1', 'addr': '172.21.15.1:6827', 'nonce': 28911}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6830', 'nonce': 28911}, {'type': 'v1', 'addr': '172.21.15.1:6831', 'nonce': 28911}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6828', 'nonce': 28911}, {'type': 'v1', 'addr': '172.21.15.1:6829', 'nonce': 28911}]}, 'public_addr': '172.21.15.1:6823/28911', 'cluster_addr': '172.21.15.1:6827/28911', 'heartbeat_back_addr': '172.21.15.1:6831/28911', 'heartbeat_front_addr': '172.21.15.1:6829/28911', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd944bf89-e183-496a-9ca3-7949741b4be4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 89, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6816', 'nonce': 28914}, {'type': 'v1', 'addr': '172.21.15.1:6817', 'nonce': 28914}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6818', 'nonce': 28914}, {'type': 'v1', 'addr': '172.21.15.1:6819', 'nonce': 28914}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6824', 'nonce': 28914}, {'type': 'v1', 'addr': '172.21.15.1:6826', 'nonce': 28914}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6820', 'nonce': 28914}, {'type': 'v1', 'addr': '172.21.15.1:6822', 'nonce': 28914}]}, 'public_addr': '172.21.15.1:6817/28914', 'cluster_addr': '172.21.15.1:6819/28914', 'heartbeat_back_addr': '172.21.15.1:6826/28914', 'heartbeat_front_addr': '172.21.15.1:6822/28914', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '15ec2dbd-d099-4aff-94b2-1bf9dbb00b56', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 89, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6803', 'nonce': 28468}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 28468}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6807', 'nonce': 28468}, {'type': 'v1', 'addr': '172.21.15.67:6809', 'nonce': 28468}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 28468}, {'type': 'v1', 'addr': '172.21.15.67:6815', 'nonce': 28468}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6811', 'nonce': 28468}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 28468}]}, 'public_addr': '172.21.15.67:6805/28468', 'cluster_addr': '172.21.15.67:6809/28468', 'heartbeat_back_addr': '172.21.15.67:6815/28468', 'heartbeat_front_addr': '172.21.15.67:6813/28468', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1e45de07-746b-4612-9a52-d05509cae9a3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 86, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6824', 'nonce': 28495}, {'type': 'v1', 'addr': '172.21.15.67:6825', 'nonce': 28495}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6826', 'nonce': 28495}, {'type': 'v1', 'addr': '172.21.15.67:6827', 'nonce': 28495}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6830', 'nonce': 28495}, {'type': 'v1', 'addr': '172.21.15.67:6831', 'nonce': 28495}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6828', 'nonce': 28495}, {'type': 'v1', 'addr': '172.21.15.67:6829', 'nonce': 28495}]}, 'public_addr': '172.21.15.67:6825/28495', 'cluster_addr': '172.21.15.67:6827/28495', 'heartbeat_back_addr': '172.21.15.67:6831/28495', 'heartbeat_front_addr': '172.21.15.67:6829/28495', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ccf785f8-751d-473e-b17c-3df9346be9f8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 89, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 28469}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 28469}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 28469}, {'type': 'v1', 'addr': '172.21.15.67:6804', 'nonce': 28469}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 28469}, {'type': 'v1', 'addr': '172.21.15.67:6812', 'nonce': 28469}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 28469}, {'type': 'v1', 'addr': '172.21.15.67:6808', 'nonce': 28469}]}, 'public_addr': '172.21.15.67:6801/28469', 'cluster_addr': '172.21.15.67:6804/28469', 'heartbeat_back_addr': '172.21.15.67:6812/28469', 'heartbeat_front_addr': '172.21.15.67:6808/28469', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f9f59987-869d-42eb-9eb6-08c39e3afe6d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 89, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 28470}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 28470}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6818', 'nonce': 28470}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 28470}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6822', 'nonce': 28470}, {'type': 'v1', 'addr': '172.21.15.67:6823', 'nonce': 28470}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 28470}, {'type': 'v1', 'addr': '172.21.15.67:6821', 'nonce': 28470}]}, 'public_addr': '172.21.15.67:6817/28470', 'cluster_addr': '172.21.15.67:6819/28470', 'heartbeat_back_addr': '172.21.15.67:6823/28470', 'heartbeat_front_addr': '172.21.15.67:6821/28470', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '69ddfe52-11df-4608-9a10-e8907c4cd87f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 89, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6808', 'nonce': 29472}, {'type': 'v1', 'addr': '172.21.15.190:6809', 'nonce': 29472}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6810', 'nonce': 29472}, {'type': 'v1', 'addr': '172.21.15.190:6811', 'nonce': 29472}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6814', 'nonce': 29472}, {'type': 'v1', 'addr': '172.21.15.190:6815', 'nonce': 29472}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6812', 'nonce': 29472}, {'type': 'v1', 'addr': '172.21.15.190:6813', 'nonce': 29472}]}, 'public_addr': '172.21.15.190:6809/29472', 'cluster_addr': '172.21.15.190:6811/29472', 'heartbeat_back_addr': '172.21.15.190:6815/29472', 'heartbeat_front_addr': '172.21.15.190:6813/29472', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'f92e1609-2a1a-4814-9ac0-44a73373d229', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 89, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6800', 'nonce': 29473}, {'type': 'v1', 'addr': '172.21.15.190:6801', 'nonce': 29473}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6802', 'nonce': 29473}, {'type': 'v1', 'addr': '172.21.15.190:6803', 'nonce': 29473}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6806', 'nonce': 29473}, {'type': 'v1', 'addr': '172.21.15.190:6807', 'nonce': 29473}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6804', 'nonce': 29473}, {'type': 'v1', 'addr': '172.21.15.190:6805', 'nonce': 29473}]}, 'public_addr': '172.21.15.190:6801/29473', 'cluster_addr': '172.21.15.190:6803/29473', 'heartbeat_back_addr': '172.21.15.190:6807/29473', 'heartbeat_front_addr': '172.21.15.190:6805/29473', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '5f2e8ca3-8c65-4595-8e82-27dc3768ab99', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 70, 'down_at': 80, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6816', 'nonce': 29474}, {'type': 'v1', 'addr': '172.21.15.190:6817', 'nonce': 29474}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6818', 'nonce': 29474}, {'type': 'v1', 'addr': '172.21.15.190:6819', 'nonce': 29474}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6822', 'nonce': 29474}, {'type': 'v1', 'addr': '172.21.15.190:6823', 'nonce': 29474}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6820', 'nonce': 29474}, {'type': 'v1', 'addr': '172.21.15.190:6821', 'nonce': 29474}]}, 'public_addr': '172.21.15.190:6817/29474', 'cluster_addr': '172.21.15.190:6819/29474', 'heartbeat_back_addr': '172.21.15.190:6823/29474', 'heartbeat_front_addr': '172.21.15.190:6821/29474', 'state': ['exists']}, {'osd': 11, 'uuid': 'cedf7df4-f875-42f5-a8cf-13cb82f05cc2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 80, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6824', 'nonce': 29475}, {'type': 'v1', 'addr': '172.21.15.190:6825', 'nonce': 29475}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6826', 'nonce': 29475}, {'type': 'v1', 'addr': '172.21.15.190:6827', 'nonce': 29475}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6830', 'nonce': 29475}, {'type': 'v1', 'addr': '172.21.15.190:6831', 'nonce': 29475}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6828', 'nonce': 29475}, {'type': 'v1', 'addr': '172.21.15.190:6829', 'nonce': 29475}]}, 'public_addr': '172.21.15.190:6825/29475', 'cluster_addr': '172.21.15.190:6827/29475', 'heartbeat_back_addr': '172.21.15.190:6831/29475', 'heartbeat_front_addr': '172.21.15.190:6829/29475', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:47:37.565633+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:47:18.975241+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:47:25.983954+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:47:26.751379+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:47:25.908552+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:47:34.158155+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:47:37.031163+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:47:18.853134+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:47:29.597174+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:47:37.409413+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:46:13.993934+0000', 'dead_epoch': 80}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-15T03:47:27.962448+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.a', 'osds': [4, 10, 8, 9, 5, 6, 2, 0]}], 'pg_upmap_items': [], 'pg_temp': [{'pgid': '3.a', 'osds': [4, 8, 8, 9, 5, 6, 2, 0]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'lrcprofile': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'k': '4', 'l': '3', 'm': '2', 'name': 'lrcprofile', 'plugin': 'lrc'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 12, 'length': 1}, {'begin': 23, 'length': 1}, {'begin': 35, 'length': 1}, {'begin': 41, 'length': 1}, {'begin': 47, 'length': 1}]}], 'new_removed_snaps': [{'pool': 3, 'snaps': [{'begin': 35, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-15T03:47:38.437 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:47:38.438 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:38.438 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:38.439 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:47:38.439 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:38.435+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:38.537 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:38.536+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:38.582 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:38.582 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:38.591 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:47:38.676 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:38.675+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:38.733 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:38.733 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:38.738 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:38.738+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:38.741 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:47:38.758 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:47:38.759 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-15T03:47:38.769 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-09-15T03:47:38.157112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":119652915,"num_objects":162,"num_object_clones":108,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":67,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":9,"num_read":432,"num_read_kb":165578,"num_write":836,"num_write_kb":113560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":184188966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":919,"ondisk_log_size":919,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":157,"num_osds":12,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":1124155392,"kb_used":10117736,"kb_used_data":1114102296,"kb_used_omap":36828,"kb_used_meta":0,"kb_avail":1114037656,"statfs":{"total":1151135121408,"available":1140774559744,"internally_reserved":66191360,"allocated":1140840751104,"data_stored":1140840751104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37712542,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":22,"commit_latency_ns":0,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17514496,"num_objects":14,"num_object_clones":14,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-44,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":24,"num_read_kb":40824,"num_write":75,"num_write_kb":11232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":56426515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000876"},"pg_stats":[{"pgid":"3.f","version":"83'47","reported_seq":308,"reported_epoch":88,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:47:34.950607+0000","last_change":"2024-09-15T03:47:27.975584+0000","last_active":"2024-09-15T03:47:34.950607+0000","last_peered":"2024-09-15T03:47:34.950607+0000","last_clean":"2024-09-15T03:47:25.960605+0000","last_became_active":"2024-09-15T03:47:27.975584+0000","last_became_peered":"2024-09-15T03:47:27.975584+0000","last_unstale":"2024-09-15T03:47:34.950607+0000","last_undegraded":"2024-09-15T03:47:27.969218+0000","last_fullsized":"2024-09-15T03:47:27.968920+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:39.670146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043542,"stat_sum":{"num_bytes":8028160,"num_objects":9,"num_object_clones":6,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":39433,"num_write":26,"num_write_kb":7840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":9994240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0,7,2,8,4,2147483647],"acting":[11,1,0,7,2,8,4,2147483647],"avail_no_missing":["11(0)","0(2)","1(1)","2(4)","4(6)","7(3)","8(5)"],"object_location_counts":[{"shards":"0(2),1(1),2(4),4(6),7(3),8(5),11(0)","objects":9}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"86'122","reported_seq":287,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T03:47:34.836804+0000","last_change":"2024-09-15T03:47:33.012409+0000","last_active":"2024-09-15T03:47:34.836804+0000","last_peered":"2024-09-15T03:47:34.836804+0000","last_clean":"2024-09-15T03:47:34.836804+0000","last_became_active":"2024-09-15T03:46:36.170536+0000","last_became_peered":"2024-09-15T03:46:36.170536+0000","last_unstale":"2024-09-15T03:47:34.836804+0000","last_undegraded":"2024-09-15T03:47:34.836804+0000","last_fullsized":"2024-09-15T03:47:34.836804+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:40.141809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020748289,"stat_sum":{"num_bytes":15351808,"num_objects":21,"num_object_clones":14,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":3,"num_read":53,"num_read_kb":16292,"num_write":48,"num_write_kb":13312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,11,7,8,5,9],"acting":[2,0,1,11,7,8,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c","length":"1"},{"start":"17","length":"1"},{"start":"29","length":"1"},{"start":"2f","length":"1"}]},{"pgid":"3.d","version":"81'46","reported_seq":245,"reported_epoch":88,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:47:34.838983+0000","last_change":"2024-09-15T03:47:27.975768+0000","last_active":"2024-09-15T03:47:34.838983+0000","last_peered":"2024-09-15T03:47:34.838983+0000","last_clean":"2024-09-15T03:47:25.958329+0000","last_became_active":"2024-09-15T03:47:27.975768+0000","last_became_peered":"2024-09-15T03:47:27.975768+0000","last_unstale":"2024-09-15T03:47:34.838983+0000","last_undegraded":"2024-09-15T03:47:27.967633+0000","last_fullsized":"2024-09-15T03:47:27.967369+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:50.627401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0090285959999999998,"stat_sum":{"num_bytes":5931008,"num_objects":9,"num_object_clones":7,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":19,"num_read_kb":8406,"num_write":22,"num_write_kb":5792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8044544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,1,3,2147483647,7,9,4],"acting":[8,11,1,3,2147483647,7,9,4],"avail_no_missing":["8(0)","1(2)","3(3)","4(7)","7(5)","9(6)","11(1)"],"object_location_counts":[{"shards":"1(2),3(3),4(7),7(5),8(0),9(6),11(1)","objects":9}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.c","version":"58'14","reported_seq":266,"reported_epoch":89,"state":"peering","last_fresh":"2024-09-15T03:47:35.840678+0000","last_change":"2024-09-15T03:47:35.841029+0000","last_active":"2024-09-15T03:47:35.840646+0000","last_peered":"2024-09-15T03:47:34.946521+0000","last_clean":"2024-09-15T03:47:25.960729+0000","last_became_active":"2024-09-15T03:47:34.847258+0000","last_became_peered":"2024-09-15T03:47:34.847258+0000","last_unstale":"2024-09-15T03:47:35.840678+0000","last_undegraded":"2024-09-15T03:47:35.840678+0000","last_fullsized":"2024-09-15T03:47:35.840678+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:03.163203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048204299999999998,"stat_sum":{"num_bytes":1720320,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":13,"num_read_kb":5045,"num_write":9,"num_write_kb":2224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,11,1,0,6,5,2],"acting":[9,7,11,1,0,6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2,5,6,7,11],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.b","version":"88'115","reported_seq":527,"reported_epoch":89,"state":"peering","last_fresh":"2024-09-15T03:47:35.841529+0000","last_change":"2024-09-15T03:47:35.842256+0000","last_active":"2024-09-15T03:47:35.841495+0000","last_peered":"2024-09-15T03:47:35.023125+0000","last_clean":"2024-09-15T03:47:25.956196+0000","last_became_active":"2024-09-15T03:47:34.854297+0000","last_became_peered":"2024-09-15T03:47:34.854297+0000","last_unstale":"2024-09-15T03:47:35.841529+0000","last_undegraded":"2024-09-15T03:47:35.841529+0000","last_fullsized":"2024-09-15T03:47:35.841529+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:44.356639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088783449999999993,"stat_sum":{"num_bytes":17186816,"num_objects":26,"num_object_clones":19,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":2,"num_read":57,"num_read_kb":30309,"num_write":152,"num_write_kb":16064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":55787520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,8,4,1,2147483647,11,3],"acting":[7,6,8,4,1,2147483647,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3,4,6,8,9,11],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"80'55","reported_seq":70,"reported_epoch":89,"state":"remapped+peering","last_fresh":"2024-09-15T03:47:35.839856+0000","last_change":"2024-09-15T03:47:35.840410+0000","last_active":"2024-09-15T03:47:35.839818+0000","last_peered":"2024-09-15T03:47:35.028347+0000","last_clean":"2024-09-15T03:47:26.974488+0000","last_became_active":"2024-09-15T03:47:34.877871+0000","last_became_peered":"2024-09-15T03:47:34.877871+0000","last_unstale":"2024-09-15T03:47:35.839856+0000","last_undegraded":"2024-09-15T03:47:35.839856+0000","last_fullsized":"2024-09-15T03:47:35.839856+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:55.144604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048920599999999995,"stat_sum":{"num_bytes":4620288,"num_objects":7,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":23,"num_write_kb":4512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2147483647,8,9,5,6,2,0],"acting":[4,8,8,9,5,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2,3,5,6,7,8,9,11],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"},{"start":"29","length":"1"},{"start":"2f","length":"1"}]},{"pgid":"3.9","version":"88'46","reported_seq":327,"reported_epoch":89,"state":"peering","last_fresh":"2024-09-15T03:47:35.840516+0000","last_change":"2024-09-15T03:47:35.840956+0000","last_active":"2024-09-15T03:47:35.840478+0000","last_peered":"2024-09-15T03:47:34.976478+0000","last_clean":"2024-09-15T03:47:25.960152+0000","last_became_active":"2024-09-15T03:47:34.848218+0000","last_became_peered":"2024-09-15T03:47:34.848218+0000","last_unstale":"2024-09-15T03:47:35.840516+0000","last_undegraded":"2024-09-15T03:47:35.840516+0000","last_fullsized":"2024-09-15T03:47:35.840516+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:59.940649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061462299999999999,"stat_sum":{"num_bytes":7749632,"num_objects":10,"num_object_clones":7,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":52,"num_write_kb":7568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":17891328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,2147483647,1,9,6,8,11],"acting":[3,2,2147483647,1,9,6,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,4,6,8,9,11],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"84'30","reported_seq":236,"reported_epoch":88,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:47:34.838248+0000","last_change":"2024-09-15T03:47:27.975434+0000","last_active":"2024-09-15T03:47:34.838248+0000","last_peered":"2024-09-15T03:47:34.838248+0000","last_clean":"2024-09-15T03:47:25.960197+0000","last_became_active":"2024-09-15T03:47:27.975434+0000","last_became_peered":"2024-09-15T03:47:27.975434+0000","last_unstale":"2024-09-15T03:47:34.838248+0000","last_undegraded":"2024-09-15T03:47:27.969197+0000","last_fullsized":"2024-09-15T03:47:27.968846+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:54.666633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043060499999999997,"stat_sum":{"num_bytes":4079616,"num_objects":7,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":19,"num_read_kb":3655,"num_write":54,"num_write_kb":3984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3063808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,11,4,1,0,2147483647],"acting":[3,7,2,11,4,1,0,2147483647],"avail_no_missing":["3(0)","0(6)","1(5)","2(2)","4(4)","7(1)","11(3)"],"object_location_counts":[{"shards":"0(6),1(5),2(2),3(0),4(4),7(1),11(3)","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":187,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T03:47:34.838456+0000","last_change":"2024-09-15T03:46:18.011841+0000","last_active":"2024-09-15T03:47:34.838456+0000","last_peered":"2024-09-15T03:47:34.838456+0000","last_clean":"2024-09-15T03:47:34.838456+0000","last_became_active":"2024-09-15T03:46:18.011130+0000","last_became_peered":"2024-09-15T03:46:18.011130+0000","last_unstale":"2024-09-15T03:47:34.838456+0000","last_undegraded":"2024-09-15T03:47:34.838456+0000","last_fullsized":"2024-09-15T03:47:34.838456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_clean_scrub_stamp":"2024-09-15T03:46:16.886982+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"22'1","reported_seq":79,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T03:47:34.840451+0000","last_change":"2024-09-15T03:46:26.776623+0000","last_active":"2024-09-15T03:47:34.840451+0000","last_peered":"2024-09-15T03:47:34.840451+0000","last_clean":"2024-09-15T03:47:34.840451+0000","last_became_active":"2024-09-15T03:46:24.038633+0000","last_became_peered":"2024-09-15T03:46:24.038633+0000","last_unstale":"2024-09-15T03:47:34.840451+0000","last_undegraded":"2024-09-15T03:47:34.840451+0000","last_fullsized":"2024-09-15T03:47:34.840451+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:45.125997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030178200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"76'35","reported_seq":337,"reported_epoch":89,"state":"peering","last_fresh":"2024-09-15T03:47:35.844138+0000","last_change":"2024-09-15T03:47:35.844555+0000","last_active":"2024-09-15T03:47:35.844106+0000","last_peered":"2024-09-15T03:47:34.966900+0000","last_clean":"2024-09-15T03:47:25.959833+0000","last_became_active":"2024-09-15T03:47:34.884265+0000","last_became_peered":"2024-09-15T03:47:34.884265+0000","last_unstale":"2024-09-15T03:47:35.844138+0000","last_undegraded":"2024-09-15T03:47:35.844138+0000","last_fullsized":"2024-09-15T03:47:35.844138+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:48.244817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065286800000000002,"stat_sum":{"num_bytes":4276224,"num_objects":7,"num_object_clones":4,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":25,"num_read_kb":6345,"num_write":19,"num_write_kb":4256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":13516800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,2147483647,7,8,6,11,1],"acting":[4,2,2147483647,7,8,6,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,6,7,8,9,11],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":86,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T03:47:34.837884+0000","last_change":"2024-09-15T03:47:27.779257+0000","last_active":"2024-09-15T03:47:34.837884+0000","last_peered":"2024-09-15T03:47:34.837884+0000","last_clean":"2024-09-15T03:47:34.837884+0000","last_became_active":"2024-09-15T03:46:24.063106+0000","last_became_peered":"2024-09-15T03:46:24.063106+0000","last_unstale":"2024-09-15T03:47:34.837884+0000","last_undegraded":"2024-09-15T03:47:34.837884+0000","last_fullsized":"2024-09-15T03:47:34.837884+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.779210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.779210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:42.717969+0000","scrub_duration":0.0021030710000000002,"objects_trimmed":0,"snaptrim_duration":0.000393075,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"76'6","reported_seq":25,"reported_epoch":88,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:47:34.843757+0000","last_change":"2024-09-15T03:47:34.021935+0000","last_active":"2024-09-15T03:47:34.843757+0000","last_peered":"2024-09-15T03:47:34.843757+0000","last_clean":"2024-09-15T03:47:26.286173+0000","last_became_active":"2024-09-15T03:47:34.006047+0000","last_became_peered":"2024-09-15T03:47:34.006047+0000","last_unstale":"2024-09-15T03:47:34.843757+0000","last_undegraded":"2024-09-15T03:47:33.996551+0000","last_fullsized":"2024-09-15T03:47:33.996160+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:51.121673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047935600000000001,"stat_sum":{"num_bytes":1327104,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":3074,"num_write":3,"num_write_kb":1296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1851392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3,2147483647,5,2,9,7],"acting":[6,11,3,2147483647,5,2,9,7],"avail_no_missing":["6(0)","0(0)","2(5)","3(2)","5(4)","7(7)","9(6)","11(1)"],"object_location_counts":[{"shards":"2(5),3(2),5(4),6(0),7(7),9(6),11(1)","objects":2}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":80,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T03:47:34.835874+0000","last_change":"2024-09-15T03:46:26.042740+0000","last_active":"2024-09-15T03:47:34.835874+0000","last_peered":"2024-09-15T03:47:34.835874+0000","last_clean":"2024-09-15T03:47:34.835874+0000","last_became_active":"2024-09-15T03:46:24.037471+0000","last_became_peered":"2024-09-15T03:46:24.037471+0000","last_unstale":"2024-09-15T03:47:34.835874+0000","last_undegraded":"2024-09-15T03:47:34.835874+0000","last_fullsized":"2024-09-15T03:47:34.835874+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026490299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"82'34","reported_seq":279,"reported_epoch":86,"state":"peering","last_fresh":"2024-09-15T03:47:32.987302+0000","last_change":"2024-09-15T03:47:32.987719+0000","last_active":"2024-09-15T03:47:32.987266+0000","last_peered":"2024-09-15T03:47:31.987335+0000","last_clean":"2024-09-15T03:47:25.959836+0000","last_became_active":"2024-09-15T03:47:27.975454+0000","last_became_peered":"2024-09-15T03:47:27.975454+0000","last_unstale":"2024-09-15T03:47:32.987302+0000","last_undegraded":"2024-09-15T03:47:32.987302+0000","last_fullsized":"2024-09-15T03:47:32.987302+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:54.531971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091292019999999995,"stat_sum":{"num_bytes":2883584,"num_objects":7,"num_object_clones":4,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":18,"num_read_kb":7495,"num_write":36,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9207808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,9,4,3,6,0,2147483647],"acting":[1,2,9,4,3,6,0,2147483647],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2,3,4,6,8,9],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"27","length":"2"}]},{"pgid":"2.2","version":"24'2","reported_seq":115,"reported_epoch":89,"state":"peering","last_fresh":"2024-09-15T03:47:35.838870+0000","last_change":"2024-09-15T03:47:35.839090+0000","last_active":"2024-09-15T03:47:35.838841+0000","last_peered":"2024-09-15T03:47:34.866215+0000","last_clean":"2024-09-15T03:47:31.987052+0000","last_became_active":"2024-09-15T03:47:34.845309+0000","last_became_peered":"2024-09-15T03:47:34.845309+0000","last_unstale":"2024-09-15T03:47:35.838870+0000","last_undegraded":"2024-09-15T03:47:35.838870+0000","last_fullsized":"2024-09-15T03:47:35.838870+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000944677,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,11],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"88'104","reported_seq":271,"reported_epoch":88,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:47:34.943802+0000","last_change":"2024-09-15T03:47:27.975952+0000","last_active":"2024-09-15T03:47:34.943802+0000","last_peered":"2024-09-15T03:47:34.943802+0000","last_clean":"2024-09-15T03:47:24.958783+0000","last_became_active":"2024-09-15T03:47:27.975952+0000","last_became_peered":"2024-09-15T03:47:27.975952+0000","last_unstale":"2024-09-15T03:47:34.943802+0000","last_undegraded":"2024-09-15T03:47:27.967398+0000","last_fullsized":"2024-09-15T03:47:27.967024+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:57.724837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047245120000000002,"stat_sum":{"num_bytes":13762560,"num_objects":19,"num_object_clones":13,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":19,"num_read_kb":11,"num_write":130,"num_write_kb":14816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,8,4,7,1,2,9,0],"acting":[2147483647,8,4,7,1,2,9,0],"avail_no_missing":["8(1)","0(7)","1(4)","2(5)","4(2)","7(3)","9(6)"],"object_location_counts":[{"shards":"0(7),1(4),2(5),4(2),7(3),8(1),9(6)","objects":19}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.5","version":"85'41","reported_seq":231,"reported_epoch":89,"state":"peering","last_fresh":"2024-09-15T03:47:35.838203+0000","last_change":"2024-09-15T03:47:35.838203+0000","last_active":"2024-09-15T03:47:34.002282+0000","last_peered":"2024-09-15T03:47:31.991178+0000","last_clean":"2024-09-15T03:47:11.874747+0000","last_became_active":"2024-09-15T03:47:27.980383+0000","last_became_peered":"2024-09-15T03:47:27.980383+0000","last_unstale":"2024-09-15T03:47:35.838203+0000","last_undegraded":"2024-09-15T03:47:35.838203+0000","last_fullsized":"2024-09-15T03:47:35.838203+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:42.161443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059856140000000002,"stat_sum":{"num_bytes":13041664,"num_objects":6,"num_object_clones":5,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":11046,"num_write":18,"num_write_kb":7648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":23789568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,6,4,9,8,7,3,0],"acting":[2147483647,6,4,9,8,7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":98,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T03:47:34.835520+0000","last_change":"2024-09-15T03:47:33.999145+0000","last_active":"2024-09-15T03:47:34.835520+0000","last_peered":"2024-09-15T03:47:34.835520+0000","last_clean":"2024-09-15T03:47:34.835520+0000","last_became_active":"2024-09-15T03:47:33.998802+0000","last_became_peered":"2024-09-15T03:47:33.998802+0000","last_unstale":"2024-09-15T03:47:34.835520+0000","last_undegraded":"2024-09-15T03:47:34.835520+0000","last_fullsized":"2024-09-15T03:47:34.835520+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:36.148421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034123999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"85'28","reported_seq":248,"reported_epoch":89,"state":"peering","last_fresh":"2024-09-15T03:47:35.838143+0000","last_change":"2024-09-15T03:47:35.838143+0000","last_active":"2024-09-15T03:47:33.997007+0000","last_peered":"2024-09-15T03:47:32.004293+0000","last_clean":"2024-09-15T03:47:25.959996+0000","last_became_active":"2024-09-15T03:47:27.977152+0000","last_became_peered":"2024-09-15T03:47:27.977152+0000","last_unstale":"2024-09-15T03:47:35.838143+0000","last_undegraded":"2024-09-15T03:47:35.838143+0000","last_fullsized":"2024-09-15T03:47:35.838143+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:46.779626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000727303,"stat_sum":{"num_bytes":5193728,"num_objects":4,"num_object_clones":3,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":6052,"num_write":10,"num_write_kb":5808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2424832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5,8,0,4,7,2147483647],"acting":[6,2,5,8,0,4,7,2147483647],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":88,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T03:47:34.838851+0000","last_change":"2024-09-15T03:47:30.025153+0000","last_active":"2024-09-15T03:47:34.838851+0000","last_peered":"2024-09-15T03:47:34.838851+0000","last_clean":"2024-09-15T03:47:34.838851+0000","last_became_active":"2024-09-15T03:46:24.038945+0000","last_became_peered":"2024-09-15T03:46:24.038945+0000","last_unstale":"2024-09-15T03:47:34.838851+0000","last_undegraded":"2024-09-15T03:47:34.838851+0000","last_fullsized":"2024-09-15T03:47:34.838851+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:30.025107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:30.025107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:36.231260+0000","scrub_duration":0.0013039429999999999,"objects_trimmed":0,"snaptrim_duration":0.00032787499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"89'79","reported_seq":309,"reported_epoch":89,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:47:35.877296+0000","last_change":"2024-09-15T03:47:27.981468+0000","last_active":"2024-09-15T03:47:35.877296+0000","last_peered":"2024-09-15T03:47:35.877296+0000","last_clean":"2024-09-15T03:47:25.960204+0000","last_became_active":"2024-09-15T03:47:27.981468+0000","last_became_peered":"2024-09-15T03:47:27.981468+0000","last_unstale":"2024-09-15T03:47:35.877296+0000","last_undegraded":"2024-09-15T03:47:27.973715+0000","last_fullsized":"2024-09-15T03:47:27.973323+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:38.655064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066553539999999996,"stat_sum":{"num_bytes":12550144,"num_objects":15,"num_object_clones":10,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":45,"num_read_kb":27825,"num_write":81,"num_write_kb":12144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9666560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,7,2,1,11,2147483647],"acting":[4,3,9,7,2,1,11,2147483647],"avail_no_missing":["4(0)","1(5)","2(4)","3(1)","7(3)","9(2)","11(6)"],"object_location_counts":[{"shards":"1(5),2(4),3(1),4(0),7(3),9(2),11(6)","objects":15}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":89,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T03:47:34.837333+0000","last_change":"2024-09-15T03:47:27.297139+0000","last_active":"2024-09-15T03:47:34.837333+0000","last_peered":"2024-09-15T03:47:34.837333+0000","last_clean":"2024-09-15T03:47:34.837333+0000","last_became_active":"2024-09-15T03:46:24.063380+0000","last_became_peered":"2024-09-15T03:46:24.063380+0000","last_unstale":"2024-09-15T03:47:34.837333+0000","last_undegraded":"2024-09-15T03:47:34.837333+0000","last_fullsized":"2024-09-15T03:47:34.837333+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.297083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.297083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:48.875998+0000","scrub_duration":0.0018328789999999999,"objects_trimmed":0,"snaptrim_duration":0.00034490600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'22","reported_seq":350,"reported_epoch":88,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:47:34.837198+0000","last_change":"2024-09-15T03:47:27.981271+0000","last_active":"2024-09-15T03:47:34.837198+0000","last_peered":"2024-09-15T03:47:34.837198+0000","last_clean":"2024-09-15T03:47:25.960095+0000","last_became_active":"2024-09-15T03:47:27.981271+0000","last_became_peered":"2024-09-15T03:47:27.981271+0000","last_unstale":"2024-09-15T03:47:34.837198+0000","last_undegraded":"2024-09-15T03:47:27.968432+0000","last_fullsized":"2024-09-15T03:47:27.968095+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:38.076960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084525500000000005,"stat_sum":{"num_bytes":1359872,"num_objects":6,"num_object_clones":4,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":515,"num_write":28,"num_write_kb":1328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":7012352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,9,8,3,11,1,2147483647],"acting":[4,7,9,8,3,11,1,2147483647],"avail_no_missing":["4(0)","1(6)","3(4)","7(1)","8(3)","9(2)","11(5)"],"object_location_counts":[{"shards":"1(6),3(4),4(0),7(1),8(3),9(2),11(5)","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":76,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T03:47:34.838449+0000","last_change":"2024-09-15T03:46:26.044851+0000","last_active":"2024-09-15T03:47:34.838449+0000","last_peered":"2024-09-15T03:47:34.838449+0000","last_clean":"2024-09-15T03:47:34.838449+0000","last_became_active":"2024-09-15T03:46:24.063335+0000","last_became_peered":"2024-09-15T03:46:24.063335+0000","last_unstale":"2024-09-15T03:47:34.838449+0000","last_undegraded":"2024-09-15T03:47:34.838449+0000","last_fullsized":"2024-09-15T03:47:34.838449+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065928000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":119062528,"num_objects":158,"num_object_clones":108,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":67,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":9,"num_read":356,"num_read_kb":165514,"num_write":711,"num_write_kb":111408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":184188928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":824,"ondisk_log_size":824,"up":128,"acting":128,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":11,"up_from":17,"seq":73014444050,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":841820,"kb_used_data":92839716,"kb_used_omap":3312,"kb_used_meta":0,"kb_avail":92837796,"statfs":{"total":95927926784,"available":95065903104,"internally_reserved":1966080,"allocated":95067869184,"data_stored":95067869184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3392038,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.57499999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.60899999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.41899999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.83699999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.96299999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.82599999999999996}]},{"osd":7,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.871},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.86399999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.79700000000000004}]},{"osd":9,"last update":"Sun Sep 15 03:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.80900000000000005}]},{"osd":10,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.46999999999999997}]}]},{"osd":10,"up_from":17,"seq":73014444047,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":826972,"kb_used_data":92854120,"kb_used_omap":1440,"kb_used_meta":0,"kb_avail":92852644,"statfs":{"total":95927926784,"available":95081107456,"internally_reserved":1511424,"allocated":95082618880,"data_stored":95082618880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1475478,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":6,"commit_latency_ns":0,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.52700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":202.59200000000001,"5min":202.59200000000001,"15min":202.59200000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":2372.2370000000001,"5min":2372.2370000000001,"15min":2372.2370000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":202.56,"5min":202.56,"15min":202.56},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":2371.5700000000002,"5min":2371.5700000000002,"15min":2371.5700000000002},"last":0.51100000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":0.57899999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.436},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.63800000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.59899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":0.72299999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.55600000000000005}]},{"osd":11,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.69999999999999996}]}]},{"osd":9,"up_from":17,"seq":73014444050,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":847500,"kb_used_data":92833988,"kb_used_omap":3468,"kb_used_meta":0,"kb_avail":92832116,"statfs":{"total":95927926784,"available":95060086784,"internally_reserved":1916928,"allocated":95062003712,"data_stored":95062003712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3551420,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6,7,8,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.65200000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.69699999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.52900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":4,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.54600000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.496}]},{"osd":6,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.55400000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.44700000000000001}]},{"osd":10,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.48599999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.57599999999999996}]}]},{"osd":8,"up_from":17,"seq":73014444050,"num_pgs":16,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":852332,"kb_used_data":92830892,"kb_used_omap":3487,"kb_used_meta":0,"kb_avail":92827284,"statfs":{"total":95927926784,"available":95055138816,"internally_reserved":3694592,"allocated":95058833408,"data_stored":95058833408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3570941,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.57699999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":42.381,"5min":42.381,"15min":42.381},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":963.98400000000004,"5min":963.98400000000004,"15min":963.98400000000004},"last":0.50800000000000001},{"interface":"front","average":{"1min":42.447000000000003,"5min":42.447000000000003,"15min":42.447000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":964.79600000000005,"5min":964.79600000000005,"15min":964.79600000000005},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.65700000000000003}]},{"osd":4,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.83099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":6,"last update":"Sun Sep 15 03:47:25 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":1.038}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.97899999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.754},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.68999999999999995}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.71699999999999997}]},{"osd":11,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.67100000000000004}]}]},{"osd":7,"up_from":17,"seq":73014444050,"num_pgs":15,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":851472,"kb_used_data":92830192,"kb_used_omap":3745,"kb_used_meta":0,"kb_avail":92828144,"statfs":{"total":95927926784,"available":95056019456,"internally_reserved":2097152,"allocated":95058116608,"data_stored":95058116608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3835033,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.74399999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.61099999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.72399999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.81699999999999995}]},{"osd":5,"last update":"Sun Sep 15 03:47:27 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.78200000000000003}]},{"osd":6,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.872}]},{"osd":8,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.92700000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.97599999999999998}]},{"osd":10,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.51100000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.88800000000000001}]}]},{"osd":6,"up_from":17,"seq":73014444050,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":845216,"kb_used_data":92858208,"kb_used_omap":2692,"kb_used_meta":0,"kb_avail":92834400,"statfs":{"total":95927926784,"available":95062425600,"internally_reserved":24379392,"allocated":95086804992,"data_stored":95086804992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2757067,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":4,"commit_latency_ns":0,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.59299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.60699999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.47099999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.47899999999999998}]},{"osd":5,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.54300000000000004}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.624},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.51600000000000001}]},{"osd":8,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.83299999999999996}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.84499999999999997}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":11,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.86599999999999999}]}]},{"osd":5,"up_from":17,"seq":73014444050,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":819148,"kb_used_data":92863324,"kb_used_omap":1292,"kb_used_meta":0,"kb_avail":92860468,"statfs":{"total":95927926784,"available":95089119232,"internally_reserved":2924544,"allocated":95092043776,"data_stored":95092043776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1323341,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.39600000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":113.717,"5min":113.717,"15min":113.717},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2037.425,"5min":2037.425,"15min":2037.425},"last":0.46899999999999997},{"interface":"front","average":{"1min":113.752,"5min":113.752,"15min":113.752},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2037.3230000000001,"5min":2037.3230000000001,"15min":2037.3230000000001},"last":0.70699999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.374},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.76700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.59399999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.72899999999999998}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.68200000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.78700000000000003}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.749},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.84599999999999997}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.83399999999999996}]},{"osd":11,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.82799999999999996}]}]},{"osd":4,"up_from":17,"seq":73014444050,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":850132,"kb_used_data":92832456,"kb_used_omap":3716,"kb_used_meta":0,"kb_avail":92829484,"statfs":{"total":95927926784,"available":95057391616,"internally_reserved":3043328,"allocated":95060434944,"data_stored":95060434944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3805401,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.47499999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":84.314999999999998,"5min":84.314999999999998,"15min":84.314999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1758.9780000000001,"5min":1758.9780000000001,"15min":1758.9780000000001},"last":0.442},{"interface":"front","average":{"1min":84.364000000000004,"5min":84.364000000000004,"15min":84.364000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1758.671,"5min":1758.671,"15min":1758.671},"last":0.495}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.38100000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.29199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":7,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.36799999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.53700000000000003}]},{"osd":9,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.435},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":0.51800000000000002}]},{"osd":10,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.59599999999999997}]},{"osd":11,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.502}]}]},{"osd":3,"up_from":17,"seq":73014444050,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":854828,"kb_used_data":92834712,"kb_used_omap":3200,"kb_used_meta":0,"kb_avail":92824788,"statfs":{"total":95927926784,"available":95052582912,"internally_reserved":10162176,"allocated":95062745088,"data_stored":95062745088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3277185,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.27500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26000000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.308}]},{"osd":4,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.371}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.38300000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.499},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.42899999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.68799999999999994}]},{"osd":9,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.70099999999999996}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":11,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.623},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.67600000000000005}]}]},{"osd":2,"up_from":17,"seq":73014444050,"num_pgs":13,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":838308,"kb_used_data":92843816,"kb_used_omap":3527,"kb_used_meta":0,"kb_avail":92841308,"statfs":{"total":95927926784,"available":95069499392,"internally_reserved":2568192,"allocated":95072067584,"data_stored":95072067584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3611972,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":1,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":34.457999999999998,"5min":34.457999999999998,"15min":34.457999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":647.92399999999998,"5min":647.92399999999998,"15min":647.92399999999998},"last":0.29699999999999999},{"interface":"front","average":{"1min":34.475000000000001,"5min":34.475000000000001,"15min":34.475000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":648.02700000000004,"5min":648.02700000000004,"15min":648.02700000000004},"last":0.42199999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.40100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.502}]},{"osd":6,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.52200000000000002}]},{"osd":7,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.57199999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.343,"15min":1.343},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":14.721,"5min":14.721,"15min":14.721},"last":0.54600000000000004},{"interface":"front","average":{"1min":1.371,"5min":1.371,"15min":1.371},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":14.970000000000001,"5min":14.970000000000001,"15min":14.970000000000001},"last":0.64800000000000002}]},{"osd":9,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.377,"15min":1.377},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.62},{"interface":"front","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":14.888,"5min":14.888,"15min":14.888},"last":0.63200000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":14.654999999999999,"5min":14.654999999999999,"15min":14.654999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.409,"5min":1.409,"15min":1.409},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":15.096,"5min":15.096,"15min":15.096},"last":0.59399999999999997}]}]},{"osd":1,"up_from":17,"seq":73014444050,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":848624,"kb_used_data":92834620,"kb_used_omap":3887,"kb_used_meta":0,"kb_avail":92830992,"statfs":{"total":95927926784,"available":95058935808,"internally_reserved":3715072,"allocated":95062650880,"data_stored":95062650880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3981170,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.56299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.44}]},{"osd":3,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.45800000000000002}]},{"osd":4,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":2.6920000000000002,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":0.71299999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.7440000000000002,"5min":2.7440000000000002,"15min":2.7440000000000002},"last":0.53200000000000003}]},{"osd":6,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":3.278,"5min":3.278,"15min":3.278},"last":0.70399999999999996}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.58599999999999997}]},{"osd":8,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":119.93899999999999,"5min":119.93899999999999,"15min":119.93899999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2861.7190000000001,"5min":2861.7190000000001,"15min":2861.7190000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.946,"5min":1.946,"15min":1.946},"last":0.79200000000000004}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":119.937,"5min":119.937,"15min":119.937},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":2861.6010000000001,"5min":2861.6010000000001,"15min":2861.6010000000001},"last":0.746},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.83999999999999997}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":119.96599999999999,"5min":119.96599999999999,"15min":119.96599999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2861.5639999999999,"5min":2861.5639999999999,"15min":2861.5639999999999},"last":0.69299999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444050,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":841384,"kb_used_data":92846252,"kb_used_omap":3058,"kb_used_meta":0,"kb_avail":92838232,"statfs":{"total":95927926784,"available":95066349568,"internally_reserved":8212480,"allocated":95074562048,"data_stored":95074562048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3131496,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.32700000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.804,"5min":1.804,"15min":1.804},"last":0.312}]},{"osd":3,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.52,"5min":1.52,"15min":1.52},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.434}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":5,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.41999999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.54500000000000004}]},{"osd":7,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.50700000000000001}]},{"osd":8,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.57999999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":10,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.64400000000000002}]},{"osd":11,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.60899999999999999}]}]}],"pool_statfs":[]}} 2024-09-15T03:47:38.773 INFO:tasks.thrashosds.thrasher:Setting 2.5 to [9, 11, 8, 7] 2024-09-15T03:47:38.773 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.5', '9', '11', '8', '7'] 2024-09-15T03:47:38.773 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.5 9 11 8 7 2024-09-15T03:47:38.839 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:38.838+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:38.842 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-15T03:47:38.842 INFO:tasks.rados.rados.0.smithi001.stdout:304: done (4 left) 2024-09-15T03:47:38.842 INFO:tasks.rados.rados.0.smithi001.stdout:308: done (3 left) 2024-09-15T03:47:38.842 INFO:tasks.rados.rados.0.smithi001.stdout:309: done (2 left) 2024-09-15T03:47:38.842 INFO:tasks.rados.rados.0.smithi001.stdout:310: done (1 left) 2024-09-15T03:47:38.842 INFO:tasks.rados.rados.0.smithi001.stdout:311: done (0 left) 2024-09-15T03:47:38.842 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 32 current snap is 36 2024-09-15T03:47:38.843 INFO:tasks.rados.rados.0.smithi001.stdout:312: seq_num 122 ranges {802816=507904,1310720=425984,1736704=442368,2179072=540672,2719744=147456} 2024-09-15T03:47:38.849 INFO:tasks.rados.rados.0.smithi001.stdout:312: writing smithi00132172-32 from 802816 to 1310720 tid 1 2024-09-15T03:47:38.851 INFO:tasks.rados.rados.0.smithi001.stdout:312: writing smithi00132172-32 from 1310720 to 1736704 tid 2 2024-09-15T03:47:38.854 INFO:tasks.rados.rados.0.smithi001.stdout:312: writing smithi00132172-32 from 1736704 to 2179072 tid 3 2024-09-15T03:47:38.860 INFO:tasks.rados.rados.0.smithi001.stdout:312: writing smithi00132172-32 from 2179072 to 2719744 tid 4 2024-09-15T03:47:38.860 INFO:tasks.rados.rados.0.smithi001.stdout:312: writing smithi00132172-32 from 2719744 to 2867200 tid 5 2024-09-15T03:47:38.860 INFO:tasks.rados.rados.0.smithi001.stdout:313: read oid 7 snap 0 2024-09-15T03:47:38.860 INFO:tasks.rados.rados.0.smithi001.stdout:313: expect (ObjNum 6 snap 0 seq_num 6) 2024-09-15T03:47:38.860 INFO:tasks.rados.rados.0.smithi001.stdout:314: snap_create 2024-09-15T03:47:38.860 INFO:tasks.rados.rados.0.smithi001.stdout:312: finishing write tid 1 to smithi00132172-32 2024-09-15T03:47:38.860 INFO:tasks.rados.rados.0.smithi001.stdout:312: finishing write tid 2 to smithi00132172-32 2024-09-15T03:47:38.860 INFO:tasks.rados.rados.0.smithi001.stdout:312: finishing write tid 3 to smithi00132172-32 2024-09-15T03:47:38.862 INFO:tasks.rados.rados.0.smithi001.stdout:312: finishing write tid 4 to smithi00132172-32 2024-09-15T03:47:38.863 INFO:tasks.rados.rados.0.smithi001.stdout:312: finishing write tid 5 to smithi00132172-32 2024-09-15T03:47:38.863 INFO:tasks.rados.rados.0.smithi001.stdout:312: finishing write tid 6 to smithi00132172-32 2024-09-15T03:47:38.865 INFO:tasks.rados.rados.0.smithi001.stdout:313: expect (ObjNum 6 snap 0 seq_num 6) 2024-09-15T03:47:38.865 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:38.865 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:38.870 INFO:tasks.rados.rados.0.smithi001.stdout:312: finishing write tid 7 to smithi00132172-32 2024-09-15T03:47:38.870 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 32 v 55 (ObjNum 122 snap 36 seq_num 122) dirty exists 2024-09-15T03:47:38.870 INFO:tasks.rados.rados.0.smithi001.stdout:312: left oid 32 (ObjNum 122 snap 36 seq_num 122) 2024-09-15T03:47:38.875 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:47:38.876 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:47:38.940 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:38.940+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:39.021 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:39.022 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:39.032 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:47:39.041 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:39.041 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:39.041 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:39.040+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:39.055 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:39.056 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:39.132 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:39.132 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:39.141 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:47:39.182 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:39.181+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:39.243 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:39.243+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:39.255 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:39.255 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:39.264 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:47:39.271 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:39.271 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:39.285 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:39.285 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:39.300 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:39.300 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:39.314 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:39.329 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:39.329 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:39.343 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:39.343 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:39.345 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:39.344+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:39.359 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:39.359 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:39.373 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:39.374 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:39.374 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-15T03:47:39.389 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:39.390 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:39.392 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T03:47:39.392 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T03:47:39.407 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:39.407 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:39.407 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:39.407 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:39.407 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:39.415 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:47:39.446 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:39.445+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:39.518 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:39.518 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:39.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:39.518 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:39.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.518 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs6 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:39.519 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108711+0000", 2024-09-15T03:47:39.519 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.406508209999998, 2024-09-15T03:47:39.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01472483, 2024-09-15T03:47:39.519 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:39.519 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:39.519 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:39.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:39.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108711+0000", 2024-09-15T03:47:39.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108711+0000", 2024-09-15T03:47:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108712+0000", 2024-09-15T03:47:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.85e-07 2024-09-15T03:47:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108745+0000", 2024-09-15T03:47:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2762000000000002e-05 2024-09-15T03:47:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108760+0000", 2024-09-15T03:47:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4567e-05 2024-09-15T03:47:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108780+0000", 2024-09-15T03:47:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0122000000000001e-05 2024-09-15T03:47:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108842+0000", 2024-09-15T03:47:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2297000000000003e-05 2024-09-15T03:47:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108851+0000", 2024-09-15T03:47:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0129999999999999e-06 2024-09-15T03:47:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108942+0000", 2024-09-15T03:47:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0713999999999999e-05 2024-09-15T03:47:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108949+0000", 2024-09-15T03:47:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1600000000000001e-06 2024-09-15T03:47:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109975+0000", 2024-09-15T03:47:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010264429999999999 2024-09-15T03:47:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109994+0000", 2024-09-15T03:47:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8374000000000001e-05 2024-09-15T03:47:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123436+0000", 2024-09-15T03:47:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013442592999999999 2024-09-15T03:47:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs6 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108809+0000", 2024-09-15T03:47:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.406410481999998, 2024-09-15T03:47:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014757497, 2024-09-15T03:47:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108809+0000", 2024-09-15T03:47:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108809+0000", 2024-09-15T03:47:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108810+0000", 2024-09-15T03:47:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.15e-07 2024-09-15T03:47:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108825+0000", 2024-09-15T03:47:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5092999999999999e-05 2024-09-15T03:47:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108835+0000", 2024-09-15T03:47:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0251000000000001e-05 2024-09-15T03:47:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108841+0000", 2024-09-15T03:47:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0510000000000002e-06 2024-09-15T03:47:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108971+0000", 2024-09-15T03:47:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012976599999999999 2024-09-15T03:47:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108976+0000", 2024-09-15T03:47:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8450000000000002e-06 2024-09-15T03:47:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109032+0000", 2024-09-15T03:47:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6119999999999998e-05 2024-09-15T03:47:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109978+0000", 2024-09-15T03:47:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00094603899999999997 2024-09-15T03:47:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110007+0000", 2024-09-15T03:47:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9062999999999999e-05 2024-09-15T03:47:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110025+0000", 2024-09-15T03:47:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7717999999999999e-05 2024-09-15T03:47:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123567+0000", 2024-09-15T03:47:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013542136 2024-09-15T03:47:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:324 3.7s0 3:e40d1694:::smithi00132172-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.671097048, 2024-09-15T03:47:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0031668960000002, 2024-09-15T03:47:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 324 2024-09-15T03:47:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:47:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844125+0000", 2024-09-15T03:47:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.142e-06 2024-09-15T03:47:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844132+0000", 2024-09-15T03:47:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0439999999999998e-06 2024-09-15T03:47:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844136+0000", 2024-09-15T03:47:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.126e-06 2024-09-15T03:47:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.847618+0000", 2024-09-15T03:47:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0034813790000000002 2024-09-15T03:47:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.847709+0000", 2024-09-15T03:47:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0835999999999996e-05 2024-09-15T03:47:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.849483+0000", 2024-09-15T03:47:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001774929 2024-09-15T03:47:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.849574+0000", 2024-09-15T03:47:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0847999999999995e-05 2024-09-15T03:47:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.847290+0000", 2024-09-15T03:47:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9977151819999999 2024-09-15T03:47:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:334 3.7s0 3:ece1472c:::smithi00132172-43:head [append 0~0] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.658369960000002, 2024-09-15T03:47:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.997592021, 2024-09-15T03:47:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 334 2024-09-15T03:47:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856851+0000", 2024-09-15T03:47:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.238e-06 2024-09-15T03:47:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856855+0000", 2024-09-15T03:47:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3640000000000001e-06 2024-09-15T03:47:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856868+0000", 2024-09-15T03:47:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2588999999999999e-05 2024-09-15T03:47:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856873+0000", 2024-09-15T03:47:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5350000000000004e-06 2024-09-15T03:47:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856916+0000", 2024-09-15T03:47:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2889000000000002e-05 2024-09-15T03:47:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856983+0000", 2024-09-15T03:47:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6746000000000001e-05 2024-09-15T03:47:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857448+0000", 2024-09-15T03:47:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046498500000000001 2024-09-15T03:47:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857720+0000", 2024-09-15T03:47:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027227600000000001 2024-09-15T03:47:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857741+0000", 2024-09-15T03:47:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1149999999999999e-05 2024-09-15T03:47:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857795+0000", 2024-09-15T03:47:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3919999999999999e-05 2024-09-15T03:47:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859582+0000", 2024-09-15T03:47:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017862640000000001 2024-09-15T03:47:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854347+0000", 2024-09-15T03:47:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9947654780000001 2024-09-15T03:47:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854442+0000", 2024-09-15T03:47:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4586999999999994e-05 2024-09-15T03:47:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:335 3.7s0 3:ece1472c:::smithi00132172-43:head [rollback 12] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.658339943000001, 2024-09-15T03:47:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9977859200000001, 2024-09-15T03:47:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 335 2024-09-15T03:47:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8099999999999999e-07 2024-09-15T03:47:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856882+0000", 2024-09-15T03:47:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.649e-06 2024-09-15T03:47:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856885+0000", 2024-09-15T03:47:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.664e-06 2024-09-15T03:47:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856888+0000", 2024-09-15T03:47:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3910000000000001e-06 2024-09-15T03:47:39.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857826+0000", 2024-09-15T03:47:39.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00093817900000000003 2024-09-15T03:47:39.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:39.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857940+0000", 2024-09-15T03:47:39.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114424 2024-09-15T03:47:39.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:39.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858504+0000", 2024-09-15T03:47:39.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056400899999999997 2024-09-15T03:47:39.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:39.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858593+0000", 2024-09-15T03:47:39.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8463999999999999e-05 2024-09-15T03:47:39.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:39.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858604+0000", 2024-09-15T03:47:39.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1567e-05 2024-09-15T03:47:39.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:39.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858642+0000", 2024-09-15T03:47:39.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7369000000000001e-05 2024-09-15T03:47:39.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:39.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859588+0000", 2024-09-15T03:47:39.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00094661799999999998 2024-09-15T03:47:39.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:39.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854594+0000", 2024-09-15T03:47:39.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9950055550000001 2024-09-15T03:47:39.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:39.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854666+0000", 2024-09-15T03:47:39.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1749999999999996e-05 2024-09-15T03:47:39.550 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.550 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:39.550 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.550 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:336 3.7s0 3:ece1472c:::smithi00132172-43:head [append 0~0] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.658308892999997, 2024-09-15T03:47:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9980017820000002, 2024-09-15T03:47:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 336 2024-09-15T03:47:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:39.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:39.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:39.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:39.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:39.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:39.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8000000000000002e-07 2024-09-15T03:47:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856913+0000", 2024-09-15T03:47:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5400000000000001e-06 2024-09-15T03:47:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856918+0000", 2024-09-15T03:47:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1540000000000001e-06 2024-09-15T03:47:39.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:39.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856921+0000", 2024-09-15T03:47:39.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2550000000000002e-06 2024-09-15T03:47:39.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858655+0000", 2024-09-15T03:47:39.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017336070000000001 2024-09-15T03:47:39.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858741+0000", 2024-09-15T03:47:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6305999999999997e-05 2024-09-15T03:47:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859193+0000", 2024-09-15T03:47:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045221200000000001 2024-09-15T03:47:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859319+0000", 2024-09-15T03:47:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012542900000000001 2024-09-15T03:47:39.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859331+0000", 2024-09-15T03:47:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2028000000000001e-05 2024-09-15T03:47:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859369+0000", 2024-09-15T03:47:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8522999999999997e-05 2024-09-15T03:47:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:39.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859649+0000", 2024-09-15T03:47:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028008200000000002 2024-09-15T03:47:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854800+0000", 2024-09-15T03:47:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9951506540000001 2024-09-15T03:47:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854913+0000", 2024-09-15T03:47:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112712 2024-09-15T03:47:39.557 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:346 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:47:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.676812523000002, 2024-09-15T03:47:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005631948, 2024-09-15T03:47:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 346 2024-09-15T03:47:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:47:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:47:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838410+0000", 2024-09-15T03:47:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4430000000000002e-06 2024-09-15T03:47:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838419+0000", 2024-09-15T03:47:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1730000000000001e-06 2024-09-15T03:47:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:39.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838444+0000", 2024-09-15T03:47:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5111000000000001e-05 2024-09-15T03:47:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838455+0000", 2024-09-15T03:47:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0839000000000001e-05 2024-09-15T03:47:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3415000000000002e-05 2024-09-15T03:47:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838527+0000", 2024-09-15T03:47:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0550000000000005e-06 2024-09-15T03:47:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840638+0000", 2024-09-15T03:47:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021105379999999999 2024-09-15T03:47:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840729+0000", 2024-09-15T03:47:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0977999999999995e-05 2024-09-15T03:47:39.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.841834+0000", 2024-09-15T03:47:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011055640000000001 2024-09-15T03:47:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842223+0000", 2024-09-15T03:47:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038899900000000001 2024-09-15T03:47:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842241+0000", 2024-09-15T03:47:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7317e-05 2024-09-15T03:47:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842298+0000", 2024-09-15T03:47:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7825000000000003e-05 2024-09-15T03:47:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844866+0000", 2024-09-15T03:47:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025675289999999998 2024-09-15T03:47:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843973+0000", 2024-09-15T03:47:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99910710599999997 2024-09-15T03:47:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844039+0000", 2024-09-15T03:47:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6056000000000006e-05 2024-09-15T03:47:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:347 3.2s0 3:48ceb02f:::smithi00132172-5:head [rollback c] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:47:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.676727726999999, 2024-09-15T03:47:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005723841, 2024-09-15T03:47:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 347 2024-09-15T03:47:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:39.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:47:39.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:39.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:47:39.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:39.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838493+0000", 2024-09-15T03:47:39.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7300000000000005e-07 2024-09-15T03:47:39.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:39.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838498+0000", 2024-09-15T03:47:39.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9300000000000002e-06 2024-09-15T03:47:39.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:39.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838508+0000", 2024-09-15T03:47:39.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9790000000000005e-06 2024-09-15T03:47:39.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:39.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838513+0000", 2024-09-15T03:47:39.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8890000000000001e-06 2024-09-15T03:47:39.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838583+0000", 2024-09-15T03:47:39.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0696000000000002e-05 2024-09-15T03:47:39.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:39.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838590+0000", 2024-09-15T03:47:39.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2990000000000002e-06 2024-09-15T03:47:39.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842313+0000", 2024-09-15T03:47:39.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037238810000000001 2024-09-15T03:47:39.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:39.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842432+0000", 2024-09-15T03:47:39.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118474 2024-09-15T03:47:39.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:39.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843108+0000", 2024-09-15T03:47:39.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00067639299999999996 2024-09-15T03:47:39.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:39.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843227+0000", 2024-09-15T03:47:39.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119092 2024-09-15T03:47:39.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:39.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843242+0000", 2024-09-15T03:47:39.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4216000000000001e-05 2024-09-15T03:47:39.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:39.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843292+0000", 2024-09-15T03:47:39.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0800999999999997e-05 2024-09-15T03:47:39.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:39.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844878+0000", 2024-09-15T03:47:39.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015852749999999999 2024-09-15T03:47:39.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:39.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844147+0000", 2024-09-15T03:47:39.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99926949200000004 2024-09-15T03:47:39.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:39.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844216+0000", 2024-09-15T03:47:39.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8551000000000001e-05 2024-09-15T03:47:39.573 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.573 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:39.573 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.573 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:348 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:39.573 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:39.573 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.676701784000002, 2024-09-15T03:47:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.006019768, 2024-09-15T03:47:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 348 2024-09-15T03:47:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:39.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:39.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:39.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:39.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8700000000000001e-07 2024-09-15T03:47:39.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:39.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838521+0000", 2024-09-15T03:47:39.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3740000000000001e-06 2024-09-15T03:47:39.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:39.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838527+0000", 2024-09-15T03:47:39.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1999999999999999e-06 2024-09-15T03:47:39.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:39.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838532+0000", 2024-09-15T03:47:39.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2059999999999998e-06 2024-09-15T03:47:39.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838609+0000", 2024-09-15T03:47:39.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7193000000000004e-05 2024-09-15T03:47:39.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:39.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838615+0000", 2024-09-15T03:47:39.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7980000000000002e-06 2024-09-15T03:47:39.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843986+0000", 2024-09-15T03:47:39.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053706450000000003 2024-09-15T03:47:39.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:39.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844106+0000", 2024-09-15T03:47:39.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120079 2024-09-15T03:47:39.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:39.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844669+0000", 2024-09-15T03:47:39.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056290300000000001 2024-09-15T03:47:39.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:39.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844804+0000", 2024-09-15T03:47:39.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013575600000000001 2024-09-15T03:47:39.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:39.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844820+0000", 2024-09-15T03:47:39.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5145e-05 2024-09-15T03:47:39.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:39.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844893+0000", 2024-09-15T03:47:39.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3270999999999997e-05 2024-09-15T03:47:39.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:39.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845043+0000", 2024-09-15T03:47:39.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150144 2024-09-15T03:47:39.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:39.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844473+0000", 2024-09-15T03:47:39.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99943048700000003 2024-09-15T03:47:39.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:39.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844538+0000", 2024-09-15T03:47:39.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4179999999999999e-05 2024-09-15T03:47:39.581 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.581 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:39.581 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.581 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:352 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~557056] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:39.581 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:47:39.581 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.670504139999998, 2024-09-15T03:47:39.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.016265065, 2024-09-15T03:47:39.581 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:39.581 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:39.581 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:39.582 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:39.582 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:39.582 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 352 2024-09-15T03:47:39.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.582 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:39.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:39.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:47:39.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:39.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:47:39.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:39.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844718+0000", 2024-09-15T03:47:39.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2340000000000001e-06 2024-09-15T03:47:39.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:39.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845237+0000", 2024-09-15T03:47:39.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051953399999999999 2024-09-15T03:47:39.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:39.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845273+0000", 2024-09-15T03:47:39.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5683999999999999e-05 2024-09-15T03:47:39.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:39.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845281+0000", 2024-09-15T03:47:39.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8439999999999998e-06 2024-09-15T03:47:39.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845365+0000", 2024-09-15T03:47:39.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4651e-05 2024-09-15T03:47:39.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:39.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845486+0000", 2024-09-15T03:47:39.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012009499999999999 2024-09-15T03:47:39.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:39.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847101+0000", 2024-09-15T03:47:39.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016157350000000001 2024-09-15T03:47:39.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:39.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847602+0000", 2024-09-15T03:47:39.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050066200000000003 2024-09-15T03:47:39.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:39.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847622+0000", 2024-09-15T03:47:39.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9749999999999999e-05 2024-09-15T03:47:39.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:39.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848616+0000", 2024-09-15T03:47:39.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099390799999999994 2024-09-15T03:47:39.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:39.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848709+0000", 2024-09-15T03:47:39.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2874000000000006e-05 2024-09-15T03:47:39.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:39.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.860888+0000", 2024-09-15T03:47:39.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.012179707 2024-09-15T03:47:39.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:39.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.860981+0000", 2024-09-15T03:47:39.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2386999999999995e-05 2024-09-15T03:47:39.587 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.587 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:39.588 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.588 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:353 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~671744] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:39.588 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:47:39.588 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.666941946999998, 2024-09-15T03:47:39.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.012912206, 2024-09-15T03:47:39.588 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:39.588 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:39.588 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:39.588 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:39.588 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:39.588 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 353 2024-09-15T03:47:39.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.589 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:39.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:39.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:47:39.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:39.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:47:39.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:39.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848279+0000", 2024-09-15T03:47:39.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.362e-06 2024-09-15T03:47:39.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:39.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849024+0000", 2024-09-15T03:47:39.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00074495500000000005 2024-09-15T03:47:39.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:39.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849039+0000", 2024-09-15T03:47:39.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5119e-05 2024-09-15T03:47:39.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:39.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849046+0000", 2024-09-15T03:47:39.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6359999999999997e-06 2024-09-15T03:47:39.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849115+0000", 2024-09-15T03:47:39.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8998999999999994e-05 2024-09-15T03:47:39.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:39.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849251+0000", 2024-09-15T03:47:39.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013589799999999999 2024-09-15T03:47:39.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:39.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850605+0000", 2024-09-15T03:47:39.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013544659999999999 2024-09-15T03:47:39.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:39.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850703+0000", 2024-09-15T03:47:39.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7576000000000002e-05 2024-09-15T03:47:39.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:39.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850719+0000", 2024-09-15T03:47:39.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5767e-05 2024-09-15T03:47:39.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:39.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851562+0000", 2024-09-15T03:47:39.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084361899999999999 2024-09-15T03:47:39.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:39.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851705+0000", 2024-09-15T03:47:39.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000142893 2024-09-15T03:47:39.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:39.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861112+0000", 2024-09-15T03:47:39.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.009407349 2024-09-15T03:47:39.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:39.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861190+0000", 2024-09-15T03:47:39.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7566999999999998e-05 2024-09-15T03:47:39.595 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.595 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:39.595 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.595 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:354 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~98304] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:39.595 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:47:39.596 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.666139510999997, 2024-09-15T03:47:39.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0177595669999999, 2024-09-15T03:47:39.596 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:39.596 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:39.596 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:39.596 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:39.596 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:39.596 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 354 2024-09-15T03:47:39.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.596 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:39.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:39.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:47:39.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:39.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:47:39.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:39.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849081+0000", 2024-09-15T03:47:39.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0610000000000001e-06 2024-09-15T03:47:39.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:39.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849157+0000", 2024-09-15T03:47:39.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5804999999999997e-05 2024-09-15T03:47:39.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:39.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849168+0000", 2024-09-15T03:47:39.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0730000000000001e-05 2024-09-15T03:47:39.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:39.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849174+0000", 2024-09-15T03:47:39.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3609999999999997e-06 2024-09-15T03:47:39.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850814+0000", 2024-09-15T03:47:39.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001640124 2024-09-15T03:47:39.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:39.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850894+0000", 2024-09-15T03:47:39.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0167999999999995e-05 2024-09-15T03:47:39.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:39.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851406+0000", 2024-09-15T03:47:39.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051203200000000005 2024-09-15T03:47:39.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:39.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851506+0000", 2024-09-15T03:47:39.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9457999999999995e-05 2024-09-15T03:47:39.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:39.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851574+0000", 2024-09-15T03:47:39.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8145000000000006e-05 2024-09-15T03:47:39.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:39.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851632+0000", 2024-09-15T03:47:39.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8108e-05 2024-09-15T03:47:39.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:39.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851712+0000", 2024-09-15T03:47:39.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9380000000000002e-05 2024-09-15T03:47:39.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:39.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.866753+0000", 2024-09-15T03:47:39.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0150417979999999 2024-09-15T03:47:39.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:39.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.866840+0000", 2024-09-15T03:47:39.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6397000000000006e-05 2024-09-15T03:47:39.602 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.602 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:39.602 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.602 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:355 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [setxattr _header (57) in=7b] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:39.602 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:47:39.602 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.666010198000002, 2024-09-15T03:47:39.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.018024966, 2024-09-15T03:47:39.603 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:39.603 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:39.603 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:39.603 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:39.603 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:39.603 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 355 2024-09-15T03:47:39.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.603 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:39.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:39.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:47:39.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:39.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:47:39.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:39.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849210+0000", 2024-09-15T03:47:39.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.33e-07 2024-09-15T03:47:39.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:39.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849215+0000", 2024-09-15T03:47:39.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4379999999999997e-06 2024-09-15T03:47:39.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:39.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849234+0000", 2024-09-15T03:47:39.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9409999999999999e-05 2024-09-15T03:47:39.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:39.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849241+0000", 2024-09-15T03:47:39.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6370000000000002e-06 2024-09-15T03:47:39.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851823+0000", 2024-09-15T03:47:39.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025817689999999998 2024-09-15T03:47:39.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:39.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851934+0000", 2024-09-15T03:47:39.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111357 2024-09-15T03:47:39.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:39.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852466+0000", 2024-09-15T03:47:39.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053248100000000001 2024-09-15T03:47:39.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:39.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852580+0000", 2024-09-15T03:47:39.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011379699999999999 2024-09-15T03:47:39.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:39.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852607+0000", 2024-09-15T03:47:39.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279e-05 2024-09-15T03:47:39.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:39.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852693+0000", 2024-09-15T03:47:39.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6792999999999994e-05 2024-09-15T03:47:39.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:39.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852869+0000", 2024-09-15T03:47:39.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017537499999999999 2024-09-15T03:47:39.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:39.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867006+0000", 2024-09-15T03:47:39.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.014136996 2024-09-15T03:47:39.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:39.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867234+0000", 2024-09-15T03:47:39.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022870099999999999 2024-09-15T03:47:39.609 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.609 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:39.609 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.609 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:356 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:39.609 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:47:39.609 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.665973434999998, 2024-09-15T03:47:39.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.020403521, 2024-09-15T03:47:39.610 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:39.610 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:39.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:39.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:39.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:39.610 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 356 2024-09-15T03:47:39.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.610 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:39.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:39.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:47:39.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:39.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:47:39.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:39.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849247+0000", 2024-09-15T03:47:39.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2699999999999999e-07 2024-09-15T03:47:39.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:39.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849250+0000", 2024-09-15T03:47:39.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2610000000000002e-06 2024-09-15T03:47:39.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:39.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849257+0000", 2024-09-15T03:47:39.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.703e-06 2024-09-15T03:47:39.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:39.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849261+0000", 2024-09-15T03:47:39.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7910000000000002e-06 2024-09-15T03:47:39.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852672+0000", 2024-09-15T03:47:39.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003411627 2024-09-15T03:47:39.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:39.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852801+0000", 2024-09-15T03:47:39.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000128918 2024-09-15T03:47:39.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867347+0000", 2024-09-15T03:47:39.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0145458380000001 2024-09-15T03:47:39.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:39.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867476+0000", 2024-09-15T03:47:39.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012915100000000001 2024-09-15T03:47:39.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:39.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.869650+0000", 2024-09-15T03:47:39.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021737050000000002 2024-09-15T03:47:39.614 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.614 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:39.614 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.615 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs3 65/64 ECSubWrite(tid=110, reqid=client.4521.0:474, at_version=65'72, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:39.615 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.850059+0000", 2024-09-15T03:47:39.615 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.665160514, 2024-09-15T03:47:39.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013621852, 2024-09-15T03:47:39.615 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:39.615 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:39.615 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:39.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:39.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.850059+0000", 2024-09-15T03:47:39.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:39.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.850059+0000", 2024-09-15T03:47:39.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:39.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.850060+0000", 2024-09-15T03:47:39.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8100000000000003e-07 2024-09-15T03:47:39.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:39.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.850200+0000", 2024-09-15T03:47:39.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014009 2024-09-15T03:47:39.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:39.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851295+0000", 2024-09-15T03:47:39.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010954579999999999 2024-09-15T03:47:39.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:39.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851302+0000", 2024-09-15T03:47:39.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5370000000000003e-06 2024-09-15T03:47:39.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851797+0000", 2024-09-15T03:47:39.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049567000000000003 2024-09-15T03:47:39.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:39.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851809+0000", 2024-09-15T03:47:39.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1270000000000001e-05 2024-09-15T03:47:39.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:39.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852166+0000", 2024-09-15T03:47:39.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035684699999999998 2024-09-15T03:47:39.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:39.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852186+0000", 2024-09-15T03:47:39.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0737000000000002e-05 2024-09-15T03:47:39.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:39.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852323+0000", 2024-09-15T03:47:39.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013634699999999999 2024-09-15T03:47:39.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:39.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852351+0000", 2024-09-15T03:47:39.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8515000000000001e-05 2024-09-15T03:47:39.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:39.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.863681+0000", 2024-09-15T03:47:39.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011329799999999999 2024-09-15T03:47:39.620 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.620 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:39.620 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.621 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/64 ECSubWrite(tid=108, reqid=client.4521.0:466, at_version=65'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:39.621 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.856519+0000", 2024-09-15T03:47:39.621 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.658701088000001, 2024-09-15T03:47:39.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015639281000000001, 2024-09-15T03:47:39.621 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:39.621 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:39.621 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:39.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:39.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856519+0000", 2024-09-15T03:47:39.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:39.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856519+0000", 2024-09-15T03:47:39.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:39.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856519+0000", 2024-09-15T03:47:39.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4200000000000005e-07 2024-09-15T03:47:39.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:39.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857060+0000", 2024-09-15T03:47:39.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054053200000000004 2024-09-15T03:47:39.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:39.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857069+0000", 2024-09-15T03:47:39.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2979999999999997e-06 2024-09-15T03:47:39.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:39.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857074+0000", 2024-09-15T03:47:39.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7570000000000004e-06 2024-09-15T03:47:39.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857122+0000", 2024-09-15T03:47:39.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7828999999999999e-05 2024-09-15T03:47:39.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:39.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857128+0000", 2024-09-15T03:47:39.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6409999999999996e-06 2024-09-15T03:47:39.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:39.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857491+0000", 2024-09-15T03:47:39.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036239300000000003 2024-09-15T03:47:39.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:39.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857503+0000", 2024-09-15T03:47:39.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2468e-05 2024-09-15T03:47:39.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:39.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857718+0000", 2024-09-15T03:47:39.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021492200000000001 2024-09-15T03:47:39.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:39.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857745+0000", 2024-09-15T03:47:39.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7189e-05 2024-09-15T03:47:39.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:39.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.872158+0000", 2024-09-15T03:47:39.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01441241 2024-09-15T03:47:39.627 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.628 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:39.628 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.628 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/64 ECSubWrite(tid=111, reqid=client.4521.0:467, at_version=65'29, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:39.628 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.858407+0000", 2024-09-15T03:47:39.628 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.656812332000001, 2024-09-15T03:47:39.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013931720999999999, 2024-09-15T03:47:39.628 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:39.628 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:39.628 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:39.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:39.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858407+0000", 2024-09-15T03:47:39.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:39.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858407+0000", 2024-09-15T03:47:39.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:39.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858408+0000", 2024-09-15T03:47:39.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5199999999999996e-07 2024-09-15T03:47:39.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:39.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858647+0000", 2024-09-15T03:47:39.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000238743 2024-09-15T03:47:39.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:39.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858659+0000", 2024-09-15T03:47:39.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2536e-05 2024-09-15T03:47:39.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:39.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858665+0000", 2024-09-15T03:47:39.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3340000000000001e-06 2024-09-15T03:47:39.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858724+0000", 2024-09-15T03:47:39.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8887e-05 2024-09-15T03:47:39.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:39.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858733+0000", 2024-09-15T03:47:39.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.781e-06 2024-09-15T03:47:39.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:39.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858903+0000", 2024-09-15T03:47:39.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016952199999999999 2024-09-15T03:47:39.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:39.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.858918+0000", 2024-09-15T03:47:39.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.472e-05 2024-09-15T03:47:39.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:39.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.860148+0000", 2024-09-15T03:47:39.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012305980000000001 2024-09-15T03:47:39.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:39.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.860176+0000", 2024-09-15T03:47:39.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7393e-05 2024-09-15T03:47:39.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:39.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.872339+0000", 2024-09-15T03:47:39.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012163455 2024-09-15T03:47:39.634 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.634 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:39.634 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.634 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:527 3.7s0 3:e4b8e19d:::smithi00132172-41:head [copy-from ver 51] snapc 23=[23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:39.634 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:47:39.634 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.566404472999999, 2024-09-15T03:47:39.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030362388000000001, 2024-09-15T03:47:39.634 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:39.634 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:39.634 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:39.634 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:39.635 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:39.635 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 527 2024-09-15T03:47:39.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.635 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:39.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:39.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:47:39.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:39.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:47:39.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:39.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948817+0000", 2024-09-15T03:47:39.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3230000000000001e-06 2024-09-15T03:47:39.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:39.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948825+0000", 2024-09-15T03:47:39.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5900000000000008e-06 2024-09-15T03:47:39.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:39.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948838+0000", 2024-09-15T03:47:39.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2712e-05 2024-09-15T03:47:39.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:39.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948842+0000", 2024-09-15T03:47:39.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4170000000000002e-06 2024-09-15T03:47:39.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948923+0000", 2024-09-15T03:47:39.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0686000000000006e-05 2024-09-15T03:47:39.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:47:39.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949020+0000", 2024-09-15T03:47:39.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6984999999999994e-05 2024-09-15T03:47:39.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954390+0000", 2024-09-15T03:47:39.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053704790000000001 2024-09-15T03:47:39.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:39.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954558+0000", 2024-09-15T03:47:39.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000167202 2024-09-15T03:47:39.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:39.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.973998+0000", 2024-09-15T03:47:39.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019440316999999999 2024-09-15T03:47:39.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:39.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974527+0000", 2024-09-15T03:47:39.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052943800000000002 2024-09-15T03:47:39.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:39.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974541+0000", 2024-09-15T03:47:39.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3774999999999999e-05 2024-09-15T03:47:39.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:39.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974841+0000", 2024-09-15T03:47:39.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029942899999999998 2024-09-15T03:47:39.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:39.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.976067+0000", 2024-09-15T03:47:39.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012265399999999999 2024-09-15T03:47:39.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:39.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.978952+0000", 2024-09-15T03:47:39.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028848839999999999 2024-09-15T03:47:39.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:39.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.979178+0000", 2024-09-15T03:47:39.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022561099999999999 2024-09-15T03:47:39.641 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.641 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:39.641 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.641 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:528 3.7s0 3:e4b8e19d:::smithi00132172-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:39.642 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:39.642 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.566370064000001, 2024-09-15T03:47:39.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026927036000000001, 2024-09-15T03:47:39.642 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:39.642 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:39.642 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:39.642 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:39.642 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:39.642 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 528 2024-09-15T03:47:39.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.642 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:39.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:39.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:39.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:39.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:39.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:39.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:39.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9700000000000003e-07 2024-09-15T03:47:39.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:39.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948852+0000", 2024-09-15T03:47:39.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.604e-06 2024-09-15T03:47:39.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:39.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948855+0000", 2024-09-15T03:47:39.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9369999999999997e-06 2024-09-15T03:47:39.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:39.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948858+0000", 2024-09-15T03:47:39.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2709999999999999e-06 2024-09-15T03:47:39.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949080+0000", 2024-09-15T03:47:39.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000222148 2024-09-15T03:47:39.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:47:39.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949156+0000", 2024-09-15T03:47:39.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6215000000000001e-05 2024-09-15T03:47:39.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954754+0000", 2024-09-15T03:47:39.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055978060000000003 2024-09-15T03:47:39.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:39.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954858+0000", 2024-09-15T03:47:39.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010419199999999999 2024-09-15T03:47:39.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974861+0000", 2024-09-15T03:47:39.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020002671999999999 2024-09-15T03:47:39.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:39.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974987+0000", 2024-09-15T03:47:39.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012638699999999999 2024-09-15T03:47:39.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:39.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.975777+0000", 2024-09-15T03:47:39.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00078950699999999995 2024-09-15T03:47:39.647 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.647 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:39.647 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.648 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:742 3.7s0 3:ece1472c:::smithi00132172-43:head [append 0~753664] snapc 32=[32,2d,2b,26,25,24,23,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T03:47:39.648 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.930340+0000", 2024-09-15T03:47:39.648 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.5848799820000004, 2024-09-15T03:47:39.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013978454, 2024-09-15T03:47:39.648 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:39.648 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:39.648 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:39.648 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:39.648 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:39.648 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 742 2024-09-15T03:47:39.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.648 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:39.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:39.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.930340+0000", 2024-09-15T03:47:39.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:39.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.930340+0000", 2024-09-15T03:47:39.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:39.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:39.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.930341+0000", 2024-09-15T03:47:39.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6759999999999999e-06 2024-09-15T03:47:39.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:39.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.931144+0000", 2024-09-15T03:47:39.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00080309900000000004 2024-09-15T03:47:39.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:39.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.931157+0000", 2024-09-15T03:47:39.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2397e-05 2024-09-15T03:47:39.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:39.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.931163+0000", 2024-09-15T03:47:39.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6790000000000002e-06 2024-09-15T03:47:39.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:39.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.931205+0000", 2024-09-15T03:47:39.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2539e-05 2024-09-15T03:47:39.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:39.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.931296+0000", 2024-09-15T03:47:39.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1080000000000002e-05 2024-09-15T03:47:39.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:39.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936342+0000", 2024-09-15T03:47:39.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050460119999999999 2024-09-15T03:47:39.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:39.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936498+0000", 2024-09-15T03:47:39.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015619200000000001 2024-09-15T03:47:39.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:39.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.937026+0000", 2024-09-15T03:47:39.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052732899999999999 2024-09-15T03:47:39.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:39.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.937498+0000", 2024-09-15T03:47:39.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047273200000000001 2024-09-15T03:47:39.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:39.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.940834+0000", 2024-09-15T03:47:39.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003335916 2024-09-15T03:47:39.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:39.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.943811+0000", 2024-09-15T03:47:39.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029768139999999999 2024-09-15T03:47:39.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:39.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:39.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:39.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.944318+0000", 2024-09-15T03:47:39.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050698900000000001 2024-09-15T03:47:39.654 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.654 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:39.654 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.654 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:39.654 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:39.654 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:39.656 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:47:39.656 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:39.655+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:39.757 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:39.756+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:39.798 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:39.798 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:39.798 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:39.798 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:39.806 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:47:39.843 INFO:teuthology.orchestra.run.smithi001.stderr:set 2.5 pg_upmap_items mapping to [9->11,8->7] 2024-09-15T03:47:39.844 INFO:tasks.rados.rados.0.smithi001.stdout:312: done (2 left) 2024-09-15T03:47:39.844 INFO:tasks.rados.rados.0.smithi001.stdout:313: done (1 left) 2024-09-15T03:47:39.844 INFO:tasks.rados.rados.0.smithi001.stdout:314: done (0 left) 2024-09-15T03:47:39.844 INFO:tasks.rados.rados.0.smithi001.stdout:315: snap_create 2024-09-15T03:47:39.857 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:39.856+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:39.908 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:39.908 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:39.916 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:47:39.958 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:39.957+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:40.059 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:40.059 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:40.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:40.059 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:40.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.060 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 38/28 ECSubWrite(tid=33, reqid=client.4521.0:222, at_version=38'14, trim_to=0'0, roll_forward_to=35'12))", 2024-09-15T03:47:40.060 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:47:40.060 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.926386612999998, 2024-09-15T03:47:40.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9949187450000001, 2024-09-15T03:47:40.060 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.060 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.060 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:47:40.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:47:40.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130996+0000", 2024-09-15T03:47:40.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8809999999999998e-06 2024-09-15T03:47:40.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131020+0000", 2024-09-15T03:47:40.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4190999999999999e-05 2024-09-15T03:47:40.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131039+0000", 2024-09-15T03:47:40.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9228999999999998e-05 2024-09-15T03:47:40.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131048+0000", 2024-09-15T03:47:40.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2380000000000003e-06 2024-09-15T03:47:40.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131122+0000", 2024-09-15T03:47:40.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3146999999999996e-05 2024-09-15T03:47:40.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131132+0000", 2024-09-15T03:47:40.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.08e-05 2024-09-15T03:47:40.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131381+0000", 2024-09-15T03:47:40.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000248936 2024-09-15T03:47:40.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131392+0000", 2024-09-15T03:47:40.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0987e-05 2024-09-15T03:47:40.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131444+0000", 2024-09-15T03:47:40.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1585000000000002e-05 2024-09-15T03:47:40.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131465+0000", 2024-09-15T03:47:40.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1365999999999998e-05 2024-09-15T03:47:40.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.125911+0000", 2024-09-15T03:47:40.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9944453850000001 2024-09-15T03:47:40.066 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.066 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.066 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.066 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 38/28 ECSubWrite(tid=34, reqid=client.4521.0:222, at_version=0'0, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T03:47:40.066 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:47:40.066 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.924987514000001, 2024-09-15T03:47:40.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.002752697, 2024-09-15T03:47:40.066 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.066 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.066 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:47:40.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:47:40.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132392+0000", 2024-09-15T03:47:40.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5899999999999995e-07 2024-09-15T03:47:40.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132415+0000", 2024-09-15T03:47:40.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3561e-05 2024-09-15T03:47:40.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132436+0000", 2024-09-15T03:47:40.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0879000000000001e-05 2024-09-15T03:47:40.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132444+0000", 2024-09-15T03:47:40.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9200000000000004e-06 2024-09-15T03:47:40.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132497+0000", 2024-09-15T03:47:40.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2299000000000002e-05 2024-09-15T03:47:40.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132508+0000", 2024-09-15T03:47:40.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1100999999999999e-05 2024-09-15T03:47:40.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132615+0000", 2024-09-15T03:47:40.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107546 2024-09-15T03:47:40.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132635+0000", 2024-09-15T03:47:40.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9374000000000001e-05 2024-09-15T03:47:40.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132677+0000", 2024-09-15T03:47:40.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2456999999999997e-05 2024-09-15T03:47:40.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132711+0000", 2024-09-15T03:47:40.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4118000000000001e-05 2024-09-15T03:47:40.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135144+0000", 2024-09-15T03:47:40.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.002432583 2024-09-15T03:47:40.071 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.071 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.072 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.072 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=45, reqid=client.4521.0:227, at_version=38'24, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:47:40.072 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:40.072 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.902060972000001, 2024-09-15T03:47:40.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.974765339, 2024-09-15T03:47:40.072 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.072 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.072 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:40.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:40.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:40.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.43e-07 2024-09-15T03:47:40.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156110+0000", 2024-09-15T03:47:40.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000792409 2024-09-15T03:47:40.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156139+0000", 2024-09-15T03:47:40.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9207e-05 2024-09-15T03:47:40.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156148+0000", 2024-09-15T03:47:40.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1269999999999992e-06 2024-09-15T03:47:40.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156248+0000", 2024-09-15T03:47:40.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9421999999999998e-05 2024-09-15T03:47:40.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156258+0000", 2024-09-15T03:47:40.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6989999999999998e-06 2024-09-15T03:47:40.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156674+0000", 2024-09-15T03:47:40.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041682600000000002 2024-09-15T03:47:40.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156705+0000", 2024-09-15T03:47:40.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0525000000000003e-05 2024-09-15T03:47:40.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157742+0000", 2024-09-15T03:47:40.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010374 2024-09-15T03:47:40.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157777+0000", 2024-09-15T03:47:40.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4878e-05 2024-09-15T03:47:40.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.130083+0000", 2024-09-15T03:47:40.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972305703 2024-09-15T03:47:40.077 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.077 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.077 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.078 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=37, reqid=client.4521.0:236, at_version=38'5, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:40.078 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:40.078 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.900447516, 2024-09-15T03:47:40.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9740836640000001, 2024-09-15T03:47:40.079 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.079 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.079 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:40.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:40.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:40.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1300000000000001e-07 2024-09-15T03:47:40.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156953+0000", 2024-09-15T03:47:40.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1585000000000001e-05 2024-09-15T03:47:40.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156974+0000", 2024-09-15T03:47:40.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0764e-05 2024-09-15T03:47:40.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156981+0000", 2024-09-15T03:47:40.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2760000000000003e-06 2024-09-15T03:47:40.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157061+0000", 2024-09-15T03:47:40.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9583e-05 2024-09-15T03:47:40.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157075+0000", 2024-09-15T03:47:40.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4019000000000001e-05 2024-09-15T03:47:40.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157351+0000", 2024-09-15T03:47:40.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027641100000000001 2024-09-15T03:47:40.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157746+0000", 2024-09-15T03:47:40.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039553199999999998 2024-09-15T03:47:40.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157838+0000", 2024-09-15T03:47:40.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1632000000000006e-05 2024-09-15T03:47:40.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157856+0000", 2024-09-15T03:47:40.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7873999999999999e-05 2024-09-15T03:47:40.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.131015+0000", 2024-09-15T03:47:40.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9731586750000001 2024-09-15T03:47:40.084 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.084 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.084 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.084 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=46, reqid=client.4521.0:228, at_version=38'25, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:47:40.084 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:40.084 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.900388734000003, 2024-09-15T03:47:40.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9732838510000001, 2024-09-15T03:47:40.084 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.084 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.084 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:40.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:40.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:40.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0999999999999998e-07 2024-09-15T03:47:40.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157149+0000", 2024-09-15T03:47:40.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000158763 2024-09-15T03:47:40.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157165+0000", 2024-09-15T03:47:40.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5883999999999999e-05 2024-09-15T03:47:40.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157175+0000", 2024-09-15T03:47:40.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4949999999999997e-06 2024-09-15T03:47:40.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157219+0000", 2024-09-15T03:47:40.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4069000000000001e-05 2024-09-15T03:47:40.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157228+0000", 2024-09-15T03:47:40.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7829999999999993e-06 2024-09-15T03:47:40.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157345+0000", 2024-09-15T03:47:40.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116655 2024-09-15T03:47:40.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157746+0000", 2024-09-15T03:47:40.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040045100000000002 2024-09-15T03:47:40.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157838+0000", 2024-09-15T03:47:40.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1955999999999998e-05 2024-09-15T03:47:40.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157888+0000", 2024-09-15T03:47:40.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0943000000000003e-05 2024-09-15T03:47:40.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.130274+0000", 2024-09-15T03:47:40.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972385142 2024-09-15T03:47:40.089 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.090 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.090 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.090 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=38, reqid=client.4521.0:237, at_version=38'6, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:40.090 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:47:40.090 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.899046163999998, 2024-09-15T03:47:40.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9764011529999999, 2024-09-15T03:47:40.090 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.090 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.090 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:47:40.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:47:40.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158333+0000", 2024-09-15T03:47:40.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8599999999999997e-07 2024-09-15T03:47:40.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158353+0000", 2024-09-15T03:47:40.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9504000000000002e-05 2024-09-15T03:47:40.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158368+0000", 2024-09-15T03:47:40.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4966999999999999e-05 2024-09-15T03:47:40.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158374+0000", 2024-09-15T03:47:40.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.798e-06 2024-09-15T03:47:40.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158430+0000", 2024-09-15T03:47:40.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5744e-05 2024-09-15T03:47:40.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158440+0000", 2024-09-15T03:47:40.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4560000000000006e-06 2024-09-15T03:47:40.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158529+0000", 2024-09-15T03:47:40.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9172999999999995e-05 2024-09-15T03:47:40.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158543+0000", 2024-09-15T03:47:40.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.395e-05 2024-09-15T03:47:40.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158654+0000", 2024-09-15T03:47:40.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011110700000000001 2024-09-15T03:47:40.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158671+0000", 2024-09-15T03:47:40.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7068999999999999e-05 2024-09-15T03:47:40.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134733+0000", 2024-09-15T03:47:40.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.976062599 2024-09-15T03:47:40.095 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.095 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.095 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.096 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=47, reqid=client.4521.0:229, at_version=38'26, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:47:40.096 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:47:40.096 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.899042201999997, 2024-09-15T03:47:40.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9727810969999999, 2024-09-15T03:47:40.096 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.096 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.096 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:47:40.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:47:40.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158337+0000", 2024-09-15T03:47:40.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.15e-07 2024-09-15T03:47:40.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158351+0000", 2024-09-15T03:47:40.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3871e-05 2024-09-15T03:47:40.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158372+0000", 2024-09-15T03:47:40.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1058999999999999e-05 2024-09-15T03:47:40.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158376+0000", 2024-09-15T03:47:40.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5249999999999999e-06 2024-09-15T03:47:40.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158420+0000", 2024-09-15T03:47:40.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3918000000000002e-05 2024-09-15T03:47:40.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158429+0000", 2024-09-15T03:47:40.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5150000000000006e-06 2024-09-15T03:47:40.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158514+0000", 2024-09-15T03:47:40.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5252000000000003e-05 2024-09-15T03:47:40.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158542+0000", 2024-09-15T03:47:40.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.798e-05 2024-09-15T03:47:40.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158653+0000", 2024-09-15T03:47:40.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111315 2024-09-15T03:47:40.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158701+0000", 2024-09-15T03:47:40.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8241000000000001e-05 2024-09-15T03:47:40.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.131117+0000", 2024-09-15T03:47:40.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972416006 2024-09-15T03:47:40.101 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.101 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.101 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.101 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=39, reqid=client.4521.0:238, at_version=38'7, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:40.101 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:47:40.101 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.898355045999999, 2024-09-15T03:47:40.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9758214509999998, 2024-09-15T03:47:40.101 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.101 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.102 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:47:40.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:47:40.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159024+0000", 2024-09-15T03:47:40.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6700000000000003e-07 2024-09-15T03:47:40.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159096+0000", 2024-09-15T03:47:40.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1386999999999999e-05 2024-09-15T03:47:40.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159113+0000", 2024-09-15T03:47:40.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7020999999999998e-05 2024-09-15T03:47:40.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159122+0000", 2024-09-15T03:47:40.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4720000000000001e-06 2024-09-15T03:47:40.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159168+0000", 2024-09-15T03:47:40.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5766000000000002e-05 2024-09-15T03:47:40.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159176+0000", 2024-09-15T03:47:40.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3070000000000003e-06 2024-09-15T03:47:40.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159258+0000", 2024-09-15T03:47:40.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1398999999999998e-05 2024-09-15T03:47:40.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159292+0000", 2024-09-15T03:47:40.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4480000000000002e-05 2024-09-15T03:47:40.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159342+0000", 2024-09-15T03:47:40.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9450999999999998e-05 2024-09-15T03:47:40.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159362+0000", 2024-09-15T03:47:40.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0633e-05 2024-09-15T03:47:40.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134845+0000", 2024-09-15T03:47:40.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.975482768 2024-09-15T03:47:40.106 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.106 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.107 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.107 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=48, reqid=client.4521.0:229, at_version=0'0, trim_to=0'0, roll_forward_to=38'26))", 2024-09-15T03:47:40.107 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:47:40.107 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.89698825, 2024-09-15T03:47:40.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.974485472, 2024-09-15T03:47:40.107 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.107 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.107 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:47:40.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:47:40.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160391+0000", 2024-09-15T03:47:40.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.136e-06 2024-09-15T03:47:40.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160405+0000", 2024-09-15T03:47:40.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3613e-05 2024-09-15T03:47:40.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160426+0000", 2024-09-15T03:47:40.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1234999999999999e-05 2024-09-15T03:47:40.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160434+0000", 2024-09-15T03:47:40.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.61e-06 2024-09-15T03:47:40.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160496+0000", 2024-09-15T03:47:40.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2384000000000004e-05 2024-09-15T03:47:40.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160507+0000", 2024-09-15T03:47:40.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1005000000000001e-05 2024-09-15T03:47:40.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160632+0000", 2024-09-15T03:47:40.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124566 2024-09-15T03:47:40.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160653+0000", 2024-09-15T03:47:40.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0707999999999999e-05 2024-09-15T03:47:40.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160701+0000", 2024-09-15T03:47:40.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8751999999999997e-05 2024-09-15T03:47:40.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160739+0000", 2024-09-15T03:47:40.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7486e-05 2024-09-15T03:47:40.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134876+0000", 2024-09-15T03:47:40.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9741369770000001 2024-09-15T03:47:40.112 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.112 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.113 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.113 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=40, reqid=client.4521.0:238, at_version=0'0, trim_to=0'0, roll_forward_to=38'7))", 2024-09-15T03:47:40.113 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:47:40.113 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.895141829000003, 2024-09-15T03:47:40.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9727862730000001, 2024-09-15T03:47:40.113 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.113 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.113 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:47:40.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:47:40.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162238+0000", 2024-09-15T03:47:40.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.672e-06 2024-09-15T03:47:40.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162253+0000", 2024-09-15T03:47:40.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4932e-05 2024-09-15T03:47:40.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162284+0000", 2024-09-15T03:47:40.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1211000000000003e-05 2024-09-15T03:47:40.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162296+0000", 2024-09-15T03:47:40.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1229999999999999e-05 2024-09-15T03:47:40.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162373+0000", 2024-09-15T03:47:40.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7146999999999998e-05 2024-09-15T03:47:40.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162387+0000", 2024-09-15T03:47:40.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4290999999999999e-05 2024-09-15T03:47:40.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162502+0000", 2024-09-15T03:47:40.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115054 2024-09-15T03:47:40.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162516+0000", 2024-09-15T03:47:40.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3597999999999999e-05 2024-09-15T03:47:40.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162558+0000", 2024-09-15T03:47:40.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2562999999999999e-05 2024-09-15T03:47:40.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162584+0000", 2024-09-15T03:47:40.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5738e-05 2024-09-15T03:47:40.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135023+0000", 2024-09-15T03:47:40.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9724388369999999 2024-09-15T03:47:40.118 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.118 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.118 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.118 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 39/28 ECSubWrite(tid=38, reqid=osd.11.0:38, at_version=39'16, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T03:47:40.118 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:40.119 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.913759529000004, 2024-09-15T03:47:40.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99210653800000004, 2024-09-15T03:47:40.119 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.119 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.119 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:40.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:40.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:40.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1800000000000002e-07 2024-09-15T03:47:40.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143648+0000", 2024-09-15T03:47:40.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.849e-05 2024-09-15T03:47:40.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143670+0000", 2024-09-15T03:47:40.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2036e-05 2024-09-15T03:47:40.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143680+0000", 2024-09-15T03:47:40.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8320000000000001e-06 2024-09-15T03:47:40.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143783+0000", 2024-09-15T03:47:40.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103569 2024-09-15T03:47:40.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143801+0000", 2024-09-15T03:47:40.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8077999999999999e-05 2024-09-15T03:47:40.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144077+0000", 2024-09-15T03:47:40.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027569500000000001 2024-09-15T03:47:40.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144459+0000", 2024-09-15T03:47:40.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038227299999999998 2024-09-15T03:47:40.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144519+0000", 2024-09-15T03:47:40.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9907000000000001e-05 2024-09-15T03:47:40.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144628+0000", 2024-09-15T03:47:40.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108958 2024-09-15T03:47:40.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135726+0000", 2024-09-15T03:47:40.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99109738199999997 2024-09-15T03:47:40.124 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.124 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.124 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.124 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 39/31 ECSubWrite(tid=41, reqid=osd.6.0:41, at_version=39'9, trim_to=0'0, roll_forward_to=38'7))", 2024-09-15T03:47:40.124 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:40.124 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.913109274, 2024-09-15T03:47:40.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99106653099999997, 2024-09-15T03:47:40.125 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.125 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.125 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:40.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:40.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:40.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.97e-07 2024-09-15T03:47:40.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144297+0000", 2024-09-15T03:47:40.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7815000000000001e-05 2024-09-15T03:47:40.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144314+0000", 2024-09-15T03:47:40.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6269e-05 2024-09-15T03:47:40.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144322+0000", 2024-09-15T03:47:40.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1880000000000002e-06 2024-09-15T03:47:40.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144767+0000", 2024-09-15T03:47:40.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044555300000000001 2024-09-15T03:47:40.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144777+0000", 2024-09-15T03:47:40.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5950000000000005e-06 2024-09-15T03:47:40.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144992+0000", 2024-09-15T03:47:40.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000215479 2024-09-15T03:47:40.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145142+0000", 2024-09-15T03:47:40.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149302 2024-09-15T03:47:40.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145249+0000", 2024-09-15T03:47:40.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106971 2024-09-15T03:47:40.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145299+0000", 2024-09-15T03:47:40.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0413e-05 2024-09-15T03:47:40.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135336+0000", 2024-09-15T03:47:40.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99003674900000005 2024-09-15T03:47:40.129 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.130 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.130 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.130 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 39/28 ECSubWrite(tid=50, reqid=osd.2.0:50, at_version=39'28, trim_to=0'0, roll_forward_to=38'26))", 2024-09-15T03:47:40.130 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:40.130 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.912811791999999, 2024-09-15T03:47:40.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99095576299999999, 2024-09-15T03:47:40.130 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.130 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.130 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:40.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:40.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:40.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.03e-07 2024-09-15T03:47:40.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144593+0000", 2024-09-15T03:47:40.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6520999999999999e-05 2024-09-15T03:47:40.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144611+0000", 2024-09-15T03:47:40.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7963000000000001e-05 2024-09-15T03:47:40.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144619+0000", 2024-09-15T03:47:40.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7409999999999992e-06 2024-09-15T03:47:40.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144821+0000", 2024-09-15T03:47:40.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000201652 2024-09-15T03:47:40.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144832+0000", 2024-09-15T03:47:40.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0827e-05 2024-09-15T03:47:40.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145099+0000", 2024-09-15T03:47:40.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026688299999999999 2024-09-15T03:47:40.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145143+0000", 2024-09-15T03:47:40.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.481e-05 2024-09-15T03:47:40.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145249+0000", 2024-09-15T03:47:40.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010618799999999999 2024-09-15T03:47:40.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145340+0000", 2024-09-15T03:47:40.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0340000000000006e-05 2024-09-15T03:47:40.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135522+0000", 2024-09-15T03:47:40.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99018263500000003 2024-09-15T03:47:40.135 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.135 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.135 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.135 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=43, reqid=client.4521.0:240, at_version=39'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:40.136 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:47:40.136 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.909471072000002, 2024-09-15T03:47:40.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99104975500000003, 2024-09-15T03:47:40.136 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.136 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.136 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:47:40.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:47:40.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147908+0000", 2024-09-15T03:47:40.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.05e-07 2024-09-15T03:47:40.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147931+0000", 2024-09-15T03:47:40.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3621e-05 2024-09-15T03:47:40.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147945+0000", 2024-09-15T03:47:40.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3648999999999999e-05 2024-09-15T03:47:40.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147951+0000", 2024-09-15T03:47:40.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0510000000000002e-06 2024-09-15T03:47:40.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148055+0000", 2024-09-15T03:47:40.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104421 2024-09-15T03:47:40.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148068+0000", 2024-09-15T03:47:40.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3019e-05 2024-09-15T03:47:40.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148375+0000", 2024-09-15T03:47:40.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030683400000000002 2024-09-15T03:47:40.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148399+0000", 2024-09-15T03:47:40.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3760999999999998e-05 2024-09-15T03:47:40.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148453+0000", 2024-09-15T03:47:40.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3851999999999999e-05 2024-09-15T03:47:40.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148481+0000", 2024-09-15T03:47:40.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8637999999999999e-05 2024-09-15T03:47:40.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.138957+0000", 2024-09-15T03:47:40.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99047580400000002 2024-09-15T03:47:40.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.141 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.141 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 39/28 ECSubWrite(tid=39, reqid=osd.11.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:47:40.141 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:47:40.141 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.909338583, 2024-09-15T03:47:40.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99452509, 2024-09-15T03:47:40.142 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.142 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.142 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:47:40.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:47:40.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148041+0000", 2024-09-15T03:47:40.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0999999999999998e-07 2024-09-15T03:47:40.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148052+0000", 2024-09-15T03:47:40.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.129e-05 2024-09-15T03:47:40.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148064+0000", 2024-09-15T03:47:40.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173e-05 2024-09-15T03:47:40.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148068+0000", 2024-09-15T03:47:40.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5680000000000001e-06 2024-09-15T03:47:40.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148136+0000", 2024-09-15T03:47:40.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7508999999999993e-05 2024-09-15T03:47:40.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148148+0000", 2024-09-15T03:47:40.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2002e-05 2024-09-15T03:47:40.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148265+0000", 2024-09-15T03:47:40.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117669 2024-09-15T03:47:40.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148289+0000", 2024-09-15T03:47:40.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3672e-05 2024-09-15T03:47:40.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148348+0000", 2024-09-15T03:47:40.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8838000000000001e-05 2024-09-15T03:47:40.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148368+0000", 2024-09-15T03:47:40.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9942999999999999e-05 2024-09-15T03:47:40.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142565+0000", 2024-09-15T03:47:40.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99419715900000005 2024-09-15T03:47:40.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.147 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.147 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.147 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=45, reqid=client.4521.0:241, at_version=39'17, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:40.147 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:47:40.147 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.906335560000002, 2024-09-15T03:47:40.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99118202200000005, 2024-09-15T03:47:40.147 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.147 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.147 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:47:40.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:47:40.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151044+0000", 2024-09-15T03:47:40.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1340000000000001e-06 2024-09-15T03:47:40.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151073+0000", 2024-09-15T03:47:40.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8894000000000002e-05 2024-09-15T03:47:40.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151095+0000", 2024-09-15T03:47:40.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1889e-05 2024-09-15T03:47:40.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151107+0000", 2024-09-15T03:47:40.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2136999999999999e-05 2024-09-15T03:47:40.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151189+0000", 2024-09-15T03:47:40.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1883000000000002e-05 2024-09-15T03:47:40.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151203+0000", 2024-09-15T03:47:40.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3825999999999999e-05 2024-09-15T03:47:40.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151305+0000", 2024-09-15T03:47:40.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101816 2024-09-15T03:47:40.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151337+0000", 2024-09-15T03:47:40.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2703000000000002e-05 2024-09-15T03:47:40.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151387+0000", 2024-09-15T03:47:40.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9366000000000001e-05 2024-09-15T03:47:40.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151405+0000", 2024-09-15T03:47:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8748000000000001e-05 2024-09-15T03:47:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142225+0000", 2024-09-15T03:47:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99081962599999995 2024-09-15T03:47:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 39/31 ECSubWrite(tid=42, reqid=osd.6.0:41, at_version=0'0, trim_to=0'0, roll_forward_to=39'9))", 2024-09-15T03:47:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:47:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.906039518999997, 2024-09-15T03:47:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99092639999999999, 2024-09-15T03:47:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:47:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:47:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151340+0000", 2024-09-15T03:47:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8299999999999996e-07 2024-09-15T03:47:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151350+0000", 2024-09-15T03:47:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0675e-05 2024-09-15T03:47:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151362+0000", 2024-09-15T03:47:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2048999999999999e-05 2024-09-15T03:47:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151369+0000", 2024-09-15T03:47:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3230000000000003e-06 2024-09-15T03:47:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151425+0000", 2024-09-15T03:47:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5964000000000002e-05 2024-09-15T03:47:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151435+0000", 2024-09-15T03:47:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9240000000000002e-06 2024-09-15T03:47:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151530+0000", 2024-09-15T03:47:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5649000000000005e-05 2024-09-15T03:47:40.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151558+0000", 2024-09-15T03:47:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7976000000000001e-05 2024-09-15T03:47:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151599+0000", 2024-09-15T03:47:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1152000000000002e-05 2024-09-15T03:47:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151617+0000", 2024-09-15T03:47:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7676000000000002e-05 2024-09-15T03:47:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142265+0000", 2024-09-15T03:47:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99064822900000005 2024-09-15T03:47:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=46, reqid=client.4521.0:242, at_version=39'18, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:40.158 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.906013883, 2024-09-15T03:47:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.991000204, 2024-09-15T03:47:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3399999999999999e-07 2024-09-15T03:47:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151393+0000", 2024-09-15T03:47:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8149000000000001e-05 2024-09-15T03:47:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151405+0000", 2024-09-15T03:47:40.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2063e-05 2024-09-15T03:47:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151415+0000", 2024-09-15T03:47:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4579999999999999e-06 2024-09-15T03:47:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151472+0000", 2024-09-15T03:47:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7018999999999997e-05 2024-09-15T03:47:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151480+0000", 2024-09-15T03:47:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1410000000000005e-06 2024-09-15T03:47:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151571+0000", 2024-09-15T03:47:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1243999999999996e-05 2024-09-15T03:47:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151602+0000", 2024-09-15T03:47:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0834000000000003e-05 2024-09-15T03:47:40.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151632+0000", 2024-09-15T03:47:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9527e-05 2024-09-15T03:47:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151685+0000", 2024-09-15T03:47:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3599e-05 2024-09-15T03:47:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142365+0000", 2024-09-15T03:47:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99067953600000003 2024-09-15T03:47:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 39/28 ECSubWrite(tid=51, reqid=osd.2.0:50, at_version=0'0, trim_to=0'0, roll_forward_to=39'28))", 2024-09-15T03:47:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:47:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.902650444999999, 2024-09-15T03:47:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98766307399999997, 2024-09-15T03:47:40.164 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:47:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:47:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154729+0000", 2024-09-15T03:47:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3799999999999996e-07 2024-09-15T03:47:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154742+0000", 2024-09-15T03:47:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2909e-05 2024-09-15T03:47:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154755+0000", 2024-09-15T03:47:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3251999999999999e-05 2024-09-15T03:47:40.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154761+0000", 2024-09-15T03:47:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6559999999999997e-06 2024-09-15T03:47:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154829+0000", 2024-09-15T03:47:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8549999999999999e-05 2024-09-15T03:47:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154840+0000", 2024-09-15T03:47:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0159999999999999e-05 2024-09-15T03:47:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154943+0000", 2024-09-15T03:47:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010347400000000001 2024-09-15T03:47:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154955+0000", 2024-09-15T03:47:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1935e-05 2024-09-15T03:47:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156442+0000", 2024-09-15T03:47:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014867859999999999 2024-09-15T03:47:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156466+0000", 2024-09-15T03:47:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4077999999999999e-05 2024-09-15T03:47:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142391+0000", 2024-09-15T03:47:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98592533599999999 2024-09-15T03:47:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=48, reqid=client.4521.0:242, at_version=0'0, trim_to=0'0, roll_forward_to=39'18))", 2024-09-15T03:47:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:47:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.900763425999997, 2024-09-15T03:47:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98592495099999999, 2024-09-15T03:47:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:47:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:47:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156616+0000", 2024-09-15T03:47:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1999999999999998e-07 2024-09-15T03:47:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156629+0000", 2024-09-15T03:47:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2870999999999999e-05 2024-09-15T03:47:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156649+0000", 2024-09-15T03:47:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9786e-05 2024-09-15T03:47:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156657+0000", 2024-09-15T03:47:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5110000000000003e-06 2024-09-15T03:47:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156723+0000", 2024-09-15T03:47:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6209000000000003e-05 2024-09-15T03:47:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156734+0000", 2024-09-15T03:47:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0475999999999999e-05 2024-09-15T03:47:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156868+0000", 2024-09-15T03:47:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013402999999999999 2024-09-15T03:47:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156889+0000", 2024-09-15T03:47:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1146e-05 2024-09-15T03:47:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156937+0000", 2024-09-15T03:47:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8031999999999997e-05 2024-09-15T03:47:40.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156970+0000", 2024-09-15T03:47:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2462999999999997e-05 2024-09-15T03:47:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142540+0000", 2024-09-15T03:47:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98557050700000004 2024-09-15T03:47:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.176 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:40.177 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:40.058+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:40.177 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:40.176+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:40.178 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:47:40.278 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:40.277+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:40.279 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:40.279 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:40.279 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:40.279 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:40.288 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:47:40.379 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:40.378+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:40.389 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:40.389 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:40.397 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:47:40.480 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:40.479+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:40.500 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:40.500 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:40.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:40.501 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:40.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.501 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=21, reqid=client.4521.0:184, at_version=33'6, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:40.501 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:40.501 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 60.390925715999998, 2024-09-15T03:47:40.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010901256999999999, 2024-09-15T03:47:40.502 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.502 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.502 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:40.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:40.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:40.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0200000000000001e-07 2024-09-15T03:47:40.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106529+0000", 2024-09-15T03:47:40.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9539e-05 2024-09-15T03:47:40.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106542+0000", 2024-09-15T03:47:40.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2731999999999999e-05 2024-09-15T03:47:40.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106549+0000", 2024-09-15T03:47:40.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3339999999999996e-06 2024-09-15T03:47:40.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106613+0000", 2024-09-15T03:47:40.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4527000000000001e-05 2024-09-15T03:47:40.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106622+0000", 2024-09-15T03:47:40.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.755e-06 2024-09-15T03:47:40.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106841+0000", 2024-09-15T03:47:40.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021906200000000001 2024-09-15T03:47:40.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106869+0000", 2024-09-15T03:47:40.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.739e-05 2024-09-15T03:47:40.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106948+0000", 2024-09-15T03:47:40.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9422e-05 2024-09-15T03:47:40.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106965+0000", 2024-09-15T03:47:40.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7010000000000001e-05 2024-09-15T03:47:40.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.117410+0000", 2024-09-15T03:47:40.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010445284000000001 2024-09-15T03:47:40.513 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.513 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.514 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.514 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:40.514 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:40.515 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 60.389012229000002, 2024-09-15T03:47:40.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015296944, 2024-09-15T03:47:40.515 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.515 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.515 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:40.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:40.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:40.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5499999999999998e-07 2024-09-15T03:47:40.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108447+0000", 2024-09-15T03:47:40.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3314000000000001e-05 2024-09-15T03:47:40.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108460+0000", 2024-09-15T03:47:40.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3444e-05 2024-09-15T03:47:40.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108464+0000", 2024-09-15T03:47:40.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3610000000000003e-06 2024-09-15T03:47:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108494+0000", 2024-09-15T03:47:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9688000000000001e-05 2024-09-15T03:47:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108502+0000", 2024-09-15T03:47:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.678e-06 2024-09-15T03:47:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108601+0000", 2024-09-15T03:47:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9748999999999997e-05 2024-09-15T03:47:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108610+0000", 2024-09-15T03:47:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2490000000000001e-06 2024-09-15T03:47:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109977+0000", 2024-09-15T03:47:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013669450000000001 2024-09-15T03:47:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109993+0000", 2024-09-15T03:47:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6688e-05 2024-09-15T03:47:40.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123720+0000", 2024-09-15T03:47:40.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013726172999999999 2024-09-15T03:47:40.525 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.525 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.525 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 60.388962567999997, 2024-09-15T03:47:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015105210000000001, 2024-09-15T03:47:40.526 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108473+0000", 2024-09-15T03:47:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.84e-07 2024-09-15T03:47:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108486+0000", 2024-09-15T03:47:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3845999999999999e-05 2024-09-15T03:47:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108491+0000", 2024-09-15T03:47:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4719999999999997e-06 2024-09-15T03:47:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108495+0000", 2024-09-15T03:47:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.822e-06 2024-09-15T03:47:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108546+0000", 2024-09-15T03:47:40.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0853000000000003e-05 2024-09-15T03:47:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108555+0000", 2024-09-15T03:47:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3319999999999998e-06 2024-09-15T03:47:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108638+0000", 2024-09-15T03:47:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3580999999999998e-05 2024-09-15T03:47:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109983+0000", 2024-09-15T03:47:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001344392 2024-09-15T03:47:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110023+0000", 2024-09-15T03:47:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9895999999999999e-05 2024-09-15T03:47:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110040+0000", 2024-09-15T03:47:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.747e-05 2024-09-15T03:47:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123577+0000", 2024-09-15T03:47:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013537262 2024-09-15T03:47:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.532 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 60.388928688999997, 2024-09-15T03:47:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015311313999999999, 2024-09-15T03:47:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8500000000000002e-07 2024-09-15T03:47:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108517+0000", 2024-09-15T03:47:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0543000000000001e-05 2024-09-15T03:47:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108524+0000", 2024-09-15T03:47:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7170000000000002e-06 2024-09-15T03:47:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108528+0000", 2024-09-15T03:47:40.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3970000000000004e-06 2024-09-15T03:47:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108632+0000", 2024-09-15T03:47:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103687 2024-09-15T03:47:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108637+0000", 2024-09-15T03:47:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4389999999999999e-06 2024-09-15T03:47:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108713+0000", 2024-09-15T03:47:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6223000000000005e-05 2024-09-15T03:47:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109984+0000", 2024-09-15T03:47:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012700719999999999 2024-09-15T03:47:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110023+0000", 2024-09-15T03:47:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9864999999999999e-05 2024-09-15T03:47:40.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110041+0000", 2024-09-15T03:47:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8040999999999999e-05 2024-09-15T03:47:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123817+0000", 2024-09-15T03:47:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013776045000000001 2024-09-15T03:47:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 60.388435080999997, 2024-09-15T03:47:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014864502, 2024-09-15T03:47:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:47:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109016+0000", 2024-09-15T03:47:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5889999999999999e-05 2024-09-15T03:47:40.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109029+0000", 2024-09-15T03:47:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2812e-05 2024-09-15T03:47:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109035+0000", 2024-09-15T03:47:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0109999999999997e-06 2024-09-15T03:47:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109073+0000", 2024-09-15T03:47:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8260000000000003e-05 2024-09-15T03:47:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109080+0000", 2024-09-15T03:47:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0190000000000001e-06 2024-09-15T03:47:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109165+0000", 2024-09-15T03:47:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4957000000000007e-05 2024-09-15T03:47:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109984+0000", 2024-09-15T03:47:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081866399999999998 2024-09-15T03:47:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110024+0000", 2024-09-15T03:47:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9644999999999997e-05 2024-09-15T03:47:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110072+0000", 2024-09-15T03:47:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8418999999999998e-05 2024-09-15T03:47:40.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123864+0000", 2024-09-15T03:47:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013792234 2024-09-15T03:47:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=34, reqid=client.4521.0:197, at_version=34'21, trim_to=0'0, roll_forward_to=34'18))", 2024-09-15T03:47:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.373423201000001, 2024-09-15T03:47:40.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012992399, 2024-09-15T03:47:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124013+0000", 2024-09-15T03:47:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6299999999999993e-07 2024-09-15T03:47:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124044+0000", 2024-09-15T03:47:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0995999999999999e-05 2024-09-15T03:47:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124062+0000", 2024-09-15T03:47:40.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8413000000000001e-05 2024-09-15T03:47:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124069+0000", 2024-09-15T03:47:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3930000000000002e-06 2024-09-15T03:47:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124150+0000", 2024-09-15T03:47:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1097e-05 2024-09-15T03:47:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124161+0000", 2024-09-15T03:47:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0720000000000001e-05 2024-09-15T03:47:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124402+0000", 2024-09-15T03:47:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000240442 2024-09-15T03:47:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125141+0000", 2024-09-15T03:47:40.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00073915500000000002 2024-09-15T03:47:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125191+0000", 2024-09-15T03:47:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9951999999999999e-05 2024-09-15T03:47:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125251+0000", 2024-09-15T03:47:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9938000000000001e-05 2024-09-15T03:47:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.137004+0000", 2024-09-15T03:47:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011753329999999999 2024-09-15T03:47:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.550 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=35, reqid=client.4521.0:197, at_version=0'0, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:47:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.371483157, 2024-09-15T03:47:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011202456, 2024-09-15T03:47:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.551 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125953+0000", 2024-09-15T03:47:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9100000000000002e-07 2024-09-15T03:47:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125964+0000", 2024-09-15T03:47:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1399999999999999e-05 2024-09-15T03:47:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125979+0000", 2024-09-15T03:47:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4810000000000001e-05 2024-09-15T03:47:40.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126004+0000", 2024-09-15T03:47:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4913e-05 2024-09-15T03:47:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126059+0000", 2024-09-15T03:47:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5668999999999998e-05 2024-09-15T03:47:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126071+0000", 2024-09-15T03:47:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1219999999999999e-05 2024-09-15T03:47:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126151+0000", 2024-09-15T03:47:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0572e-05 2024-09-15T03:47:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126163+0000", 2024-09-15T03:47:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2101e-05 2024-09-15T03:47:40.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.127945+0000", 2024-09-15T03:47:40.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001781803 2024-09-15T03:47:40.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.127982+0000", 2024-09-15T03:47:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7129999999999999e-05 2024-09-15T03:47:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.137154+0000", 2024-09-15T03:47:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0091719469999999997 2024-09-15T03:47:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.557 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:243 3.3s0 3:c7c3ef16:::smithi00132172-25:head [create,setxattr iZnWfebab`ptjURR\\f`psW]tpS`qpUaeWkY[Oqep (1330) in=40b,setxattr f[mZ_qQmcZejQ`fbdeic_tkPUaTYcnm[WprjVmc] (12488) in=40b,setxattr tkZgQ]S_pdlhZnqrVaZRa`oQQr_USmUfhado^dXT (1121) in=40b,setxattr srWjTqXhcth`hfZmQY_fg]UqNjWofa^dfgOV`sYP (455) in=40b,setxattr ^_mdiRroj_TtQ^gcNj`XtkW[ooqY`PWXqYmcOk_n (138) in=40b,setxattr jQlajdtcalhkcfYaakZfNb[^_TjkfcS`Vrk]q_d[ (806) in=40b,setxattr ZfYgkVcrsXcnpchgfcbldNrUTasnQNmfZW]Nbh\\o (561) in=40b,setxattr _hkYOqjPNhmhUlSNktfeQpsqSWjO\\fbf[fpTlZiO (1227) in=40b,setxattr ]mdV^Xdc\\sViXdkXNU[js_Qhh[b[sbfnZVXkdids (11788) in=40b,setxattr tb^mbOrNhfrhtce^g (455) in=17b,setxattr koTkVgrfU\\m`StnZ`Rirjhbhg`PhVqm^a^Xgaaqn (35) in=40b,setxattr aUtZPUPXPnQmT]\\\\n\\lZpnXakdNkTldVQoQhPUf] (559) in=40b,setxattr mrhR]^U`Tteo[Topmql`SNma\\pqfk`if]OppnYt` (770) in=40b,setxattr Oetk]RrnWeXbVctVtVrhXf_RfTdelqX^lWp[sTkp (1226) in=40b,setxattr WtedXoOd`VqnbksOqQkatk`gdTnW^lXbUtrlYgkl (3576) in=40b,setxattr dddOWN\\j\\XXg^fr[rZcskeVX]jOQ^ZmoWepk[]\\a (17824) in=40b,setxattr TeRO^djPgNZiemQ_aRXgP[]jgUZa\\^\\iUgesaqaN (208) in=40b,setxattr teqm]mUobZbWjcdQ_jTa[f[k[Sd\\lh]n\\osigqkN (7436) in=40b,setxattr dphaol[XkeitbkomWYaQctWfataUhbSiNntkWo`P (1893) in=40b,setxattr oQsnhO]ZklSVYQ[WUpaelY[p[Qie^UepOt`Z[q (455) in=38b] snapc 7=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:47:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.328519268999997, 2024-09-15T03:47:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012365956000000001, 2024-09-15T03:47:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 243 2024-09-15T03:47:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168917+0000", 2024-09-15T03:47:40.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.725e-06 2024-09-15T03:47:40.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169030+0000", 2024-09-15T03:47:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113035 2024-09-15T03:47:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169047+0000", 2024-09-15T03:47:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6701000000000002e-05 2024-09-15T03:47:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169054+0000", 2024-09-15T03:47:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1369999999999996e-06 2024-09-15T03:47:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169140+0000", 2024-09-15T03:47:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6069999999999994e-05 2024-09-15T03:47:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169270+0000", 2024-09-15T03:47:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130071 2024-09-15T03:47:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.170661+0000", 2024-09-15T03:47:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001390666 2024-09-15T03:47:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171751+0000", 2024-09-15T03:47:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010899079999999999 2024-09-15T03:47:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171812+0000", 2024-09-15T03:47:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0797000000000001e-05 2024-09-15T03:47:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172279+0000", 2024-09-15T03:47:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046704500000000003 2024-09-15T03:47:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172299+0000", 2024-09-15T03:47:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0387999999999999e-05 2024-09-15T03:47:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.176110+0000", 2024-09-15T03:47:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038112340000000001 2024-09-15T03:47:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.181282+0000", 2024-09-15T03:47:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051711789999999997 2024-09-15T03:47:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:258 3.3s0 3:c9b663a2:::smithi00132172-7:head [delete] snapc 7=[6,5,3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e40)", 2024-09-15T03:47:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:47.146116+0000", 2024-09-15T03:47:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.351318450999997, 2024-09-15T03:47:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096428380000000008, 2024-09-15T03:47:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 258 2024-09-15T03:47:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146116+0000", 2024-09-15T03:47:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146116+0000", 2024-09-15T03:47:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146119+0000", 2024-09-15T03:47:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.227e-06 2024-09-15T03:47:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146127+0000", 2024-09-15T03:47:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1529999999999997e-06 2024-09-15T03:47:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146146+0000", 2024-09-15T03:47:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9318999999999999e-05 2024-09-15T03:47:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146154+0000", 2024-09-15T03:47:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7200000000000006e-06 2024-09-15T03:47:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146233+0000", 2024-09-15T03:47:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8725000000000003e-05 2024-09-15T03:47:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146244+0000", 2024-09-15T03:47:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1725e-05 2024-09-15T03:47:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.149207+0000", 2024-09-15T03:47:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029627999999999998 2024-09-15T03:47:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.149326+0000", 2024-09-15T03:47:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011869199999999999 2024-09-15T03:47:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.150266+0000", 2024-09-15T03:47:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00094048500000000002 2024-09-15T03:47:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.150626+0000", 2024-09-15T03:47:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035946699999999998 2024-09-15T03:47:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.150751+0000", 2024-09-15T03:47:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012544800000000001 2024-09-15T03:47:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.151656+0000", 2024-09-15T03:47:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00090447499999999996 2024-09-15T03:47:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.153489+0000", 2024-09-15T03:47:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001833115 2024-09-15T03:47:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.155171+0000", 2024-09-15T03:47:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001682455 2024-09-15T03:47:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.155759+0000", 2024-09-15T03:47:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00058803200000000005 2024-09-15T03:47:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/28 ECSubWrite(tid=15, reqid=client.4521.0:337, at_version=49'10, trim_to=0'0, roll_forward_to=43'8))", 2024-09-15T03:47:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.639529359999997, 2024-09-15T03:47:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010500893000000001, 2024-09-15T03:47:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857906+0000", 2024-09-15T03:47:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.63e-07 2024-09-15T03:47:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857928+0000", 2024-09-15T03:47:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2155999999999999e-05 2024-09-15T03:47:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857937+0000", 2024-09-15T03:47:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2359999999999993e-06 2024-09-15T03:47:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857941+0000", 2024-09-15T03:47:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0960000000000003e-06 2024-09-15T03:47:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857999+0000", 2024-09-15T03:47:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7407000000000001e-05 2024-09-15T03:47:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858011+0000", 2024-09-15T03:47:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1759999999999999e-05 2024-09-15T03:47:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858240+0000", 2024-09-15T03:47:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022949299999999999 2024-09-15T03:47:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858251+0000", 2024-09-15T03:47:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1168999999999999e-05 2024-09-15T03:47:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858332+0000", 2024-09-15T03:47:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1204000000000004e-05 2024-09-15T03:47:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858360+0000", 2024-09-15T03:47:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7767999999999999e-05 2024-09-15T03:47:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.868406+0000", 2024-09-15T03:47:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010046141 2024-09-15T03:47:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:349 3.3s0 3:c7c3ef16:::smithi00132172-25:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 42.658953334000003, 2024-09-15T03:47:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005145406, 2024-09-15T03:47:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 349 2024-09-15T03:47:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838485+0000", 2024-09-15T03:47:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0620000000000001e-06 2024-09-15T03:47:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838494+0000", 2024-09-15T03:47:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3179999999999996e-06 2024-09-15T03:47:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3689999999999998e-05 2024-09-15T03:47:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838812+0000", 2024-09-15T03:47:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029437300000000001 2024-09-15T03:47:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838905+0000", 2024-09-15T03:47:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3196999999999996e-05 2024-09-15T03:47:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838914+0000", 2024-09-15T03:47:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0890000000000006e-06 2024-09-15T03:47:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840773+0000", 2024-09-15T03:47:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018584820000000001 2024-09-15T03:47:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840889+0000", 2024-09-15T03:47:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115986 2024-09-15T03:47:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842016+0000", 2024-09-15T03:47:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001126878 2024-09-15T03:47:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842320+0000", 2024-09-15T03:47:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030409799999999999 2024-09-15T03:47:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842360+0000", 2024-09-15T03:47:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0750000000000001e-05 2024-09-15T03:47:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842448+0000", 2024-09-15T03:47:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7842000000000004e-05 2024-09-15T03:47:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844580+0000", 2024-09-15T03:47:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021313719999999999 2024-09-15T03:47:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843574+0000", 2024-09-15T03:47:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99899468199999997 2024-09-15T03:47:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843627+0000", 2024-09-15T03:47:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2587000000000003e-05 2024-09-15T03:47:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:350 3.3s0 3:c7c3ef16:::smithi00132172-25:head [rollback 14] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 42.658601673, 2024-09-15T03:47:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0050383839999999, 2024-09-15T03:47:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 350 2024-09-15T03:47:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838834+0000", 2024-09-15T03:47:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0200000000000001e-07 2024-09-15T03:47:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838839+0000", 2024-09-15T03:47:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9790000000000001e-06 2024-09-15T03:47:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838852+0000", 2024-09-15T03:47:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3138e-05 2024-09-15T03:47:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838858+0000", 2024-09-15T03:47:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8320000000000002e-06 2024-09-15T03:47:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839003+0000", 2024-09-15T03:47:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014513500000000001 2024-09-15T03:47:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839011+0000", 2024-09-15T03:47:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7719999999999994e-06 2024-09-15T03:47:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842395+0000", 2024-09-15T03:47:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033840110000000001 2024-09-15T03:47:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842580+0000", 2024-09-15T03:47:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000185067 2024-09-15T03:47:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843394+0000", 2024-09-15T03:47:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081377299999999995 2024-09-15T03:47:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843510+0000", 2024-09-15T03:47:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0001165 2024-09-15T03:47:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843524+0000", 2024-09-15T03:47:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4022999999999999e-05 2024-09-15T03:47:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843579+0000", 2024-09-15T03:47:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4654999999999998e-05 2024-09-15T03:47:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844583+0000", 2024-09-15T03:47:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010042429999999999 2024-09-15T03:47:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843772+0000", 2024-09-15T03:47:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99918911200000005 2024-09-15T03:47:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843872+0000", 2024-09-15T03:47:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9341999999999999e-05 2024-09-15T03:47:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:351 3.3s0 3:c7c3ef16:::smithi00132172-25:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 42.658544040000002, 2024-09-15T03:47:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0051167679999999, 2024-09-15T03:47:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 351 2024-09-15T03:47:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0800000000000004e-07 2024-09-15T03:47:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838895+0000", 2024-09-15T03:47:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.241e-06 2024-09-15T03:47:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838904+0000", 2024-09-15T03:47:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9549999999999998e-06 2024-09-15T03:47:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838909+0000", 2024-09-15T03:47:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7869999999999998e-06 2024-09-15T03:47:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839045+0000", 2024-09-15T03:47:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013556399999999999 2024-09-15T03:47:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839052+0000", 2024-09-15T03:47:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8199999999999999e-06 2024-09-15T03:47:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843587+0000", 2024-09-15T03:47:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045349739999999998 2024-09-15T03:47:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843708+0000", 2024-09-15T03:47:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120776 2024-09-15T03:47:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844246+0000", 2024-09-15T03:47:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053838200000000001 2024-09-15T03:47:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844335+0000", 2024-09-15T03:47:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9240999999999995e-05 2024-09-15T03:47:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844347+0000", 2024-09-15T03:47:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2187e-05 2024-09-15T03:47:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844460+0000", 2024-09-15T03:47:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112571 2024-09-15T03:47:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844584+0000", 2024-09-15T03:47:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012403800000000001 2024-09-15T03:47:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843972+0000", 2024-09-15T03:47:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99938787299999998 2024-09-15T03:47:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844008+0000", 2024-09-15T03:47:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5750999999999997e-05 2024-09-15T03:47:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 42.651410089000002, 2024-09-15T03:47:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013010173, 2024-09-15T03:47:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846026+0000", 2024-09-15T03:47:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1230000000000001e-06 2024-09-15T03:47:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846047+0000", 2024-09-15T03:47:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0774000000000001e-05 2024-09-15T03:47:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846064+0000", 2024-09-15T03:47:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7560000000000001e-05 2024-09-15T03:47:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846072+0000", 2024-09-15T03:47:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5020000000000003e-06 2024-09-15T03:47:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846187+0000", 2024-09-15T03:47:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115717 2024-09-15T03:47:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846201+0000", 2024-09-15T03:47:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3329999999999999e-05 2024-09-15T03:47:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846303+0000", 2024-09-15T03:47:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101807 2024-09-15T03:47:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846312+0000", 2024-09-15T03:47:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5529999999999998e-06 2024-09-15T03:47:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846382+0000", 2024-09-15T03:47:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0146000000000002e-05 2024-09-15T03:47:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846520+0000", 2024-09-15T03:47:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013814099999999999 2024-09-15T03:47:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.859035+0000", 2024-09-15T03:47:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012514519999999999 2024-09-15T03:47:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 42.651313342000002, 2024-09-15T03:47:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013044694000000001, 2024-09-15T03:47:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846122+0000", 2024-09-15T03:47:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0399999999999996e-07 2024-09-15T03:47:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846174+0000", 2024-09-15T03:47:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1672999999999998e-05 2024-09-15T03:47:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846184+0000", 2024-09-15T03:47:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0514e-05 2024-09-15T03:47:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846190+0000", 2024-09-15T03:47:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2820000000000002e-06 2024-09-15T03:47:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846341+0000", 2024-09-15T03:47:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150803 2024-09-15T03:47:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846351+0000", 2024-09-15T03:47:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3899999999999999e-06 2024-09-15T03:47:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846450+0000", 2024-09-15T03:47:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9839000000000004e-05 2024-09-15T03:47:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846482+0000", 2024-09-15T03:47:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1089e-05 2024-09-15T03:47:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846615+0000", 2024-09-15T03:47:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013305899999999999 2024-09-15T03:47:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846634+0000", 2024-09-15T03:47:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8899999999999999e-05 2024-09-15T03:47:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.859166+0000", 2024-09-15T03:47:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012532641000000001 2024-09-15T03:47:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 60/28 ECSubWrite(tid=73, reqid=client.4521.0:424, at_version=60'41, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.597004495, 2024-09-15T03:47:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011124821, 2024-09-15T03:47:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900431+0000", 2024-09-15T03:47:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2900000000000003e-07 2024-09-15T03:47:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900451+0000", 2024-09-15T03:47:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0066999999999999e-05 2024-09-15T03:47:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900466+0000", 2024-09-15T03:47:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5062000000000001e-05 2024-09-15T03:47:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900474+0000", 2024-09-15T03:47:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6310000000000003e-06 2024-09-15T03:47:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900524+0000", 2024-09-15T03:47:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0037000000000002e-05 2024-09-15T03:47:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900535+0000", 2024-09-15T03:47:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1209000000000001e-05 2024-09-15T03:47:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900659+0000", 2024-09-15T03:47:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012441000000000001 2024-09-15T03:47:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900971+0000", 2024-09-15T03:47:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031183799999999999 2024-09-15T03:47:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901144+0000", 2024-09-15T03:47:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172261 2024-09-15T03:47:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901734+0000", 2024-09-15T03:47:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059069700000000005 2024-09-15T03:47:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.911555+0000", 2024-09-15T03:47:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0098208800000000006 2024-09-15T03:47:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 60/28 ECSubWrite(tid=74, reqid=client.4521.0:425, at_version=60'42, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:06.901240+0000", 2024-09-15T03:47:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.596194916000002, 2024-09-15T03:47:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010429678, 2024-09-15T03:47:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901240+0000", 2024-09-15T03:47:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901240+0000", 2024-09-15T03:47:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901241+0000", 2024-09-15T03:47:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7999999999999993e-07 2024-09-15T03:47:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901273+0000", 2024-09-15T03:47:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2087e-05 2024-09-15T03:47:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901302+0000", 2024-09-15T03:47:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8705e-05 2024-09-15T03:47:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901311+0000", 2024-09-15T03:47:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9190000000000005e-06 2024-09-15T03:47:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901797+0000", 2024-09-15T03:47:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048606499999999999 2024-09-15T03:47:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901808+0000", 2024-09-15T03:47:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1075e-05 2024-09-15T03:47:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901934+0000", 2024-09-15T03:47:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012674299999999999 2024-09-15T03:47:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901964+0000", 2024-09-15T03:47:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9833999999999999e-05 2024-09-15T03:47:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.902006+0000", 2024-09-15T03:47:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1823999999999998e-05 2024-09-15T03:47:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.902020+0000", 2024-09-15T03:47:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.363e-05 2024-09-15T03:47:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.911670+0000", 2024-09-15T03:47:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096498160000000003 2024-09-15T03:47:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPush(2.2 88/87 [PushOp(2:5cea7035:::rbd_info:head, version: 24'2, data_included: [0~19], data_size: 19, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:5cea7035:::rbd_info:head@24'2, size: 19, copy_subset: [0~19], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:19, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-15T03:47:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.846903+0000", 2024-09-15T03:47:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.6505313690000003, 2024-09-15T03:47:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013354107, 2024-09-15T03:47:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846903+0000", 2024-09-15T03:47:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846903+0000", 2024-09-15T03:47:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846904+0000", 2024-09-15T03:47:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4099999999999998e-07 2024-09-15T03:47:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846920+0000", 2024-09-15T03:47:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6061000000000001e-05 2024-09-15T03:47:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846945+0000", 2024-09-15T03:47:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4519e-05 2024-09-15T03:47:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846958+0000", 2024-09-15T03:47:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3546999999999999e-05 2024-09-15T03:47:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.847077+0000", 2024-09-15T03:47:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011864 2024-09-15T03:47:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for flush", 2024-09-15T03:47:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.847094+0000", 2024-09-15T03:47:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7017e-05 2024-09-15T03:47:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.859991+0000", 2024-09-15T03:47:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012897136 2024-09-15T03:47:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.860001+0000", 2024-09-15T03:47:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5089999999999999e-06 2024-09-15T03:47:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.860258+0000", 2024-09-15T03:47:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000256937 2024-09-15T03:47:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 88/87 ECSubWrite(tid=200, reqid=client.4521.0:737, at_version=88'109, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.883279+0000", 2024-09-15T03:47:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.6141554459999998, 2024-09-15T03:47:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010263239, 2024-09-15T03:47:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883279+0000", 2024-09-15T03:47:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883279+0000", 2024-09-15T03:47:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883280+0000", 2024-09-15T03:47:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5799999999999999e-07 2024-09-15T03:47:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883328+0000", 2024-09-15T03:47:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8198999999999997e-05 2024-09-15T03:47:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883341+0000", 2024-09-15T03:47:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2557e-05 2024-09-15T03:47:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883347+0000", 2024-09-15T03:47:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5359999999999998e-06 2024-09-15T03:47:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883406+0000", 2024-09-15T03:47:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8971999999999997e-05 2024-09-15T03:47:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883417+0000", 2024-09-15T03:47:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0346000000000001e-05 2024-09-15T03:47:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883682+0000", 2024-09-15T03:47:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026550500000000003 2024-09-15T03:47:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883705+0000", 2024-09-15T03:47:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2779e-05 2024-09-15T03:47:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883736+0000", 2024-09-15T03:47:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0626999999999997e-05 2024-09-15T03:47:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883752+0000", 2024-09-15T03:47:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6180000000000001e-05 2024-09-15T03:47:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.893543+0000", 2024-09-15T03:47:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0097908800000000001 2024-09-15T03:47:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 88/87 ECSubWrite(tid=210, reqid=client.4521.0:740, at_version=88'113, trim_to=0'0, roll_forward_to=88'111))", 2024-09-15T03:47:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.915188+0000", 2024-09-15T03:47:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.5822472440000004, 2024-09-15T03:47:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012691618999999999, 2024-09-15T03:47:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915188+0000", 2024-09-15T03:47:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915188+0000", 2024-09-15T03:47:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915188+0000", 2024-09-15T03:47:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3399999999999999e-07 2024-09-15T03:47:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915216+0000", 2024-09-15T03:47:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8067000000000001e-05 2024-09-15T03:47:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915233+0000", 2024-09-15T03:47:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.683e-05 2024-09-15T03:47:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915239+0000", 2024-09-15T03:47:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5720000000000002e-06 2024-09-15T03:47:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915313+0000", 2024-09-15T03:47:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4066000000000005e-05 2024-09-15T03:47:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915325+0000", 2024-09-15T03:47:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.252e-05 2024-09-15T03:47:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915603+0000", 2024-09-15T03:47:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027800699999999998 2024-09-15T03:47:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915645+0000", 2024-09-15T03:47:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1777000000000003e-05 2024-09-15T03:47:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915687+0000", 2024-09-15T03:47:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2181000000000002e-05 2024-09-15T03:47:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.917623+0000", 2024-09-15T03:47:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019357210000000001 2024-09-15T03:47:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.927879+0000", 2024-09-15T03:47:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010256243999999999 2024-09-15T03:47:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:40.665 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:40.667 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:47:40.667 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:40.666+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:40.768 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:40.767+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:40.809 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:40.810 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:40.810 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:40.810 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:40.818 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:47:40.845 INFO:tasks.rados.rados.0.smithi001.stdout:315: done (0 left) 2024-09-15T03:47:40.845 INFO:tasks.rados.rados.0.smithi001.stdout:316: snap_create 2024-09-15T03:47:40.859 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T03:47:40.869 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:40.868+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:40.919 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:40.919 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:40.919 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:40.919 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:40.919 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:40.927 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:47:40.969 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:40.969+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:41.029 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:197 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [delete] snapc 3=[3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e34)", 2024-09-15T03:47:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.915407467000001, 2024-09-15T03:47:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021934704999999999, 2024-09-15T03:47:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:41.030 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197 2024-09-15T03:47:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:41.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112266+0000", 2024-09-15T03:47:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5000000000000002e-07 2024-09-15T03:47:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112269+0000", 2024-09-15T03:47:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.066e-06 2024-09-15T03:47:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112276+0000", 2024-09-15T03:47:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-09-15T03:47:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112281+0000", 2024-09-15T03:47:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3970000000000004e-06 2024-09-15T03:47:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123281+0000", 2024-09-15T03:47:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011000575 2024-09-15T03:47:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123395+0000", 2024-09-15T03:47:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113398 2024-09-15T03:47:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123963+0000", 2024-09-15T03:47:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000568101 2024-09-15T03:47:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124220+0000", 2024-09-15T03:47:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025704900000000003 2024-09-15T03:47:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124229+0000", 2024-09-15T03:47:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4860000000000003e-06 2024-09-15T03:47:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124277+0000", 2024-09-15T03:47:41.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7574000000000002e-05 2024-09-15T03:47:41.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:41.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124312+0000", 2024-09-15T03:47:41.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5256000000000003e-05 2024-09-15T03:47:41.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:41.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125716+0000", 2024-09-15T03:47:41.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014043720000000001 2024-09-15T03:47:41.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:41.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.134200+0000", 2024-09-15T03:47:41.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084835120000000003 2024-09-15T03:47:41.037 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.037 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:41.037 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.037 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:220 3.bs0 3:d99ddedc:::smithi00132172-44:head [copy-from ver 22] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:41.037 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:41.037 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 55.900027797, 2024-09-15T03:47:41.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024273429999999999, 2024-09-15T03:47:41.037 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:41.037 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:41.038 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:41.038 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:41.038 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.038 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 220 2024-09-15T03:47:41.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.038 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:41.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:41.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:41.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127647+0000", 2024-09-15T03:47:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1799999999999999e-06 2024-09-15T03:47:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127656+0000", 2024-09-15T03:47:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3759999999999997e-06 2024-09-15T03:47:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127674+0000", 2024-09-15T03:47:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7374e-05 2024-09-15T03:47:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127680+0000", 2024-09-15T03:47:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6949999999999997e-06 2024-09-15T03:47:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127736+0000", 2024-09-15T03:47:41.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6011999999999998e-05 2024-09-15T03:47:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127744+0000", 2024-09-15T03:47:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2870000000000004e-06 2024-09-15T03:47:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130338+0000", 2024-09-15T03:47:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025940249999999998 2024-09-15T03:47:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:41.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130423+0000", 2024-09-15T03:47:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5303000000000006e-05 2024-09-15T03:47:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.147840+0000", 2024-09-15T03:47:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017416580000000001 2024-09-15T03:47:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148318+0000", 2024-09-15T03:47:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047822899999999999 2024-09-15T03:47:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148329+0000", 2024-09-15T03:47:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0611000000000001e-05 2024-09-15T03:47:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148488+0000", 2024-09-15T03:47:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015938399999999999 2024-09-15T03:47:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148870+0000", 2024-09-15T03:47:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000382405 2024-09-15T03:47:41.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.151092+0000", 2024-09-15T03:47:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022212529999999999 2024-09-15T03:47:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.151918+0000", 2024-09-15T03:47:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082671600000000002 2024-09-15T03:47:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:221 3.bs0 3:d99ddedc:::smithi00132172-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 55.899983642000002, 2024-09-15T03:47:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021194474000000001, 2024-09-15T03:47:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 221 2024-09-15T03:47:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127690+0000", 2024-09-15T03:47:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:47:41.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127692+0000", 2024-09-15T03:47:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6589999999999999e-06 2024-09-15T03:47:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127700+0000", 2024-09-15T03:47:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4769999999999997e-06 2024-09-15T03:47:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127704+0000", 2024-09-15T03:47:41.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3819999999999997e-06 2024-09-15T03:47:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127805+0000", 2024-09-15T03:47:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100639 2024-09-15T03:47:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127809+0000", 2024-09-15T03:47:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8870000000000002e-06 2024-09-15T03:47:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130527+0000", 2024-09-15T03:47:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002718579 2024-09-15T03:47:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130558+0000", 2024-09-15T03:47:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0827999999999997e-05 2024-09-15T03:47:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148546+0000", 2024-09-15T03:47:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017988356 2024-09-15T03:47:41.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148683+0000", 2024-09-15T03:47:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013650899999999999 2024-09-15T03:47:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148884+0000", 2024-09-15T03:47:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020061499999999999 2024-09-15T03:47:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:357 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.178632108000002, 2024-09-15T03:47:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.02017917, 2024-09-15T03:47:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 357 2024-09-15T03:47:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.051 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849043+0000", 2024-09-15T03:47:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.295e-06 2024-09-15T03:47:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849051+0000", 2024-09-15T03:47:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4079999999999998e-06 2024-09-15T03:47:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849069+0000", 2024-09-15T03:47:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7462999999999999e-05 2024-09-15T03:47:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849077+0000", 2024-09-15T03:47:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8869999999999992e-06 2024-09-15T03:47:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849194+0000", 2024-09-15T03:47:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011723899999999999 2024-09-15T03:47:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849296+0000", 2024-09-15T03:47:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102011 2024-09-15T03:47:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861106+0000", 2024-09-15T03:47:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0118097610000001 2024-09-15T03:47:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861222+0000", 2024-09-15T03:47:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116561 2024-09-15T03:47:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.869220+0000", 2024-09-15T03:47:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007997545 2024-09-15T03:47:41.055 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:358 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.178577162000003, 2024-09-15T03:47:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0228638050000001, 2024-09-15T03:47:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:41.056 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 358 2024-09-15T03:47:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4099999999999997e-07 2024-09-15T03:47:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849101+0000", 2024-09-15T03:47:41.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8180000000000003e-06 2024-09-15T03:47:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849110+0000", 2024-09-15T03:47:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1409999999999994e-06 2024-09-15T03:47:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849116+0000", 2024-09-15T03:47:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2390000000000002e-06 2024-09-15T03:47:41.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849326+0000", 2024-09-15T03:47:41.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000210029 2024-09-15T03:47:41.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:41.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849421+0000", 2024-09-15T03:47:41.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5775999999999999e-05 2024-09-15T03:47:41.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861647+0000", 2024-09-15T03:47:41.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0122257139999999 2024-09-15T03:47:41.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:41.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861777+0000", 2024-09-15T03:47:41.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130087 2024-09-15T03:47:41.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:41.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.871959+0000", 2024-09-15T03:47:41.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010182160000000001 2024-09-15T03:47:41.061 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.061 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:41.062 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.062 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:359 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:41.062 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:41.062 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.178552009999997, 2024-09-15T03:47:41.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0240468149999999, 2024-09-15T03:47:41.062 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:41.062 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:41.062 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:41.062 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:41.062 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.064 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 359 2024-09-15T03:47:41.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.064 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:41.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:41.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:41.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:41.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:41.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:41.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:41.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:47:41.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:41.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849124+0000", 2024-09-15T03:47:41.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2679999999999999e-06 2024-09-15T03:47:41.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:41.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849132+0000", 2024-09-15T03:47:41.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4030000000000001e-06 2024-09-15T03:47:41.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849136+0000", 2024-09-15T03:47:41.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2289999999999997e-06 2024-09-15T03:47:41.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849446+0000", 2024-09-15T03:47:41.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031043299999999999 2024-09-15T03:47:41.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:41.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849553+0000", 2024-09-15T03:47:41.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106415 2024-09-15T03:47:41.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.863404+0000", 2024-09-15T03:47:41.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0138511029999999 2024-09-15T03:47:41.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:41.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.863475+0000", 2024-09-15T03:47:41.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0814000000000003e-05 2024-09-15T03:47:41.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:41.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.873168+0000", 2024-09-15T03:47:41.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00969274 2024-09-15T03:47:41.068 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.068 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:41.068 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:474 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~786432] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.152627334999998, 2024-09-15T03:47:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98604943700000003, 2024-09-15T03:47:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 474 2024-09-15T03:47:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:41.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875048+0000", 2024-09-15T03:47:41.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1600000000000001e-06 2024-09-15T03:47:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875844+0000", 2024-09-15T03:47:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079686299999999998 2024-09-15T03:47:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875857+0000", 2024-09-15T03:47:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2889e-05 2024-09-15T03:47:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875864+0000", 2024-09-15T03:47:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6139999999999997e-06 2024-09-15T03:47:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875916+0000", 2024-09-15T03:47:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1992e-05 2024-09-15T03:47:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875918+0000", 2024-09-15T03:47:41.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3300000000000001e-06 2024-09-15T03:47:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.845918+0000", 2024-09-15T03:47:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96999959199999997 2024-09-15T03:47:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.846229+0000", 2024-09-15T03:47:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000311601 2024-09-15T03:47:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:41.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848445+0000", 2024-09-15T03:47:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022160499999999998 2024-09-15T03:47:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848880+0000", 2024-09-15T03:47:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043432600000000002 2024-09-15T03:47:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848952+0000", 2024-09-15T03:47:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2509000000000007e-05 2024-09-15T03:47:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.849311+0000", 2024-09-15T03:47:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000358611 2024-09-15T03:47:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851131+0000", 2024-09-15T03:47:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001820115 2024-09-15T03:47:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861048+0000", 2024-09-15T03:47:41.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099164830000000002 2024-09-15T03:47:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861095+0000", 2024-09-15T03:47:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7302000000000002e-05 2024-09-15T03:47:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:475 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~491520] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:41.076 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.149860864000001, 2024-09-15T03:47:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98407319000000004, 2024-09-15T03:47:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 475 2024-09-15T03:47:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877813+0000", 2024-09-15T03:47:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8800000000000002e-07 2024-09-15T03:47:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878260+0000", 2024-09-15T03:47:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044732099999999998 2024-09-15T03:47:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878270+0000", 2024-09-15T03:47:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9799999999999993e-06 2024-09-15T03:47:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878276+0000", 2024-09-15T03:47:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2269999999999998e-06 2024-09-15T03:47:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878313+0000", 2024-09-15T03:47:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6570999999999999e-05 2024-09-15T03:47:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878315+0000", 2024-09-15T03:47:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8729999999999999e-06 2024-09-15T03:47:41.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851316+0000", 2024-09-15T03:47:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97300154800000005 2024-09-15T03:47:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851440+0000", 2024-09-15T03:47:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124287 2024-09-15T03:47:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852732+0000", 2024-09-15T03:47:41.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001291174 2024-09-15T03:47:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852991+0000", 2024-09-15T03:47:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025905299999999997 2024-09-15T03:47:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853003+0000", 2024-09-15T03:47:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2908000000000001e-05 2024-09-15T03:47:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853104+0000", 2024-09-15T03:47:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010011800000000001 2024-09-15T03:47:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854342+0000", 2024-09-15T03:47:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012380189999999999 2024-09-15T03:47:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861206+0000", 2024-09-15T03:47:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068648499999999996 2024-09-15T03:47:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861885+0000", 2024-09-15T03:47:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00067857300000000002 2024-09-15T03:47:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:476 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~278528] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.148233851000001, 2024-09-15T03:47:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98268067299999995, 2024-09-15T03:47:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 476 2024-09-15T03:47:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.13e-07 2024-09-15T03:47:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879706+0000", 2024-09-15T03:47:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026656100000000001 2024-09-15T03:47:41.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879715+0000", 2024-09-15T03:47:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9330000000000007e-06 2024-09-15T03:47:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879720+0000", 2024-09-15T03:47:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0880000000000002e-06 2024-09-15T03:47:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879743+0000", 2024-09-15T03:47:41.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2654e-05 2024-09-15T03:47:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879743+0000", 2024-09-15T03:47:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.99e-07 2024-09-15T03:47:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853170+0000", 2024-09-15T03:47:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97342687900000002 2024-09-15T03:47:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853297+0000", 2024-09-15T03:47:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127157 2024-09-15T03:47:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854150+0000", 2024-09-15T03:47:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085288900000000001 2024-09-15T03:47:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854292+0000", 2024-09-15T03:47:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014199300000000001 2024-09-15T03:47:41.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854559+0000", 2024-09-15T03:47:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026668999999999998 2024-09-15T03:47:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855998+0000", 2024-09-15T03:47:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001438714 2024-09-15T03:47:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:41.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856547+0000", 2024-09-15T03:47:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054880399999999996 2024-09-15T03:47:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861858+0000", 2024-09-15T03:47:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053110320000000003 2024-09-15T03:47:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.862120+0000", 2024-09-15T03:47:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026196699999999999 2024-09-15T03:47:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:41.091 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:477 3.bs0 3:d52faf55:::smithi00132172-40:head [setxattr _header (57) in=7b] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.147945914000001, 2024-09-15T03:47:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98562198000000001, 2024-09-15T03:47:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.092 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 477 2024-09-15T03:47:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2800000000000002e-07 2024-09-15T03:47:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879730+0000", 2024-09-15T03:47:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0589999999999998e-06 2024-09-15T03:47:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879737+0000", 2024-09-15T03:47:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0419999999999997e-06 2024-09-15T03:47:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879741+0000", 2024-09-15T03:47:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9960000000000004e-06 2024-09-15T03:47:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879756+0000", 2024-09-15T03:47:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4681999999999999e-05 2024-09-15T03:47:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879756+0000", 2024-09-15T03:47:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.22e-07 2024-09-15T03:47:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854590+0000", 2024-09-15T03:47:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97483319000000002 2024-09-15T03:47:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854718+0000", 2024-09-15T03:47:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012842100000000001 2024-09-15T03:47:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855416+0000", 2024-09-15T03:47:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069759200000000003 2024-09-15T03:47:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855551+0000", 2024-09-15T03:47:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013517600000000001 2024-09-15T03:47:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:41.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856002+0000", 2024-09-15T03:47:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045169399999999999 2024-09-15T03:47:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856065+0000", 2024-09-15T03:47:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3028999999999996e-05 2024-09-15T03:47:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856554+0000", 2024-09-15T03:47:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000488486 2024-09-15T03:47:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865278+0000", 2024-09-15T03:47:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087241929999999999 2024-09-15T03:47:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865349+0000", 2024-09-15T03:47:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0669999999999999e-05 2024-09-15T03:47:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.099 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:478 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.147927118999998, 2024-09-15T03:47:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99471090500000003, 2024-09-15T03:47:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 478 2024-09-15T03:47:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2599999999999998e-07 2024-09-15T03:47:41.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879748+0000", 2024-09-15T03:47:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9609999999999999e-06 2024-09-15T03:47:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879755+0000", 2024-09-15T03:47:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1879999999999996e-06 2024-09-15T03:47:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879760+0000", 2024-09-15T03:47:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2560000000000002e-06 2024-09-15T03:47:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879797+0000", 2024-09-15T03:47:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6452999999999998e-05 2024-09-15T03:47:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879799+0000", 2024-09-15T03:47:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6619999999999999e-06 2024-09-15T03:47:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855637+0000", 2024-09-15T03:47:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97583850999999999 2024-09-15T03:47:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855763+0000", 2024-09-15T03:47:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012612699999999999 2024-09-15T03:47:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865707+0000", 2024-09-15T03:47:41.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099439660000000003 2024-09-15T03:47:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865803+0000", 2024-09-15T03:47:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5946999999999995e-05 2024-09-15T03:47:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.874456+0000", 2024-09-15T03:47:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0086534090000000008 2024-09-15T03:47:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:41.105 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:486 3.bs0 3:df8b18a3:::smithi00132172-17:head [delete] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e64)", 2024-09-15T03:47:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.145114639999999, 2024-09-15T03:47:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98362617100000005, 2024-09-15T03:47:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 486 2024-09-15T03:47:41.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:41.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882559+0000", 2024-09-15T03:47:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0359999999999999e-06 2024-09-15T03:47:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882564+0000", 2024-09-15T03:47:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6530000000000002e-06 2024-09-15T03:47:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882574+0000", 2024-09-15T03:47:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0464e-05 2024-09-15T03:47:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882578+0000", 2024-09-15T03:47:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2459999999999997e-06 2024-09-15T03:47:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882624+0000", 2024-09-15T03:47:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5698000000000002e-05 2024-09-15T03:47:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882627+0000", 2024-09-15T03:47:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3350000000000001e-06 2024-09-15T03:47:41.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856574+0000", 2024-09-15T03:47:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97394739399999997 2024-09-15T03:47:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856853+0000", 2024-09-15T03:47:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027918399999999998 2024-09-15T03:47:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.866184+0000", 2024-09-15T03:47:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093311609999999993 2024-09-15T03:47:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:495 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [copy-from ver 51] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-15T03:47:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 29.139213591000001, 2024-09-15T03:47:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023555178, 2024-09-15T03:47:41.111 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 495 2024-09-15T03:47:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:41.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888461+0000", 2024-09-15T03:47:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.719e-06 2024-09-15T03:47:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888471+0000", 2024-09-15T03:47:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0161e-05 2024-09-15T03:47:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888500+0000", 2024-09-15T03:47:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8989e-05 2024-09-15T03:47:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888508+0000", 2024-09-15T03:47:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5299999999999999e-06 2024-09-15T03:47:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888554+0000", 2024-09-15T03:47:41.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6489000000000002e-05 2024-09-15T03:47:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888809+0000", 2024-09-15T03:47:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000255467 2024-09-15T03:47:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.905724+0000", 2024-09-15T03:47:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016914348999999999 2024-09-15T03:47:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906244+0000", 2024-09-15T03:47:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052054900000000003 2024-09-15T03:47:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906262+0000", 2024-09-15T03:47:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7499999999999998e-05 2024-09-15T03:47:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.907367+0000", 2024-09-15T03:47:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011049460000000001 2024-09-15T03:47:41.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.907384+0000", 2024-09-15T03:47:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7000999999999999e-05 2024-09-15T03:47:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.911965+0000", 2024-09-15T03:47:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045809470000000001 2024-09-15T03:47:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.912014+0000", 2024-09-15T03:47:41.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9530999999999997e-05 2024-09-15T03:47:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:496 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-15T03:47:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.888524+0000", 2024-09-15T03:47:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 29.139148499000001, 2024-09-15T03:47:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01823953, 2024-09-15T03:47:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 496 2024-09-15T03:47:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888524+0000", 2024-09-15T03:47:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888524+0000", 2024-09-15T03:47:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888525+0000", 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2699999999999999e-07 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888529+0000", 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7840000000000001e-06 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888540+0000", 2024-09-15T03:47:41.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1477e-05 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888546+0000", 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4990000000000002e-06 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888971+0000", 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042491300000000002 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.889024+0000", 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3046e-05 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906517+0000", 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017492909000000001 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906638+0000", 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012138000000000001 2024-09-15T03:47:41.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906764+0000", 2024-09-15T03:47:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125795 2024-09-15T03:47:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:689 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~5144576,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.088846+0000", 2024-09-15T03:47:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.938827171, 2024-09-15T03:47:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020311780000000002, 2024-09-15T03:47:41.123 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 689 2024-09-15T03:47:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088846+0000", 2024-09-15T03:47:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088846+0000", 2024-09-15T03:47:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088847+0000", 2024-09-15T03:47:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9300000000000006e-07 2024-09-15T03:47:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088852+0000", 2024-09-15T03:47:41.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1850000000000003e-06 2024-09-15T03:47:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088863+0000", 2024-09-15T03:47:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1418e-05 2024-09-15T03:47:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088873+0000", 2024-09-15T03:47:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4020000000000008e-06 2024-09-15T03:47:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.089560+0000", 2024-09-15T03:47:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068774899999999998 2024-09-15T03:47:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090073+0000", 2024-09-15T03:47:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051222799999999999 2024-09-15T03:47:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.109157+0000", 2024-09-15T03:47:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019084805 2024-09-15T03:47:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:41.127 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:690 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~5144576,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.938794033000001, 2024-09-15T03:47:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025340499999999998, 2024-09-15T03:47:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 690 2024-09-15T03:47:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.128 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:41.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7999999999999996e-07 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088882+0000", 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1839999999999999e-06 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088892+0000", 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3540000000000006e-06 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088896+0000", 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5210000000000004e-06 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.093968+0000", 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050715170000000002 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.094098+0000", 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013070000000000001 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.114219+0000", 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020120743999999999 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:691 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~5144576,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.088900+0000", 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.938772289999999, 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031407274999999998, 2024-09-15T03:47:41.132 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 691 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088900+0000", 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088900+0000", 2024-09-15T03:47:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088901+0000", 2024-09-15T03:47:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9099999999999999e-07 2024-09-15T03:47:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088904+0000", 2024-09-15T03:47:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0299999999999998e-06 2024-09-15T03:47:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088911+0000", 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.708e-06 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088915+0000", 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0579999999999999e-06 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.096143+0000", 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007228099 2024-09-15T03:47:41.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.096202+0000", 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9667999999999999e-05 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.120308+0000", 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024105320999999999 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(osd.0.17:2 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [assert-version v51,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.090507+0000", 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.937165823000001, 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025010186, 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090507+0000", 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090507+0000", 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090508+0000", 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.176e-06 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:41.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090512+0000", 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2989999999999998e-06 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090522+0000", 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6549999999999999e-06 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090526+0000", 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8500000000000004e-06 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.094385+0000", 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038591670000000002 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.094541+0000", 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015590999999999999 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.115517+0000", 2024-09-15T03:47:41.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020976129 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:749 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~737280,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.935286+0000", 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.0923862829999997, 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021703044000000001, 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:41.141 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 749 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935286+0000", 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935286+0000", 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935287+0000", 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.51e-07 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935290+0000", 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8710000000000001e-06 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935297+0000", 2024-09-15T03:47:41.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3390000000000004e-06 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935302+0000", 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5560000000000001e-06 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936361+0000", 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010593200000000001 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936486+0000", 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124657 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.952979+0000", 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016493154999999999 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.953132+0000", 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015346500000000001 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.956989+0000", 2024-09-15T03:47:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038571299999999998 2024-09-15T03:47:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:750 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [create,setxattr \\^tkqj^qnPPfYPq\\WZoebWOkaebalTko_OhaYsrZ (1787) in=40b,setxattr cg]RoaNaVsOQTrQfp[VeRj_tmeegdoc`PtRYpskN (1998) in=40b,setxattr \\_N_s\\f^o_Ojh]gqWPkfsZ\\WTSbY]SOlX`mlV[^q (6944) in=40b,setxattr nsdiO]arkaqtYt (33) in=14b,setxattr ti`p[nirg_rWRRSe[oZhhn]b]ZsTimedg]]p[sQY (1049) in=40b,setxattr i[YNharOrm`bbP]UtYhjlqVWP\\XnccYlfhV_deTf (383) in=40b,setxattr ka_q[S]a]naRePWgcgUWmZSaW^RVV\\_rNXdZefal (875) in=40b,setxattr gaoaanUtdfqUrimS^_NfUtarRjNSVRf^_PbUpdtf (1645) in=40b,setxattr qNb^mqtq^Vsg]m\\aWdRTbbgOlZYptsW]O[_R`Vaa (1155) in=40b,setxattr gUPP_ttp`PdrngYsWrVdfZZVgff^Q`YRSjgOXTao (417) in=40b,setxattr mi`p\\f[PUQf^[[skf^\\Za_iXO_oTscOWNnQqcR[[ (4560) in=40b,setxattr N^_ka_abmi]seatXYWSQtibSdeOsfV\\OXsmmXlYm (1893) in=40b,setxattr aoqZmYaUpNW (33) in=11b,setxattr Z[bbsdRZTfkrP`]akaXifm`oV[YTOPrPaeeXV[fl (1611) in=40b,setxattr oZ`SalVqYt^T_[b_a]r[rlaaWOa_WZQ^^opf\\aN_ (243) in=40b,setxattr k\\V^enpkstj^VpoopX\\PcoTY]eogbs`pQQW]ofrm (347) in=40b,setxattr cfXfpnr`SogqjNa^cTVYtpgdciPp_eTUg]Ujn\\at (1926) in=40b,setxattr t`ojSZgRjbb[aaUXSepgQf_nc`kaOetle^n^`Vfd (1087) in=40b,setxattr ONYdltaQhe\\a\\da\\ehQRpsS^Ugdgmf\\WWigfUarV (631) in=40b,setxattr [Tj`Rbkt]qjatpSf_fPsanNr^ipYnOQlPnUZlmRU (4068) in=40b] snapc 32=[32,2d,2b,26,25,24,23,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T03:47:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.935912+0000", 2024-09-15T03:47:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.0917609590000001, 2024-09-15T03:47:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03915623, 2024-09-15T03:47:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 750 2024-09-15T03:47:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935912+0000", 2024-09-15T03:47:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935912+0000", 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935913+0000", 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7599999999999996e-07 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935984+0000", 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1136000000000005e-05 2024-09-15T03:47:41.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935997+0000", 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3482000000000001e-05 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936004+0000", 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3260000000000001e-06 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936878+0000", 2024-09-15T03:47:41.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087483999999999999 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.937895+0000", 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010163209999999999 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.965388+0000", 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027492986000000001 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:41.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.965630+0000", 2024-09-15T03:47:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000242012 2024-09-15T03:47:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.966983+0000", 2024-09-15T03:47:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013535349999999999 2024-09-15T03:47:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.967360+0000", 2024-09-15T03:47:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037704399999999999 2024-09-15T03:47:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.967371+0000", 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0988999999999999e-05 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.967436+0000", 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4679999999999997e-05 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.969080+0000", 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001644017 2024-09-15T03:47:41.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.969532+0000", 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045157699999999999 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.975068+0000", 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055364089999999999 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:41.154 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:41.155 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:47:41.156 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:41.154+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:41.256 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:41.256+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:41.264 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:41.264 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:41.264 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:41.265 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:41.273 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:47:41.357 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:41.357+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:41.371 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:41.372 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:41.372 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:41.372 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:41.372 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:41.380 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:47:41.394 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:47:41.458 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:41.457+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:41.478 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:41.478 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:41.478 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:41.478 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:47:41.478 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.478 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:47:41.479 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:41.479 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 83.573966562999999, 2024-09-15T03:47:41.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10988437500000001, 2024-09-15T03:47:41.479 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:41.479 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:41.479 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:41.479 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:47:41.479 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:47:41.479 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1 2024-09-15T03:47:41.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.479 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:41.479 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.480 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:41.480 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:41.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.480 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:41.480 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:41.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.480 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:41.480 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902355+0000", 2024-09-15T03:47:41.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.429e-06 2024-09-15T03:47:41.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.481 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.481 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:41.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902362+0000", 2024-09-15T03:47:41.481 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7290000000000003e-06 2024-09-15T03:47:41.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.481 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.481 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:41.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902373+0000", 2024-09-15T03:47:41.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0533999999999999e-05 2024-09-15T03:47:41.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.482 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902379+0000", 2024-09-15T03:47:41.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4350000000000002e-06 2024-09-15T03:47:41.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.482 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902428+0000", 2024-09-15T03:47:41.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8571e-05 2024-09-15T03:47:41.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.482 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:47:41.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902434+0000", 2024-09-15T03:47:41.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5610000000000004e-06 2024-09-15T03:47:41.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011307+0000", 2024-09-15T03:47:41.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.108872306 2024-09-15T03:47:41.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:41.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011448+0000", 2024-09-15T03:47:41.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014137199999999999 2024-09-15T03:47:41.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.484 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:47:41.484 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011569+0000", 2024-09-15T03:47:41.484 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120925 2024-09-15T03:47:41.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.484 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:41.484 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011695+0000", 2024-09-15T03:47:41.484 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012586800000000001 2024-09-15T03:47:41.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.484 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:41.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011703+0000", 2024-09-15T03:47:41.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4190000000000002e-06 2024-09-15T03:47:41.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:41.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011739+0000", 2024-09-15T03:47:41.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5474e-05 2024-09-15T03:47:41.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:47:41.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011940+0000", 2024-09-15T03:47:41.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020108699999999999 2024-09-15T03:47:41.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.486 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:47:41.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012179+0000", 2024-09-15T03:47:41.486 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023950299999999999 2024-09-15T03:47:41.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.486 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:41.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012215+0000", 2024-09-15T03:47:41.486 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5332e-05 2024-09-15T03:47:41.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.486 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:41.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012238+0000", 2024-09-15T03:47:41.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3830000000000001e-05 2024-09-15T03:47:41.487 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.487 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:41.487 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.487 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.487 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4263.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2024-09-15T03:47:41.487 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:41.487 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 77.432569877000006, 2024-09-15T03:47:41.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019495788999999999, 2024-09-15T03:47:41.487 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:41.487 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:41.487 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:41.488 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4263", 2024-09-15T03:47:41.488 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/646873808", 2024-09-15T03:47:41.488 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2 2024-09-15T03:47:41.488 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.488 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:41.488 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.488 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:41.488 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:41.488 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.488 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.488 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.488 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:41.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:41.489 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.489 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.489 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:41.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043752+0000", 2024-09-15T03:47:41.489 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.815e-06 2024-09-15T03:47:41.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.489 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.489 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:41.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043757+0000", 2024-09-15T03:47:41.489 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4240000000000003e-06 2024-09-15T03:47:41.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.489 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.490 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:41.490 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043766+0000", 2024-09-15T03:47:41.490 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8629999999999997e-06 2024-09-15T03:47:41.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.490 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.490 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043773+0000", 2024-09-15T03:47:41.490 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0099999999999998e-06 2024-09-15T03:47:41.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.490 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.490 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043830+0000", 2024-09-15T03:47:41.490 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7089999999999997e-05 2024-09-15T03:47:41.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.491 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.491 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:47:41.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043837+0000", 2024-09-15T03:47:41.491 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.08e-06 2024-09-15T03:47:41.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.491 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.491 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063103+0000", 2024-09-15T03:47:41.491 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019265619000000001 2024-09-15T03:47:41.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.491 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.491 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:41.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063246+0000", 2024-09-15T03:47:41.492 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000143888 2024-09-15T03:47:41.492 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.492 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:41.492 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.492 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.492 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:47:41.492 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:41.492 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 55.305496488000003, 2024-09-15T03:47:41.492 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011878227, 2024-09-15T03:47:41.492 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:41.492 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:41.492 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:41.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170825+0000", 2024-09-15T03:47:41.494 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9599999999999999e-07 2024-09-15T03:47:41.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.494 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.494 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:41.494 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171100+0000", 2024-09-15T03:47:41.494 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027548500000000001 2024-09-15T03:47:41.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.494 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.494 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:41.494 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171112+0000", 2024-09-15T03:47:41.494 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2152e-05 2024-09-15T03:47:41.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.494 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.495 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.495 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171119+0000", 2024-09-15T03:47:41.495 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5050000000000004e-06 2024-09-15T03:47:41.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.495 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.495 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.495 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171162+0000", 2024-09-15T03:47:41.495 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2908000000000003e-05 2024-09-15T03:47:41.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.495 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.495 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:41.495 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171169+0000", 2024-09-15T03:47:41.495 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9739999999999996e-06 2024-09-15T03:47:41.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.495 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.496 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:41.496 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171398+0000", 2024-09-15T03:47:41.496 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022954799999999999 2024-09-15T03:47:41.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.496 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.496 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:41.496 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173640+0000", 2024-09-15T03:47:41.496 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022417000000000001 2024-09-15T03:47:41.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.496 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.496 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:41.496 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173705+0000", 2024-09-15T03:47:41.497 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4907999999999996e-05 2024-09-15T03:47:41.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.497 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.497 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:41.497 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173742+0000", 2024-09-15T03:47:41.497 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6940999999999997e-05 2024-09-15T03:47:41.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.497 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.497 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:41.497 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.182702+0000", 2024-09-15T03:47:41.497 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0089604100000000002 2024-09-15T03:47:41.497 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.497 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:41.498 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.498 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.498 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.498 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 39/31 ECSubWrite(tid=43, reqid=client.4521.0:252, at_version=39'27, trim_to=0'0, roll_forward_to=39'25))", 2024-09-15T03:47:41.498 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:41.498 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 55.297779169999998, 2024-09-15T03:47:41.498 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011286757, 2024-09-15T03:47:41.498 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:41.498 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:41.498 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:41.498 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.498 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:41.498 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:41.499 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.499 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.499 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.499 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:41.499 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:41.499 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.499 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.499 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.499 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:41.499 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178542+0000", 2024-09-15T03:47:41.499 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9400000000000004e-07 2024-09-15T03:47:41.499 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.499 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.499 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:41.500 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178754+0000", 2024-09-15T03:47:41.500 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021228800000000001 2024-09-15T03:47:41.500 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.500 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.500 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:41.500 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178765+0000", 2024-09-15T03:47:41.500 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0176e-05 2024-09-15T03:47:41.500 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.500 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.500 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.500 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178769+0000", 2024-09-15T03:47:41.500 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1359999999999999e-06 2024-09-15T03:47:41.500 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.501 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.501 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.501 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178810+0000", 2024-09-15T03:47:41.501 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0825999999999998e-05 2024-09-15T03:47:41.501 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.501 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.501 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:41.501 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178820+0000", 2024-09-15T03:47:41.501 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0237999999999999e-05 2024-09-15T03:47:41.501 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.501 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.501 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:41.501 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179033+0000", 2024-09-15T03:47:41.502 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021361500000000001 2024-09-15T03:47:41.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.502 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.502 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:41.502 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179040+0000", 2024-09-15T03:47:41.502 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0509999999999999e-06 2024-09-15T03:47:41.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.502 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.502 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:41.502 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179102+0000", 2024-09-15T03:47:41.502 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1977000000000006e-05 2024-09-15T03:47:41.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.502 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.502 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:41.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179114+0000", 2024-09-15T03:47:41.503 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1223000000000001e-05 2024-09-15T03:47:41.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.503 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.503 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:41.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.189828+0000", 2024-09-15T03:47:41.503 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010714433000000001 2024-09-15T03:47:41.503 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.503 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:41.503 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.503 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.503 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 51/31 ECSubWrite(tid=67, reqid=client.4521.0:350, at_version=51'33, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:41.503 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:41.504 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 43.632479129000004, 2024-09-15T03:47:41.504 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.015569239, 2024-09-15T03:47:41.504 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:41.504 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:41.504 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:41.504 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.504 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:41.504 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:41.504 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.504 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.504 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.504 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:41.504 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:41.504 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.505 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.505 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:41.505 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843843+0000", 2024-09-15T03:47:41.505 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.195e-06 2024-09-15T03:47:41.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.505 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.505 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:41.505 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843880+0000", 2024-09-15T03:47:41.505 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7527999999999997e-05 2024-09-15T03:47:41.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.505 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.505 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:41.506 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843898+0000", 2024-09-15T03:47:41.506 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7696000000000001e-05 2024-09-15T03:47:41.506 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.506 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.506 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.506 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843907+0000", 2024-09-15T03:47:41.506 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8820000000000007e-06 2024-09-15T03:47:41.506 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.506 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.506 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.506 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843978+0000", 2024-09-15T03:47:41.506 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1537000000000003e-05 2024-09-15T03:47:41.506 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.506 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.507 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:41.507 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843989+0000", 2024-09-15T03:47:41.507 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0720000000000001e-05 2024-09-15T03:47:41.507 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.507 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.507 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:41.507 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844102+0000", 2024-09-15T03:47:41.507 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000113406 2024-09-15T03:47:41.507 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.507 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.507 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:41.507 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844131+0000", 2024-09-15T03:47:41.507 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8762e-05 2024-09-15T03:47:41.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.508 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.508 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:41.508 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844180+0000", 2024-09-15T03:47:41.508 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9329999999999997e-05 2024-09-15T03:47:41.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.508 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.508 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:41.508 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844201+0000", 2024-09-15T03:47:41.508 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0772999999999999e-05 2024-09-15T03:47:41.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.508 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.508 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:41.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.859411+0000", 2024-09-15T03:47:41.509 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01520941 2024-09-15T03:47:41.509 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.509 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:41.509 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.509 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.509 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:41.509 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:41.509 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 43.631600218000003, 2024-09-15T03:47:41.509 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012156452, 2024-09-15T03:47:41.509 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:41.509 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:41.510 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:41.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.510 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:41.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:41.510 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.510 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:41.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:41.510 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.510 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:41.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844721+0000", 2024-09-15T03:47:41.511 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1100000000000005e-07 2024-09-15T03:47:41.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.511 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.511 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:41.511 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844743+0000", 2024-09-15T03:47:41.511 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1999000000000001e-05 2024-09-15T03:47:41.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.511 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.511 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:41.511 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844758+0000", 2024-09-15T03:47:41.511 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4718999999999999e-05 2024-09-15T03:47:41.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.511 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.511 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.512 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844772+0000", 2024-09-15T03:47:41.512 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4637000000000001e-05 2024-09-15T03:47:41.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.512 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.512 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.512 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844842+0000", 2024-09-15T03:47:41.512 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9596000000000002e-05 2024-09-15T03:47:41.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.512 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.512 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:41.512 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844853+0000", 2024-09-15T03:47:41.512 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0912e-05 2024-09-15T03:47:41.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.513 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.513 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:41.513 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844966+0000", 2024-09-15T03:47:41.513 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011276 2024-09-15T03:47:41.513 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.513 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.513 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:41.513 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844977+0000", 2024-09-15T03:47:41.513 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1676e-05 2024-09-15T03:47:41.513 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.513 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.513 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:41.513 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847414+0000", 2024-09-15T03:47:41.514 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024363760000000001 2024-09-15T03:47:41.514 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.514 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.514 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:41.514 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847438+0000", 2024-09-15T03:47:41.514 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3737e-05 2024-09-15T03:47:41.514 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.514 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.514 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:41.514 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856877+0000", 2024-09-15T03:47:41.514 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094392290000000004 2024-09-15T03:47:41.514 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.514 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:41.514 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.515 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.515 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.515 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:41.515 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:41.515 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 43.631530822999999, 2024-09-15T03:47:41.515 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012258211, 2024-09-15T03:47:41.515 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:41.515 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:41.515 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:41.515 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.515 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:41.515 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:41.515 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.515 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.516 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:41.516 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:41.516 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.516 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:41.516 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:41.516 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.58e-07 2024-09-15T03:47:41.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.516 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:41.516 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844818+0000", 2024-09-15T03:47:41.517 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.796e-05 2024-09-15T03:47:41.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.517 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.517 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:41.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844829+0000", 2024-09-15T03:47:41.517 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0895e-05 2024-09-15T03:47:41.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.517 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.517 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844836+0000", 2024-09-15T03:47:41.517 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.618e-06 2024-09-15T03:47:41.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.517 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.517 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.518 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845023+0000", 2024-09-15T03:47:41.518 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018743500000000001 2024-09-15T03:47:41.518 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.518 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.518 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:41.518 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845035+0000", 2024-09-15T03:47:41.518 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1958e-05 2024-09-15T03:47:41.518 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.518 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.518 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:41.518 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845147+0000", 2024-09-15T03:47:41.518 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112281 2024-09-15T03:47:41.518 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.519 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.519 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:41.519 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847417+0000", 2024-09-15T03:47:41.519 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022693969999999998 2024-09-15T03:47:41.519 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.519 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.519 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:41.519 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847462+0000", 2024-09-15T03:47:41.519 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5466999999999997e-05 2024-09-15T03:47:41.519 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.519 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.519 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:41.519 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847503+0000", 2024-09-15T03:47:41.520 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1022999999999997e-05 2024-09-15T03:47:41.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.520 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:41.520 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.857048+0000", 2024-09-15T03:47:41.520 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095447190000000001 2024-09-15T03:47:41.520 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.520 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:41.520 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.520 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 51/31 ECSubWrite(tid=68, reqid=client.4521.0:351, at_version=51'34, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:41.520 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:41.520 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 43.631112705, 2024-09-15T03:47:41.521 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.014544793, 2024-09-15T03:47:41.521 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:41.521 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:41.521 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:41.521 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.521 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:41.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:41.521 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.521 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.521 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:41.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:41.521 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:41.522 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845209+0000", 2024-09-15T03:47:41.522 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5300000000000003e-07 2024-09-15T03:47:41.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:41.522 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845227+0000", 2024-09-15T03:47:41.522 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8759000000000001e-05 2024-09-15T03:47:41.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:41.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845241+0000", 2024-09-15T03:47:41.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3740000000000001e-05 2024-09-15T03:47:41.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.523 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845247+0000", 2024-09-15T03:47:41.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2779999999999999e-06 2024-09-15T03:47:41.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.523 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845308+0000", 2024-09-15T03:47:41.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0702000000000003e-05 2024-09-15T03:47:41.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.524 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.524 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:41.524 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845317+0000", 2024-09-15T03:47:41.524 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3770000000000002e-06 2024-09-15T03:47:41.524 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.524 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.524 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:41.524 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845413+0000", 2024-09-15T03:47:41.524 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5356e-05 2024-09-15T03:47:41.524 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.524 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.524 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:41.524 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847421+0000", 2024-09-15T03:47:41.525 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020077200000000002 2024-09-15T03:47:41.525 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.525 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.525 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:41.525 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847466+0000", 2024-09-15T03:47:41.525 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5754999999999998e-05 2024-09-15T03:47:41.525 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.525 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.525 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:41.525 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847572+0000", 2024-09-15T03:47:41.525 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010594000000000001 2024-09-15T03:47:41.525 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.525 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.526 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:41.526 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.859753+0000", 2024-09-15T03:47:41.526 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012180412999999999 2024-09-15T03:47:41.526 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.526 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:41.526 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.526 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.526 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.526 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 60/28 ECSubWrite(tid=73, reqid=client.4521.0:424, at_version=60'41, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:41.526 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:41.526 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 34.575909451000001, 2024-09-15T03:47:41.526 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01137208, 2024-09-15T03:47:41.526 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:41.526 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:41.527 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:41.527 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.527 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:41.527 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:41.527 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.527 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.527 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.527 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:41.527 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:41.527 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.527 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.527 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.527 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:41.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900412+0000", 2024-09-15T03:47:41.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7599999999999996e-07 2024-09-15T03:47:41.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.528 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.528 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:41.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:41.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7986999999999999e-05 2024-09-15T03:47:41.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.528 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.528 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:41.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900445+0000", 2024-09-15T03:47:41.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.468e-05 2024-09-15T03:47:41.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.529 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.529 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.529 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900451+0000", 2024-09-15T03:47:41.529 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5799999999999997e-06 2024-09-15T03:47:41.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.529 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.529 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.529 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900507+0000", 2024-09-15T03:47:41.529 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5855e-05 2024-09-15T03:47:41.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.529 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.529 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:41.529 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900530+0000", 2024-09-15T03:47:41.530 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2929e-05 2024-09-15T03:47:41.530 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.530 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.530 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:41.530 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900640+0000", 2024-09-15T03:47:41.530 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000110536 2024-09-15T03:47:41.530 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.530 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.530 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:41.530 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900648+0000", 2024-09-15T03:47:41.530 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9470000000000003e-06 2024-09-15T03:47:41.530 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.530 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.531 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:41.531 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900764+0000", 2024-09-15T03:47:41.531 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011518099999999999 2024-09-15T03:47:41.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.531 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.531 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:41.531 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900803+0000", 2024-09-15T03:47:41.531 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9941999999999998e-05 2024-09-15T03:47:41.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.531 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.531 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:41.531 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.911783+0000", 2024-09-15T03:47:41.531 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010979667 2024-09-15T03:47:41.532 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.532 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:41.532 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.532 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:517 3.3s1 3:caaee79c:::smithi00132172-10:head [read 0~3145728,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:41.532 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.947050+0000", 2024-09-15T03:47:41.532 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 25.529270518000001, 2024-09-15T03:47:41.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012110134, 2024-09-15T03:47:41.532 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:41.532 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:41.532 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:41.532 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:41.533 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.533 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 517 2024-09-15T03:47:41.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.533 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:41.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:41.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947050+0000", 2024-09-15T03:47:41.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:41.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947050+0000", 2024-09-15T03:47:41.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:41.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947052+0000", 2024-09-15T03:47:41.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9099999999999999e-06 2024-09-15T03:47:41.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.534 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:41.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947060+0000", 2024-09-15T03:47:41.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8529999999999992e-06 2024-09-15T03:47:41.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.534 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:41.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947074+0000", 2024-09-15T03:47:41.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4344e-05 2024-09-15T03:47:41.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.535 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.535 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.535 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947082+0000", 2024-09-15T03:47:41.535 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9249999999999995e-06 2024-09-15T03:47:41.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.535 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.535 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.535 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947138+0000", 2024-09-15T03:47:41.535 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6054000000000002e-05 2024-09-15T03:47:41.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.535 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.535 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:41.535 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947339+0000", 2024-09-15T03:47:41.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000200829 2024-09-15T03:47:41.536 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.536 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:41.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.959160+0000", 2024-09-15T03:47:41.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011821219000000001 2024-09-15T03:47:41.536 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.536 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:41.536 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.536 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.536 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:518 3.3s1 3:caaee79c:::smithi00132172-10:head [read 0~3145728,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:41.536 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.947091+0000", 2024-09-15T03:47:41.536 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 25.529229376, 2024-09-15T03:47:41.537 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017608233000000001, 2024-09-15T03:47:41.537 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:41.537 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:41.537 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:41.537 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:41.537 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.537 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 518 2024-09-15T03:47:41.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.537 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:41.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.537 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:41.537 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947091+0000", 2024-09-15T03:47:41.537 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:41.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947091+0000", 2024-09-15T03:47:41.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:41.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947092+0000", 2024-09-15T03:47:41.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7800000000000002e-07 2024-09-15T03:47:41.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:41.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947095+0000", 2024-09-15T03:47:41.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2959999999999999e-06 2024-09-15T03:47:41.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.539 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:41.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947102+0000", 2024-09-15T03:47:41.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7469999999999999e-06 2024-09-15T03:47:41.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.539 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947106+0000", 2024-09-15T03:47:41.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6550000000000003e-06 2024-09-15T03:47:41.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947622+0000", 2024-09-15T03:47:41.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000515727 2024-09-15T03:47:41.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:41.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947751+0000", 2024-09-15T03:47:41.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012920600000000001 2024-09-15T03:47:41.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:41.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.964699+0000", 2024-09-15T03:47:41.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.016948123999999998 2024-09-15T03:47:41.541 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.541 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:41.541 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.541 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:519 3.3s1 3:caaee79c:::smithi00132172-10:head [read 0~3145728,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:41.541 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:41.541 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 25.52919567, 2024-09-15T03:47:41.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.021743243999999998, 2024-09-15T03:47:41.541 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:41.541 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:41.542 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:41.542 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:41.542 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.542 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 519 2024-09-15T03:47:41.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.542 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:41.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.542 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:41.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:41.542 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.543 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:41.543 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:41.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.543 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.543 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.543 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:41.543 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:41.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5999999999999999e-07 2024-09-15T03:47:41.543 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.543 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.543 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:41.543 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947130+0000", 2024-09-15T03:47:41.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7720000000000002e-06 2024-09-15T03:47:41.543 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:41.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947137+0000", 2024-09-15T03:47:41.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2370000000000003e-06 2024-09-15T03:47:41.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947141+0000", 2024-09-15T03:47:41.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8809999999999998e-06 2024-09-15T03:47:41.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.948008+0000", 2024-09-15T03:47:41.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00086656700000000005 2024-09-15T03:47:41.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.545 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.545 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:41.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.948113+0000", 2024-09-15T03:47:41.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104904 2024-09-15T03:47:41.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.545 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.545 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:41.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.968868+0000", 2024-09-15T03:47:41.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.020755422999999999 2024-09-15T03:47:41.545 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.545 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:41.545 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.546 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.546 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.3s1 69/68 ECSubReadReply(tid=163, attrs_read=0))", 2024-09-15T03:47:41.546 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:41.546 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 25.522619926000001, 2024-09-15T03:47:41.546 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011102266, 2024-09-15T03:47:41.546 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:41.546 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:41.546 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:41.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.546 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:41.546 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:41.546 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.546 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.547 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:41.547 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:41.547 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.547 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.547 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:41.547 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:41.547 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1199999999999998e-07 2024-09-15T03:47:41.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.547 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.547 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:41.547 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956300+0000", 2024-09-15T03:47:41.547 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025987459999999999 2024-09-15T03:47:41.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:41.548 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956314+0000", 2024-09-15T03:47:41.548 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4469e-05 2024-09-15T03:47:41.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.548 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956319+0000", 2024-09-15T03:47:41.548 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.746e-06 2024-09-15T03:47:41.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.548 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.959591+0000", 2024-09-15T03:47:41.548 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032721510000000001 2024-09-15T03:47:41.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.549 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.549 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:41.549 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.964803+0000", 2024-09-15T03:47:41.549 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0052117420000000001 2024-09-15T03:47:41.549 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.549 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:41.549 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.549 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.549 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.3s1 69/68 ECSubReadReply(tid=164, attrs_read=0))", 2024-09-15T03:47:41.549 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:41.549 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 25.519995493, 2024-09-15T03:47:41.549 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012633276000000001, 2024-09-15T03:47:41.549 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:41.550 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:41.550 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:41.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.550 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:41.550 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:41.550 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.550 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:41.550 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:41.550 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.550 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:41.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:41.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5900000000000003e-07 2024-09-15T03:47:41.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:41.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957786+0000", 2024-09-15T03:47:41.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001460755 2024-09-15T03:47:41.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:41.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957807+0000", 2024-09-15T03:47:41.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0937e-05 2024-09-15T03:47:41.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957814+0000", 2024-09-15T03:47:41.552 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5989999999999998e-06 2024-09-15T03:47:41.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.964911+0000", 2024-09-15T03:47:41.552 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0070976370000000004 2024-09-15T03:47:41.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:41.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.968958+0000", 2024-09-15T03:47:41.552 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.004046989 2024-09-15T03:47:41.553 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.553 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:41.553 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.553 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.553 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.553 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:567 3.ds0 3:b166213b:::smithi00132172-21:head [rollback 3] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:41.553 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.947576+0000", 2024-09-15T03:47:41.553 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 20.528744565, 2024-09-15T03:47:41.553 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013871319999999999, 2024-09-15T03:47:41.553 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:41.553 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:41.553 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:41.553 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:41.554 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.554 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 567 2024-09-15T03:47:41.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.554 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:41.554 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.554 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:41.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947576+0000", 2024-09-15T03:47:41.554 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.554 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.554 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:41.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947576+0000", 2024-09-15T03:47:41.554 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.555 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.555 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.555 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:41.555 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947577+0000", 2024-09-15T03:47:41.555 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.37e-07 2024-09-15T03:47:41.555 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.555 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.555 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:41.555 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947580+0000", 2024-09-15T03:47:41.555 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5710000000000002e-06 2024-09-15T03:47:41.555 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.555 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.555 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:41.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947588+0000", 2024-09-15T03:47:41.556 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8939999999999993e-06 2024-09-15T03:47:41.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.556 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.556 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947593+0000", 2024-09-15T03:47:41.556 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7049999999999998e-06 2024-09-15T03:47:41.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.556 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.556 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948070+0000", 2024-09-15T03:47:41.556 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047685099999999998 2024-09-15T03:47:41.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:41.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948177+0000", 2024-09-15T03:47:41.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010765299999999999 2024-09-15T03:47:41.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:41.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948986+0000", 2024-09-15T03:47:41.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000808977 2024-09-15T03:47:41.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:41.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949117+0000", 2024-09-15T03:47:41.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000131032 2024-09-15T03:47:41.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:41.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949133+0000", 2024-09-15T03:47:41.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5798999999999999e-05 2024-09-15T03:47:41.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:41.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950936+0000", 2024-09-15T03:47:41.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00180316 2024-09-15T03:47:41.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:41.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950994+0000", 2024-09-15T03:47:41.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7884999999999999e-05 2024-09-15T03:47:41.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:41.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951532+0000", 2024-09-15T03:47:41.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00053787699999999997 2024-09-15T03:47:41.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:41.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961447+0000", 2024-09-15T03:47:41.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0099152790000000008 2024-09-15T03:47:41.559 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.559 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:41.559 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.560 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.560 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:568 3.ds0 3:b166213b:::smithi00132172-21:head [append 0~0] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:41.560 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.948048+0000", 2024-09-15T03:47:41.560 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 20.528272640000001, 2024-09-15T03:47:41.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013579269, 2024-09-15T03:47:41.560 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:41.560 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:41.560 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:41.560 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:41.560 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.560 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 568 2024-09-15T03:47:41.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.561 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:41.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.561 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:41.561 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948048+0000", 2024-09-15T03:47:41.561 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.561 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:41.561 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948048+0000", 2024-09-15T03:47:41.561 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.561 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:41.562 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948049+0000", 2024-09-15T03:47:41.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7199999999999998e-07 2024-09-15T03:47:41.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.562 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:41.562 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948056+0000", 2024-09-15T03:47:41.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0280000000000003e-06 2024-09-15T03:47:41.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.562 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:41.562 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948066+0000", 2024-09-15T03:47:41.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0141e-05 2024-09-15T03:47:41.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948072+0000", 2024-09-15T03:47:41.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1519999999999997e-06 2024-09-15T03:47:41.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949221+0000", 2024-09-15T03:47:41.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001149125 2024-09-15T03:47:41.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:41.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949313+0000", 2024-09-15T03:47:41.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1469000000000001e-05 2024-09-15T03:47:41.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:41.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949818+0000", 2024-09-15T03:47:41.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00050549700000000004 2024-09-15T03:47:41.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:41.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950002+0000", 2024-09-15T03:47:41.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018414700000000001 2024-09-15T03:47:41.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:41.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950940+0000", 2024-09-15T03:47:41.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00093786100000000001 2024-09-15T03:47:41.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:41.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951004+0000", 2024-09-15T03:47:41.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3859e-05 2024-09-15T03:47:41.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:41.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951020+0000", 2024-09-15T03:47:41.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.628e-05 2024-09-15T03:47:41.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:41.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951753+0000", 2024-09-15T03:47:41.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00073276999999999999 2024-09-15T03:47:41.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:41.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961627+0000", 2024-09-15T03:47:41.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098741679999999991 2024-09-15T03:47:41.566 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.566 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:41.566 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.567 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:677 3.3s1 3:caaee79c:::smithi00132172-10:head [copy-from ver 31] snapc 2f=[2d,2b,29,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:41.567 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:30.087209+0000", 2024-09-15T03:47:41.567 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.389111449, 2024-09-15T03:47:41.567 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.025257667000000001, 2024-09-15T03:47:41.567 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:41.567 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:41.567 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:41.567 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:41.567 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.567 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 677 2024-09-15T03:47:41.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.567 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:41.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.568 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:41.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087209+0000", 2024-09-15T03:47:41.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.568 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:41.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087209+0000", 2024-09-15T03:47:41.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.568 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:41.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087211+0000", 2024-09-15T03:47:41.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2500000000000001e-06 2024-09-15T03:47:41.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.569 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:41.569 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087222+0000", 2024-09-15T03:47:41.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.113e-05 2024-09-15T03:47:41.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.569 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:41.569 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087252+0000", 2024-09-15T03:47:41.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9488999999999998e-05 2024-09-15T03:47:41.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.569 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.569 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087264+0000", 2024-09-15T03:47:41.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1875999999999999e-05 2024-09-15T03:47:41.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087368+0000", 2024-09-15T03:47:41.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104035 2024-09-15T03:47:41.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:41.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087560+0000", 2024-09-15T03:47:41.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019211400000000001 2024-09-15T03:47:41.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:41.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.104984+0000", 2024-09-15T03:47:41.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017424489000000001 2024-09-15T03:47:41.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:41.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105388+0000", 2024-09-15T03:47:41.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000403686 2024-09-15T03:47:41.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:41.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105404+0000", 2024-09-15T03:47:41.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6268000000000001e-05 2024-09-15T03:47:41.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:41.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105641+0000", 2024-09-15T03:47:41.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023648399999999999 2024-09-15T03:47:41.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:41.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105662+0000", 2024-09-15T03:47:41.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0602999999999999e-05 2024-09-15T03:47:41.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.573 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.573 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:41.573 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.110231+0000", 2024-09-15T03:47:41.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0045697089999999999 2024-09-15T03:47:41.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.573 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.573 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:41.573 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.112467+0000", 2024-09-15T03:47:41.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002235534 2024-09-15T03:47:41.573 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.574 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:41.574 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.574 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.574 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:678 3.3s1 3:caaee79c:::smithi00132172-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:41.574 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:30.087279+0000", 2024-09-15T03:47:41.574 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.389041993999999, 2024-09-15T03:47:41.574 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.018991965, 2024-09-15T03:47:41.574 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:41.574 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:41.574 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:41.574 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:41.574 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.574 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 678 2024-09-15T03:47:41.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.575 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:41.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.575 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:41.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087279+0000", 2024-09-15T03:47:41.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.575 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:41.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087279+0000", 2024-09-15T03:47:41.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:41.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087280+0000", 2024-09-15T03:47:41.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.147e-06 2024-09-15T03:47:41.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:41.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087285+0000", 2024-09-15T03:47:41.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2059999999999998e-06 2024-09-15T03:47:41.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:41.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087303+0000", 2024-09-15T03:47:41.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7646e-05 2024-09-15T03:47:41.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.577 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087313+0000", 2024-09-15T03:47:41.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0862999999999999e-05 2024-09-15T03:47:41.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.577 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087765+0000", 2024-09-15T03:47:41.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045120400000000003 2024-09-15T03:47:41.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.577 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:41.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087962+0000", 2024-09-15T03:47:41.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000197095 2024-09-15T03:47:41.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.578 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105788+0000", 2024-09-15T03:47:41.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017826143999999999 2024-09-15T03:47:41.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.578 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:41.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.106034+0000", 2024-09-15T03:47:41.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024586999999999998 2024-09-15T03:47:41.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:41.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.106270+0000", 2024-09-15T03:47:41.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023678999999999999 2024-09-15T03:47:41.579 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.579 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:41.579 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.579 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:688 3.3s1 3:c7c3ef16:::smithi00132172-25:head [create,setxattr og^`]cfbTbdPrNph_pkYgeiTrgatpl] (805) in=31b,setxattr `Trb[\\aaaksnX\\dQOYP^TPXpYtmQ_maX\\[hbePkT (981) in=40b,setxattr UkQ^W[[sWh]fSrWg]^\\oZpWaamVkVUnZYbR[Vql\\ (663) in=40b,setxattr ofr[Os^]tjr^jicnNpPaUeXQTfldVQPdSPjcgWcT (7228) in=40b,setxattr Vl`NPsTVpnj_gO_srrbrZ\\eq (805) in=24b,setxattr ^_mcpljcT[qX_UeeYZY\\_ZTSsSsojP\\esUkVphT] (1963) in=40b,setxattr [oY^bSn[ealaPaOhprWW_SNaa]VhaXN[jmo`YYgt (138) in=40b,setxattr tT[SSseaPa\\UPS`\\g\\pRqsd`^lo__arabQlQmob` (663) in=40b,setxattr o^\\]jXsesZhoknr\\dagnjidgsPOhkfg[ffVe[PQW (1259) in=40b,setxattr c^VX_^n^gsh^P`aVbXpaS\\cgkpZO]OmXmchTtWZd (1612) in=40b,setxattr _^aeiTb[Uhh`rNNUOZOPt_Z_Qmeps`XXQ^qidetf (629) in=40b,setxattr ms[glaQ`N\\pqbicVTmgp^\\Rg]h`fefatkfg\\js_[ (139) in=40b,setxattr O\\gsPtYmrReq]if[pkOgPtqS`]Nrq_eYpsefkSgl (700) in=40b,setxattr RgefRtSsOT`jTQb^h\\igU`_lbscW`fgkXroNdmea (1751) in=40b,setxattr sgaV_cSaelbWQYill^k`RsVajcXpUSfW]eNd_nat (807) in=40b,setxattr P\\lm\\r`\\jpssZmkPqcQSbje_]]`YOSZTeRUUSSY[ (1365) in=40b,setxattr aa^ePena\\opOdeWXTj[XU]N^jQS]bens\\Tc_g]ea (16140) in=40b,setxattr R[qckQQO`n[[egbgcZsdtUbdPdnPNlOmsbP_OXOQ (1227) in=40b,setxattr SpRUpSaQZ]nlQ]kpXZnOWSUS]tsmf`qZiPTkq]Yi (209) in=40b,setxattr Y[Y[X`O_XaVOb]NZ_PgmhRX]Vds_dfbp]h[SPkYb (417) in=40b] snapc 2f=[2d,2b,29,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:41.579 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:30.088531+0000", 2024-09-15T03:47:41.579 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.387789259, 2024-09-15T03:47:41.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013293503999999999, 2024-09-15T03:47:41.580 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:41.580 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:41.580 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:41.580 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:41.580 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:41.580 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 688 2024-09-15T03:47:41.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.580 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:41.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.580 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:41.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088531+0000", 2024-09-15T03:47:41.580 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:41.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088531+0000", 2024-09-15T03:47:41.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:41.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088533+0000", 2024-09-15T03:47:41.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5379999999999999e-06 2024-09-15T03:47:41.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:41.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088662+0000", 2024-09-15T03:47:41.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000129002 2024-09-15T03:47:41.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.582 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:41.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088688+0000", 2024-09-15T03:47:41.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6084999999999998e-05 2024-09-15T03:47:41.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.582 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088696+0000", 2024-09-15T03:47:41.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0490000000000004e-06 2024-09-15T03:47:41.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.582 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088746+0000", 2024-09-15T03:47:41.583 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0074000000000002e-05 2024-09-15T03:47:41.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.583 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:41.583 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.089084+0000", 2024-09-15T03:47:41.583 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033810200000000001 2024-09-15T03:47:41.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.583 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:41.583 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090498+0000", 2024-09-15T03:47:41.583 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001414205 2024-09-15T03:47:41.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.583 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:41.584 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090936+0000", 2024-09-15T03:47:41.584 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043802799999999997 2024-09-15T03:47:41.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.584 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.584 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:41.584 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090951+0000", 2024-09-15T03:47:41.584 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4953999999999999e-05 2024-09-15T03:47:41.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.584 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.584 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:41.584 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.091069+0000", 2024-09-15T03:47:41.584 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000117236 2024-09-15T03:47:41.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.584 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:41.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.091128+0000", 2024-09-15T03:47:41.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9580999999999998e-05 2024-09-15T03:47:41.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:41.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.097312+0000", 2024-09-15T03:47:41.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0061838129999999998 2024-09-15T03:47:41.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:41.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.101825+0000", 2024-09-15T03:47:41.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0045128369999999996 2024-09-15T03:47:41.585 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.586 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:41.586 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.586 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s4 83/80 ECSubWrite(tid=24, reqid=client.4521.0:675, at_version=83'37, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:41.586 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:30.126290+0000", 2024-09-15T03:47:41.586 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.350030415999999, 2024-09-15T03:47:41.586 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.014228881000000001, 2024-09-15T03:47:41.586 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:41.586 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:41.586 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:41.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.586 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:41.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.126290+0000", 2024-09-15T03:47:41.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.587 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.587 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:41.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.126290+0000", 2024-09-15T03:47:41.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:41.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.587 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.587 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:41.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.126291+0000", 2024-09-15T03:47:41.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6499999999999998e-07 2024-09-15T03:47:41.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.588 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:41.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128867+0000", 2024-09-15T03:47:41.588 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025759749999999999 2024-09-15T03:47:41.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.588 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:41.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128881+0000", 2024-09-15T03:47:41.588 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3823e-05 2024-09-15T03:47:41.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.588 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:41.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128888+0000", 2024-09-15T03:47:41.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.605e-06 2024-09-15T03:47:41.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.589 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:41.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128949+0000", 2024-09-15T03:47:41.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0949999999999998e-05 2024-09-15T03:47:41.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.589 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:41.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128957+0000", 2024-09-15T03:47:41.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4760000000000001e-06 2024-09-15T03:47:41.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.589 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.590 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:41.590 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.129323+0000", 2024-09-15T03:47:41.590 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036638900000000001 2024-09-15T03:47:41.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.590 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.590 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:41.590 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.129331+0000", 2024-09-15T03:47:41.590 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1319999999999994e-06 2024-09-15T03:47:41.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.590 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.590 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:41.590 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.129567+0000", 2024-09-15T03:47:41.590 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000235393 2024-09-15T03:47:41.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:41.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.129586+0000", 2024-09-15T03:47:41.591 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9517999999999998e-05 2024-09-15T03:47:41.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:41.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:41.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:41.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.140519+0000", 2024-09-15T03:47:41.591 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010932855 2024-09-15T03:47:41.591 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.591 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:41.591 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.591 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:41.592 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:41.592 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:41.593 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:41.593 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:41.594 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:41.594 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:41.594 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:47:41.595 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:41.595 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:41.596 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:41.596 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:41.596 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:41.596 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:41.597 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:41.597 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:41.592+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:41.600 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:41.600 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:41.614 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:41.615 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:41.629 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:41.629 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:41.643 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:41.644 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:41.644 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-15T03:47:41.658 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:41.659 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:41.661 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T03:47:41.662 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T03:47:41.694 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:41.694 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:41.695 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:41.694+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:41.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:47:41.795 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:41.795+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:41.799 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:41.799 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:41.807 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:47:41.847 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-15T03:47:41.847 INFO:tasks.rados.rados.0.smithi001.stdout:316: done (0 left) 2024-09-15T03:47:41.847 INFO:tasks.rados.rados.0.smithi001.stdout:317: snap_remove snap 25 2024-09-15T03:47:41.896 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:41.896+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:41.904 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:41.905 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:41.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:47:41.998 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:41.997+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:42.011 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:42.011 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:42.020 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:47:42.098 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:42.098+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:42.116 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:42.116 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:42.124 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:47:42.199 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:42.199+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:42.221 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:42.221 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:42.229 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:47:42.300 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:42.299+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:42.343 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:42.344 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:42.352 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:47:42.401 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:42.400+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:42.454 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:42.454 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:42.463 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:47:42.502 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:42.501+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:42.602 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:42.602+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:42.604 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:42.604 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:42.613 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:47:42.705 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:42.704+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:42.713 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:42.713 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:42.721 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:47:42.806 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:42.805+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:42.821 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:42.821 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:42.830 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:47:42.848 INFO:tasks.rados.rados.0.smithi001.stdout:317: done (0 left) 2024-09-15T03:47:42.848 INFO:tasks.rados.rados.0.smithi001.stdout:318: read oid 4 snap 1 2024-09-15T03:47:42.848 INFO:tasks.rados.rados.0.smithi001.stdout:318: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-15T03:47:42.848 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 27 current snap is 39 2024-09-15T03:47:42.848 INFO:tasks.rados.rados.0.smithi001.stdout:319: seq_num 123 ranges {5210112=524288,5734400=573440,6307840=688128,6995968=294912} 2024-09-15T03:47:42.880 INFO:tasks.rados.rados.0.smithi001.stdout:319: writing smithi00132172-27 from 5210112 to 5734400 tid 1 2024-09-15T03:47:42.883 INFO:tasks.rados.rados.0.smithi001.stdout:319: writing smithi00132172-27 from 5734400 to 6307840 tid 2 2024-09-15T03:47:42.887 INFO:tasks.rados.rados.0.smithi001.stdout:319: writing smithi00132172-27 from 6307840 to 6995968 tid 3 2024-09-15T03:47:42.889 INFO:tasks.rados.rados.0.smithi001.stdout:319: writing smithi00132172-27 from 6995968 to 7290880 tid 4 2024-09-15T03:47:42.889 INFO:tasks.rados.rados.0.smithi001.stdout:320: snap_create 2024-09-15T03:47:42.889 INFO:tasks.rados.rados.0.smithi001.stdout:318: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-15T03:47:42.891 INFO:tasks.rados.rados.0.smithi001.stdout:319: finishing write tid 1 to smithi00132172-27 2024-09-15T03:47:42.891 INFO:tasks.rados.rados.0.smithi001.stdout:319: finishing write tid 2 to smithi00132172-27 2024-09-15T03:47:42.893 INFO:tasks.rados.rados.0.smithi001.stdout:319: finishing write tid 3 to smithi00132172-27 2024-09-15T03:47:42.894 INFO:tasks.rados.rados.0.smithi001.stdout:319: finishing write tid 4 to smithi00132172-27 2024-09-15T03:47:42.894 INFO:tasks.rados.rados.0.smithi001.stdout:319: finishing write tid 5 to smithi00132172-27 2024-09-15T03:47:42.896 INFO:tasks.rados.rados.0.smithi001.stdout:319: finishing write tid 6 to smithi00132172-27 2024-09-15T03:47:42.896 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 27 v 59 (ObjNum 123 snap 39 seq_num 123) dirty exists 2024-09-15T03:47:42.896 INFO:tasks.rados.rados.0.smithi001.stdout:319: left oid 27 (ObjNum 123 snap 39 seq_num 123) 2024-09-15T03:47:42.909 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:42.908+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:42.955 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:42.955 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:42.963 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:47:43.010 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:43.010+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:43.105 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:43.105 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:43.114 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:47:43.114 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:43.112+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:43.214 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:43.214 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:43.214 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:43.213+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:43.222 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:47:43.315 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:43.315+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:43.322 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:43.322 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:43.331 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:47:43.416 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:43.415+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:43.472 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:43.472 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:43.480 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:47:43.525 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:43.524+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:43.580 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:43.581 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:43.589 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:47:43.617 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:43.616+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:43.663 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:47:43.718 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:43.717+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:43.731 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:43.731 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:43.740 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:47:43.790 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:43.790 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:43.804 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:43.804 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:43.818 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:43.860 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:43.861 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:43.861 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:43.861 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:43.862 INFO:tasks.rados.rados.0.smithi001.stdout:318: done (2 left) 2024-09-15T03:47:43.862 INFO:tasks.rados.rados.0.smithi001.stdout:319: done (1 left) 2024-09-15T03:47:43.863 INFO:tasks.rados.rados.0.smithi001.stdout:320: done (0 left) 2024-09-15T03:47:43.863 INFO:tasks.rados.rados.0.smithi001.stdout:321: read oid 48 snap -1 2024-09-15T03:47:43.863 INFO:tasks.rados.rados.0.smithi001.stdout:321: expect (ObjNum 47 snap 0 seq_num 47) 2024-09-15T03:47:43.863 INFO:tasks.rados.rados.0.smithi001.stdout:322: setattr oid 21 current snap is 40 2024-09-15T03:47:43.863 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 43 current snap is 40 2024-09-15T03:47:43.863 INFO:tasks.rados.rados.0.smithi001.stdout:323: seq_num 124 ranges {2719744=540672,3260416=720896,3981312=786432,4767744=49152} 2024-09-15T03:47:43.863 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T03:47:43.865 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:43.863+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:43.865 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:43.865 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:43.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.865 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:815 3.7s0 3:eb927994:::smithi00132172-48:head [read 0~802816,checksum in=4b,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e97)", 2024-09-15T03:47:43.865 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:43.851739+0000", 2024-09-15T03:47:43.865 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0078001310000000001, 2024-09-15T03:47:43.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0078333110000000008, 2024-09-15T03:47:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 815 2024-09-15T03:47:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851739+0000", 2024-09-15T03:47:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851739+0000", 2024-09-15T03:47:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851740+0000", 2024-09-15T03:47:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2240000000000001e-06 2024-09-15T03:47:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851747+0000", 2024-09-15T03:47:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0369999999999997e-06 2024-09-15T03:47:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851760+0000", 2024-09-15T03:47:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3298999999999999e-05 2024-09-15T03:47:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851767+0000", 2024-09-15T03:47:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5459999999999997e-06 2024-09-15T03:47:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851841+0000", 2024-09-15T03:47:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4814999999999994e-05 2024-09-15T03:47:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851853+0000", 2024-09-15T03:47:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2001000000000001e-05 2024-09-15T03:47:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.857989+0000", 2024-09-15T03:47:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006135415 2024-09-15T03:47:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.858087+0000", 2024-09-15T03:47:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8566999999999993e-05 2024-09-15T03:47:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:816 3.7s0 3:eb927994:::smithi00132172-48:head [read 0~802816,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e97)", 2024-09-15T03:47:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:43.851776+0000", 2024-09-15T03:47:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0077626279999999997, 2024-09-15T03:47:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0078967559999999996, 2024-09-15T03:47:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 816 2024-09-15T03:47:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851776+0000", 2024-09-15T03:47:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851776+0000", 2024-09-15T03:47:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851776+0000", 2024-09-15T03:47:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8999999999999998e-07 2024-09-15T03:47:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851779+0000", 2024-09-15T03:47:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2079999999999999e-06 2024-09-15T03:47:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851784+0000", 2024-09-15T03:47:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1390000000000003e-06 2024-09-15T03:47:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851791+0000", 2024-09-15T03:47:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3810000000000001e-06 2024-09-15T03:47:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851920+0000", 2024-09-15T03:47:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012913599999999999 2024-09-15T03:47:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851925+0000", 2024-09-15T03:47:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.916e-06 2024-09-15T03:47:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.858399+0000", 2024-09-15T03:47:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064739410000000004 2024-09-15T03:47:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.858505+0000", 2024-09-15T03:47:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000105641 2024-09-15T03:47:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:817 3.7s0 3:eb927994:::smithi00132172-48:head [read 0~802816,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e97)", 2024-09-15T03:47:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:43.851834+0000", 2024-09-15T03:47:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0077050929999999997, 2024-09-15T03:47:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0079290239999999998, 2024-09-15T03:47:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 817 2024-09-15T03:47:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851834+0000", 2024-09-15T03:47:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851834+0000", 2024-09-15T03:47:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851834+0000", 2024-09-15T03:47:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9299999999999999e-07 2024-09-15T03:47:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851836+0000", 2024-09-15T03:47:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.401e-06 2024-09-15T03:47:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851840+0000", 2024-09-15T03:47:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0759999999999996e-06 2024-09-15T03:47:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851843+0000", 2024-09-15T03:47:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8729999999999999e-06 2024-09-15T03:47:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851949+0000", 2024-09-15T03:47:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106165 2024-09-15T03:47:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.851953+0000", 2024-09-15T03:47:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6839999999999998e-06 2024-09-15T03:47:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.858974+0000", 2024-09-15T03:47:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0070214370000000002 2024-09-15T03:47:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859173+0000", 2024-09-15T03:47:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019860400000000001 2024-09-15T03:47:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpRead(3.7s0 97/80 ECSubRead(tid=307, to_read={3:eb927994:::smithi00132172-48:head=0,200704,0}, subchunks={3:eb927994:::smithi00132172-48:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:47:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:43.858864+0000", 2024-09-15T03:47:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.00067479499999999995, 2024-09-15T03:47:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096687400000000005, 2024-09-15T03:47:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.858864+0000", 2024-09-15T03:47:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.858864+0000", 2024-09-15T03:47:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.858864+0000", 2024-09-15T03:47:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.858864+0000", 2024-09-15T03:47:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.858905+0000", 2024-09-15T03:47:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1292999999999999e-05 2024-09-15T03:47:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.858912+0000", 2024-09-15T03:47:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2570000000000001e-06 2024-09-15T03:47:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859814+0000", 2024-09-15T03:47:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00090161899999999999 2024-09-15T03:47:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpReadReply(3.7s0 96/80 ECSubReadReply(tid=306, attrs_read=0))", 2024-09-15T03:47:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:43.858850+0000", 2024-09-15T03:47:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.00068897499999999998, 2024-09-15T03:47:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010079990000000001, 2024-09-15T03:47:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "queued for pg", 2024-09-15T03:47:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.858850+0000", 2024-09-15T03:47:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.858850+0000", 2024-09-15T03:47:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.858850+0000", 2024-09-15T03:47:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6899999999999997e-07 2024-09-15T03:47:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859126+0000", 2024-09-15T03:47:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027551700000000003 2024-09-15T03:47:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859139+0000", 2024-09-15T03:47:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2648e-05 2024-09-15T03:47:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859145+0000", 2024-09-15T03:47:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2110000000000003e-06 2024-09-15T03:47:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpReadReply(3.7s0 97/80 ECSubReadReply(tid=306, attrs_read=0))", 2024-09-15T03:47:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:43.858770+0000", 2024-09-15T03:47:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.00076855399999999996, 2024-09-15T03:47:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001115132, 2024-09-15T03:47:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "queued for pg", 2024-09-15T03:47:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.858770+0000", 2024-09-15T03:47:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.858770+0000", 2024-09-15T03:47:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.858771+0000", 2024-09-15T03:47:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1999999999999998e-07 2024-09-15T03:47:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859162+0000", 2024-09-15T03:47:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039101700000000001 2024-09-15T03:47:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859172+0000", 2024-09-15T03:47:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8130000000000007e-06 2024-09-15T03:47:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859177+0000", 2024-09-15T03:47:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9710000000000003e-06 2024-09-15T03:47:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpRead(3.7s0 97/80 ECSubRead(tid=308, to_read={3:eb927994:::smithi00132172-48:head=0,200704,0}, subchunks={3:eb927994:::smithi00132172-48:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:47:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:43.859435+0000", 2024-09-15T03:47:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.000103842, 2024-09-15T03:47:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047623400000000002, 2024-09-15T03:47:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "queued for pg", 2024-09-15T03:47:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859435+0000", 2024-09-15T03:47:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859435+0000", 2024-09-15T03:47:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859435+0000", 2024-09-15T03:47:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859435+0000", 2024-09-15T03:47:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859462+0000", 2024-09-15T03:47:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7155e-05 2024-09-15T03:47:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859470+0000", 2024-09-15T03:47:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1030000000000002e-06 2024-09-15T03:47:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpReadReply(3.7s0 97/80 ECSubReadReply(tid=306, attrs_read=0))", 2024-09-15T03:47:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:43.859268+0000", 2024-09-15T03:47:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.00027082199999999999, 2024-09-15T03:47:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000665454, 2024-09-15T03:47:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "queued for pg", 2024-09-15T03:47:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859268+0000", 2024-09-15T03:47:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859268+0000", 2024-09-15T03:47:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859268+0000", 2024-09-15T03:47:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2200000000000004e-07 2024-09-15T03:47:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859546+0000", 2024-09-15T03:47:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027771000000000002 2024-09-15T03:47:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859556+0000", 2024-09-15T03:47:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.482e-06 2024-09-15T03:47:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859561+0000", 2024-09-15T03:47:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4820000000000002e-06 2024-09-15T03:47:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpReadReply(3.7s0 97/80 ECSubReadReply(tid=307, attrs_read=0))", 2024-09-15T03:47:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:43.859572+0000", 2024-09-15T03:47:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4294967295.9999671, 2024-09-15T03:47:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038776499999999998, 2024-09-15T03:47:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "queued for pg", 2024-09-15T03:47:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859572+0000", 2024-09-15T03:47:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859572+0000", 2024-09-15T03:47:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859572+0000", 2024-09-15T03:47:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2600000000000002e-07 2024-09-15T03:47:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859725+0000", 2024-09-15T03:47:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015263399999999999 2024-09-15T03:47:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859734+0000", 2024-09-15T03:47:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4909999999999994e-06 2024-09-15T03:47:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859738+0000", 2024-09-15T03:47:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5449999999999999e-06 2024-09-15T03:47:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpReadReply(3.7s0 97/80 ECSubReadReply(tid=306, attrs_read=0))", 2024-09-15T03:47:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:43.859747+0000", 2024-09-15T03:47:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4294967295.9997911, 2024-09-15T03:47:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025044800000000001, 2024-09-15T03:47:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "queued for pg", 2024-09-15T03:47:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859747+0000", 2024-09-15T03:47:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859747+0000", 2024-09-15T03:47:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859747+0000", 2024-09-15T03:47:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859747+0000", 2024-09-15T03:47:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859764+0000", 2024-09-15T03:47:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6795000000000001e-05 2024-09-15T03:47:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859769+0000", 2024-09-15T03:47:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8509999999999998e-06 2024-09-15T03:47:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpReadReply(3.7s0 97/80 ECSubReadReply(tid=307, attrs_read=0))", 2024-09-15T03:47:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:43.859962+0000", 2024-09-15T03:47:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4294967295.999577, 2024-09-15T03:47:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3505999999999996e-05, 2024-09-15T03:47:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "queued for pg", 2024-09-15T03:47:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859962+0000", 2024-09-15T03:47:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859962+0000", 2024-09-15T03:47:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859962+0000", 2024-09-15T03:47:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859962+0000", 2024-09-15T03:47:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859970+0000", 2024-09-15T03:47:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6540000000000005e-06 2024-09-15T03:47:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.859973+0000", 2024-09-15T03:47:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6730000000000001e-06 2024-09-15T03:47:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.905 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:43.905 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.905 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:43.905 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-15T03:47:43.905 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 11 2024-09-15T03:47:43.905 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:43.906 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:43.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:47:43.908 INFO:tasks.rados.rados.0.smithi001.stdout:323: writing smithi00132172-43 from 2719744 to 3260416 tid 1 2024-09-15T03:47:43.908 INFO:tasks.rados.rados.0.smithi001.stdout:323: writing smithi00132172-43 from 3260416 to 3981312 tid 2 2024-09-15T03:47:43.908 INFO:tasks.rados.rados.0.smithi001.stdout:323: writing smithi00132172-43 from 3981312 to 4767744 tid 3 2024-09-15T03:47:43.908 INFO:tasks.rados.rados.0.smithi001.stdout:323: writing smithi00132172-43 from 4767744 to 4816896 tid 4 2024-09-15T03:47:43.908 INFO:tasks.rados.rados.0.smithi001.stdout:324: read oid 20 snap 0 2024-09-15T03:47:43.908 INFO:tasks.rados.rados.0.smithi001.stdout:324: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-15T03:47:43.908 INFO:tasks.rados.rados.0.smithi001.stdout:325: copy_from oid 35 from oid 28 current snap is 40 2024-09-15T03:47:43.909 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 21 v 48 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-15T03:47:43.909 INFO:tasks.rados.rados.0.smithi001.stdout:321: expect (ObjNum 47 snap 0 seq_num 47) 2024-09-15T03:47:43.909 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:43.909 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:43.910 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:43.910 INFO:tasks.rados.rados.0.smithi001.stdout:323: finishing write tid 1 to smithi00132172-43 2024-09-15T03:47:43.910 INFO:tasks.rados.rados.0.smithi001.stdout:323: finishing write tid 2 to smithi00132172-43 2024-09-15T03:47:43.910 INFO:tasks.rados.rados.0.smithi001.stdout:325: finishing copy_from to smithi00132172-35 2024-09-15T03:47:43.910 INFO:tasks.rados.rados.0.smithi001.stdout:325: got expected ENOENT (src dne) 2024-09-15T03:47:43.910 INFO:tasks.rados.rados.0.smithi001.stdout:325: finishing copy_from racing read to smithi00132172-35 2024-09-15T03:47:43.911 INFO:tasks.rados.rados.0.smithi001.stdout:323: finishing write tid 3 to smithi00132172-43 2024-09-15T03:47:43.911 INFO:tasks.rados.rados.0.smithi001.stdout:323: finishing write tid 4 to smithi00132172-43 2024-09-15T03:47:43.911 INFO:tasks.rados.rados.0.smithi001.stdout:323: finishing write tid 5 to smithi00132172-43 2024-09-15T03:47:43.911 INFO:tasks.rados.rados.0.smithi001.stdout:324: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-15T03:47:43.913 INFO:tasks.rados.rados.0.smithi001.stdout:323: finishing write tid 6 to smithi00132172-43 2024-09-15T03:47:43.913 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 43 v 87 (ObjNum 124 snap 40 seq_num 124) dirty exists 2024-09-15T03:47:43.913 INFO:tasks.rados.rados.0.smithi001.stdout:323: left oid 43 (ObjNum 124 snap 40 seq_num 124) 2024-09-15T03:47:43.913 INFO:tasks.rados.rados.0.smithi001.stdout:321: done (4 left) 2024-09-15T03:47:43.913 INFO:tasks.rados.rados.0.smithi001.stdout:322: done (3 left) 2024-09-15T03:47:43.913 INFO:tasks.rados.rados.0.smithi001.stdout:323: done (2 left) 2024-09-15T03:47:43.913 INFO:tasks.rados.rados.0.smithi001.stdout:324: done (1 left) 2024-09-15T03:47:43.913 INFO:tasks.rados.rados.0.smithi001.stdout:325: done (0 left) 2024-09-15T03:47:43.913 INFO:tasks.rados.rados.0.smithi001.stdout:326: copy_from oid 28 from oid 22 current snap is 40 2024-09-15T03:47:43.913 INFO:tasks.rados.rados.0.smithi001.stdout:327: read oid 43 snap -1 2024-09-15T03:47:43.913 INFO:tasks.rados.rados.0.smithi001.stdout:327: expect (ObjNum 124 snap 40 seq_num 124) 2024-09-15T03:47:43.913 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 42 current snap is 40 2024-09-15T03:47:43.913 INFO:tasks.rados.rados.0.smithi001.stdout:328: seq_num 125 ranges {0=557056,557056=475136,1032192=507904,1540096=475136,2015232=98304} 2024-09-15T03:47:43.917 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:43.918 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:43.918 INFO:tasks.rados.rados.0.smithi001.stdout:328: writing smithi00132172-42 from 0 to 557056 tid 1 2024-09-15T03:47:43.920 INFO:tasks.rados.rados.0.smithi001.stdout:328: writing smithi00132172-42 from 557056 to 1032192 tid 2 2024-09-15T03:47:43.924 INFO:tasks.rados.rados.0.smithi001.stdout:328: writing smithi00132172-42 from 1032192 to 1540096 tid 3 2024-09-15T03:47:43.929 INFO:tasks.rados.rados.0.smithi001.stdout:328: writing smithi00132172-42 from 1540096 to 2015232 tid 4 2024-09-15T03:47:43.930 INFO:tasks.rados.rados.0.smithi001.stdout:328: writing smithi00132172-42 from 2015232 to 2113536 tid 5 2024-09-15T03:47:43.930 INFO:tasks.rados.rados.0.smithi001.stdout:329: snap_create 2024-09-15T03:47:43.930 INFO:tasks.rados.rados.0.smithi001.stdout:328: finishing write tid 1 to smithi00132172-42 2024-09-15T03:47:43.930 INFO:tasks.rados.rados.0.smithi001.stdout:328: finishing write tid 2 to smithi00132172-42 2024-09-15T03:47:43.930 INFO:tasks.rados.rados.0.smithi001.stdout:328: finishing write tid 3 to smithi00132172-42 2024-09-15T03:47:43.936 INFO:tasks.rados.rados.0.smithi001.stdout:328: finishing write tid 4 to smithi00132172-42 2024-09-15T03:47:43.936 INFO:tasks.rados.rados.0.smithi001.stdout:328: finishing write tid 5 to smithi00132172-42 2024-09-15T03:47:43.937 INFO:tasks.rados.rados.0.smithi001.stdout:326: finishing copy_from racing read to smithi00132172-28 2024-09-15T03:47:43.937 INFO:tasks.rados.rados.0.smithi001.stdout:328: finishing write tid 6 to smithi00132172-42 2024-09-15T03:47:43.940 INFO:tasks.rados.rados.0.smithi001.stdout:328: finishing write tid 7 to smithi00132172-42 2024-09-15T03:47:43.940 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 42 v 40 (ObjNum 125 snap 40 seq_num 125) dirty exists 2024-09-15T03:47:43.940 INFO:tasks.rados.rados.0.smithi001.stdout:328: left oid 42 (ObjNum 125 snap 40 seq_num 125) 2024-09-15T03:47:43.940 INFO:tasks.rados.rados.0.smithi001.stdout:326: finishing copy_from to smithi00132172-28 2024-09-15T03:47:43.940 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 28 v 107 (ObjNum 112 snap 32 seq_num 112) dirty exists 2024-09-15T03:47:43.944 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:43.945 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:43.945 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-15T03:47:43.945 INFO:tasks.rados.rados.0.smithi001.stdout:327: expect (ObjNum 124 snap 40 seq_num 124) 2024-09-15T03:47:43.959 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:43.960 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:43.963 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T03:47:43.963 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T03:47:43.965 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:43.964+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:44.020 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:44.020 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:44.021 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:44.021 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:44.021 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:44.029 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:47:44.067 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:44.066+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:44.168 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:44.167+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:44.172 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:44.172 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:44.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:44.172 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:44.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.172 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs6 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:44.172 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108711+0000", 2024-09-15T03:47:44.172 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.061017234000005, 2024-09-15T03:47:44.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01472483, 2024-09-15T03:47:44.173 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.173 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.173 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108711+0000", 2024-09-15T03:47:44.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108711+0000", 2024-09-15T03:47:44.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108712+0000", 2024-09-15T03:47:44.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.85e-07 2024-09-15T03:47:44.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108745+0000", 2024-09-15T03:47:44.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2762000000000002e-05 2024-09-15T03:47:44.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108760+0000", 2024-09-15T03:47:44.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4567e-05 2024-09-15T03:47:44.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108780+0000", 2024-09-15T03:47:44.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0122000000000001e-05 2024-09-15T03:47:44.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108842+0000", 2024-09-15T03:47:44.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2297000000000003e-05 2024-09-15T03:47:44.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108851+0000", 2024-09-15T03:47:44.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0129999999999999e-06 2024-09-15T03:47:44.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108942+0000", 2024-09-15T03:47:44.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0713999999999999e-05 2024-09-15T03:47:44.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108949+0000", 2024-09-15T03:47:44.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1600000000000001e-06 2024-09-15T03:47:44.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109975+0000", 2024-09-15T03:47:44.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010264429999999999 2024-09-15T03:47:44.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109994+0000", 2024-09-15T03:47:44.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8374000000000001e-05 2024-09-15T03:47:44.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123436+0000", 2024-09-15T03:47:44.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013442592999999999 2024-09-15T03:47:44.178 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.178 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.178 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.178 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs6 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:44.178 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108809+0000", 2024-09-15T03:47:44.178 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.060919506000005, 2024-09-15T03:47:44.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014757497, 2024-09-15T03:47:44.178 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.178 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.179 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108809+0000", 2024-09-15T03:47:44.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108809+0000", 2024-09-15T03:47:44.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108810+0000", 2024-09-15T03:47:44.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.15e-07 2024-09-15T03:47:44.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108825+0000", 2024-09-15T03:47:44.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5092999999999999e-05 2024-09-15T03:47:44.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108835+0000", 2024-09-15T03:47:44.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0251000000000001e-05 2024-09-15T03:47:44.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108841+0000", 2024-09-15T03:47:44.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0510000000000002e-06 2024-09-15T03:47:44.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108971+0000", 2024-09-15T03:47:44.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012976599999999999 2024-09-15T03:47:44.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108976+0000", 2024-09-15T03:47:44.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8450000000000002e-06 2024-09-15T03:47:44.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109032+0000", 2024-09-15T03:47:44.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6119999999999998e-05 2024-09-15T03:47:44.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109978+0000", 2024-09-15T03:47:44.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00094603899999999997 2024-09-15T03:47:44.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110007+0000", 2024-09-15T03:47:44.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9062999999999999e-05 2024-09-15T03:47:44.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110025+0000", 2024-09-15T03:47:44.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7717999999999999e-05 2024-09-15T03:47:44.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123567+0000", 2024-09-15T03:47:44.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013542136 2024-09-15T03:47:44.184 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.184 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.184 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.184 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:324 3.7s0 3:e40d1694:::smithi00132172-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:44.185 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:44.185 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 48.325606071999999, 2024-09-15T03:47:44.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0031668960000002, 2024-09-15T03:47:44.185 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.185 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:44.185 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:44.185 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:44.185 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:44.185 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 324 2024-09-15T03:47:44.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.185 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:44.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:44.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:44.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:47:44.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844125+0000", 2024-09-15T03:47:44.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.142e-06 2024-09-15T03:47:44.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844132+0000", 2024-09-15T03:47:44.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0439999999999998e-06 2024-09-15T03:47:44.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844136+0000", 2024-09-15T03:47:44.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.126e-06 2024-09-15T03:47:44.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.847618+0000", 2024-09-15T03:47:44.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0034813790000000002 2024-09-15T03:47:44.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:44.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.847709+0000", 2024-09-15T03:47:44.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0835999999999996e-05 2024-09-15T03:47:44.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.849483+0000", 2024-09-15T03:47:44.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001774929 2024-09-15T03:47:44.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:44.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.849574+0000", 2024-09-15T03:47:44.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0847999999999995e-05 2024-09-15T03:47:44.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.847290+0000", 2024-09-15T03:47:44.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9977151819999999 2024-09-15T03:47:44.189 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.190 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.190 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.190 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:334 3.7s0 3:ece1472c:::smithi00132172-43:head [append 0~0] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:44.190 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:44.190 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 48.312878984000001, 2024-09-15T03:47:44.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.997592021, 2024-09-15T03:47:44.190 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.190 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:44.190 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:44.190 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:44.190 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:44.191 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 334 2024-09-15T03:47:44.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.191 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:44.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:44.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856851+0000", 2024-09-15T03:47:44.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.238e-06 2024-09-15T03:47:44.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856855+0000", 2024-09-15T03:47:44.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3640000000000001e-06 2024-09-15T03:47:44.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856868+0000", 2024-09-15T03:47:44.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2588999999999999e-05 2024-09-15T03:47:44.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856873+0000", 2024-09-15T03:47:44.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5350000000000004e-06 2024-09-15T03:47:44.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856916+0000", 2024-09-15T03:47:44.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2889000000000002e-05 2024-09-15T03:47:44.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:44.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856983+0000", 2024-09-15T03:47:44.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6746000000000001e-05 2024-09-15T03:47:44.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857448+0000", 2024-09-15T03:47:44.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046498500000000001 2024-09-15T03:47:44.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857720+0000", 2024-09-15T03:47:44.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027227600000000001 2024-09-15T03:47:44.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857741+0000", 2024-09-15T03:47:44.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1149999999999999e-05 2024-09-15T03:47:44.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857795+0000", 2024-09-15T03:47:44.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3919999999999999e-05 2024-09-15T03:47:44.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859582+0000", 2024-09-15T03:47:44.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017862640000000001 2024-09-15T03:47:44.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:44.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854347+0000", 2024-09-15T03:47:44.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9947654780000001 2024-09-15T03:47:44.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854442+0000", 2024-09-15T03:47:44.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4586999999999994e-05 2024-09-15T03:47:44.196 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.196 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.196 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.197 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:335 3.7s0 3:ece1472c:::smithi00132172-43:head [rollback 12] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:44.197 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:44.197 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 48.312848967000001, 2024-09-15T03:47:44.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9977859200000001, 2024-09-15T03:47:44.197 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.197 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:44.197 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:44.197 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:44.197 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:44.197 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 335 2024-09-15T03:47:44.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.198 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:44.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:44.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:44.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8099999999999999e-07 2024-09-15T03:47:44.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856882+0000", 2024-09-15T03:47:44.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.649e-06 2024-09-15T03:47:44.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856885+0000", 2024-09-15T03:47:44.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.664e-06 2024-09-15T03:47:44.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856888+0000", 2024-09-15T03:47:44.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3910000000000001e-06 2024-09-15T03:47:44.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857826+0000", 2024-09-15T03:47:44.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00093817900000000003 2024-09-15T03:47:44.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:44.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857940+0000", 2024-09-15T03:47:44.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114424 2024-09-15T03:47:44.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858504+0000", 2024-09-15T03:47:44.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056400899999999997 2024-09-15T03:47:44.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858593+0000", 2024-09-15T03:47:44.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8463999999999999e-05 2024-09-15T03:47:44.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858604+0000", 2024-09-15T03:47:44.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1567e-05 2024-09-15T03:47:44.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858642+0000", 2024-09-15T03:47:44.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7369000000000001e-05 2024-09-15T03:47:44.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859588+0000", 2024-09-15T03:47:44.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00094661799999999998 2024-09-15T03:47:44.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:44.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854594+0000", 2024-09-15T03:47:44.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9950055550000001 2024-09-15T03:47:44.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854666+0000", 2024-09-15T03:47:44.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1749999999999996e-05 2024-09-15T03:47:44.203 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.203 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.203 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.203 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:336 3.7s0 3:ece1472c:::smithi00132172-43:head [append 0~0] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:44.204 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:44.204 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 48.312817916999997, 2024-09-15T03:47:44.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9980017820000002, 2024-09-15T03:47:44.204 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.204 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:44.204 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:44.204 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:44.204 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:44.204 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 336 2024-09-15T03:47:44.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.204 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:44.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:44.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:44.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8000000000000002e-07 2024-09-15T03:47:44.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856913+0000", 2024-09-15T03:47:44.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5400000000000001e-06 2024-09-15T03:47:44.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856918+0000", 2024-09-15T03:47:44.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1540000000000001e-06 2024-09-15T03:47:44.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856921+0000", 2024-09-15T03:47:44.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2550000000000002e-06 2024-09-15T03:47:44.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858655+0000", 2024-09-15T03:47:44.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017336070000000001 2024-09-15T03:47:44.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:44.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858741+0000", 2024-09-15T03:47:44.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6305999999999997e-05 2024-09-15T03:47:44.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859193+0000", 2024-09-15T03:47:44.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045221200000000001 2024-09-15T03:47:44.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859319+0000", 2024-09-15T03:47:44.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012542900000000001 2024-09-15T03:47:44.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859331+0000", 2024-09-15T03:47:44.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2028000000000001e-05 2024-09-15T03:47:44.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859369+0000", 2024-09-15T03:47:44.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8522999999999997e-05 2024-09-15T03:47:44.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859649+0000", 2024-09-15T03:47:44.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028008200000000002 2024-09-15T03:47:44.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:44.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854800+0000", 2024-09-15T03:47:44.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9951506540000001 2024-09-15T03:47:44.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854913+0000", 2024-09-15T03:47:44.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112712 2024-09-15T03:47:44.210 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.210 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.210 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.210 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:346 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:44.210 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:47:44.211 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.331321547000002, 2024-09-15T03:47:44.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005631948, 2024-09-15T03:47:44.211 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.211 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:44.211 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:44.211 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:44.211 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:44.211 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 346 2024-09-15T03:47:44.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.211 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:47:44.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:47:44.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838410+0000", 2024-09-15T03:47:44.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4430000000000002e-06 2024-09-15T03:47:44.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838419+0000", 2024-09-15T03:47:44.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1730000000000001e-06 2024-09-15T03:47:44.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838444+0000", 2024-09-15T03:47:44.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5111000000000001e-05 2024-09-15T03:47:44.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838455+0000", 2024-09-15T03:47:44.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0839000000000001e-05 2024-09-15T03:47:44.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:44.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3415000000000002e-05 2024-09-15T03:47:44.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:44.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838527+0000", 2024-09-15T03:47:44.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0550000000000005e-06 2024-09-15T03:47:44.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840638+0000", 2024-09-15T03:47:44.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021105379999999999 2024-09-15T03:47:44.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:44.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840729+0000", 2024-09-15T03:47:44.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0977999999999995e-05 2024-09-15T03:47:44.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.841834+0000", 2024-09-15T03:47:44.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011055640000000001 2024-09-15T03:47:44.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842223+0000", 2024-09-15T03:47:44.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038899900000000001 2024-09-15T03:47:44.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842241+0000", 2024-09-15T03:47:44.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7317e-05 2024-09-15T03:47:44.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842298+0000", 2024-09-15T03:47:44.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7825000000000003e-05 2024-09-15T03:47:44.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844866+0000", 2024-09-15T03:47:44.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025675289999999998 2024-09-15T03:47:44.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:44.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843973+0000", 2024-09-15T03:47:44.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99910710599999997 2024-09-15T03:47:44.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844039+0000", 2024-09-15T03:47:44.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6056000000000006e-05 2024-09-15T03:47:44.218 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.218 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.218 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.218 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:347 3.2s0 3:48ceb02f:::smithi00132172-5:head [rollback c] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:44.218 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:47:44.218 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.331236750999999, 2024-09-15T03:47:44.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005723841, 2024-09-15T03:47:44.218 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.218 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:44.218 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:44.218 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 347 2024-09-15T03:47:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:47:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:47:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838493+0000", 2024-09-15T03:47:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7300000000000005e-07 2024-09-15T03:47:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838498+0000", 2024-09-15T03:47:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9300000000000002e-06 2024-09-15T03:47:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838508+0000", 2024-09-15T03:47:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9790000000000005e-06 2024-09-15T03:47:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838513+0000", 2024-09-15T03:47:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8890000000000001e-06 2024-09-15T03:47:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838583+0000", 2024-09-15T03:47:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0696000000000002e-05 2024-09-15T03:47:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838590+0000", 2024-09-15T03:47:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2990000000000002e-06 2024-09-15T03:47:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842313+0000", 2024-09-15T03:47:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037238810000000001 2024-09-15T03:47:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842432+0000", 2024-09-15T03:47:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118474 2024-09-15T03:47:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843108+0000", 2024-09-15T03:47:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00067639299999999996 2024-09-15T03:47:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843227+0000", 2024-09-15T03:47:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119092 2024-09-15T03:47:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843242+0000", 2024-09-15T03:47:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4216000000000001e-05 2024-09-15T03:47:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843292+0000", 2024-09-15T03:47:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0800999999999997e-05 2024-09-15T03:47:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844878+0000", 2024-09-15T03:47:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015852749999999999 2024-09-15T03:47:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844147+0000", 2024-09-15T03:47:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99926949200000004 2024-09-15T03:47:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844216+0000", 2024-09-15T03:47:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8551000000000001e-05 2024-09-15T03:47:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:348 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.331210808000002, 2024-09-15T03:47:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.006019768, 2024-09-15T03:47:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 348 2024-09-15T03:47:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8700000000000001e-07 2024-09-15T03:47:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838521+0000", 2024-09-15T03:47:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3740000000000001e-06 2024-09-15T03:47:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838527+0000", 2024-09-15T03:47:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1999999999999999e-06 2024-09-15T03:47:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838532+0000", 2024-09-15T03:47:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2059999999999998e-06 2024-09-15T03:47:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838609+0000", 2024-09-15T03:47:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7193000000000004e-05 2024-09-15T03:47:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838615+0000", 2024-09-15T03:47:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7980000000000002e-06 2024-09-15T03:47:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843986+0000", 2024-09-15T03:47:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053706450000000003 2024-09-15T03:47:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844106+0000", 2024-09-15T03:47:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120079 2024-09-15T03:47:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844669+0000", 2024-09-15T03:47:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056290300000000001 2024-09-15T03:47:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844804+0000", 2024-09-15T03:47:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013575600000000001 2024-09-15T03:47:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844820+0000", 2024-09-15T03:47:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5145e-05 2024-09-15T03:47:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844893+0000", 2024-09-15T03:47:44.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3270999999999997e-05 2024-09-15T03:47:44.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845043+0000", 2024-09-15T03:47:44.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150144 2024-09-15T03:47:44.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:44.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844473+0000", 2024-09-15T03:47:44.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99943048700000003 2024-09-15T03:47:44.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844538+0000", 2024-09-15T03:47:44.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4179999999999999e-05 2024-09-15T03:47:44.233 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.233 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.233 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.233 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:352 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~557056] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:44.233 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:47:44.233 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.325013163999998, 2024-09-15T03:47:44.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.016265065, 2024-09-15T03:47:44.234 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.234 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:44.234 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:44.234 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:44.234 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:44.234 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 352 2024-09-15T03:47:44.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.234 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:47:44.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:47:44.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844718+0000", 2024-09-15T03:47:44.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2340000000000001e-06 2024-09-15T03:47:44.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845237+0000", 2024-09-15T03:47:44.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051953399999999999 2024-09-15T03:47:44.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845273+0000", 2024-09-15T03:47:44.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5683999999999999e-05 2024-09-15T03:47:44.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845281+0000", 2024-09-15T03:47:44.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8439999999999998e-06 2024-09-15T03:47:44.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845365+0000", 2024-09-15T03:47:44.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4651e-05 2024-09-15T03:47:44.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:44.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845486+0000", 2024-09-15T03:47:44.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012009499999999999 2024-09-15T03:47:44.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847101+0000", 2024-09-15T03:47:44.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016157350000000001 2024-09-15T03:47:44.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847602+0000", 2024-09-15T03:47:44.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050066200000000003 2024-09-15T03:47:44.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847622+0000", 2024-09-15T03:47:44.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9749999999999999e-05 2024-09-15T03:47:44.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848616+0000", 2024-09-15T03:47:44.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099390799999999994 2024-09-15T03:47:44.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848709+0000", 2024-09-15T03:47:44.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2874000000000006e-05 2024-09-15T03:47:44.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:44.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.860888+0000", 2024-09-15T03:47:44.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.012179707 2024-09-15T03:47:44.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.860981+0000", 2024-09-15T03:47:44.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2386999999999995e-05 2024-09-15T03:47:44.240 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.240 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.240 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.240 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:353 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~671744] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:44.240 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:47:44.240 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.321450970999997, 2024-09-15T03:47:44.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.012912206, 2024-09-15T03:47:44.240 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.240 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:44.240 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:44.241 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:44.241 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:44.241 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 353 2024-09-15T03:47:44.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.241 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:47:44.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:47:44.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848279+0000", 2024-09-15T03:47:44.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.362e-06 2024-09-15T03:47:44.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849024+0000", 2024-09-15T03:47:44.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00074495500000000005 2024-09-15T03:47:44.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849039+0000", 2024-09-15T03:47:44.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5119e-05 2024-09-15T03:47:44.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849046+0000", 2024-09-15T03:47:44.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6359999999999997e-06 2024-09-15T03:47:44.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849115+0000", 2024-09-15T03:47:44.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8998999999999994e-05 2024-09-15T03:47:44.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:44.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849251+0000", 2024-09-15T03:47:44.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013589799999999999 2024-09-15T03:47:44.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850605+0000", 2024-09-15T03:47:44.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013544659999999999 2024-09-15T03:47:44.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850703+0000", 2024-09-15T03:47:44.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7576000000000002e-05 2024-09-15T03:47:44.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850719+0000", 2024-09-15T03:47:44.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5767e-05 2024-09-15T03:47:44.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851562+0000", 2024-09-15T03:47:44.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084361899999999999 2024-09-15T03:47:44.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851705+0000", 2024-09-15T03:47:44.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000142893 2024-09-15T03:47:44.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:44.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861112+0000", 2024-09-15T03:47:44.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.009407349 2024-09-15T03:47:44.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861190+0000", 2024-09-15T03:47:44.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7566999999999998e-05 2024-09-15T03:47:44.246 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.246 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.247 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.247 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:354 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~98304] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:44.247 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:47:44.247 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.320648534999997, 2024-09-15T03:47:44.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0177595669999999, 2024-09-15T03:47:44.247 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.247 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:44.247 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:44.247 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:44.247 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:44.247 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 354 2024-09-15T03:47:44.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.248 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:47:44.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:47:44.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849081+0000", 2024-09-15T03:47:44.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0610000000000001e-06 2024-09-15T03:47:44.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849157+0000", 2024-09-15T03:47:44.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5804999999999997e-05 2024-09-15T03:47:44.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849168+0000", 2024-09-15T03:47:44.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0730000000000001e-05 2024-09-15T03:47:44.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849174+0000", 2024-09-15T03:47:44.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3609999999999997e-06 2024-09-15T03:47:44.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850814+0000", 2024-09-15T03:47:44.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001640124 2024-09-15T03:47:44.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:44.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850894+0000", 2024-09-15T03:47:44.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0167999999999995e-05 2024-09-15T03:47:44.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851406+0000", 2024-09-15T03:47:44.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051203200000000005 2024-09-15T03:47:44.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851506+0000", 2024-09-15T03:47:44.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9457999999999995e-05 2024-09-15T03:47:44.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851574+0000", 2024-09-15T03:47:44.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8145000000000006e-05 2024-09-15T03:47:44.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851632+0000", 2024-09-15T03:47:44.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8108e-05 2024-09-15T03:47:44.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851712+0000", 2024-09-15T03:47:44.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9380000000000002e-05 2024-09-15T03:47:44.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:44.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.866753+0000", 2024-09-15T03:47:44.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0150417979999999 2024-09-15T03:47:44.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.866840+0000", 2024-09-15T03:47:44.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6397000000000006e-05 2024-09-15T03:47:44.253 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.253 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.253 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.254 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:355 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [setxattr _header (57) in=7b] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:44.254 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:47:44.254 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.320519222000001, 2024-09-15T03:47:44.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.018024966, 2024-09-15T03:47:44.254 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.254 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:44.254 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:44.254 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:44.254 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:44.254 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 355 2024-09-15T03:47:44.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.254 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:47:44.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:47:44.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849210+0000", 2024-09-15T03:47:44.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.33e-07 2024-09-15T03:47:44.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849215+0000", 2024-09-15T03:47:44.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4379999999999997e-06 2024-09-15T03:47:44.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849234+0000", 2024-09-15T03:47:44.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9409999999999999e-05 2024-09-15T03:47:44.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849241+0000", 2024-09-15T03:47:44.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6370000000000002e-06 2024-09-15T03:47:44.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851823+0000", 2024-09-15T03:47:44.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025817689999999998 2024-09-15T03:47:44.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:44.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851934+0000", 2024-09-15T03:47:44.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111357 2024-09-15T03:47:44.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852466+0000", 2024-09-15T03:47:44.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053248100000000001 2024-09-15T03:47:44.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852580+0000", 2024-09-15T03:47:44.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011379699999999999 2024-09-15T03:47:44.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852607+0000", 2024-09-15T03:47:44.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279e-05 2024-09-15T03:47:44.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852693+0000", 2024-09-15T03:47:44.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6792999999999994e-05 2024-09-15T03:47:44.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852869+0000", 2024-09-15T03:47:44.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017537499999999999 2024-09-15T03:47:44.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:44.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867006+0000", 2024-09-15T03:47:44.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.014136996 2024-09-15T03:47:44.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867234+0000", 2024-09-15T03:47:44.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022870099999999999 2024-09-15T03:47:44.260 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.260 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.260 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.260 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:356 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:44.260 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:47:44.261 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.320482458999997, 2024-09-15T03:47:44.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.020403521, 2024-09-15T03:47:44.261 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.261 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:44.261 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:44.261 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:44.261 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:44.261 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 356 2024-09-15T03:47:44.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.262 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:47:44.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:47:44.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849247+0000", 2024-09-15T03:47:44.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2699999999999999e-07 2024-09-15T03:47:44.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849250+0000", 2024-09-15T03:47:44.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2610000000000002e-06 2024-09-15T03:47:44.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849257+0000", 2024-09-15T03:47:44.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.703e-06 2024-09-15T03:47:44.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849261+0000", 2024-09-15T03:47:44.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7910000000000002e-06 2024-09-15T03:47:44.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852672+0000", 2024-09-15T03:47:44.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003411627 2024-09-15T03:47:44.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:44.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852801+0000", 2024-09-15T03:47:44.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000128918 2024-09-15T03:47:44.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867347+0000", 2024-09-15T03:47:44.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0145458380000001 2024-09-15T03:47:44.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:44.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867476+0000", 2024-09-15T03:47:44.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012915100000000001 2024-09-15T03:47:44.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.869650+0000", 2024-09-15T03:47:44.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021737050000000002 2024-09-15T03:47:44.266 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.266 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.266 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.267 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/64 ECSubWrite(tid=108, reqid=client.4521.0:466, at_version=65'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:44.267 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.856519+0000", 2024-09-15T03:47:44.267 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.313210112, 2024-09-15T03:47:44.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015639281000000001, 2024-09-15T03:47:44.267 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.267 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.267 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856519+0000", 2024-09-15T03:47:44.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856519+0000", 2024-09-15T03:47:44.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856519+0000", 2024-09-15T03:47:44.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4200000000000005e-07 2024-09-15T03:47:44.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857060+0000", 2024-09-15T03:47:44.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054053200000000004 2024-09-15T03:47:44.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857069+0000", 2024-09-15T03:47:44.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2979999999999997e-06 2024-09-15T03:47:44.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857074+0000", 2024-09-15T03:47:44.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7570000000000004e-06 2024-09-15T03:47:44.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857122+0000", 2024-09-15T03:47:44.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7828999999999999e-05 2024-09-15T03:47:44.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857128+0000", 2024-09-15T03:47:44.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6409999999999996e-06 2024-09-15T03:47:44.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857491+0000", 2024-09-15T03:47:44.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036239300000000003 2024-09-15T03:47:44.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857503+0000", 2024-09-15T03:47:44.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2468e-05 2024-09-15T03:47:44.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857718+0000", 2024-09-15T03:47:44.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021492200000000001 2024-09-15T03:47:44.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857745+0000", 2024-09-15T03:47:44.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7189e-05 2024-09-15T03:47:44.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.872158+0000", 2024-09-15T03:47:44.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01441241 2024-09-15T03:47:44.272 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.272 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.272 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.272 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:527 3.7s0 3:e4b8e19d:::smithi00132172-41:head [copy-from ver 51] snapc 23=[23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:44.272 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:47:44.273 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.220913497000002, 2024-09-15T03:47:44.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030362388000000001, 2024-09-15T03:47:44.273 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.273 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:44.273 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:44.273 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:44.273 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:44.273 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 527 2024-09-15T03:47:44.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.273 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:47:44.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:47:44.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948817+0000", 2024-09-15T03:47:44.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3230000000000001e-06 2024-09-15T03:47:44.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948825+0000", 2024-09-15T03:47:44.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5900000000000008e-06 2024-09-15T03:47:44.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948838+0000", 2024-09-15T03:47:44.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2712e-05 2024-09-15T03:47:44.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948842+0000", 2024-09-15T03:47:44.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4170000000000002e-06 2024-09-15T03:47:44.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948923+0000", 2024-09-15T03:47:44.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0686000000000006e-05 2024-09-15T03:47:44.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:47:44.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949020+0000", 2024-09-15T03:47:44.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6984999999999994e-05 2024-09-15T03:47:44.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954390+0000", 2024-09-15T03:47:44.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053704790000000001 2024-09-15T03:47:44.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:44.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954558+0000", 2024-09-15T03:47:44.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000167202 2024-09-15T03:47:44.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.973998+0000", 2024-09-15T03:47:44.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019440316999999999 2024-09-15T03:47:44.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974527+0000", 2024-09-15T03:47:44.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052943800000000002 2024-09-15T03:47:44.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974541+0000", 2024-09-15T03:47:44.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3774999999999999e-05 2024-09-15T03:47:44.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974841+0000", 2024-09-15T03:47:44.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029942899999999998 2024-09-15T03:47:44.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.976067+0000", 2024-09-15T03:47:44.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012265399999999999 2024-09-15T03:47:44.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:44.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.978952+0000", 2024-09-15T03:47:44.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028848839999999999 2024-09-15T03:47:44.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.979178+0000", 2024-09-15T03:47:44.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022561099999999999 2024-09-15T03:47:44.280 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.280 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.280 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.280 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:528 3.7s0 3:e4b8e19d:::smithi00132172-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:44.280 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:44.280 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.220879088, 2024-09-15T03:47:44.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026927036000000001, 2024-09-15T03:47:44.280 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.280 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:44.281 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:44.281 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:44.281 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:44.281 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 528 2024-09-15T03:47:44.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.281 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:44.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:44.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:44.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9700000000000003e-07 2024-09-15T03:47:44.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948852+0000", 2024-09-15T03:47:44.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.604e-06 2024-09-15T03:47:44.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948855+0000", 2024-09-15T03:47:44.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9369999999999997e-06 2024-09-15T03:47:44.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948858+0000", 2024-09-15T03:47:44.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2709999999999999e-06 2024-09-15T03:47:44.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949080+0000", 2024-09-15T03:47:44.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000222148 2024-09-15T03:47:44.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:47:44.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949156+0000", 2024-09-15T03:47:44.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6215000000000001e-05 2024-09-15T03:47:44.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954754+0000", 2024-09-15T03:47:44.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055978060000000003 2024-09-15T03:47:44.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:44.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954858+0000", 2024-09-15T03:47:44.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010419199999999999 2024-09-15T03:47:44.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974861+0000", 2024-09-15T03:47:44.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020002671999999999 2024-09-15T03:47:44.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:44.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974987+0000", 2024-09-15T03:47:44.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012638699999999999 2024-09-15T03:47:44.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.975777+0000", 2024-09-15T03:47:44.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00078950699999999995 2024-09-15T03:47:44.287 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.287 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.287 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.287 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:742 3.7s0 3:ece1472c:::smithi00132172-43:head [append 0~753664] snapc 32=[32,2d,2b,26,25,24,23,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T03:47:44.287 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.930340+0000", 2024-09-15T03:47:44.287 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.2393890059999997, 2024-09-15T03:47:44.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013978454, 2024-09-15T03:47:44.287 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.287 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:44.287 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:44.287 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:44.287 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:44.288 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 742 2024-09-15T03:47:44.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.288 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.930340+0000", 2024-09-15T03:47:44.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.930340+0000", 2024-09-15T03:47:44.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.930341+0000", 2024-09-15T03:47:44.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6759999999999999e-06 2024-09-15T03:47:44.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.931144+0000", 2024-09-15T03:47:44.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00080309900000000004 2024-09-15T03:47:44.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.931157+0000", 2024-09-15T03:47:44.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2397e-05 2024-09-15T03:47:44.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.931163+0000", 2024-09-15T03:47:44.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6790000000000002e-06 2024-09-15T03:47:44.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.931205+0000", 2024-09-15T03:47:44.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2539e-05 2024-09-15T03:47:44.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:44.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.931296+0000", 2024-09-15T03:47:44.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1080000000000002e-05 2024-09-15T03:47:44.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936342+0000", 2024-09-15T03:47:44.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050460119999999999 2024-09-15T03:47:44.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936498+0000", 2024-09-15T03:47:44.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015619200000000001 2024-09-15T03:47:44.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.937026+0000", 2024-09-15T03:47:44.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052732899999999999 2024-09-15T03:47:44.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.937498+0000", 2024-09-15T03:47:44.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047273200000000001 2024-09-15T03:47:44.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.940834+0000", 2024-09-15T03:47:44.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003335916 2024-09-15T03:47:44.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:44.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.943811+0000", 2024-09-15T03:47:44.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029768139999999999 2024-09-15T03:47:44.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.944318+0000", 2024-09-15T03:47:44.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050698900000000001 2024-09-15T03:47:44.294 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.294 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.294 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.294 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:833 3.7s0 3:ece1472c:::smithi00132172-43:head [read 0~4816896,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e97)", 2024-09-15T03:47:44.294 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:43.913147+0000", 2024-09-15T03:47:44.294 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.25658197300000002, 2024-09-15T03:47:44.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014828559999999999, 2024-09-15T03:47:44.294 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.295 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:44.295 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:44.295 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:44.295 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:44.295 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 833 2024-09-15T03:47:44.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.295 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913147+0000", 2024-09-15T03:47:44.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913147+0000", 2024-09-15T03:47:44.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913147+0000", 2024-09-15T03:47:44.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7000000000000001e-07 2024-09-15T03:47:44.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913149+0000", 2024-09-15T03:47:44.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8929999999999999e-06 2024-09-15T03:47:44.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913152+0000", 2024-09-15T03:47:44.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.472e-06 2024-09-15T03:47:44.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913154+0000", 2024-09-15T03:47:44.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0940000000000002e-06 2024-09-15T03:47:44.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913477+0000", 2024-09-15T03:47:44.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032283000000000002 2024-09-15T03:47:44.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:44.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913585+0000", 2024-09-15T03:47:44.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010744899999999999 2024-09-15T03:47:44.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.927975+0000", 2024-09-15T03:47:44.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014390551999999999 2024-09-15T03:47:44.298 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.299 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.299 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.299 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:834 3.7s0 3:ece1472c:::smithi00132172-43:head [read 0~4816896,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e97)", 2024-09-15T03:47:44.299 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:43.913171+0000", 2024-09-15T03:47:44.299 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.256558171, 2024-09-15T03:47:44.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017820687000000002, 2024-09-15T03:47:44.299 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.299 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:44.299 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:44.299 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:44.299 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:44.299 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 834 2024-09-15T03:47:44.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.300 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913171+0000", 2024-09-15T03:47:44.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913171+0000", 2024-09-15T03:47:44.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913171+0000", 2024-09-15T03:47:44.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6e-07 2024-09-15T03:47:44.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913173+0000", 2024-09-15T03:47:44.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7719999999999999e-06 2024-09-15T03:47:44.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913176+0000", 2024-09-15T03:47:44.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8020000000000002e-06 2024-09-15T03:47:44.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913180+0000", 2024-09-15T03:47:44.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.692e-06 2024-09-15T03:47:44.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.914761+0000", 2024-09-15T03:47:44.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001580936 2024-09-15T03:47:44.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:44.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.914879+0000", 2024-09-15T03:47:44.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117518 2024-09-15T03:47:44.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.930991+0000", 2024-09-15T03:47:44.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016112707 2024-09-15T03:47:44.303 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.303 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.303 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.303 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.303 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.303 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:44.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:47:44.306 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:44.304+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:44.406 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:44.406+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:44.407 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:44.407 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:44.407 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:44.407 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:44.415 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:47:44.507 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:44.506+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:44.515 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:44.515 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:44.515 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:44.515 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:44.515 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:44.524 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:47:44.608 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:44.607+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:44.627 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:44.627 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:44.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:44.628 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:44.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.628 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 38/28 ECSubWrite(tid=33, reqid=client.4521.0:222, at_version=38'14, trim_to=0'0, roll_forward_to=35'12))", 2024-09-15T03:47:44.628 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:47:44.629 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.493858441, 2024-09-15T03:47:44.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9949187450000001, 2024-09-15T03:47:44.629 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.629 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.629 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:47:44.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:47:44.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130996+0000", 2024-09-15T03:47:44.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8809999999999998e-06 2024-09-15T03:47:44.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131020+0000", 2024-09-15T03:47:44.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4190999999999999e-05 2024-09-15T03:47:44.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131039+0000", 2024-09-15T03:47:44.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9228999999999998e-05 2024-09-15T03:47:44.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131048+0000", 2024-09-15T03:47:44.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2380000000000003e-06 2024-09-15T03:47:44.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131122+0000", 2024-09-15T03:47:44.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3146999999999996e-05 2024-09-15T03:47:44.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131132+0000", 2024-09-15T03:47:44.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.08e-05 2024-09-15T03:47:44.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131381+0000", 2024-09-15T03:47:44.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000248936 2024-09-15T03:47:44.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131392+0000", 2024-09-15T03:47:44.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0987e-05 2024-09-15T03:47:44.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131444+0000", 2024-09-15T03:47:44.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1585000000000002e-05 2024-09-15T03:47:44.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131465+0000", 2024-09-15T03:47:44.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1365999999999998e-05 2024-09-15T03:47:44.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.125911+0000", 2024-09-15T03:47:44.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9944453850000001 2024-09-15T03:47:44.642 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.642 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.642 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.642 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 38/28 ECSubWrite(tid=34, reqid=client.4521.0:222, at_version=0'0, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T03:47:44.643 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:47:44.643 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.492459341999997, 2024-09-15T03:47:44.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.002752697, 2024-09-15T03:47:44.643 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.643 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.643 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:47:44.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:47:44.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132392+0000", 2024-09-15T03:47:44.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5899999999999995e-07 2024-09-15T03:47:44.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132415+0000", 2024-09-15T03:47:44.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3561e-05 2024-09-15T03:47:44.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132436+0000", 2024-09-15T03:47:44.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0879000000000001e-05 2024-09-15T03:47:44.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132444+0000", 2024-09-15T03:47:44.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9200000000000004e-06 2024-09-15T03:47:44.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132497+0000", 2024-09-15T03:47:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2299000000000002e-05 2024-09-15T03:47:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132508+0000", 2024-09-15T03:47:44.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1100999999999999e-05 2024-09-15T03:47:44.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132615+0000", 2024-09-15T03:47:44.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107546 2024-09-15T03:47:44.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132635+0000", 2024-09-15T03:47:44.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9374000000000001e-05 2024-09-15T03:47:44.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132677+0000", 2024-09-15T03:47:44.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2456999999999997e-05 2024-09-15T03:47:44.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132711+0000", 2024-09-15T03:47:44.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4118000000000001e-05 2024-09-15T03:47:44.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135144+0000", 2024-09-15T03:47:44.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.002432583 2024-09-15T03:47:44.654 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.654 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.655 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.655 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=45, reqid=client.4521.0:227, at_version=38'24, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:47:44.655 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:44.655 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.469532800000003, 2024-09-15T03:47:44.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.974765339, 2024-09-15T03:47:44.656 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.656 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.656 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:44.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:44.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:44.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.43e-07 2024-09-15T03:47:44.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156110+0000", 2024-09-15T03:47:44.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000792409 2024-09-15T03:47:44.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156139+0000", 2024-09-15T03:47:44.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9207e-05 2024-09-15T03:47:44.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156148+0000", 2024-09-15T03:47:44.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1269999999999992e-06 2024-09-15T03:47:44.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156248+0000", 2024-09-15T03:47:44.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9421999999999998e-05 2024-09-15T03:47:44.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156258+0000", 2024-09-15T03:47:44.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6989999999999998e-06 2024-09-15T03:47:44.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156674+0000", 2024-09-15T03:47:44.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041682600000000002 2024-09-15T03:47:44.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156705+0000", 2024-09-15T03:47:44.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0525000000000003e-05 2024-09-15T03:47:44.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157742+0000", 2024-09-15T03:47:44.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010374 2024-09-15T03:47:44.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157777+0000", 2024-09-15T03:47:44.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4878e-05 2024-09-15T03:47:44.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.130083+0000", 2024-09-15T03:47:44.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972305703 2024-09-15T03:47:44.669 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.669 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.669 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.670 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=37, reqid=client.4521.0:236, at_version=38'5, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:44.670 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:44.670 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.467919344000002, 2024-09-15T03:47:44.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9740836640000001, 2024-09-15T03:47:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:44.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:44.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1300000000000001e-07 2024-09-15T03:47:44.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156953+0000", 2024-09-15T03:47:44.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1585000000000001e-05 2024-09-15T03:47:44.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156974+0000", 2024-09-15T03:47:44.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0764e-05 2024-09-15T03:47:44.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156981+0000", 2024-09-15T03:47:44.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2760000000000003e-06 2024-09-15T03:47:44.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157061+0000", 2024-09-15T03:47:44.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9583e-05 2024-09-15T03:47:44.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157075+0000", 2024-09-15T03:47:44.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4019000000000001e-05 2024-09-15T03:47:44.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157351+0000", 2024-09-15T03:47:44.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027641100000000001 2024-09-15T03:47:44.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157746+0000", 2024-09-15T03:47:44.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039553199999999998 2024-09-15T03:47:44.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157838+0000", 2024-09-15T03:47:44.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1632000000000006e-05 2024-09-15T03:47:44.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157856+0000", 2024-09-15T03:47:44.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7873999999999999e-05 2024-09-15T03:47:44.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.131015+0000", 2024-09-15T03:47:44.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9731586750000001 2024-09-15T03:47:44.682 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.682 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.682 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.683 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=46, reqid=client.4521.0:228, at_version=38'25, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:47:44.683 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:44.683 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.467860561999998, 2024-09-15T03:47:44.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9732838510000001, 2024-09-15T03:47:44.684 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.684 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.684 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:44.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:44.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:44.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0999999999999998e-07 2024-09-15T03:47:44.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157149+0000", 2024-09-15T03:47:44.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000158763 2024-09-15T03:47:44.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157165+0000", 2024-09-15T03:47:44.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5883999999999999e-05 2024-09-15T03:47:44.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157175+0000", 2024-09-15T03:47:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4949999999999997e-06 2024-09-15T03:47:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157219+0000", 2024-09-15T03:47:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4069000000000001e-05 2024-09-15T03:47:44.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157228+0000", 2024-09-15T03:47:44.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7829999999999993e-06 2024-09-15T03:47:44.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157345+0000", 2024-09-15T03:47:44.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116655 2024-09-15T03:47:44.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157746+0000", 2024-09-15T03:47:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040045100000000002 2024-09-15T03:47:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157838+0000", 2024-09-15T03:47:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1955999999999998e-05 2024-09-15T03:47:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157888+0000", 2024-09-15T03:47:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0943000000000003e-05 2024-09-15T03:47:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.130274+0000", 2024-09-15T03:47:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972385142 2024-09-15T03:47:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=38, reqid=client.4521.0:237, at_version=38'6, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:47:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.466517992, 2024-09-15T03:47:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9764011529999999, 2024-09-15T03:47:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:47:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:47:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158333+0000", 2024-09-15T03:47:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8599999999999997e-07 2024-09-15T03:47:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158353+0000", 2024-09-15T03:47:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9504000000000002e-05 2024-09-15T03:47:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158368+0000", 2024-09-15T03:47:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4966999999999999e-05 2024-09-15T03:47:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158374+0000", 2024-09-15T03:47:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.798e-06 2024-09-15T03:47:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158430+0000", 2024-09-15T03:47:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5744e-05 2024-09-15T03:47:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158440+0000", 2024-09-15T03:47:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4560000000000006e-06 2024-09-15T03:47:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158529+0000", 2024-09-15T03:47:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9172999999999995e-05 2024-09-15T03:47:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158543+0000", 2024-09-15T03:47:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.395e-05 2024-09-15T03:47:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158654+0000", 2024-09-15T03:47:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011110700000000001 2024-09-15T03:47:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158671+0000", 2024-09-15T03:47:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7068999999999999e-05 2024-09-15T03:47:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134733+0000", 2024-09-15T03:47:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.976062599 2024-09-15T03:47:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=47, reqid=client.4521.0:229, at_version=38'26, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:47:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:47:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.466514029999999, 2024-09-15T03:47:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9727810969999999, 2024-09-15T03:47:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:47:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:47:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158337+0000", 2024-09-15T03:47:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.15e-07 2024-09-15T03:47:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158351+0000", 2024-09-15T03:47:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3871e-05 2024-09-15T03:47:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158372+0000", 2024-09-15T03:47:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1058999999999999e-05 2024-09-15T03:47:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158376+0000", 2024-09-15T03:47:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5249999999999999e-06 2024-09-15T03:47:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158420+0000", 2024-09-15T03:47:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3918000000000002e-05 2024-09-15T03:47:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158429+0000", 2024-09-15T03:47:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5150000000000006e-06 2024-09-15T03:47:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158514+0000", 2024-09-15T03:47:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5252000000000003e-05 2024-09-15T03:47:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158542+0000", 2024-09-15T03:47:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.798e-05 2024-09-15T03:47:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158653+0000", 2024-09-15T03:47:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111315 2024-09-15T03:47:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158701+0000", 2024-09-15T03:47:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8241000000000001e-05 2024-09-15T03:47:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.131117+0000", 2024-09-15T03:47:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972416006 2024-09-15T03:47:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=39, reqid=client.4521.0:238, at_version=38'7, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:47:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.465826874000001, 2024-09-15T03:47:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9758214509999998, 2024-09-15T03:47:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:47:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:47:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159024+0000", 2024-09-15T03:47:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6700000000000003e-07 2024-09-15T03:47:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159096+0000", 2024-09-15T03:47:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1386999999999999e-05 2024-09-15T03:47:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159113+0000", 2024-09-15T03:47:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7020999999999998e-05 2024-09-15T03:47:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159122+0000", 2024-09-15T03:47:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4720000000000001e-06 2024-09-15T03:47:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159168+0000", 2024-09-15T03:47:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5766000000000002e-05 2024-09-15T03:47:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159176+0000", 2024-09-15T03:47:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3070000000000003e-06 2024-09-15T03:47:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159258+0000", 2024-09-15T03:47:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1398999999999998e-05 2024-09-15T03:47:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159292+0000", 2024-09-15T03:47:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4480000000000002e-05 2024-09-15T03:47:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159342+0000", 2024-09-15T03:47:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9450999999999998e-05 2024-09-15T03:47:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159362+0000", 2024-09-15T03:47:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0633e-05 2024-09-15T03:47:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134845+0000", 2024-09-15T03:47:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.975482768 2024-09-15T03:47:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=48, reqid=client.4521.0:229, at_version=0'0, trim_to=0'0, roll_forward_to=38'26))", 2024-09-15T03:47:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:47:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.464460078000002, 2024-09-15T03:47:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.974485472, 2024-09-15T03:47:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:47:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:47:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160391+0000", 2024-09-15T03:47:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.136e-06 2024-09-15T03:47:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160405+0000", 2024-09-15T03:47:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3613e-05 2024-09-15T03:47:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160426+0000", 2024-09-15T03:47:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1234999999999999e-05 2024-09-15T03:47:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160434+0000", 2024-09-15T03:47:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.61e-06 2024-09-15T03:47:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160496+0000", 2024-09-15T03:47:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2384000000000004e-05 2024-09-15T03:47:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160507+0000", 2024-09-15T03:47:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1005000000000001e-05 2024-09-15T03:47:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160632+0000", 2024-09-15T03:47:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124566 2024-09-15T03:47:44.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160653+0000", 2024-09-15T03:47:44.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0707999999999999e-05 2024-09-15T03:47:44.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160701+0000", 2024-09-15T03:47:44.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8751999999999997e-05 2024-09-15T03:47:44.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160739+0000", 2024-09-15T03:47:44.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7486e-05 2024-09-15T03:47:44.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134876+0000", 2024-09-15T03:47:44.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9741369770000001 2024-09-15T03:47:44.718 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.718 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.718 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.719 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=40, reqid=client.4521.0:238, at_version=0'0, trim_to=0'0, roll_forward_to=38'7))", 2024-09-15T03:47:44.719 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:47:44.719 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.462613656999999, 2024-09-15T03:47:44.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9727862730000001, 2024-09-15T03:47:44.719 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.719 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.719 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:47:44.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:47:44.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162238+0000", 2024-09-15T03:47:44.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.672e-06 2024-09-15T03:47:44.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162253+0000", 2024-09-15T03:47:44.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4932e-05 2024-09-15T03:47:44.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162284+0000", 2024-09-15T03:47:44.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1211000000000003e-05 2024-09-15T03:47:44.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162296+0000", 2024-09-15T03:47:44.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1229999999999999e-05 2024-09-15T03:47:44.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162373+0000", 2024-09-15T03:47:44.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7146999999999998e-05 2024-09-15T03:47:44.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162387+0000", 2024-09-15T03:47:44.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4290999999999999e-05 2024-09-15T03:47:44.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162502+0000", 2024-09-15T03:47:44.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115054 2024-09-15T03:47:44.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162516+0000", 2024-09-15T03:47:44.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3597999999999999e-05 2024-09-15T03:47:44.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162558+0000", 2024-09-15T03:47:44.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2562999999999999e-05 2024-09-15T03:47:44.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162584+0000", 2024-09-15T03:47:44.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5738e-05 2024-09-15T03:47:44.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135023+0000", 2024-09-15T03:47:44.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9724388369999999 2024-09-15T03:47:44.724 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.724 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.724 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.725 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 39/28 ECSubWrite(tid=38, reqid=osd.11.0:38, at_version=39'16, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T03:47:44.725 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:44.725 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.481231356999999, 2024-09-15T03:47:44.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99210653800000004, 2024-09-15T03:47:44.725 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.725 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.725 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:44.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:44.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:44.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1800000000000002e-07 2024-09-15T03:47:44.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143648+0000", 2024-09-15T03:47:44.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.849e-05 2024-09-15T03:47:44.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143670+0000", 2024-09-15T03:47:44.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2036e-05 2024-09-15T03:47:44.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143680+0000", 2024-09-15T03:47:44.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8320000000000001e-06 2024-09-15T03:47:44.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143783+0000", 2024-09-15T03:47:44.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103569 2024-09-15T03:47:44.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143801+0000", 2024-09-15T03:47:44.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8077999999999999e-05 2024-09-15T03:47:44.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144077+0000", 2024-09-15T03:47:44.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027569500000000001 2024-09-15T03:47:44.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144459+0000", 2024-09-15T03:47:44.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038227299999999998 2024-09-15T03:47:44.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144519+0000", 2024-09-15T03:47:44.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9907000000000001e-05 2024-09-15T03:47:44.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144628+0000", 2024-09-15T03:47:44.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108958 2024-09-15T03:47:44.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135726+0000", 2024-09-15T03:47:44.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99109738199999997 2024-09-15T03:47:44.730 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.730 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.730 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.730 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 39/31 ECSubWrite(tid=41, reqid=osd.6.0:41, at_version=39'9, trim_to=0'0, roll_forward_to=38'7))", 2024-09-15T03:47:44.731 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:44.731 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.480581102000002, 2024-09-15T03:47:44.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99106653099999997, 2024-09-15T03:47:44.731 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.731 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.731 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:44.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:44.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:44.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.97e-07 2024-09-15T03:47:44.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144297+0000", 2024-09-15T03:47:44.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7815000000000001e-05 2024-09-15T03:47:44.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144314+0000", 2024-09-15T03:47:44.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6269e-05 2024-09-15T03:47:44.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144322+0000", 2024-09-15T03:47:44.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1880000000000002e-06 2024-09-15T03:47:44.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144767+0000", 2024-09-15T03:47:44.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044555300000000001 2024-09-15T03:47:44.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144777+0000", 2024-09-15T03:47:44.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5950000000000005e-06 2024-09-15T03:47:44.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144992+0000", 2024-09-15T03:47:44.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000215479 2024-09-15T03:47:44.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145142+0000", 2024-09-15T03:47:44.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149302 2024-09-15T03:47:44.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145249+0000", 2024-09-15T03:47:44.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106971 2024-09-15T03:47:44.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145299+0000", 2024-09-15T03:47:44.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0413e-05 2024-09-15T03:47:44.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135336+0000", 2024-09-15T03:47:44.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99003674900000005 2024-09-15T03:47:44.737 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.737 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.737 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.737 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 39/28 ECSubWrite(tid=50, reqid=osd.2.0:50, at_version=39'28, trim_to=0'0, roll_forward_to=38'26))", 2024-09-15T03:47:44.737 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:44.737 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.480283620000002, 2024-09-15T03:47:44.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99095576299999999, 2024-09-15T03:47:44.738 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.738 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.738 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:44.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:44.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:44.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.03e-07 2024-09-15T03:47:44.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144593+0000", 2024-09-15T03:47:44.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6520999999999999e-05 2024-09-15T03:47:44.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144611+0000", 2024-09-15T03:47:44.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7963000000000001e-05 2024-09-15T03:47:44.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144619+0000", 2024-09-15T03:47:44.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7409999999999992e-06 2024-09-15T03:47:44.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144821+0000", 2024-09-15T03:47:44.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000201652 2024-09-15T03:47:44.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144832+0000", 2024-09-15T03:47:44.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0827e-05 2024-09-15T03:47:44.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145099+0000", 2024-09-15T03:47:44.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026688299999999999 2024-09-15T03:47:44.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145143+0000", 2024-09-15T03:47:44.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.481e-05 2024-09-15T03:47:44.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145249+0000", 2024-09-15T03:47:44.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010618799999999999 2024-09-15T03:47:44.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145340+0000", 2024-09-15T03:47:44.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0340000000000006e-05 2024-09-15T03:47:44.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135522+0000", 2024-09-15T03:47:44.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99018263500000003 2024-09-15T03:47:44.743 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.744 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.744 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.744 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=43, reqid=client.4521.0:240, at_version=39'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:44.744 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:47:44.744 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.476942899999997, 2024-09-15T03:47:44.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99104975500000003, 2024-09-15T03:47:44.744 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.744 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.744 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:47:44.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:47:44.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147908+0000", 2024-09-15T03:47:44.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.05e-07 2024-09-15T03:47:44.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147931+0000", 2024-09-15T03:47:44.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3621e-05 2024-09-15T03:47:44.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147945+0000", 2024-09-15T03:47:44.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3648999999999999e-05 2024-09-15T03:47:44.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147951+0000", 2024-09-15T03:47:44.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0510000000000002e-06 2024-09-15T03:47:44.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148055+0000", 2024-09-15T03:47:44.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104421 2024-09-15T03:47:44.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148068+0000", 2024-09-15T03:47:44.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3019e-05 2024-09-15T03:47:44.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148375+0000", 2024-09-15T03:47:44.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030683400000000002 2024-09-15T03:47:44.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148399+0000", 2024-09-15T03:47:44.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3760999999999998e-05 2024-09-15T03:47:44.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148453+0000", 2024-09-15T03:47:44.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3851999999999999e-05 2024-09-15T03:47:44.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148481+0000", 2024-09-15T03:47:44.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8637999999999999e-05 2024-09-15T03:47:44.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.138957+0000", 2024-09-15T03:47:44.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99047580400000002 2024-09-15T03:47:44.749 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.750 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.750 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.750 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 39/28 ECSubWrite(tid=39, reqid=osd.11.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:47:44.750 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:47:44.750 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.476810411000002, 2024-09-15T03:47:44.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99452509, 2024-09-15T03:47:44.750 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.750 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.750 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:47:44.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:47:44.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148041+0000", 2024-09-15T03:47:44.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0999999999999998e-07 2024-09-15T03:47:44.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148052+0000", 2024-09-15T03:47:44.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.129e-05 2024-09-15T03:47:44.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148064+0000", 2024-09-15T03:47:44.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173e-05 2024-09-15T03:47:44.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148068+0000", 2024-09-15T03:47:44.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5680000000000001e-06 2024-09-15T03:47:44.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148136+0000", 2024-09-15T03:47:44.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7508999999999993e-05 2024-09-15T03:47:44.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148148+0000", 2024-09-15T03:47:44.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2002e-05 2024-09-15T03:47:44.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148265+0000", 2024-09-15T03:47:44.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117669 2024-09-15T03:47:44.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148289+0000", 2024-09-15T03:47:44.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3672e-05 2024-09-15T03:47:44.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148348+0000", 2024-09-15T03:47:44.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8838000000000001e-05 2024-09-15T03:47:44.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148368+0000", 2024-09-15T03:47:44.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9942999999999999e-05 2024-09-15T03:47:44.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142565+0000", 2024-09-15T03:47:44.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99419715900000005 2024-09-15T03:47:44.757 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.757 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.757 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.758 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=45, reqid=client.4521.0:241, at_version=39'17, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:44.758 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:47:44.758 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.473807387999997, 2024-09-15T03:47:44.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99118202200000005, 2024-09-15T03:47:44.758 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.758 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.758 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:47:44.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:47:44.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151044+0000", 2024-09-15T03:47:44.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1340000000000001e-06 2024-09-15T03:47:44.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151073+0000", 2024-09-15T03:47:44.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8894000000000002e-05 2024-09-15T03:47:44.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151095+0000", 2024-09-15T03:47:44.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1889e-05 2024-09-15T03:47:44.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151107+0000", 2024-09-15T03:47:44.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2136999999999999e-05 2024-09-15T03:47:44.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151189+0000", 2024-09-15T03:47:44.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1883000000000002e-05 2024-09-15T03:47:44.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151203+0000", 2024-09-15T03:47:44.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3825999999999999e-05 2024-09-15T03:47:44.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151305+0000", 2024-09-15T03:47:44.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101816 2024-09-15T03:47:44.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151337+0000", 2024-09-15T03:47:44.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2703000000000002e-05 2024-09-15T03:47:44.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151387+0000", 2024-09-15T03:47:44.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9366000000000001e-05 2024-09-15T03:47:44.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151405+0000", 2024-09-15T03:47:44.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8748000000000001e-05 2024-09-15T03:47:44.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142225+0000", 2024-09-15T03:47:44.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99081962599999995 2024-09-15T03:47:44.763 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.763 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.763 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.764 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 39/31 ECSubWrite(tid=42, reqid=osd.6.0:41, at_version=0'0, trim_to=0'0, roll_forward_to=39'9))", 2024-09-15T03:47:44.764 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:47:44.764 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.473511346999999, 2024-09-15T03:47:44.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99092639999999999, 2024-09-15T03:47:44.764 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.764 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.764 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:47:44.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:47:44.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151340+0000", 2024-09-15T03:47:44.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8299999999999996e-07 2024-09-15T03:47:44.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151350+0000", 2024-09-15T03:47:44.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0675e-05 2024-09-15T03:47:44.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151362+0000", 2024-09-15T03:47:44.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2048999999999999e-05 2024-09-15T03:47:44.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151369+0000", 2024-09-15T03:47:44.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3230000000000003e-06 2024-09-15T03:47:44.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151425+0000", 2024-09-15T03:47:44.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5964000000000002e-05 2024-09-15T03:47:44.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151435+0000", 2024-09-15T03:47:44.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9240000000000002e-06 2024-09-15T03:47:44.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151530+0000", 2024-09-15T03:47:44.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5649000000000005e-05 2024-09-15T03:47:44.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151558+0000", 2024-09-15T03:47:44.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7976000000000001e-05 2024-09-15T03:47:44.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151599+0000", 2024-09-15T03:47:44.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1152000000000002e-05 2024-09-15T03:47:44.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151617+0000", 2024-09-15T03:47:44.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7676000000000002e-05 2024-09-15T03:47:44.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142265+0000", 2024-09-15T03:47:44.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99064822900000005 2024-09-15T03:47:44.769 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.769 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.769 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.769 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=46, reqid=client.4521.0:242, at_version=39'18, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:44.770 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:44.770 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.473485711000002, 2024-09-15T03:47:44.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.991000204, 2024-09-15T03:47:44.770 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.770 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.770 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:44.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:44.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:44.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3399999999999999e-07 2024-09-15T03:47:44.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151393+0000", 2024-09-15T03:47:44.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8149000000000001e-05 2024-09-15T03:47:44.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151405+0000", 2024-09-15T03:47:44.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2063e-05 2024-09-15T03:47:44.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151415+0000", 2024-09-15T03:47:44.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4579999999999999e-06 2024-09-15T03:47:44.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151472+0000", 2024-09-15T03:47:44.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7018999999999997e-05 2024-09-15T03:47:44.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151480+0000", 2024-09-15T03:47:44.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1410000000000005e-06 2024-09-15T03:47:44.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151571+0000", 2024-09-15T03:47:44.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1243999999999996e-05 2024-09-15T03:47:44.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151602+0000", 2024-09-15T03:47:44.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0834000000000003e-05 2024-09-15T03:47:44.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151632+0000", 2024-09-15T03:47:44.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9527e-05 2024-09-15T03:47:44.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151685+0000", 2024-09-15T03:47:44.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3599e-05 2024-09-15T03:47:44.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142365+0000", 2024-09-15T03:47:44.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99067953600000003 2024-09-15T03:47:44.775 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.775 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.775 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.775 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 39/28 ECSubWrite(tid=51, reqid=osd.2.0:50, at_version=0'0, trim_to=0'0, roll_forward_to=39'28))", 2024-09-15T03:47:44.775 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:47:44.775 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.470122273000001, 2024-09-15T03:47:44.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98766307399999997, 2024-09-15T03:47:44.776 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.776 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.776 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:47:44.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:47:44.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154729+0000", 2024-09-15T03:47:44.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3799999999999996e-07 2024-09-15T03:47:44.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154742+0000", 2024-09-15T03:47:44.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2909e-05 2024-09-15T03:47:44.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154755+0000", 2024-09-15T03:47:44.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3251999999999999e-05 2024-09-15T03:47:44.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154761+0000", 2024-09-15T03:47:44.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6559999999999997e-06 2024-09-15T03:47:44.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154829+0000", 2024-09-15T03:47:44.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8549999999999999e-05 2024-09-15T03:47:44.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154840+0000", 2024-09-15T03:47:44.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0159999999999999e-05 2024-09-15T03:47:44.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154943+0000", 2024-09-15T03:47:44.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010347400000000001 2024-09-15T03:47:44.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154955+0000", 2024-09-15T03:47:44.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1935e-05 2024-09-15T03:47:44.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156442+0000", 2024-09-15T03:47:44.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014867859999999999 2024-09-15T03:47:44.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156466+0000", 2024-09-15T03:47:44.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4077999999999999e-05 2024-09-15T03:47:44.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142391+0000", 2024-09-15T03:47:44.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98592533599999999 2024-09-15T03:47:44.781 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.781 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.781 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.781 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=48, reqid=client.4521.0:242, at_version=0'0, trim_to=0'0, roll_forward_to=39'18))", 2024-09-15T03:47:44.781 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:47:44.781 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.468235254, 2024-09-15T03:47:44.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98592495099999999, 2024-09-15T03:47:44.782 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:44.782 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:44.782 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:44.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:44.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:47:44.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:44.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:47:44.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:44.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:44.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156616+0000", 2024-09-15T03:47:44.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1999999999999998e-07 2024-09-15T03:47:44.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:44.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156629+0000", 2024-09-15T03:47:44.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2870999999999999e-05 2024-09-15T03:47:44.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:44.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156649+0000", 2024-09-15T03:47:44.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9786e-05 2024-09-15T03:47:44.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:44.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156657+0000", 2024-09-15T03:47:44.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5110000000000003e-06 2024-09-15T03:47:44.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:44.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156723+0000", 2024-09-15T03:47:44.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6209000000000003e-05 2024-09-15T03:47:44.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:44.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156734+0000", 2024-09-15T03:47:44.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0475999999999999e-05 2024-09-15T03:47:44.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:44.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156868+0000", 2024-09-15T03:47:44.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013402999999999999 2024-09-15T03:47:44.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:44.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156889+0000", 2024-09-15T03:47:44.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1146e-05 2024-09-15T03:47:44.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:44.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156937+0000", 2024-09-15T03:47:44.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8031999999999997e-05 2024-09-15T03:47:44.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:44.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156970+0000", 2024-09-15T03:47:44.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2462999999999997e-05 2024-09-15T03:47:44.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:44.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:44.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:44.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142540+0000", 2024-09-15T03:47:44.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98557050700000004 2024-09-15T03:47:44.787 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.787 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.787 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.787 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:44.787 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:44.787 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:44.789 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:47:44.790 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:44.788+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:44.855 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-15T03:47:44.856 INFO:tasks.rados.rados.0.smithi001.stdout:326: done (3 left) 2024-09-15T03:47:44.856 INFO:tasks.rados.rados.0.smithi001.stdout:327: done (2 left) 2024-09-15T03:47:44.856 INFO:tasks.rados.rados.0.smithi001.stdout:328: done (1 left) 2024-09-15T03:47:44.856 INFO:tasks.rados.rados.0.smithi001.stdout:329: done (0 left) 2024-09-15T03:47:44.856 INFO:tasks.rados.rados.0.smithi001.stdout:330: delete oid 18 current snap is 41 2024-09-15T03:47:44.859 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [10] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] 2024-09-15T03:47:44.859 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T03:47:44.859 INFO:tasks.thrashosds.thrasher:inject_pause on osd.11 2024-09-15T03:47:44.859 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-15T03:47:44.859 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-15T03:47:44.859 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok config set filestore_inject_stall 3 2024-09-15T03:47:44.863 INFO:tasks.rados.rados.0.smithi001.stdout:330: done (0 left) 2024-09-15T03:47:44.863 INFO:tasks.rados.rados.0.smithi001.stdout:331: read oid 13 snap -1 2024-09-15T03:47:44.864 INFO:tasks.rados.rados.0.smithi001.stdout:331: expect (ObjNum 68 snap 6 seq_num 68) 2024-09-15T03:47:44.864 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 23 current snap is 41 2024-09-15T03:47:44.864 INFO:tasks.rados.rados.0.smithi001.stdout:332: seq_num 127 ranges {376832=589824,966656=753664,1720320=475136,2195456=327680} 2024-09-15T03:47:44.869 INFO:tasks.rados.rados.0.smithi001.stdout:332: writing smithi00132172-23 from 376832 to 966656 tid 1 2024-09-15T03:47:44.873 INFO:tasks.rados.rados.0.smithi001.stdout:332: writing smithi00132172-23 from 966656 to 1720320 tid 2 2024-09-15T03:47:44.876 INFO:tasks.rados.rados.0.smithi001.stdout:332: writing smithi00132172-23 from 1720320 to 2195456 tid 3 2024-09-15T03:47:44.878 INFO:tasks.rados.rados.0.smithi001.stdout:332: writing smithi00132172-23 from 2195456 to 2523136 tid 4 2024-09-15T03:47:44.878 INFO:tasks.rados.rados.0.smithi001.stdout:333: copy_from oid 50 from oid 5 current snap is 41 2024-09-15T03:47:44.878 INFO:tasks.rados.rados.0.smithi001.stdout:331: expect (ObjNum 68 snap 6 seq_num 68) 2024-09-15T03:47:44.903 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:44.904 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:44.904 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:44.903+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:44.912 INFO:tasks.rados.rados.0.smithi001.stdout:332: finishing write tid 1 to smithi00132172-23 2024-09-15T03:47:44.912 INFO:tasks.rados.rados.0.smithi001.stdout:332: finishing write tid 2 to smithi00132172-23 2024-09-15T03:47:44.912 INFO:tasks.rados.rados.0.smithi001.stdout:331: done (2 left) 2024-09-15T03:47:44.912 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 2 current snap is 41 2024-09-15T03:47:44.912 INFO:tasks.rados.rados.0.smithi001.stdout:334: seq_num 128 ranges {32768=606208,638976=507904,1146880=573440,1720320=475136} 2024-09-15T03:47:44.913 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:47:44.915 INFO:tasks.rados.rados.0.smithi001.stdout:334: writing smithi00132172-2 from 32768 to 638976 tid 1 2024-09-15T03:47:44.918 INFO:tasks.rados.rados.0.smithi001.stdout:334: writing smithi00132172-2 from 638976 to 1146880 tid 2 2024-09-15T03:47:44.921 INFO:tasks.rados.rados.0.smithi001.stdout:334: writing smithi00132172-2 from 1146880 to 1720320 tid 3 2024-09-15T03:47:44.924 INFO:tasks.rados.rados.0.smithi001.stdout:334: writing smithi00132172-2 from 1720320 to 2195456 tid 4 2024-09-15T03:47:44.924 INFO:tasks.rados.rados.0.smithi001.stdout:335: delete oid 22 current snap is 41 2024-09-15T03:47:44.925 INFO:tasks.rados.rados.0.smithi001.stdout:332: finishing write tid 3 to smithi00132172-23 2024-09-15T03:47:44.925 INFO:tasks.rados.rados.0.smithi001.stdout:332: finishing write tid 4 to smithi00132172-23 2024-09-15T03:47:44.925 INFO:tasks.rados.rados.0.smithi001.stdout:332: finishing write tid 5 to smithi00132172-23 2024-09-15T03:47:44.925 INFO:tasks.rados.rados.0.smithi001.stdout:332: finishing write tid 6 to smithi00132172-23 2024-09-15T03:47:44.925 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 23 v 41 (ObjNum 127 snap 41 seq_num 127) dirty exists 2024-09-15T03:47:44.925 INFO:tasks.rados.rados.0.smithi001.stdout:332: left oid 23 (ObjNum 127 snap 41 seq_num 127) 2024-09-15T03:47:44.925 INFO:tasks.rados.rados.0.smithi001.stdout:333: finishing copy_from racing read to smithi00132172-50 2024-09-15T03:47:44.925 INFO:tasks.rados.rados.0.smithi001.stdout:333: finishing copy_from to smithi00132172-50 2024-09-15T03:47:44.925 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 50 v 33 (ObjNum 105 snap 29 seq_num 105) dirty exists 2024-09-15T03:47:44.925 INFO:tasks.rados.rados.0.smithi001.stdout:334: finishing write tid 1 to smithi00132172-2 2024-09-15T03:47:44.926 INFO:tasks.rados.rados.0.smithi001.stdout:334: finishing write tid 2 to smithi00132172-2 2024-09-15T03:47:44.928 INFO:tasks.rados.rados.0.smithi001.stdout:334: finishing write tid 3 to smithi00132172-2 2024-09-15T03:47:44.928 INFO:tasks.rados.rados.0.smithi001.stdout:334: finishing write tid 4 to smithi00132172-2 2024-09-15T03:47:44.928 INFO:tasks.rados.rados.0.smithi001.stdout:334: finishing write tid 5 to smithi00132172-2 2024-09-15T03:47:44.929 INFO:tasks.rados.rados.0.smithi001.stdout:332: done (3 left) 2024-09-15T03:47:44.929 INFO:tasks.rados.rados.0.smithi001.stdout:333: done (2 left) 2024-09-15T03:47:44.929 INFO:tasks.rados.rados.0.smithi001.stdout:335: done (1 left) 2024-09-15T03:47:44.929 INFO:tasks.rados.rados.0.smithi001.stdout:336: read oid 22 snap 35 2024-09-15T03:47:44.929 INFO:tasks.rados.rados.0.smithi001.stdout:336: expect (ObjNum 112 snap 32 seq_num 112) 2024-09-15T03:47:44.929 INFO:tasks.rados.rados.0.smithi001.stdout:337: snap_remove snap 32 2024-09-15T03:47:44.974 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:44.975 INFO:teuthology.orchestra.run.smithi190.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-15T03:47:44.975 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:44.990 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:44.990+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:45.018 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:45.019 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:45.019 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:45.019 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:45.019 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:45.027 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:47:45.091 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:45.089+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:45.129 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:45.129 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:45.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:45.130 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:45.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.130 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=21, reqid=client.4521.0:184, at_version=33'6, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:45.130 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:45.130 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 65.020807177999998, 2024-09-15T03:47:45.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010901256999999999, 2024-09-15T03:47:45.130 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.130 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:45.130 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:45.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:45.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:45.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0200000000000001e-07 2024-09-15T03:47:45.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106529+0000", 2024-09-15T03:47:45.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9539e-05 2024-09-15T03:47:45.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106542+0000", 2024-09-15T03:47:45.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2731999999999999e-05 2024-09-15T03:47:45.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106549+0000", 2024-09-15T03:47:45.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3339999999999996e-06 2024-09-15T03:47:45.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106613+0000", 2024-09-15T03:47:45.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4527000000000001e-05 2024-09-15T03:47:45.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106622+0000", 2024-09-15T03:47:45.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.755e-06 2024-09-15T03:47:45.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106841+0000", 2024-09-15T03:47:45.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021906200000000001 2024-09-15T03:47:45.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106869+0000", 2024-09-15T03:47:45.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.739e-05 2024-09-15T03:47:45.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106948+0000", 2024-09-15T03:47:45.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9422e-05 2024-09-15T03:47:45.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106965+0000", 2024-09-15T03:47:45.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7010000000000001e-05 2024-09-15T03:47:45.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.117410+0000", 2024-09-15T03:47:45.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010445284000000001 2024-09-15T03:47:45.136 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.136 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.136 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.136 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:45.136 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:45.136 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 65.018893691000002, 2024-09-15T03:47:45.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015296944, 2024-09-15T03:47:45.136 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.136 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:45.137 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:45.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:45.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:45.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5499999999999998e-07 2024-09-15T03:47:45.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108447+0000", 2024-09-15T03:47:45.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3314000000000001e-05 2024-09-15T03:47:45.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108460+0000", 2024-09-15T03:47:45.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3444e-05 2024-09-15T03:47:45.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108464+0000", 2024-09-15T03:47:45.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3610000000000003e-06 2024-09-15T03:47:45.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108494+0000", 2024-09-15T03:47:45.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9688000000000001e-05 2024-09-15T03:47:45.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108502+0000", 2024-09-15T03:47:45.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.678e-06 2024-09-15T03:47:45.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108601+0000", 2024-09-15T03:47:45.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9748999999999997e-05 2024-09-15T03:47:45.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108610+0000", 2024-09-15T03:47:45.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2490000000000001e-06 2024-09-15T03:47:45.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109977+0000", 2024-09-15T03:47:45.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013669450000000001 2024-09-15T03:47:45.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109993+0000", 2024-09-15T03:47:45.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6688e-05 2024-09-15T03:47:45.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123720+0000", 2024-09-15T03:47:45.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013726172999999999 2024-09-15T03:47:45.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.142 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.142 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.142 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:45.142 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:45.142 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 65.018844029999997, 2024-09-15T03:47:45.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015105210000000001, 2024-09-15T03:47:45.142 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.142 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:45.142 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:45.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:45.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108473+0000", 2024-09-15T03:47:45.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.84e-07 2024-09-15T03:47:45.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108486+0000", 2024-09-15T03:47:45.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3845999999999999e-05 2024-09-15T03:47:45.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108491+0000", 2024-09-15T03:47:45.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4719999999999997e-06 2024-09-15T03:47:45.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108495+0000", 2024-09-15T03:47:45.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.822e-06 2024-09-15T03:47:45.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108546+0000", 2024-09-15T03:47:45.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0853000000000003e-05 2024-09-15T03:47:45.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108555+0000", 2024-09-15T03:47:45.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3319999999999998e-06 2024-09-15T03:47:45.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108638+0000", 2024-09-15T03:47:45.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3580999999999998e-05 2024-09-15T03:47:45.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109983+0000", 2024-09-15T03:47:45.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001344392 2024-09-15T03:47:45.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110023+0000", 2024-09-15T03:47:45.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9895999999999999e-05 2024-09-15T03:47:45.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110040+0000", 2024-09-15T03:47:45.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.747e-05 2024-09-15T03:47:45.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123577+0000", 2024-09-15T03:47:45.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013537262 2024-09-15T03:47:45.147 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.147 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.148 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.148 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:45.148 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:45.148 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 65.018810150999997, 2024-09-15T03:47:45.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015311313999999999, 2024-09-15T03:47:45.148 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.148 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:45.148 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:45.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:45.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:45.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8500000000000002e-07 2024-09-15T03:47:45.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108517+0000", 2024-09-15T03:47:45.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0543000000000001e-05 2024-09-15T03:47:45.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108524+0000", 2024-09-15T03:47:45.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7170000000000002e-06 2024-09-15T03:47:45.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108528+0000", 2024-09-15T03:47:45.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3970000000000004e-06 2024-09-15T03:47:45.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108632+0000", 2024-09-15T03:47:45.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103687 2024-09-15T03:47:45.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108637+0000", 2024-09-15T03:47:45.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4389999999999999e-06 2024-09-15T03:47:45.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108713+0000", 2024-09-15T03:47:45.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6223000000000005e-05 2024-09-15T03:47:45.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109984+0000", 2024-09-15T03:47:45.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012700719999999999 2024-09-15T03:47:45.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110023+0000", 2024-09-15T03:47:45.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9864999999999999e-05 2024-09-15T03:47:45.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110041+0000", 2024-09-15T03:47:45.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8040999999999999e-05 2024-09-15T03:47:45.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123817+0000", 2024-09-15T03:47:45.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013776045000000001 2024-09-15T03:47:45.153 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.153 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.154 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.154 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:45.154 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:45.154 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 65.018316542999997, 2024-09-15T03:47:45.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014864502, 2024-09-15T03:47:45.154 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.154 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:45.154 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:45.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:45.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:45.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:47:45.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109016+0000", 2024-09-15T03:47:45.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5889999999999999e-05 2024-09-15T03:47:45.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109029+0000", 2024-09-15T03:47:45.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2812e-05 2024-09-15T03:47:45.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109035+0000", 2024-09-15T03:47:45.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0109999999999997e-06 2024-09-15T03:47:45.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109073+0000", 2024-09-15T03:47:45.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8260000000000003e-05 2024-09-15T03:47:45.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109080+0000", 2024-09-15T03:47:45.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0190000000000001e-06 2024-09-15T03:47:45.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109165+0000", 2024-09-15T03:47:45.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4957000000000007e-05 2024-09-15T03:47:45.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109984+0000", 2024-09-15T03:47:45.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081866399999999998 2024-09-15T03:47:45.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110024+0000", 2024-09-15T03:47:45.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9644999999999997e-05 2024-09-15T03:47:45.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110072+0000", 2024-09-15T03:47:45.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8418999999999998e-05 2024-09-15T03:47:45.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123864+0000", 2024-09-15T03:47:45.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013792234 2024-09-15T03:47:45.159 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=34, reqid=client.4521.0:197, at_version=34'21, trim_to=0'0, roll_forward_to=34'18))", 2024-09-15T03:47:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.003304662999994, 2024-09-15T03:47:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012992399, 2024-09-15T03:47:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124013+0000", 2024-09-15T03:47:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6299999999999993e-07 2024-09-15T03:47:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124044+0000", 2024-09-15T03:47:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0995999999999999e-05 2024-09-15T03:47:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124062+0000", 2024-09-15T03:47:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8413000000000001e-05 2024-09-15T03:47:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124069+0000", 2024-09-15T03:47:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3930000000000002e-06 2024-09-15T03:47:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124150+0000", 2024-09-15T03:47:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1097e-05 2024-09-15T03:47:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124161+0000", 2024-09-15T03:47:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0720000000000001e-05 2024-09-15T03:47:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124402+0000", 2024-09-15T03:47:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000240442 2024-09-15T03:47:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125141+0000", 2024-09-15T03:47:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00073915500000000002 2024-09-15T03:47:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125191+0000", 2024-09-15T03:47:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9951999999999999e-05 2024-09-15T03:47:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125251+0000", 2024-09-15T03:47:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9938000000000001e-05 2024-09-15T03:47:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.137004+0000", 2024-09-15T03:47:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011753329999999999 2024-09-15T03:47:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=35, reqid=client.4521.0:197, at_version=0'0, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:47:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.001364619, 2024-09-15T03:47:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011202456, 2024-09-15T03:47:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125953+0000", 2024-09-15T03:47:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9100000000000002e-07 2024-09-15T03:47:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125964+0000", 2024-09-15T03:47:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1399999999999999e-05 2024-09-15T03:47:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125979+0000", 2024-09-15T03:47:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4810000000000001e-05 2024-09-15T03:47:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126004+0000", 2024-09-15T03:47:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4913e-05 2024-09-15T03:47:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126059+0000", 2024-09-15T03:47:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5668999999999998e-05 2024-09-15T03:47:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126071+0000", 2024-09-15T03:47:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1219999999999999e-05 2024-09-15T03:47:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126151+0000", 2024-09-15T03:47:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0572e-05 2024-09-15T03:47:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126163+0000", 2024-09-15T03:47:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2101e-05 2024-09-15T03:47:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.127945+0000", 2024-09-15T03:47:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001781803 2024-09-15T03:47:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.127982+0000", 2024-09-15T03:47:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7129999999999999e-05 2024-09-15T03:47:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.137154+0000", 2024-09-15T03:47:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0091719469999999997 2024-09-15T03:47:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:243 3.3s0 3:c7c3ef16:::smithi00132172-25:head [create,setxattr iZnWfebab`ptjURR\\f`psW]tpS`qpUaeWkY[Oqep (1330) in=40b,setxattr f[mZ_qQmcZejQ`fbdeic_tkPUaTYcnm[WprjVmc] (12488) in=40b,setxattr tkZgQ]S_pdlhZnqrVaZRa`oQQr_USmUfhado^dXT (1121) in=40b,setxattr srWjTqXhcth`hfZmQY_fg]UqNjWofa^dfgOV`sYP (455) in=40b,setxattr ^_mdiRroj_TtQ^gcNj`XtkW[ooqY`PWXqYmcOk_n (138) in=40b,setxattr jQlajdtcalhkcfYaakZfNb[^_TjkfcS`Vrk]q_d[ (806) in=40b,setxattr ZfYgkVcrsXcnpchgfcbldNrUTasnQNmfZW]Nbh\\o (561) in=40b,setxattr _hkYOqjPNhmhUlSNktfeQpsqSWjO\\fbf[fpTlZiO (1227) in=40b,setxattr ]mdV^Xdc\\sViXdkXNU[js_Qhh[b[sbfnZVXkdids (11788) in=40b,setxattr tb^mbOrNhfrhtce^g (455) in=17b,setxattr koTkVgrfU\\m`StnZ`Rirjhbhg`PhVqm^a^Xgaaqn (35) in=40b,setxattr aUtZPUPXPnQmT]\\\\n\\lZpnXakdNkTldVQoQhPUf] (559) in=40b,setxattr mrhR]^U`Tteo[Topmql`SNma\\pqfk`if]OppnYt` (770) in=40b,setxattr Oetk]RrnWeXbVctVtVrhXf_RfTdelqX^lWp[sTkp (1226) in=40b,setxattr WtedXoOd`VqnbksOqQkatk`gdTnW^lXbUtrlYgkl (3576) in=40b,setxattr dddOWN\\j\\XXg^fr[rZcskeVX]jOQ^ZmoWepk[]\\a (17824) in=40b,setxattr TeRO^djPgNZiemQ_aRXgP[]jgUZa\\^\\iUgesaqaN (208) in=40b,setxattr teqm]mUobZbWjcdQ_jTa[f[k[Sd\\lh]n\\osigqkN (7436) in=40b,setxattr dphaol[XkeitbkomWYaQctWfataUhbSiNntkWo`P (1893) in=40b,setxattr oQsnhO]ZklSVYQ[WUpaelY[p[Qie^UepOt`Z[q (455) in=38b] snapc 7=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:47:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.958400730999998, 2024-09-15T03:47:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012365956000000001, 2024-09-15T03:47:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 243 2024-09-15T03:47:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168917+0000", 2024-09-15T03:47:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.725e-06 2024-09-15T03:47:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169030+0000", 2024-09-15T03:47:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113035 2024-09-15T03:47:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169047+0000", 2024-09-15T03:47:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6701000000000002e-05 2024-09-15T03:47:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169054+0000", 2024-09-15T03:47:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1369999999999996e-06 2024-09-15T03:47:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169140+0000", 2024-09-15T03:47:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6069999999999994e-05 2024-09-15T03:47:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169270+0000", 2024-09-15T03:47:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130071 2024-09-15T03:47:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.170661+0000", 2024-09-15T03:47:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001390666 2024-09-15T03:47:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171751+0000", 2024-09-15T03:47:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010899079999999999 2024-09-15T03:47:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171812+0000", 2024-09-15T03:47:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0797000000000001e-05 2024-09-15T03:47:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172279+0000", 2024-09-15T03:47:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046704500000000003 2024-09-15T03:47:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172299+0000", 2024-09-15T03:47:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0387999999999999e-05 2024-09-15T03:47:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.176110+0000", 2024-09-15T03:47:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038112340000000001 2024-09-15T03:47:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.181282+0000", 2024-09-15T03:47:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051711789999999997 2024-09-15T03:47:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:258 3.3s0 3:c9b663a2:::smithi00132172-7:head [delete] snapc 7=[6,5,3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e40)", 2024-09-15T03:47:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:47.146116+0000", 2024-09-15T03:47:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.981199912999998, 2024-09-15T03:47:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096428380000000008, 2024-09-15T03:47:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 258 2024-09-15T03:47:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146116+0000", 2024-09-15T03:47:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146116+0000", 2024-09-15T03:47:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146119+0000", 2024-09-15T03:47:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.227e-06 2024-09-15T03:47:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146127+0000", 2024-09-15T03:47:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1529999999999997e-06 2024-09-15T03:47:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146146+0000", 2024-09-15T03:47:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9318999999999999e-05 2024-09-15T03:47:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146154+0000", 2024-09-15T03:47:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7200000000000006e-06 2024-09-15T03:47:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146233+0000", 2024-09-15T03:47:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8725000000000003e-05 2024-09-15T03:47:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.146244+0000", 2024-09-15T03:47:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1725e-05 2024-09-15T03:47:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.149207+0000", 2024-09-15T03:47:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029627999999999998 2024-09-15T03:47:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.149326+0000", 2024-09-15T03:47:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011869199999999999 2024-09-15T03:47:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.150266+0000", 2024-09-15T03:47:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00094048500000000002 2024-09-15T03:47:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.150626+0000", 2024-09-15T03:47:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035946699999999998 2024-09-15T03:47:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.150751+0000", 2024-09-15T03:47:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012544800000000001 2024-09-15T03:47:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.151656+0000", 2024-09-15T03:47:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00090447499999999996 2024-09-15T03:47:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.153489+0000", 2024-09-15T03:47:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001833115 2024-09-15T03:47:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.155171+0000", 2024-09-15T03:47:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001682455 2024-09-15T03:47:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.155759+0000", 2024-09-15T03:47:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00058803200000000005 2024-09-15T03:47:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/28 ECSubWrite(tid=15, reqid=client.4521.0:337, at_version=49'10, trim_to=0'0, roll_forward_to=43'8))", 2024-09-15T03:47:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 49.269410821999998, 2024-09-15T03:47:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010500893000000001, 2024-09-15T03:47:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857906+0000", 2024-09-15T03:47:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.63e-07 2024-09-15T03:47:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857928+0000", 2024-09-15T03:47:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2155999999999999e-05 2024-09-15T03:47:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857937+0000", 2024-09-15T03:47:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2359999999999993e-06 2024-09-15T03:47:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857941+0000", 2024-09-15T03:47:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0960000000000003e-06 2024-09-15T03:47:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857999+0000", 2024-09-15T03:47:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7407000000000001e-05 2024-09-15T03:47:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858011+0000", 2024-09-15T03:47:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1759999999999999e-05 2024-09-15T03:47:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858240+0000", 2024-09-15T03:47:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022949299999999999 2024-09-15T03:47:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858251+0000", 2024-09-15T03:47:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1168999999999999e-05 2024-09-15T03:47:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858332+0000", 2024-09-15T03:47:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1204000000000004e-05 2024-09-15T03:47:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858360+0000", 2024-09-15T03:47:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7767999999999999e-05 2024-09-15T03:47:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.868406+0000", 2024-09-15T03:47:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010046141 2024-09-15T03:47:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:349 3.3s0 3:c7c3ef16:::smithi00132172-25:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.288834796000003, 2024-09-15T03:47:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005145406, 2024-09-15T03:47:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 349 2024-09-15T03:47:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838485+0000", 2024-09-15T03:47:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0620000000000001e-06 2024-09-15T03:47:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838494+0000", 2024-09-15T03:47:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3179999999999996e-06 2024-09-15T03:47:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3689999999999998e-05 2024-09-15T03:47:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838812+0000", 2024-09-15T03:47:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029437300000000001 2024-09-15T03:47:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838905+0000", 2024-09-15T03:47:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3196999999999996e-05 2024-09-15T03:47:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838914+0000", 2024-09-15T03:47:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0890000000000006e-06 2024-09-15T03:47:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840773+0000", 2024-09-15T03:47:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018584820000000001 2024-09-15T03:47:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840889+0000", 2024-09-15T03:47:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115986 2024-09-15T03:47:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842016+0000", 2024-09-15T03:47:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001126878 2024-09-15T03:47:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842320+0000", 2024-09-15T03:47:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030409799999999999 2024-09-15T03:47:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842360+0000", 2024-09-15T03:47:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0750000000000001e-05 2024-09-15T03:47:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842448+0000", 2024-09-15T03:47:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7842000000000004e-05 2024-09-15T03:47:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844580+0000", 2024-09-15T03:47:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021313719999999999 2024-09-15T03:47:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843574+0000", 2024-09-15T03:47:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99899468199999997 2024-09-15T03:47:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843627+0000", 2024-09-15T03:47:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2587000000000003e-05 2024-09-15T03:47:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:350 3.3s0 3:c7c3ef16:::smithi00132172-25:head [rollback 14] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.288483135, 2024-09-15T03:47:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0050383839999999, 2024-09-15T03:47:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 350 2024-09-15T03:47:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838834+0000", 2024-09-15T03:47:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0200000000000001e-07 2024-09-15T03:47:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838839+0000", 2024-09-15T03:47:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9790000000000001e-06 2024-09-15T03:47:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838852+0000", 2024-09-15T03:47:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3138e-05 2024-09-15T03:47:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838858+0000", 2024-09-15T03:47:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8320000000000002e-06 2024-09-15T03:47:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839003+0000", 2024-09-15T03:47:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014513500000000001 2024-09-15T03:47:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839011+0000", 2024-09-15T03:47:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7719999999999994e-06 2024-09-15T03:47:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842395+0000", 2024-09-15T03:47:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033840110000000001 2024-09-15T03:47:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842580+0000", 2024-09-15T03:47:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000185067 2024-09-15T03:47:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843394+0000", 2024-09-15T03:47:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081377299999999995 2024-09-15T03:47:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843510+0000", 2024-09-15T03:47:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0001165 2024-09-15T03:47:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843524+0000", 2024-09-15T03:47:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4022999999999999e-05 2024-09-15T03:47:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843579+0000", 2024-09-15T03:47:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4654999999999998e-05 2024-09-15T03:47:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844583+0000", 2024-09-15T03:47:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010042429999999999 2024-09-15T03:47:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843772+0000", 2024-09-15T03:47:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99918911200000005 2024-09-15T03:47:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843872+0000", 2024-09-15T03:47:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9341999999999999e-05 2024-09-15T03:47:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:351 3.3s0 3:c7c3ef16:::smithi00132172-25:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.288425502000003, 2024-09-15T03:47:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0051167679999999, 2024-09-15T03:47:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 351 2024-09-15T03:47:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0800000000000004e-07 2024-09-15T03:47:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838895+0000", 2024-09-15T03:47:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.241e-06 2024-09-15T03:47:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838904+0000", 2024-09-15T03:47:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9549999999999998e-06 2024-09-15T03:47:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838909+0000", 2024-09-15T03:47:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7869999999999998e-06 2024-09-15T03:47:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839045+0000", 2024-09-15T03:47:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013556399999999999 2024-09-15T03:47:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839052+0000", 2024-09-15T03:47:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8199999999999999e-06 2024-09-15T03:47:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843587+0000", 2024-09-15T03:47:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045349739999999998 2024-09-15T03:47:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843708+0000", 2024-09-15T03:47:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120776 2024-09-15T03:47:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844246+0000", 2024-09-15T03:47:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053838200000000001 2024-09-15T03:47:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844335+0000", 2024-09-15T03:47:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9240999999999995e-05 2024-09-15T03:47:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844347+0000", 2024-09-15T03:47:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2187e-05 2024-09-15T03:47:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844460+0000", 2024-09-15T03:47:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112571 2024-09-15T03:47:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844584+0000", 2024-09-15T03:47:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012403800000000001 2024-09-15T03:47:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843972+0000", 2024-09-15T03:47:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99938787299999998 2024-09-15T03:47:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844008+0000", 2024-09-15T03:47:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5750999999999997e-05 2024-09-15T03:47:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.281291551000002, 2024-09-15T03:47:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013010173, 2024-09-15T03:47:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846026+0000", 2024-09-15T03:47:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1230000000000001e-06 2024-09-15T03:47:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846047+0000", 2024-09-15T03:47:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0774000000000001e-05 2024-09-15T03:47:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846064+0000", 2024-09-15T03:47:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7560000000000001e-05 2024-09-15T03:47:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846072+0000", 2024-09-15T03:47:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5020000000000003e-06 2024-09-15T03:47:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846187+0000", 2024-09-15T03:47:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115717 2024-09-15T03:47:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846201+0000", 2024-09-15T03:47:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3329999999999999e-05 2024-09-15T03:47:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846303+0000", 2024-09-15T03:47:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101807 2024-09-15T03:47:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846312+0000", 2024-09-15T03:47:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5529999999999998e-06 2024-09-15T03:47:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846382+0000", 2024-09-15T03:47:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0146000000000002e-05 2024-09-15T03:47:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846520+0000", 2024-09-15T03:47:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013814099999999999 2024-09-15T03:47:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.859035+0000", 2024-09-15T03:47:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012514519999999999 2024-09-15T03:47:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.281194804000002, 2024-09-15T03:47:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013044694000000001, 2024-09-15T03:47:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846122+0000", 2024-09-15T03:47:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0399999999999996e-07 2024-09-15T03:47:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846174+0000", 2024-09-15T03:47:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1672999999999998e-05 2024-09-15T03:47:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846184+0000", 2024-09-15T03:47:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0514e-05 2024-09-15T03:47:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846190+0000", 2024-09-15T03:47:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2820000000000002e-06 2024-09-15T03:47:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846341+0000", 2024-09-15T03:47:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150803 2024-09-15T03:47:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846351+0000", 2024-09-15T03:47:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3899999999999999e-06 2024-09-15T03:47:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846450+0000", 2024-09-15T03:47:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9839000000000004e-05 2024-09-15T03:47:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846482+0000", 2024-09-15T03:47:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1089e-05 2024-09-15T03:47:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846615+0000", 2024-09-15T03:47:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013305899999999999 2024-09-15T03:47:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846634+0000", 2024-09-15T03:47:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8899999999999999e-05 2024-09-15T03:47:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.859166+0000", 2024-09-15T03:47:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012532641000000001 2024-09-15T03:47:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 60/28 ECSubWrite(tid=73, reqid=client.4521.0:424, at_version=60'41, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.226885957, 2024-09-15T03:47:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011124821, 2024-09-15T03:47:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900431+0000", 2024-09-15T03:47:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2900000000000003e-07 2024-09-15T03:47:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900451+0000", 2024-09-15T03:47:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0066999999999999e-05 2024-09-15T03:47:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900466+0000", 2024-09-15T03:47:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5062000000000001e-05 2024-09-15T03:47:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900474+0000", 2024-09-15T03:47:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6310000000000003e-06 2024-09-15T03:47:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900524+0000", 2024-09-15T03:47:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0037000000000002e-05 2024-09-15T03:47:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900535+0000", 2024-09-15T03:47:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1209000000000001e-05 2024-09-15T03:47:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900659+0000", 2024-09-15T03:47:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012441000000000001 2024-09-15T03:47:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900971+0000", 2024-09-15T03:47:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031183799999999999 2024-09-15T03:47:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901144+0000", 2024-09-15T03:47:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172261 2024-09-15T03:47:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901734+0000", 2024-09-15T03:47:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059069700000000005 2024-09-15T03:47:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.911555+0000", 2024-09-15T03:47:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0098208800000000006 2024-09-15T03:47:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 60/28 ECSubWrite(tid=74, reqid=client.4521.0:425, at_version=60'42, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:06.901240+0000", 2024-09-15T03:47:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.226076378000002, 2024-09-15T03:47:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010429678, 2024-09-15T03:47:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901240+0000", 2024-09-15T03:47:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901240+0000", 2024-09-15T03:47:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901241+0000", 2024-09-15T03:47:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7999999999999993e-07 2024-09-15T03:47:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901273+0000", 2024-09-15T03:47:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2087e-05 2024-09-15T03:47:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901302+0000", 2024-09-15T03:47:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8705e-05 2024-09-15T03:47:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901311+0000", 2024-09-15T03:47:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9190000000000005e-06 2024-09-15T03:47:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901797+0000", 2024-09-15T03:47:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048606499999999999 2024-09-15T03:47:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901808+0000", 2024-09-15T03:47:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1075e-05 2024-09-15T03:47:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901934+0000", 2024-09-15T03:47:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012674299999999999 2024-09-15T03:47:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901964+0000", 2024-09-15T03:47:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9833999999999999e-05 2024-09-15T03:47:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.902006+0000", 2024-09-15T03:47:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1823999999999998e-05 2024-09-15T03:47:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.902020+0000", 2024-09-15T03:47:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.363e-05 2024-09-15T03:47:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.911670+0000", 2024-09-15T03:47:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096498160000000003 2024-09-15T03:47:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPush(2.2 88/87 [PushOp(2:5cea7035:::rbd_info:head, version: 24'2, data_included: [0~19], data_size: 19, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:5cea7035:::rbd_info:head@24'2, size: 19, copy_subset: [0~19], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:19, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-15T03:47:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.846903+0000", 2024-09-15T03:47:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.280412831, 2024-09-15T03:47:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013354107, 2024-09-15T03:47:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846903+0000", 2024-09-15T03:47:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846903+0000", 2024-09-15T03:47:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846904+0000", 2024-09-15T03:47:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4099999999999998e-07 2024-09-15T03:47:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846920+0000", 2024-09-15T03:47:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6061000000000001e-05 2024-09-15T03:47:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846945+0000", 2024-09-15T03:47:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4519e-05 2024-09-15T03:47:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846958+0000", 2024-09-15T03:47:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3546999999999999e-05 2024-09-15T03:47:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.847077+0000", 2024-09-15T03:47:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011864 2024-09-15T03:47:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for flush", 2024-09-15T03:47:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.847094+0000", 2024-09-15T03:47:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7017e-05 2024-09-15T03:47:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.859991+0000", 2024-09-15T03:47:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012897136 2024-09-15T03:47:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.860001+0000", 2024-09-15T03:47:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5089999999999999e-06 2024-09-15T03:47:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.860258+0000", 2024-09-15T03:47:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000256937 2024-09-15T03:47:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 88/87 ECSubWrite(tid=200, reqid=client.4521.0:737, at_version=88'109, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.883279+0000", 2024-09-15T03:47:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.244036908, 2024-09-15T03:47:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010263239, 2024-09-15T03:47:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883279+0000", 2024-09-15T03:47:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883279+0000", 2024-09-15T03:47:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883280+0000", 2024-09-15T03:47:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5799999999999999e-07 2024-09-15T03:47:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883328+0000", 2024-09-15T03:47:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8198999999999997e-05 2024-09-15T03:47:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883341+0000", 2024-09-15T03:47:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2557e-05 2024-09-15T03:47:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883347+0000", 2024-09-15T03:47:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5359999999999998e-06 2024-09-15T03:47:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883406+0000", 2024-09-15T03:47:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8971999999999997e-05 2024-09-15T03:47:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883417+0000", 2024-09-15T03:47:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0346000000000001e-05 2024-09-15T03:47:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883682+0000", 2024-09-15T03:47:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026550500000000003 2024-09-15T03:47:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883705+0000", 2024-09-15T03:47:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2779e-05 2024-09-15T03:47:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883736+0000", 2024-09-15T03:47:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0626999999999997e-05 2024-09-15T03:47:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.883752+0000", 2024-09-15T03:47:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6180000000000001e-05 2024-09-15T03:47:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.893543+0000", 2024-09-15T03:47:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0097908800000000001 2024-09-15T03:47:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 88/87 ECSubWrite(tid=210, reqid=client.4521.0:740, at_version=88'113, trim_to=0'0, roll_forward_to=88'111))", 2024-09-15T03:47:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.915188+0000", 2024-09-15T03:47:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.212128706, 2024-09-15T03:47:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012691618999999999, 2024-09-15T03:47:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915188+0000", 2024-09-15T03:47:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915188+0000", 2024-09-15T03:47:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915188+0000", 2024-09-15T03:47:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3399999999999999e-07 2024-09-15T03:47:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915216+0000", 2024-09-15T03:47:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8067000000000001e-05 2024-09-15T03:47:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915233+0000", 2024-09-15T03:47:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.683e-05 2024-09-15T03:47:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915239+0000", 2024-09-15T03:47:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5720000000000002e-06 2024-09-15T03:47:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915313+0000", 2024-09-15T03:47:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4066000000000005e-05 2024-09-15T03:47:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915325+0000", 2024-09-15T03:47:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.252e-05 2024-09-15T03:47:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915603+0000", 2024-09-15T03:47:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027800699999999998 2024-09-15T03:47:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915645+0000", 2024-09-15T03:47:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1777000000000003e-05 2024-09-15T03:47:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915687+0000", 2024-09-15T03:47:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2181000000000002e-05 2024-09-15T03:47:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.917623+0000", 2024-09-15T03:47:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019357210000000001 2024-09-15T03:47:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.927879+0000", 2024-09-15T03:47:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010256243999999999 2024-09-15T03:47:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.258 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:45.260 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:47:45.260 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:45.259+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:45.360 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:45.359+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:45.401 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:45.401 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:45.401 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:45.402 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:45.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:47:45.461 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:45.461+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:45.552 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:45.552 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:45.552 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:45.553 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:45.553 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:45.560 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:47:45.562 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:45.561+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:45.663 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:45.663 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:45.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:45.663 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:45.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.663 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:197 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [delete] snapc 3=[3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e34)", 2024-09-15T03:47:45.663 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:45.664 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.548620505000002, 2024-09-15T03:47:45.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021934704999999999, 2024-09-15T03:47:45.664 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.664 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:45.664 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.664 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.664 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.664 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197 2024-09-15T03:47:45.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.664 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:45.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:45.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112266+0000", 2024-09-15T03:47:45.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5000000000000002e-07 2024-09-15T03:47:45.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112269+0000", 2024-09-15T03:47:45.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.066e-06 2024-09-15T03:47:45.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112276+0000", 2024-09-15T03:47:45.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-09-15T03:47:45.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112281+0000", 2024-09-15T03:47:45.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3970000000000004e-06 2024-09-15T03:47:45.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123281+0000", 2024-09-15T03:47:45.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011000575 2024-09-15T03:47:45.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123395+0000", 2024-09-15T03:47:45.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113398 2024-09-15T03:47:45.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123963+0000", 2024-09-15T03:47:45.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000568101 2024-09-15T03:47:45.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124220+0000", 2024-09-15T03:47:45.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025704900000000003 2024-09-15T03:47:45.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124229+0000", 2024-09-15T03:47:45.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4860000000000003e-06 2024-09-15T03:47:45.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124277+0000", 2024-09-15T03:47:45.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7574000000000002e-05 2024-09-15T03:47:45.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124312+0000", 2024-09-15T03:47:45.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5256000000000003e-05 2024-09-15T03:47:45.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:45.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125716+0000", 2024-09-15T03:47:45.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014043720000000001 2024-09-15T03:47:45.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.134200+0000", 2024-09-15T03:47:45.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084835120000000003 2024-09-15T03:47:45.671 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.671 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.671 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.671 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:220 3.bs0 3:d99ddedc:::smithi00132172-44:head [copy-from ver 22] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:45.671 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:45.672 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 60.533240835000001, 2024-09-15T03:47:45.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024273429999999999, 2024-09-15T03:47:45.672 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.672 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:45.672 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.672 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.672 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.672 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 220 2024-09-15T03:47:45.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.672 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:45.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:45.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127647+0000", 2024-09-15T03:47:45.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1799999999999999e-06 2024-09-15T03:47:45.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127656+0000", 2024-09-15T03:47:45.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3759999999999997e-06 2024-09-15T03:47:45.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127674+0000", 2024-09-15T03:47:45.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7374e-05 2024-09-15T03:47:45.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127680+0000", 2024-09-15T03:47:45.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6949999999999997e-06 2024-09-15T03:47:45.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127736+0000", 2024-09-15T03:47:45.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6011999999999998e-05 2024-09-15T03:47:45.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:45.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127744+0000", 2024-09-15T03:47:45.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2870000000000004e-06 2024-09-15T03:47:45.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130338+0000", 2024-09-15T03:47:45.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025940249999999998 2024-09-15T03:47:45.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130423+0000", 2024-09-15T03:47:45.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5303000000000006e-05 2024-09-15T03:47:45.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.147840+0000", 2024-09-15T03:47:45.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017416580000000001 2024-09-15T03:47:45.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148318+0000", 2024-09-15T03:47:45.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047822899999999999 2024-09-15T03:47:45.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148329+0000", 2024-09-15T03:47:45.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0611000000000001e-05 2024-09-15T03:47:45.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148488+0000", 2024-09-15T03:47:45.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015938399999999999 2024-09-15T03:47:45.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148870+0000", 2024-09-15T03:47:45.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000382405 2024-09-15T03:47:45.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:45.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.151092+0000", 2024-09-15T03:47:45.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022212529999999999 2024-09-15T03:47:45.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.151918+0000", 2024-09-15T03:47:45.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082671600000000002 2024-09-15T03:47:45.679 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.679 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.679 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.679 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:221 3.bs0 3:d99ddedc:::smithi00132172-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:45.679 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:45.679 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 60.533196680000003, 2024-09-15T03:47:45.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021194474000000001, 2024-09-15T03:47:45.679 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.679 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:45.680 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.680 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.680 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.680 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 221 2024-09-15T03:47:45.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.680 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:45.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:45.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127690+0000", 2024-09-15T03:47:45.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:47:45.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127692+0000", 2024-09-15T03:47:45.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6589999999999999e-06 2024-09-15T03:47:45.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127700+0000", 2024-09-15T03:47:45.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4769999999999997e-06 2024-09-15T03:47:45.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127704+0000", 2024-09-15T03:47:45.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3819999999999997e-06 2024-09-15T03:47:45.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127805+0000", 2024-09-15T03:47:45.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100639 2024-09-15T03:47:45.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:45.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127809+0000", 2024-09-15T03:47:45.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8870000000000002e-06 2024-09-15T03:47:45.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130527+0000", 2024-09-15T03:47:45.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002718579 2024-09-15T03:47:45.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:45.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130558+0000", 2024-09-15T03:47:45.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0827999999999997e-05 2024-09-15T03:47:45.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148546+0000", 2024-09-15T03:47:45.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017988356 2024-09-15T03:47:45.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148683+0000", 2024-09-15T03:47:45.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013650899999999999 2024-09-15T03:47:45.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148884+0000", 2024-09-15T03:47:45.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020061499999999999 2024-09-15T03:47:45.685 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.685 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.685 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.685 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:357 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:45.685 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:45.685 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.811845146000003, 2024-09-15T03:47:45.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.02017917, 2024-09-15T03:47:45.686 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.686 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:45.686 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.686 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.686 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.686 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 357 2024-09-15T03:47:45.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.686 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:45.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:45.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849043+0000", 2024-09-15T03:47:45.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.295e-06 2024-09-15T03:47:45.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849051+0000", 2024-09-15T03:47:45.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4079999999999998e-06 2024-09-15T03:47:45.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849069+0000", 2024-09-15T03:47:45.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7462999999999999e-05 2024-09-15T03:47:45.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849077+0000", 2024-09-15T03:47:45.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8869999999999992e-06 2024-09-15T03:47:45.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849194+0000", 2024-09-15T03:47:45.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011723899999999999 2024-09-15T03:47:45.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:45.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849296+0000", 2024-09-15T03:47:45.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102011 2024-09-15T03:47:45.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861106+0000", 2024-09-15T03:47:45.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0118097610000001 2024-09-15T03:47:45.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861222+0000", 2024-09-15T03:47:45.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116561 2024-09-15T03:47:45.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.869220+0000", 2024-09-15T03:47:45.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007997545 2024-09-15T03:47:45.691 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.691 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.691 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.691 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:358 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:45.691 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:45.691 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.811790199999997, 2024-09-15T03:47:45.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0228638050000001, 2024-09-15T03:47:45.691 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.691 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:45.692 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.692 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.692 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.692 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 358 2024-09-15T03:47:45.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.692 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:45.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:45.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:45.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4099999999999997e-07 2024-09-15T03:47:45.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849101+0000", 2024-09-15T03:47:45.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8180000000000003e-06 2024-09-15T03:47:45.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849110+0000", 2024-09-15T03:47:45.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1409999999999994e-06 2024-09-15T03:47:45.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849116+0000", 2024-09-15T03:47:45.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2390000000000002e-06 2024-09-15T03:47:45.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849326+0000", 2024-09-15T03:47:45.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000210029 2024-09-15T03:47:45.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:45.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849421+0000", 2024-09-15T03:47:45.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5775999999999999e-05 2024-09-15T03:47:45.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861647+0000", 2024-09-15T03:47:45.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0122257139999999 2024-09-15T03:47:45.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861777+0000", 2024-09-15T03:47:45.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130087 2024-09-15T03:47:45.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.871959+0000", 2024-09-15T03:47:45.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010182160000000001 2024-09-15T03:47:45.696 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.696 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.696 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.697 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:359 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:45.697 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:45.697 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.811765047999998, 2024-09-15T03:47:45.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0240468149999999, 2024-09-15T03:47:45.697 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.697 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:45.697 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.697 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.697 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.697 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 359 2024-09-15T03:47:45.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.698 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:45.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:45.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:45.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:47:45.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849124+0000", 2024-09-15T03:47:45.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2679999999999999e-06 2024-09-15T03:47:45.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849132+0000", 2024-09-15T03:47:45.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4030000000000001e-06 2024-09-15T03:47:45.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849136+0000", 2024-09-15T03:47:45.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2289999999999997e-06 2024-09-15T03:47:45.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849446+0000", 2024-09-15T03:47:45.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031043299999999999 2024-09-15T03:47:45.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:45.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849553+0000", 2024-09-15T03:47:45.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106415 2024-09-15T03:47:45.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.863404+0000", 2024-09-15T03:47:45.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0138511029999999 2024-09-15T03:47:45.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.863475+0000", 2024-09-15T03:47:45.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0814000000000003e-05 2024-09-15T03:47:45.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.873168+0000", 2024-09-15T03:47:45.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00969274 2024-09-15T03:47:45.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.702 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.702 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:474 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~786432] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:45.702 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:45.702 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 34.785840372999999, 2024-09-15T03:47:45.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98604943700000003, 2024-09-15T03:47:45.702 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.703 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:45.703 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.703 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.703 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.703 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 474 2024-09-15T03:47:45.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.703 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:45.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:45.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875048+0000", 2024-09-15T03:47:45.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1600000000000001e-06 2024-09-15T03:47:45.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875844+0000", 2024-09-15T03:47:45.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079686299999999998 2024-09-15T03:47:45.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875857+0000", 2024-09-15T03:47:45.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2889e-05 2024-09-15T03:47:45.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875864+0000", 2024-09-15T03:47:45.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6139999999999997e-06 2024-09-15T03:47:45.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875916+0000", 2024-09-15T03:47:45.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1992e-05 2024-09-15T03:47:45.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:45.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875918+0000", 2024-09-15T03:47:45.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3300000000000001e-06 2024-09-15T03:47:45.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.845918+0000", 2024-09-15T03:47:45.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96999959199999997 2024-09-15T03:47:45.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.846229+0000", 2024-09-15T03:47:45.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000311601 2024-09-15T03:47:45.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848445+0000", 2024-09-15T03:47:45.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022160499999999998 2024-09-15T03:47:45.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848880+0000", 2024-09-15T03:47:45.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043432600000000002 2024-09-15T03:47:45.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848952+0000", 2024-09-15T03:47:45.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2509000000000007e-05 2024-09-15T03:47:45.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.849311+0000", 2024-09-15T03:47:45.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000358611 2024-09-15T03:47:45.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851131+0000", 2024-09-15T03:47:45.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001820115 2024-09-15T03:47:45.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:45.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861048+0000", 2024-09-15T03:47:45.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099164830000000002 2024-09-15T03:47:45.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861095+0000", 2024-09-15T03:47:45.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7302000000000002e-05 2024-09-15T03:47:45.709 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.710 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.710 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.710 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:475 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~491520] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:45.710 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:45.710 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 34.783073901999998, 2024-09-15T03:47:45.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98407319000000004, 2024-09-15T03:47:45.710 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.710 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:45.710 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.710 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.710 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.711 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 475 2024-09-15T03:47:45.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.711 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:45.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:45.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877813+0000", 2024-09-15T03:47:45.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8800000000000002e-07 2024-09-15T03:47:45.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878260+0000", 2024-09-15T03:47:45.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044732099999999998 2024-09-15T03:47:45.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878270+0000", 2024-09-15T03:47:45.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9799999999999993e-06 2024-09-15T03:47:45.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878276+0000", 2024-09-15T03:47:45.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2269999999999998e-06 2024-09-15T03:47:45.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878313+0000", 2024-09-15T03:47:45.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6570999999999999e-05 2024-09-15T03:47:45.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:45.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878315+0000", 2024-09-15T03:47:45.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8729999999999999e-06 2024-09-15T03:47:45.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851316+0000", 2024-09-15T03:47:45.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97300154800000005 2024-09-15T03:47:45.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851440+0000", 2024-09-15T03:47:45.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124287 2024-09-15T03:47:45.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852732+0000", 2024-09-15T03:47:45.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001291174 2024-09-15T03:47:45.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852991+0000", 2024-09-15T03:47:45.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025905299999999997 2024-09-15T03:47:45.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853003+0000", 2024-09-15T03:47:45.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2908000000000001e-05 2024-09-15T03:47:45.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853104+0000", 2024-09-15T03:47:45.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010011800000000001 2024-09-15T03:47:45.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854342+0000", 2024-09-15T03:47:45.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012380189999999999 2024-09-15T03:47:45.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:45.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861206+0000", 2024-09-15T03:47:45.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068648499999999996 2024-09-15T03:47:45.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861885+0000", 2024-09-15T03:47:45.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00067857300000000002 2024-09-15T03:47:45.717 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.717 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.717 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.718 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:476 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~278528] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:45.718 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:45.718 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 34.781446889000001, 2024-09-15T03:47:45.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98268067299999995, 2024-09-15T03:47:45.718 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.718 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:45.718 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.718 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.718 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.718 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 476 2024-09-15T03:47:45.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.718 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:45.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:45.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:45.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.13e-07 2024-09-15T03:47:45.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879706+0000", 2024-09-15T03:47:45.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026656100000000001 2024-09-15T03:47:45.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879715+0000", 2024-09-15T03:47:45.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9330000000000007e-06 2024-09-15T03:47:45.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879720+0000", 2024-09-15T03:47:45.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0880000000000002e-06 2024-09-15T03:47:45.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879743+0000", 2024-09-15T03:47:45.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2654e-05 2024-09-15T03:47:45.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:45.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879743+0000", 2024-09-15T03:47:45.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.99e-07 2024-09-15T03:47:45.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853170+0000", 2024-09-15T03:47:45.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97342687900000002 2024-09-15T03:47:45.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853297+0000", 2024-09-15T03:47:45.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127157 2024-09-15T03:47:45.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854150+0000", 2024-09-15T03:47:45.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085288900000000001 2024-09-15T03:47:45.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854292+0000", 2024-09-15T03:47:45.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014199300000000001 2024-09-15T03:47:45.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854559+0000", 2024-09-15T03:47:45.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026668999999999998 2024-09-15T03:47:45.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855998+0000", 2024-09-15T03:47:45.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001438714 2024-09-15T03:47:45.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856547+0000", 2024-09-15T03:47:45.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054880399999999996 2024-09-15T03:47:45.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:45.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861858+0000", 2024-09-15T03:47:45.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053110320000000003 2024-09-15T03:47:45.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.862120+0000", 2024-09-15T03:47:45.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026196699999999999 2024-09-15T03:47:45.725 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.725 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.725 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.725 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:477 3.bs0 3:d52faf55:::smithi00132172-40:head [setxattr _header (57) in=7b] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:45.725 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:45.725 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 34.781158951999998, 2024-09-15T03:47:45.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98562198000000001, 2024-09-15T03:47:45.726 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.726 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:45.726 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.726 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.726 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.726 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 477 2024-09-15T03:47:45.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.726 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:45.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:45.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:45.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2800000000000002e-07 2024-09-15T03:47:45.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879730+0000", 2024-09-15T03:47:45.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0589999999999998e-06 2024-09-15T03:47:45.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879737+0000", 2024-09-15T03:47:45.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0419999999999997e-06 2024-09-15T03:47:45.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879741+0000", 2024-09-15T03:47:45.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9960000000000004e-06 2024-09-15T03:47:45.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879756+0000", 2024-09-15T03:47:45.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4681999999999999e-05 2024-09-15T03:47:45.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:45.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879756+0000", 2024-09-15T03:47:45.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.22e-07 2024-09-15T03:47:45.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854590+0000", 2024-09-15T03:47:45.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97483319000000002 2024-09-15T03:47:45.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854718+0000", 2024-09-15T03:47:45.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012842100000000001 2024-09-15T03:47:45.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855416+0000", 2024-09-15T03:47:45.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069759200000000003 2024-09-15T03:47:45.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855551+0000", 2024-09-15T03:47:45.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013517600000000001 2024-09-15T03:47:45.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856002+0000", 2024-09-15T03:47:45.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045169399999999999 2024-09-15T03:47:45.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856065+0000", 2024-09-15T03:47:45.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3028999999999996e-05 2024-09-15T03:47:45.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856554+0000", 2024-09-15T03:47:45.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000488486 2024-09-15T03:47:45.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:45.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865278+0000", 2024-09-15T03:47:45.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087241929999999999 2024-09-15T03:47:45.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865349+0000", 2024-09-15T03:47:45.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0669999999999999e-05 2024-09-15T03:47:45.733 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.733 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.733 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.733 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:478 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:45.733 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:45.733 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 34.781140157000003, 2024-09-15T03:47:45.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99471090500000003, 2024-09-15T03:47:45.733 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.733 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:45.733 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.734 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.734 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.734 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 478 2024-09-15T03:47:45.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.734 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:45.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:45.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:45.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2599999999999998e-07 2024-09-15T03:47:45.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879748+0000", 2024-09-15T03:47:45.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9609999999999999e-06 2024-09-15T03:47:45.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879755+0000", 2024-09-15T03:47:45.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1879999999999996e-06 2024-09-15T03:47:45.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879760+0000", 2024-09-15T03:47:45.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2560000000000002e-06 2024-09-15T03:47:45.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879797+0000", 2024-09-15T03:47:45.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6452999999999998e-05 2024-09-15T03:47:45.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:45.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879799+0000", 2024-09-15T03:47:45.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6619999999999999e-06 2024-09-15T03:47:45.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855637+0000", 2024-09-15T03:47:45.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97583850999999999 2024-09-15T03:47:45.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:45.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855763+0000", 2024-09-15T03:47:45.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012612699999999999 2024-09-15T03:47:45.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865707+0000", 2024-09-15T03:47:45.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099439660000000003 2024-09-15T03:47:45.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865803+0000", 2024-09-15T03:47:45.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5946999999999995e-05 2024-09-15T03:47:45.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.874456+0000", 2024-09-15T03:47:45.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0086534090000000008 2024-09-15T03:47:45.739 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.739 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.739 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.740 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:486 3.bs0 3:df8b18a3:::smithi00132172-17:head [delete] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e64)", 2024-09-15T03:47:45.740 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:45.740 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 34.778327677999997, 2024-09-15T03:47:45.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98362617100000005, 2024-09-15T03:47:45.740 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.740 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:45.740 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.740 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.740 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.740 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 486 2024-09-15T03:47:45.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.740 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:45.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:45.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882559+0000", 2024-09-15T03:47:45.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0359999999999999e-06 2024-09-15T03:47:45.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882564+0000", 2024-09-15T03:47:45.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6530000000000002e-06 2024-09-15T03:47:45.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882574+0000", 2024-09-15T03:47:45.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0464e-05 2024-09-15T03:47:45.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882578+0000", 2024-09-15T03:47:45.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2459999999999997e-06 2024-09-15T03:47:45.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882624+0000", 2024-09-15T03:47:45.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5698000000000002e-05 2024-09-15T03:47:45.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:45.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882627+0000", 2024-09-15T03:47:45.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3350000000000001e-06 2024-09-15T03:47:45.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856574+0000", 2024-09-15T03:47:45.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97394739399999997 2024-09-15T03:47:45.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856853+0000", 2024-09-15T03:47:45.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027918399999999998 2024-09-15T03:47:45.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.866184+0000", 2024-09-15T03:47:45.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093311609999999993 2024-09-15T03:47:45.745 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.745 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.745 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.745 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:495 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [copy-from ver 51] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-15T03:47:45.745 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:45.745 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.772426629000002, 2024-09-15T03:47:45.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023555178, 2024-09-15T03:47:45.745 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.745 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:45.745 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.745 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.746 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.746 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 495 2024-09-15T03:47:45.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.746 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:45.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:45.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888461+0000", 2024-09-15T03:47:45.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.719e-06 2024-09-15T03:47:45.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888471+0000", 2024-09-15T03:47:45.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0161e-05 2024-09-15T03:47:45.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888500+0000", 2024-09-15T03:47:45.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8989e-05 2024-09-15T03:47:45.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888508+0000", 2024-09-15T03:47:45.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5299999999999999e-06 2024-09-15T03:47:45.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888554+0000", 2024-09-15T03:47:45.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6489000000000002e-05 2024-09-15T03:47:45.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888809+0000", 2024-09-15T03:47:45.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000255467 2024-09-15T03:47:45.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.905724+0000", 2024-09-15T03:47:45.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016914348999999999 2024-09-15T03:47:45.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906244+0000", 2024-09-15T03:47:45.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052054900000000003 2024-09-15T03:47:45.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906262+0000", 2024-09-15T03:47:45.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7499999999999998e-05 2024-09-15T03:47:45.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.907367+0000", 2024-09-15T03:47:45.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011049460000000001 2024-09-15T03:47:45.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.907384+0000", 2024-09-15T03:47:45.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7000999999999999e-05 2024-09-15T03:47:45.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:45.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.911965+0000", 2024-09-15T03:47:45.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045809470000000001 2024-09-15T03:47:45.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.912014+0000", 2024-09-15T03:47:45.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9530999999999997e-05 2024-09-15T03:47:45.751 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.751 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.752 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.752 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:496 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-09-15T03:47:45.752 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.888524+0000", 2024-09-15T03:47:45.752 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.772361537000002, 2024-09-15T03:47:45.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01823953, 2024-09-15T03:47:45.752 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.752 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:45.752 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.752 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.752 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.752 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 496 2024-09-15T03:47:45.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.753 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888524+0000", 2024-09-15T03:47:45.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888524+0000", 2024-09-15T03:47:45.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888525+0000", 2024-09-15T03:47:45.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2699999999999999e-07 2024-09-15T03:47:45.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888529+0000", 2024-09-15T03:47:45.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7840000000000001e-06 2024-09-15T03:47:45.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888540+0000", 2024-09-15T03:47:45.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1477e-05 2024-09-15T03:47:45.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888546+0000", 2024-09-15T03:47:45.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4990000000000002e-06 2024-09-15T03:47:45.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888971+0000", 2024-09-15T03:47:45.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042491300000000002 2024-09-15T03:47:45.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:45.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.889024+0000", 2024-09-15T03:47:45.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3046e-05 2024-09-15T03:47:45.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906517+0000", 2024-09-15T03:47:45.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017492909000000001 2024-09-15T03:47:45.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906638+0000", 2024-09-15T03:47:45.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012138000000000001 2024-09-15T03:47:45.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906764+0000", 2024-09-15T03:47:45.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125795 2024-09-15T03:47:45.757 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.757 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.757 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.758 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:689 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~5144576,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:45.758 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.088846+0000", 2024-09-15T03:47:45.758 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.572040209000001, 2024-09-15T03:47:45.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020311780000000002, 2024-09-15T03:47:45.758 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.758 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:45.758 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.758 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.758 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.758 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 689 2024-09-15T03:47:45.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.758 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088846+0000", 2024-09-15T03:47:45.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088846+0000", 2024-09-15T03:47:45.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088847+0000", 2024-09-15T03:47:45.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9300000000000006e-07 2024-09-15T03:47:45.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088852+0000", 2024-09-15T03:47:45.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1850000000000003e-06 2024-09-15T03:47:45.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088863+0000", 2024-09-15T03:47:45.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1418e-05 2024-09-15T03:47:45.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088873+0000", 2024-09-15T03:47:45.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4020000000000008e-06 2024-09-15T03:47:45.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.089560+0000", 2024-09-15T03:47:45.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068774899999999998 2024-09-15T03:47:45.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090073+0000", 2024-09-15T03:47:45.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051222799999999999 2024-09-15T03:47:45.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.109157+0000", 2024-09-15T03:47:45.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019084805 2024-09-15T03:47:45.762 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.762 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.762 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.762 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:690 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~5144576,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:45.763 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:45.763 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.572007071, 2024-09-15T03:47:45.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025340499999999998, 2024-09-15T03:47:45.763 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.763 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:45.763 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.763 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.763 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.763 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 690 2024-09-15T03:47:45.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.763 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:45.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:45.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:45.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7999999999999996e-07 2024-09-15T03:47:45.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088882+0000", 2024-09-15T03:47:45.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1839999999999999e-06 2024-09-15T03:47:45.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088892+0000", 2024-09-15T03:47:45.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3540000000000006e-06 2024-09-15T03:47:45.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088896+0000", 2024-09-15T03:47:45.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5210000000000004e-06 2024-09-15T03:47:45.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.093968+0000", 2024-09-15T03:47:45.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050715170000000002 2024-09-15T03:47:45.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.094098+0000", 2024-09-15T03:47:45.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013070000000000001 2024-09-15T03:47:45.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.114219+0000", 2024-09-15T03:47:45.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020120743999999999 2024-09-15T03:47:45.767 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.767 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.767 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.767 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:691 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~5144576,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:45.767 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.088900+0000", 2024-09-15T03:47:45.767 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.571985328, 2024-09-15T03:47:45.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031407274999999998, 2024-09-15T03:47:45.767 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.767 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:45.768 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.768 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.768 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.768 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 691 2024-09-15T03:47:45.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.768 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088900+0000", 2024-09-15T03:47:45.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088900+0000", 2024-09-15T03:47:45.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088901+0000", 2024-09-15T03:47:45.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9099999999999999e-07 2024-09-15T03:47:45.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088904+0000", 2024-09-15T03:47:45.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0299999999999998e-06 2024-09-15T03:47:45.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088911+0000", 2024-09-15T03:47:45.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.708e-06 2024-09-15T03:47:45.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088915+0000", 2024-09-15T03:47:45.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0579999999999999e-06 2024-09-15T03:47:45.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.096143+0000", 2024-09-15T03:47:45.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007228099 2024-09-15T03:47:45.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.096202+0000", 2024-09-15T03:47:45.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9667999999999999e-05 2024-09-15T03:47:45.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.120308+0000", 2024-09-15T03:47:45.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024105320999999999 2024-09-15T03:47:45.771 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.771 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.771 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.772 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(osd.0.17:2 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [assert-version v51,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:45.772 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.090507+0000", 2024-09-15T03:47:45.772 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.570378861, 2024-09-15T03:47:45.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025010186, 2024-09-15T03:47:45.772 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.772 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:45.772 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090507+0000", 2024-09-15T03:47:45.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090507+0000", 2024-09-15T03:47:45.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090508+0000", 2024-09-15T03:47:45.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.176e-06 2024-09-15T03:47:45.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090512+0000", 2024-09-15T03:47:45.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2989999999999998e-06 2024-09-15T03:47:45.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090522+0000", 2024-09-15T03:47:45.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6549999999999999e-06 2024-09-15T03:47:45.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090526+0000", 2024-09-15T03:47:45.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8500000000000004e-06 2024-09-15T03:47:45.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.094385+0000", 2024-09-15T03:47:45.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038591670000000002 2024-09-15T03:47:45.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.094541+0000", 2024-09-15T03:47:45.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015590999999999999 2024-09-15T03:47:45.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.115517+0000", 2024-09-15T03:47:45.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020976129 2024-09-15T03:47:45.776 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.776 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.776 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.776 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:749 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~737280,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-15T03:47:45.776 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.935286+0000", 2024-09-15T03:47:45.776 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.725599321000001, 2024-09-15T03:47:45.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021703044000000001, 2024-09-15T03:47:45.776 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.776 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:45.776 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.776 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.777 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.777 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 749 2024-09-15T03:47:45.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.777 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935286+0000", 2024-09-15T03:47:45.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935286+0000", 2024-09-15T03:47:45.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935287+0000", 2024-09-15T03:47:45.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.51e-07 2024-09-15T03:47:45.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935290+0000", 2024-09-15T03:47:45.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8710000000000001e-06 2024-09-15T03:47:45.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935297+0000", 2024-09-15T03:47:45.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3390000000000004e-06 2024-09-15T03:47:45.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935302+0000", 2024-09-15T03:47:45.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5560000000000001e-06 2024-09-15T03:47:45.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936361+0000", 2024-09-15T03:47:45.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010593200000000001 2024-09-15T03:47:45.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:47:45.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936486+0000", 2024-09-15T03:47:45.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124657 2024-09-15T03:47:45.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.952979+0000", 2024-09-15T03:47:45.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016493154999999999 2024-09-15T03:47:45.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.953132+0000", 2024-09-15T03:47:45.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015346500000000001 2024-09-15T03:47:45.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.956989+0000", 2024-09-15T03:47:45.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038571299999999998 2024-09-15T03:47:45.781 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.781 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.781 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.781 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:750 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [create,setxattr \\^tkqj^qnPPfYPq\\WZoebWOkaebalTko_OhaYsrZ (1787) in=40b,setxattr cg]RoaNaVsOQTrQfp[VeRj_tmeegdoc`PtRYpskN (1998) in=40b,setxattr \\_N_s\\f^o_Ojh]gqWPkfsZ\\WTSbY]SOlX`mlV[^q (6944) in=40b,setxattr nsdiO]arkaqtYt (33) in=14b,setxattr ti`p[nirg_rWRRSe[oZhhn]b]ZsTimedg]]p[sQY (1049) in=40b,setxattr i[YNharOrm`bbP]UtYhjlqVWP\\XnccYlfhV_deTf (383) in=40b,setxattr ka_q[S]a]naRePWgcgUWmZSaW^RVV\\_rNXdZefal (875) in=40b,setxattr gaoaanUtdfqUrimS^_NfUtarRjNSVRf^_PbUpdtf (1645) in=40b,setxattr qNb^mqtq^Vsg]m\\aWdRTbbgOlZYptsW]O[_R`Vaa (1155) in=40b,setxattr gUPP_ttp`PdrngYsWrVdfZZVgff^Q`YRSjgOXTao (417) in=40b,setxattr mi`p\\f[PUQf^[[skf^\\Za_iXO_oTscOWNnQqcR[[ (4560) in=40b,setxattr N^_ka_abmi]seatXYWSQtibSdeOsfV\\OXsmmXlYm (1893) in=40b,setxattr aoqZmYaUpNW (33) in=11b,setxattr Z[bbsdRZTfkrP`]akaXifm`oV[YTOPrPaeeXV[fl (1611) in=40b,setxattr oZ`SalVqYt^T_[b_a]r[rlaaWOa_WZQ^^opf\\aN_ (243) in=40b,setxattr k\\V^enpkstj^VpoopX\\PcoTY]eogbs`pQQW]ofrm (347) in=40b,setxattr cfXfpnr`SogqjNa^cTVYtpgdciPp_eTUg]Ujn\\at (1926) in=40b,setxattr t`ojSZgRjbb[aaUXSepgQf_nc`kaOetle^n^`Vfd (1087) in=40b,setxattr ONYdltaQhe\\a\\da\\ehQRpsS^Ugdgmf\\WWigfUarV (631) in=40b,setxattr [Tj`Rbkt]qjatpSf_fPsanNr^ipYnOQlPnUZlmRU (4068) in=40b] snapc 32=[32,2d,2b,26,25,24,23,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T03:47:45.781 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.935912+0000", 2024-09-15T03:47:45.781 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.724973996999999, 2024-09-15T03:47:45.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03915623, 2024-09-15T03:47:45.782 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:45.782 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:45.782 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:45.782 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:45.782 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:45.782 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 750 2024-09-15T03:47:45.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.782 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:45.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:45.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935912+0000", 2024-09-15T03:47:45.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:45.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935912+0000", 2024-09-15T03:47:45.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:45.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:45.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935913+0000", 2024-09-15T03:47:45.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7599999999999996e-07 2024-09-15T03:47:45.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:45.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935984+0000", 2024-09-15T03:47:45.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1136000000000005e-05 2024-09-15T03:47:45.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:45.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935997+0000", 2024-09-15T03:47:45.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3482000000000001e-05 2024-09-15T03:47:45.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:45.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936004+0000", 2024-09-15T03:47:45.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3260000000000001e-06 2024-09-15T03:47:45.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936878+0000", 2024-09-15T03:47:45.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087483999999999999 2024-09-15T03:47:45.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:47:45.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.937895+0000", 2024-09-15T03:47:45.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010163209999999999 2024-09-15T03:47:45.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:45.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.965388+0000", 2024-09-15T03:47:45.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027492986000000001 2024-09-15T03:47:45.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:45.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.965630+0000", 2024-09-15T03:47:45.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000242012 2024-09-15T03:47:45.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:45.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.966983+0000", 2024-09-15T03:47:45.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013535349999999999 2024-09-15T03:47:45.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:45.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.967360+0000", 2024-09-15T03:47:45.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037704399999999999 2024-09-15T03:47:45.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:45.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.967371+0000", 2024-09-15T03:47:45.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0988999999999999e-05 2024-09-15T03:47:45.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:45.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.967436+0000", 2024-09-15T03:47:45.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4679999999999997e-05 2024-09-15T03:47:45.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:45.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.969080+0000", 2024-09-15T03:47:45.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001644017 2024-09-15T03:47:45.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:45.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.969532+0000", 2024-09-15T03:47:45.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045157699999999999 2024-09-15T03:47:45.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:45.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:45.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:45.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.975068+0000", 2024-09-15T03:47:45.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055364089999999999 2024-09-15T03:47:45.789 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.789 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.789 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.789 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:45.789 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:45.789 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:45.790 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:45.662+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:45.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:47:45.791 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:45.789+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:45.861 INFO:tasks.rados.rados.0.smithi001.stdout:337: done (2 left) 2024-09-15T03:47:45.861 INFO:tasks.rados.rados.0.smithi001.stdout:338: rollback oid 27 current snap is 41 2024-09-15T03:47:45.861 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 27 to 30 2024-09-15T03:47:45.861 INFO:tasks.rados.rados.0.smithi001.stdout:334: finishing write tid 6 to smithi00132172-2 2024-09-15T03:47:45.861 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 2 v 110 (ObjNum 128 snap 41 seq_num 128) dirty exists 2024-09-15T03:47:45.861 INFO:tasks.rados.rados.0.smithi001.stdout:334: left oid 2 (ObjNum 128 snap 41 seq_num 128) 2024-09-15T03:47:45.861 INFO:tasks.rados.rados.0.smithi001.stdout:336: expect (ObjNum 112 snap 32 seq_num 112) 2024-09-15T03:47:45.892 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:45.890+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:45.904 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:45.904 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:45.905 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:45.905 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:45.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:47:45.927 INFO:tasks.rados.rados.0.smithi001.stdout:338: finishing rollback tid 0 to smithi00132172-27 2024-09-15T03:47:45.927 INFO:tasks.rados.rados.0.smithi001.stdout:338: finishing rollback tid 1 to smithi00132172-27 2024-09-15T03:47:45.927 INFO:tasks.rados.rados.0.smithi001.stdout:338: finishing rollback tid 2 to smithi00132172-27 2024-09-15T03:47:45.927 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 27 v 62 (ObjNum 85 snap 18 seq_num 85) dirty exists 2024-09-15T03:47:45.927 INFO:tasks.rados.rados.0.smithi001.stdout:334: done (2 left) 2024-09-15T03:47:45.928 INFO:tasks.rados.rados.0.smithi001.stdout:336: done (1 left) 2024-09-15T03:47:45.928 INFO:tasks.rados.rados.0.smithi001.stdout:338: done (0 left) 2024-09-15T03:47:45.928 INFO:tasks.rados.rados.0.smithi001.stdout:339: rmattr oid 39 current snap is 41 2024-09-15T03:47:45.928 INFO:tasks.rados.rados.0.smithi001.stdout:339: done (0 left) 2024-09-15T03:47:45.928 INFO:tasks.rados.rados.0.smithi001.stdout:340: setattr oid 7 current snap is 41 2024-09-15T03:47:45.928 INFO:tasks.rados.rados.0.smithi001.stdout:341: delete oid 17 current snap is 41 2024-09-15T03:47:45.930 INFO:tasks.rados.rados.0.smithi001.stdout:341: done (1 left) 2024-09-15T03:47:45.931 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 1 current snap is 41 2024-09-15T03:47:45.931 INFO:tasks.rados.rados.0.smithi001.stdout:342: seq_num 131 ranges {0=655360,655360=425984,1081344=638976,1720320=491520} 2024-09-15T03:47:45.934 INFO:tasks.rados.rados.0.smithi001.stdout:342: writing smithi00132172-1 from 0 to 655360 tid 1 2024-09-15T03:47:45.937 INFO:tasks.rados.rados.0.smithi001.stdout:342: writing smithi00132172-1 from 655360 to 1081344 tid 2 2024-09-15T03:47:45.941 INFO:tasks.rados.rados.0.smithi001.stdout:342: writing smithi00132172-1 from 1081344 to 1720320 tid 3 2024-09-15T03:47:45.944 INFO:tasks.rados.rados.0.smithi001.stdout:342: writing smithi00132172-1 from 1720320 to 2211840 tid 4 2024-09-15T03:47:45.944 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 46 current snap is 41 2024-09-15T03:47:45.944 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 7 v 112 (ObjNum 113 snap 32 seq_num 113) dirty exists 2024-09-15T03:47:45.944 INFO:tasks.rados.rados.0.smithi001.stdout:342: finishing write tid 1 to smithi00132172-1 2024-09-15T03:47:45.944 INFO:tasks.rados.rados.0.smithi001.stdout:342: finishing write tid 2 to smithi00132172-1 2024-09-15T03:47:45.945 INFO:tasks.rados.rados.0.smithi001.stdout:342: finishing write tid 3 to smithi00132172-1 2024-09-15T03:47:45.945 INFO:tasks.rados.rados.0.smithi001.stdout:343: seq_num 132 ranges {770048=671744,1441792=573440,2015232=753664,2768896=229376} 2024-09-15T03:47:45.952 INFO:tasks.rados.rados.0.smithi001.stdout:343: writing smithi00132172-46 from 770048 to 1441792 tid 1 2024-09-15T03:47:45.955 INFO:tasks.rados.rados.0.smithi001.stdout:343: writing smithi00132172-46 from 1441792 to 2015232 tid 2 2024-09-15T03:47:45.959 INFO:tasks.rados.rados.0.smithi001.stdout:343: writing smithi00132172-46 from 2015232 to 2768896 tid 3 2024-09-15T03:47:45.961 INFO:tasks.rados.rados.0.smithi001.stdout:343: writing smithi00132172-46 from 2768896 to 2998272 tid 4 2024-09-15T03:47:45.962 INFO:tasks.rados.rados.0.smithi001.stdout:340: done (2 left) 2024-09-15T03:47:45.962 INFO:tasks.rados.rados.0.smithi001.stdout:344: snap_create 2024-09-15T03:47:45.962 INFO:tasks.rados.rados.0.smithi001.stdout:342: finishing write tid 4 to smithi00132172-1 2024-09-15T03:47:45.962 INFO:tasks.rados.rados.0.smithi001.stdout:342: finishing write tid 5 to smithi00132172-1 2024-09-15T03:47:45.962 INFO:tasks.rados.rados.0.smithi001.stdout:342: finishing write tid 6 to smithi00132172-1 2024-09-15T03:47:45.962 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 1 v 46 (ObjNum 131 snap 41 seq_num 131) dirty exists 2024-09-15T03:47:45.962 INFO:tasks.rados.rados.0.smithi001.stdout:342: left oid 1 (ObjNum 131 snap 41 seq_num 131) 2024-09-15T03:47:45.962 INFO:tasks.rados.rados.0.smithi001.stdout:343: finishing write tid 1 to smithi00132172-46 2024-09-15T03:47:45.962 INFO:tasks.rados.rados.0.smithi001.stdout:343: finishing write tid 2 to smithi00132172-46 2024-09-15T03:47:45.963 INFO:tasks.rados.rados.0.smithi001.stdout:343: finishing write tid 3 to smithi00132172-46 2024-09-15T03:47:45.963 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:47:45.964 INFO:tasks.rados.rados.0.smithi001.stdout:343: finishing write tid 4 to smithi00132172-46 2024-09-15T03:47:45.964 INFO:tasks.rados.rados.0.smithi001.stdout:343: finishing write tid 5 to smithi00132172-46 2024-09-15T03:47:45.966 INFO:tasks.rados.rados.0.smithi001.stdout:343: finishing write tid 6 to smithi00132172-46 2024-09-15T03:47:45.966 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 46 v 118 (ObjNum 132 snap 41 seq_num 132) dirty exists 2024-09-15T03:47:45.966 INFO:tasks.rados.rados.0.smithi001.stdout:343: left oid 46 (ObjNum 132 snap 41 seq_num 132) 2024-09-15T03:47:45.993 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:45.992+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:46.017 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:46.017 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:46.017 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:46.018 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:46.018 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:46.025 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:47:46.081 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:46.081 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:46.094 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:46.093+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:46.095 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:46.095 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:46.109 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:46.109 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:46.123 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:46.123 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:46.138 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:46.139 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:46.153 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:46.153 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:46.164 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:46.164 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:46.164 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:46.164 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:47:46.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.165 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:47:46.165 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:46.165 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 88.259011376999993, 2024-09-15T03:47:46.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10988437500000001, 2024-09-15T03:47:46.165 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:46.165 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:46.165 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:46.165 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:47:46.165 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:47:46.165 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1 2024-09-15T03:47:46.165 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.165 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:46.165 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.165 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:46.166 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:46.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.166 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:46.166 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:46.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.166 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:46.166 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902355+0000", 2024-09-15T03:47:46.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.429e-06 2024-09-15T03:47:46.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.167 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.167 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:46.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902362+0000", 2024-09-15T03:47:46.167 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7290000000000003e-06 2024-09-15T03:47:46.167 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.167 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.167 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:46.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902373+0000", 2024-09-15T03:47:46.167 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0533999999999999e-05 2024-09-15T03:47:46.167 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.167 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.167 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:46.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902379+0000", 2024-09-15T03:47:46.167 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4350000000000002e-06 2024-09-15T03:47:46.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.168 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902428+0000", 2024-09-15T03:47:46.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8571e-05 2024-09-15T03:47:46.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.168 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:47:46.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902434+0000", 2024-09-15T03:47:46.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5610000000000004e-06 2024-09-15T03:47:46.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011307+0000", 2024-09-15T03:47:46.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.108872306 2024-09-15T03:47:46.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:46.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011448+0000", 2024-09-15T03:47:46.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014137199999999999 2024-09-15T03:47:46.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.170 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:47:46.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011569+0000", 2024-09-15T03:47:46.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120925 2024-09-15T03:47:46.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.170 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.170 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:46.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011695+0000", 2024-09-15T03:47:46.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012586800000000001 2024-09-15T03:47:46.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.170 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.170 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:46.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011703+0000", 2024-09-15T03:47:46.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4190000000000002e-06 2024-09-15T03:47:46.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.171 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:46.171 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011739+0000", 2024-09-15T03:47:46.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5474e-05 2024-09-15T03:47:46.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.171 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:47:46.171 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011940+0000", 2024-09-15T03:47:46.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020108699999999999 2024-09-15T03:47:46.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.171 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:47:46.172 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012179+0000", 2024-09-15T03:47:46.172 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023950299999999999 2024-09-15T03:47:46.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.172 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.172 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:46.172 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012215+0000", 2024-09-15T03:47:46.172 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5332e-05 2024-09-15T03:47:46.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.172 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.172 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:46.172 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012238+0000", 2024-09-15T03:47:46.172 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3830000000000001e-05 2024-09-15T03:47:46.172 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.173 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:46.173 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.173 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.173 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4263.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2024-09-15T03:47:46.173 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:46.173 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 82.117614691, 2024-09-15T03:47:46.173 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019495788999999999, 2024-09-15T03:47:46.173 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:46.173 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:46.173 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:46.173 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4263", 2024-09-15T03:47:46.173 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/646873808", 2024-09-15T03:47:46.173 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2 2024-09-15T03:47:46.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.174 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:46.174 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.174 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:46.174 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:46.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:46.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:46.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:46.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043752+0000", 2024-09-15T03:47:46.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.815e-06 2024-09-15T03:47:46.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:46.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043757+0000", 2024-09-15T03:47:46.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4240000000000003e-06 2024-09-15T03:47:46.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:46.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043766+0000", 2024-09-15T03:47:46.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8629999999999997e-06 2024-09-15T03:47:46.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:46.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043773+0000", 2024-09-15T03:47:46.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0099999999999998e-06 2024-09-15T03:47:46.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.177 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.177 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043830+0000", 2024-09-15T03:47:46.177 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7089999999999997e-05 2024-09-15T03:47:46.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.177 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:47:46.177 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043837+0000", 2024-09-15T03:47:46.177 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.08e-06 2024-09-15T03:47:46.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.177 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.177 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063103+0000", 2024-09-15T03:47:46.178 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019265619000000001 2024-09-15T03:47:46.178 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.178 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.178 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:46.178 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063246+0000", 2024-09-15T03:47:46.178 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000143888 2024-09-15T03:47:46.178 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.178 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:46.178 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.178 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.178 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.178 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:47:46.178 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:46.178 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 59.990541301999997, 2024-09-15T03:47:46.179 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011878227, 2024-09-15T03:47:46.179 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:46.179 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:46.179 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:46.179 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.179 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:46.179 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:46.179 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.179 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.180 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.180 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:46.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:46.180 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.180 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.180 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:46.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170825+0000", 2024-09-15T03:47:46.180 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9599999999999999e-07 2024-09-15T03:47:46.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.180 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.180 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:46.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171100+0000", 2024-09-15T03:47:46.181 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027548500000000001 2024-09-15T03:47:46.181 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.181 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:46.181 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171112+0000", 2024-09-15T03:47:46.181 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2152e-05 2024-09-15T03:47:46.181 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.181 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:46.181 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171119+0000", 2024-09-15T03:47:46.181 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5050000000000004e-06 2024-09-15T03:47:46.181 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.181 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.182 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171162+0000", 2024-09-15T03:47:46.182 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2908000000000003e-05 2024-09-15T03:47:46.182 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.182 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.182 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:46.182 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171169+0000", 2024-09-15T03:47:46.182 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9739999999999996e-06 2024-09-15T03:47:46.182 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.182 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.182 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:46.182 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171398+0000", 2024-09-15T03:47:46.182 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022954799999999999 2024-09-15T03:47:46.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.183 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.183 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:46.183 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173640+0000", 2024-09-15T03:47:46.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022417000000000001 2024-09-15T03:47:46.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.183 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.183 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:46.183 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173705+0000", 2024-09-15T03:47:46.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4907999999999996e-05 2024-09-15T03:47:46.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.183 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.183 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:46.183 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173742+0000", 2024-09-15T03:47:46.184 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6940999999999997e-05 2024-09-15T03:47:46.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.184 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:46.184 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.182702+0000", 2024-09-15T03:47:46.184 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0089604100000000002 2024-09-15T03:47:46.184 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.184 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:46.184 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.184 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 39/31 ECSubWrite(tid=43, reqid=client.4521.0:252, at_version=39'27, trim_to=0'0, roll_forward_to=39'25))", 2024-09-15T03:47:46.184 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:46.185 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 59.982823983999999, 2024-09-15T03:47:46.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011286757, 2024-09-15T03:47:46.185 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:46.185 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:46.185 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:46.185 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.185 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:46.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:46.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.186 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.186 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:46.186 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:46.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.186 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.186 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:46.186 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178542+0000", 2024-09-15T03:47:46.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9400000000000004e-07 2024-09-15T03:47:46.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:46.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178754+0000", 2024-09-15T03:47:46.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021228800000000001 2024-09-15T03:47:46.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:46.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178765+0000", 2024-09-15T03:47:46.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0176e-05 2024-09-15T03:47:46.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:46.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178769+0000", 2024-09-15T03:47:46.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1359999999999999e-06 2024-09-15T03:47:46.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178810+0000", 2024-09-15T03:47:46.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0825999999999998e-05 2024-09-15T03:47:46.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:46.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178820+0000", 2024-09-15T03:47:46.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0237999999999999e-05 2024-09-15T03:47:46.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:46.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179033+0000", 2024-09-15T03:47:46.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021361500000000001 2024-09-15T03:47:46.189 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.189 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.189 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:46.189 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179040+0000", 2024-09-15T03:47:46.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0509999999999999e-06 2024-09-15T03:47:46.189 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.189 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.189 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:46.189 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179102+0000", 2024-09-15T03:47:46.190 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1977000000000006e-05 2024-09-15T03:47:46.190 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.190 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:46.190 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179114+0000", 2024-09-15T03:47:46.190 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1223000000000001e-05 2024-09-15T03:47:46.190 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.190 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:46.190 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.189828+0000", 2024-09-15T03:47:46.190 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010714433000000001 2024-09-15T03:47:46.190 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.191 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:46.191 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.191 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 51/31 ECSubWrite(tid=67, reqid=client.4521.0:350, at_version=51'33, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:46.191 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:46.191 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 48.317523942999998, 2024-09-15T03:47:46.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.015569239, 2024-09-15T03:47:46.191 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:46.191 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:46.191 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:46.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:46.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:46.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:46.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:46.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:46.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843843+0000", 2024-09-15T03:47:46.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.195e-06 2024-09-15T03:47:46.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:46.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843880+0000", 2024-09-15T03:47:46.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7527999999999997e-05 2024-09-15T03:47:46.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:46.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843898+0000", 2024-09-15T03:47:46.196 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7696000000000001e-05 2024-09-15T03:47:46.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.196 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:46.196 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843907+0000", 2024-09-15T03:47:46.196 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8820000000000007e-06 2024-09-15T03:47:46.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.196 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.196 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843978+0000", 2024-09-15T03:47:46.196 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1537000000000003e-05 2024-09-15T03:47:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843989+0000", 2024-09-15T03:47:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0720000000000001e-05 2024-09-15T03:47:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844102+0000", 2024-09-15T03:47:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000113406 2024-09-15T03:47:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844131+0000", 2024-09-15T03:47:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8762e-05 2024-09-15T03:47:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844180+0000", 2024-09-15T03:47:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9329999999999997e-05 2024-09-15T03:47:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844201+0000", 2024-09-15T03:47:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0772999999999999e-05 2024-09-15T03:47:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.199 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:46.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.859411+0000", 2024-09-15T03:47:46.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01520941 2024-09-15T03:47:46.199 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.199 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:46.199 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.199 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:46.199 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:46.199 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 48.316645031999997, 2024-09-15T03:47:46.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012156452, 2024-09-15T03:47:46.199 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:46.200 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:46.200 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:46.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.200 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:46.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:46.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.200 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:46.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:46.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.201 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:46.201 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844721+0000", 2024-09-15T03:47:46.201 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1100000000000005e-07 2024-09-15T03:47:46.201 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.201 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.201 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:46.201 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844743+0000", 2024-09-15T03:47:46.201 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1999000000000001e-05 2024-09-15T03:47:46.201 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.201 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.201 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:46.201 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844758+0000", 2024-09-15T03:47:46.201 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4718999999999999e-05 2024-09-15T03:47:46.201 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:46.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844772+0000", 2024-09-15T03:47:46.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4637000000000001e-05 2024-09-15T03:47:46.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844842+0000", 2024-09-15T03:47:46.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9596000000000002e-05 2024-09-15T03:47:46.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:46.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844853+0000", 2024-09-15T03:47:46.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0912e-05 2024-09-15T03:47:46.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:46.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844966+0000", 2024-09-15T03:47:46.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011276 2024-09-15T03:47:46.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:46.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844977+0000", 2024-09-15T03:47:46.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1676e-05 2024-09-15T03:47:46.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:46.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847414+0000", 2024-09-15T03:47:46.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024363760000000001 2024-09-15T03:47:46.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:46.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847438+0000", 2024-09-15T03:47:46.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3737e-05 2024-09-15T03:47:46.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:46.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856877+0000", 2024-09-15T03:47:46.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094392290000000004 2024-09-15T03:47:46.204 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.205 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:46.205 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.205 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:46.205 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:46.205 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 48.316575637, 2024-09-15T03:47:46.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012258211, 2024-09-15T03:47:46.205 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:46.205 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:46.205 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:46.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:46.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:46.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:46.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:46.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:46.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:46.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.58e-07 2024-09-15T03:47:46.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:46.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844818+0000", 2024-09-15T03:47:46.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.796e-05 2024-09-15T03:47:46.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.207 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:46.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844829+0000", 2024-09-15T03:47:46.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0895e-05 2024-09-15T03:47:46.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.207 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:46.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844836+0000", 2024-09-15T03:47:46.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.618e-06 2024-09-15T03:47:46.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.208 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845023+0000", 2024-09-15T03:47:46.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018743500000000001 2024-09-15T03:47:46.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.208 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:46.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845035+0000", 2024-09-15T03:47:46.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1958e-05 2024-09-15T03:47:46.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.208 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:46.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845147+0000", 2024-09-15T03:47:46.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112281 2024-09-15T03:47:46.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:46.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847417+0000", 2024-09-15T03:47:46.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022693969999999998 2024-09-15T03:47:46.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:46.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847462+0000", 2024-09-15T03:47:46.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5466999999999997e-05 2024-09-15T03:47:46.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:46.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847503+0000", 2024-09-15T03:47:46.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1022999999999997e-05 2024-09-15T03:47:46.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:46.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.857048+0000", 2024-09-15T03:47:46.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095447190000000001 2024-09-15T03:47:46.210 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.210 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:46.210 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.211 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 51/31 ECSubWrite(tid=68, reqid=client.4521.0:351, at_version=51'34, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:46.211 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:46.211 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 48.316157519000001, 2024-09-15T03:47:46.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.014544793, 2024-09-15T03:47:46.211 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:46.211 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:46.211 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:46.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:46.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:46.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.212 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:46.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:46.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.212 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.212 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:46.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845209+0000", 2024-09-15T03:47:46.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5300000000000003e-07 2024-09-15T03:47:46.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.212 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.212 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:46.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845227+0000", 2024-09-15T03:47:46.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8759000000000001e-05 2024-09-15T03:47:46.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.213 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:46.213 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845241+0000", 2024-09-15T03:47:46.213 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3740000000000001e-05 2024-09-15T03:47:46.213 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.213 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:46.213 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845247+0000", 2024-09-15T03:47:46.213 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2779999999999999e-06 2024-09-15T03:47:46.213 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.213 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.213 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845308+0000", 2024-09-15T03:47:46.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0702000000000003e-05 2024-09-15T03:47:46.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:46.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845317+0000", 2024-09-15T03:47:46.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3770000000000002e-06 2024-09-15T03:47:46.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:46.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845413+0000", 2024-09-15T03:47:46.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5356e-05 2024-09-15T03:47:46.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.215 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:46.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847421+0000", 2024-09-15T03:47:46.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020077200000000002 2024-09-15T03:47:46.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.215 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.215 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:46.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847466+0000", 2024-09-15T03:47:46.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5754999999999998e-05 2024-09-15T03:47:46.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.215 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.215 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:46.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847572+0000", 2024-09-15T03:47:46.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010594000000000001 2024-09-15T03:47:46.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:46.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.859753+0000", 2024-09-15T03:47:46.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012180412999999999 2024-09-15T03:47:46.216 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.216 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:46.216 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.216 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 60/28 ECSubWrite(tid=73, reqid=client.4521.0:424, at_version=60'41, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:46.216 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:46.216 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 39.260954265000002, 2024-09-15T03:47:46.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01137208, 2024-09-15T03:47:46.217 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:46.217 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:46.217 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:46.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:46.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:46.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:46.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:46.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.218 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:46.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900412+0000", 2024-09-15T03:47:46.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7599999999999996e-07 2024-09-15T03:47:46.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.218 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.218 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:46.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:46.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7986999999999999e-05 2024-09-15T03:47:46.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.218 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.218 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:46.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900445+0000", 2024-09-15T03:47:46.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.468e-05 2024-09-15T03:47:46.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.219 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.219 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:46.219 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900451+0000", 2024-09-15T03:47:46.219 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5799999999999997e-06 2024-09-15T03:47:46.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.219 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.219 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.219 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900507+0000", 2024-09-15T03:47:46.219 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5855e-05 2024-09-15T03:47:46.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.219 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.219 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:46.219 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900530+0000", 2024-09-15T03:47:46.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2929e-05 2024-09-15T03:47:46.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:46.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900640+0000", 2024-09-15T03:47:46.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000110536 2024-09-15T03:47:46.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:46.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900648+0000", 2024-09-15T03:47:46.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9470000000000003e-06 2024-09-15T03:47:46.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.221 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:46.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900764+0000", 2024-09-15T03:47:46.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011518099999999999 2024-09-15T03:47:46.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.221 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.221 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:46.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900803+0000", 2024-09-15T03:47:46.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9941999999999998e-05 2024-09-15T03:47:46.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.221 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.221 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:46.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.911783+0000", 2024-09-15T03:47:46.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010979667 2024-09-15T03:47:46.222 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.222 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:46.222 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.222 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:517 3.3s1 3:caaee79c:::smithi00132172-10:head [read 0~3145728,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:46.222 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.947050+0000", 2024-09-15T03:47:46.222 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 30.214315332000002, 2024-09-15T03:47:46.222 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012110134, 2024-09-15T03:47:46.222 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:46.222 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:46.222 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:46.222 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:46.223 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:46.223 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 517 2024-09-15T03:47:46.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.223 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:46.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:46.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947050+0000", 2024-09-15T03:47:46.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:46.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947050+0000", 2024-09-15T03:47:46.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:46.224 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947052+0000", 2024-09-15T03:47:46.224 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9099999999999999e-06 2024-09-15T03:47:46.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:46.224 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947060+0000", 2024-09-15T03:47:46.224 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8529999999999992e-06 2024-09-15T03:47:46.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:46.224 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947074+0000", 2024-09-15T03:47:46.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4344e-05 2024-09-15T03:47:46.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:46.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947082+0000", 2024-09-15T03:47:46.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9249999999999995e-06 2024-09-15T03:47:46.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947138+0000", 2024-09-15T03:47:46.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6054000000000002e-05 2024-09-15T03:47:46.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:46.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947339+0000", 2024-09-15T03:47:46.226 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000200829 2024-09-15T03:47:46.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.226 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:46.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.959160+0000", 2024-09-15T03:47:46.226 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011821219000000001 2024-09-15T03:47:46.226 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.226 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:46.226 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.226 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:518 3.3s1 3:caaee79c:::smithi00132172-10:head [read 0~3145728,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:46.227 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.947091+0000", 2024-09-15T03:47:46.227 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 30.214274190000001, 2024-09-15T03:47:46.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017608233000000001, 2024-09-15T03:47:46.227 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:46.227 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:46.227 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:46.227 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:46.227 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:46.227 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 518 2024-09-15T03:47:46.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.227 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:46.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:46.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947091+0000", 2024-09-15T03:47:46.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:46.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947091+0000", 2024-09-15T03:47:46.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:46.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947092+0000", 2024-09-15T03:47:46.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7800000000000002e-07 2024-09-15T03:47:46.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:46.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947095+0000", 2024-09-15T03:47:46.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2959999999999999e-06 2024-09-15T03:47:46.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:46.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947102+0000", 2024-09-15T03:47:46.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7469999999999999e-06 2024-09-15T03:47:46.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:46.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947106+0000", 2024-09-15T03:47:46.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6550000000000003e-06 2024-09-15T03:47:46.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.230 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.230 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947622+0000", 2024-09-15T03:47:46.230 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000515727 2024-09-15T03:47:46.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.230 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:46.230 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947751+0000", 2024-09-15T03:47:46.230 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012920600000000001 2024-09-15T03:47:46.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.230 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:46.230 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.964699+0000", 2024-09-15T03:47:46.231 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.016948123999999998 2024-09-15T03:47:46.231 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.231 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:46.231 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.231 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.231 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.231 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:519 3.3s1 3:caaee79c:::smithi00132172-10:head [read 0~3145728,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:46.231 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:46.231 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 30.214240484000001, 2024-09-15T03:47:46.231 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.021743243999999998, 2024-09-15T03:47:46.231 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:46.231 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:46.232 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:46.232 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:46.232 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:46.232 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 519 2024-09-15T03:47:46.232 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.232 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:46.232 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.232 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:46.232 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:46.232 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.232 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.232 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.233 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:46.233 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:46.233 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.233 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.233 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.233 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:46.233 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:46.233 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5999999999999999e-07 2024-09-15T03:47:46.233 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.233 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.233 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:46.233 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947130+0000", 2024-09-15T03:47:46.234 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7720000000000002e-06 2024-09-15T03:47:46.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.234 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.234 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:46.234 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947137+0000", 2024-09-15T03:47:46.234 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2370000000000003e-06 2024-09-15T03:47:46.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.234 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.234 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:46.234 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947141+0000", 2024-09-15T03:47:46.235 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8809999999999998e-06 2024-09-15T03:47:46.235 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.235 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.235 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.235 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.948008+0000", 2024-09-15T03:47:46.235 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00086656700000000005 2024-09-15T03:47:46.235 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.235 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.235 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:46.235 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.948113+0000", 2024-09-15T03:47:46.235 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104904 2024-09-15T03:47:46.235 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.235 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.235 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:46.236 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.968868+0000", 2024-09-15T03:47:46.236 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.020755422999999999 2024-09-15T03:47:46.236 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.236 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:46.236 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.236 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.236 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.236 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.3s1 69/68 ECSubReadReply(tid=163, attrs_read=0))", 2024-09-15T03:47:46.236 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:46.236 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 30.207664739999998, 2024-09-15T03:47:46.236 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011102266, 2024-09-15T03:47:46.236 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:46.236 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:46.237 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:46.237 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.237 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:46.237 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:46.237 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.237 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.237 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:46.237 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:46.237 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.237 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.237 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:46.237 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:46.238 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1199999999999998e-07 2024-09-15T03:47:46.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.238 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.238 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:46.238 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956300+0000", 2024-09-15T03:47:46.238 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025987459999999999 2024-09-15T03:47:46.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.238 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.238 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:46.238 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956314+0000", 2024-09-15T03:47:46.238 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4469e-05 2024-09-15T03:47:46.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.238 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.239 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:46.239 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956319+0000", 2024-09-15T03:47:46.239 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.746e-06 2024-09-15T03:47:46.239 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.239 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.239 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.239 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.959591+0000", 2024-09-15T03:47:46.239 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032721510000000001 2024-09-15T03:47:46.239 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.239 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.239 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:46.239 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.964803+0000", 2024-09-15T03:47:46.239 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0052117420000000001 2024-09-15T03:47:46.239 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.240 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:46.240 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.240 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.240 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.240 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.3s1 69/68 ECSubReadReply(tid=164, attrs_read=0))", 2024-09-15T03:47:46.240 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:46.240 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 30.205040307000001, 2024-09-15T03:47:46.240 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012633276000000001, 2024-09-15T03:47:46.240 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:46.240 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:46.240 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:46.240 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.240 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:46.240 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:46.241 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.241 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.241 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.241 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:46.241 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:46.241 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.241 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.241 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.241 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:46.241 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:46.241 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5900000000000003e-07 2024-09-15T03:47:46.241 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.241 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.242 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:46.242 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957786+0000", 2024-09-15T03:47:46.242 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001460755 2024-09-15T03:47:46.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.242 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.242 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:46.242 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957807+0000", 2024-09-15T03:47:46.242 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0937e-05 2024-09-15T03:47:46.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.242 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.242 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:46.242 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957814+0000", 2024-09-15T03:47:46.242 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5989999999999998e-06 2024-09-15T03:47:46.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.243 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.243 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.243 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.964911+0000", 2024-09-15T03:47:46.243 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0070976370000000004 2024-09-15T03:47:46.243 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.243 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.243 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:46.243 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.968958+0000", 2024-09-15T03:47:46.243 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.004046989 2024-09-15T03:47:46.243 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.243 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:46.243 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.243 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.244 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.244 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:567 3.ds0 3:b166213b:::smithi00132172-21:head [rollback 3] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:46.244 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.947576+0000", 2024-09-15T03:47:46.244 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 25.213789379000001, 2024-09-15T03:47:46.244 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013871319999999999, 2024-09-15T03:47:46.244 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:46.244 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:46.244 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:46.244 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:46.244 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:46.244 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 567 2024-09-15T03:47:46.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.244 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:46.245 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.245 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:46.245 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947576+0000", 2024-09-15T03:47:46.245 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.245 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.245 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:46.245 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947576+0000", 2024-09-15T03:47:46.245 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.245 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.245 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:46.245 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947577+0000", 2024-09-15T03:47:46.246 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.37e-07 2024-09-15T03:47:46.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.246 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.246 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:46.246 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947580+0000", 2024-09-15T03:47:46.246 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5710000000000002e-06 2024-09-15T03:47:46.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.246 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.246 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:46.246 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947588+0000", 2024-09-15T03:47:46.246 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8939999999999993e-06 2024-09-15T03:47:46.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.246 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.247 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:46.247 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947593+0000", 2024-09-15T03:47:46.247 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7049999999999998e-06 2024-09-15T03:47:46.247 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.247 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.247 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.247 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948070+0000", 2024-09-15T03:47:46.247 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047685099999999998 2024-09-15T03:47:46.247 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.247 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.247 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:46.247 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948177+0000", 2024-09-15T03:47:46.247 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010765299999999999 2024-09-15T03:47:46.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.248 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.248 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:46.248 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948986+0000", 2024-09-15T03:47:46.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000808977 2024-09-15T03:47:46.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.248 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.248 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:46.248 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949117+0000", 2024-09-15T03:47:46.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000131032 2024-09-15T03:47:46.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.248 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.248 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:46.248 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949133+0000", 2024-09-15T03:47:46.249 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5798999999999999e-05 2024-09-15T03:47:46.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.249 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.249 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:46.249 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950936+0000", 2024-09-15T03:47:46.249 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00180316 2024-09-15T03:47:46.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.249 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.249 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:46.249 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950994+0000", 2024-09-15T03:47:46.249 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7884999999999999e-05 2024-09-15T03:47:46.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.249 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.250 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:46.250 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951532+0000", 2024-09-15T03:47:46.250 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00053787699999999997 2024-09-15T03:47:46.250 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.250 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.250 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:46.250 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961447+0000", 2024-09-15T03:47:46.250 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0099152790000000008 2024-09-15T03:47:46.250 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.250 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:46.250 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.250 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.250 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.250 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:568 3.ds0 3:b166213b:::smithi00132172-21:head [append 0~0] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:46.251 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.948048+0000", 2024-09-15T03:47:46.251 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 25.213317453999998, 2024-09-15T03:47:46.251 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013579269, 2024-09-15T03:47:46.251 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:46.251 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:46.251 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:46.251 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:46.251 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:46.251 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 568 2024-09-15T03:47:46.251 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.251 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:46.251 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.251 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:46.251 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948048+0000", 2024-09-15T03:47:46.252 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.252 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.252 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.252 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:46.252 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948048+0000", 2024-09-15T03:47:46.252 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.252 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.252 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.252 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:46.252 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948049+0000", 2024-09-15T03:47:46.252 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7199999999999998e-07 2024-09-15T03:47:46.252 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.252 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.252 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:46.253 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948056+0000", 2024-09-15T03:47:46.253 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0280000000000003e-06 2024-09-15T03:47:46.253 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.253 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.253 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:46.253 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948066+0000", 2024-09-15T03:47:46.253 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0141e-05 2024-09-15T03:47:46.253 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.253 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.253 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:46.253 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948072+0000", 2024-09-15T03:47:46.253 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1519999999999997e-06 2024-09-15T03:47:46.253 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.254 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.254 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949221+0000", 2024-09-15T03:47:46.254 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001149125 2024-09-15T03:47:46.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.254 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.254 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:46.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949313+0000", 2024-09-15T03:47:46.254 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1469000000000001e-05 2024-09-15T03:47:46.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.254 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.254 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:46.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949818+0000", 2024-09-15T03:47:46.254 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00050549700000000004 2024-09-15T03:47:46.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.255 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:46.255 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950002+0000", 2024-09-15T03:47:46.255 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018414700000000001 2024-09-15T03:47:46.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.255 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:46.255 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950940+0000", 2024-09-15T03:47:46.255 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00093786100000000001 2024-09-15T03:47:46.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.255 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:46.256 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951004+0000", 2024-09-15T03:47:46.256 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3859e-05 2024-09-15T03:47:46.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.256 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.256 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:46.256 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951020+0000", 2024-09-15T03:47:46.256 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.628e-05 2024-09-15T03:47:46.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.256 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.256 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:46.256 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951753+0000", 2024-09-15T03:47:46.256 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00073276999999999999 2024-09-15T03:47:46.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.257 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.257 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:46.257 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961627+0000", 2024-09-15T03:47:46.257 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098741679999999991 2024-09-15T03:47:46.257 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.257 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:46.257 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.257 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.257 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.257 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:677 3.3s1 3:caaee79c:::smithi00132172-10:head [copy-from ver 31] snapc 2f=[2d,2b,29,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:46.257 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:30.087209+0000", 2024-09-15T03:47:46.257 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.074156262999999, 2024-09-15T03:47:46.257 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.025257667000000001, 2024-09-15T03:47:46.257 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:46.258 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:46.258 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:46.258 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:46.258 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:46.258 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 677 2024-09-15T03:47:46.258 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.258 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:46.258 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.258 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:46.258 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087209+0000", 2024-09-15T03:47:46.258 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.258 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:46.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087209+0000", 2024-09-15T03:47:46.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:46.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087211+0000", 2024-09-15T03:47:46.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2500000000000001e-06 2024-09-15T03:47:46.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:46.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087222+0000", 2024-09-15T03:47:46.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.113e-05 2024-09-15T03:47:46.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:46.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087252+0000", 2024-09-15T03:47:46.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9488999999999998e-05 2024-09-15T03:47:46.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:46.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087264+0000", 2024-09-15T03:47:46.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1875999999999999e-05 2024-09-15T03:47:46.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.261 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087368+0000", 2024-09-15T03:47:46.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104035 2024-09-15T03:47:46.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.261 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.261 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:46.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087560+0000", 2024-09-15T03:47:46.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019211400000000001 2024-09-15T03:47:46.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.261 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.261 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:46.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.104984+0000", 2024-09-15T03:47:46.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017424489000000001 2024-09-15T03:47:46.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.262 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.262 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:46.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105388+0000", 2024-09-15T03:47:46.262 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000403686 2024-09-15T03:47:46.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.262 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.262 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:46.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105404+0000", 2024-09-15T03:47:46.262 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6268000000000001e-05 2024-09-15T03:47:46.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.262 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.262 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:46.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105641+0000", 2024-09-15T03:47:46.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023648399999999999 2024-09-15T03:47:46.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.263 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:46.263 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105662+0000", 2024-09-15T03:47:46.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0602999999999999e-05 2024-09-15T03:47:46.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.263 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:46.263 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.110231+0000", 2024-09-15T03:47:46.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0045697089999999999 2024-09-15T03:47:46.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.264 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:46.264 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.112467+0000", 2024-09-15T03:47:46.264 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002235534 2024-09-15T03:47:46.264 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.264 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:46.264 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.264 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.264 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:678 3.3s1 3:caaee79c:::smithi00132172-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:46.264 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:30.087279+0000", 2024-09-15T03:47:46.264 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.074086808000001, 2024-09-15T03:47:46.264 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.018991965, 2024-09-15T03:47:46.264 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:46.265 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:46.265 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:46.265 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:46.265 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:46.265 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 678 2024-09-15T03:47:46.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.265 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:46.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:46.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087279+0000", 2024-09-15T03:47:46.265 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:46.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087279+0000", 2024-09-15T03:47:46.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.266 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.266 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:46.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087280+0000", 2024-09-15T03:47:46.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.147e-06 2024-09-15T03:47:46.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.266 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.266 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:46.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087285+0000", 2024-09-15T03:47:46.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2059999999999998e-06 2024-09-15T03:47:46.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.267 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:46.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087303+0000", 2024-09-15T03:47:46.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7646e-05 2024-09-15T03:47:46.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.267 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:46.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087313+0000", 2024-09-15T03:47:46.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0862999999999999e-05 2024-09-15T03:47:46.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.267 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087765+0000", 2024-09-15T03:47:46.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045120400000000003 2024-09-15T03:47:46.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:46.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087962+0000", 2024-09-15T03:47:46.268 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000197095 2024-09-15T03:47:46.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105788+0000", 2024-09-15T03:47:46.268 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017826143999999999 2024-09-15T03:47:46.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:46.269 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.106034+0000", 2024-09-15T03:47:46.269 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024586999999999998 2024-09-15T03:47:46.269 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.269 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.269 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:46.269 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.106270+0000", 2024-09-15T03:47:46.269 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023678999999999999 2024-09-15T03:47:46.269 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.269 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:46.269 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.269 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.269 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.269 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:688 3.3s1 3:c7c3ef16:::smithi00132172-25:head [create,setxattr og^`]cfbTbdPrNph_pkYgeiTrgatpl] (805) in=31b,setxattr `Trb[\\aaaksnX\\dQOYP^TPXpYtmQ_maX\\[hbePkT (981) in=40b,setxattr UkQ^W[[sWh]fSrWg]^\\oZpWaamVkVUnZYbR[Vql\\ (663) in=40b,setxattr ofr[Os^]tjr^jicnNpPaUeXQTfldVQPdSPjcgWcT (7228) in=40b,setxattr Vl`NPsTVpnj_gO_srrbrZ\\eq (805) in=24b,setxattr ^_mcpljcT[qX_UeeYZY\\_ZTSsSsojP\\esUkVphT] (1963) in=40b,setxattr [oY^bSn[ealaPaOhprWW_SNaa]VhaXN[jmo`YYgt (138) in=40b,setxattr tT[SSseaPa\\UPS`\\g\\pRqsd`^lo__arabQlQmob` (663) in=40b,setxattr o^\\]jXsesZhoknr\\dagnjidgsPOhkfg[ffVe[PQW (1259) in=40b,setxattr c^VX_^n^gsh^P`aVbXpaS\\cgkpZO]OmXmchTtWZd (1612) in=40b,setxattr _^aeiTb[Uhh`rNNUOZOPt_Z_Qmeps`XXQ^qidetf (629) in=40b,setxattr ms[glaQ`N\\pqbicVTmgp^\\Rg]h`fefatkfg\\js_[ (139) in=40b,setxattr O\\gsPtYmrReq]if[pkOgPtqS`]Nrq_eYpsefkSgl (700) in=40b,setxattr RgefRtSsOT`jTQb^h\\igU`_lbscW`fgkXroNdmea (1751) in=40b,setxattr sgaV_cSaelbWQYill^k`RsVajcXpUSfW]eNd_nat (807) in=40b,setxattr P\\lm\\r`\\jpssZmkPqcQSbje_]]`YOSZTeRUUSSY[ (1365) in=40b,setxattr aa^ePena\\opOdeWXTj[XU]N^jQS]bens\\Tc_g]ea (16140) in=40b,setxattr R[qckQQO`n[[egbgcZsdtUbdPdnPNlOmsbP_OXOQ (1227) in=40b,setxattr SpRUpSaQZ]nlQ]kpXZnOWSUS]tsmf`qZiPTkq]Yi (209) in=40b,setxattr Y[Y[X`O_XaVOb]NZ_PgmhRX]Vds_dfbp]h[SPkYb (417) in=40b] snapc 2f=[2d,2b,29,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:46.270 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:30.088531+0000", 2024-09-15T03:47:46.270 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.072834072999999, 2024-09-15T03:47:46.270 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013293503999999999, 2024-09-15T03:47:46.270 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:46.270 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:46.270 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:46.270 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:46.270 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:46.270 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 688 2024-09-15T03:47:46.270 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.271 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:46.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.271 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:46.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088531+0000", 2024-09-15T03:47:46.271 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:46.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088531+0000", 2024-09-15T03:47:46.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:46.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088533+0000", 2024-09-15T03:47:46.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5379999999999999e-06 2024-09-15T03:47:46.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:46.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088662+0000", 2024-09-15T03:47:46.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000129002 2024-09-15T03:47:46.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:46.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088688+0000", 2024-09-15T03:47:46.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6084999999999998e-05 2024-09-15T03:47:46.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:46.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088696+0000", 2024-09-15T03:47:46.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0490000000000004e-06 2024-09-15T03:47:46.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.274 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.274 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.274 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088746+0000", 2024-09-15T03:47:46.274 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0074000000000002e-05 2024-09-15T03:47:46.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.274 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.274 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:46.274 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.089084+0000", 2024-09-15T03:47:46.274 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033810200000000001 2024-09-15T03:47:46.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.275 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.275 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:46.275 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090498+0000", 2024-09-15T03:47:46.275 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001414205 2024-09-15T03:47:46.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.275 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.275 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:46.275 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090936+0000", 2024-09-15T03:47:46.275 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043802799999999997 2024-09-15T03:47:46.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.275 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.276 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:46.276 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090951+0000", 2024-09-15T03:47:46.276 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4953999999999999e-05 2024-09-15T03:47:46.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.276 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.276 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:46.276 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.091069+0000", 2024-09-15T03:47:46.276 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000117236 2024-09-15T03:47:46.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.276 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.276 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:46.276 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.091128+0000", 2024-09-15T03:47:46.277 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9580999999999998e-05 2024-09-15T03:47:46.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.277 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.277 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:46.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.097312+0000", 2024-09-15T03:47:46.277 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0061838129999999998 2024-09-15T03:47:46.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.277 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.277 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:46.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.101825+0000", 2024-09-15T03:47:46.277 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0045128369999999996 2024-09-15T03:47:46.277 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.278 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:46.278 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.278 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.278 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s4 83/80 ECSubWrite(tid=24, reqid=client.4521.0:675, at_version=83'37, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:46.278 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:30.126290+0000", 2024-09-15T03:47:46.278 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.03507523, 2024-09-15T03:47:46.278 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.014228881000000001, 2024-09-15T03:47:46.278 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:46.278 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:46.278 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:46.278 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.278 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:46.279 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.126290+0000", 2024-09-15T03:47:46.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.279 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:46.279 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.126290+0000", 2024-09-15T03:47:46.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:46.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.279 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:46.279 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.126291+0000", 2024-09-15T03:47:46.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6499999999999998e-07 2024-09-15T03:47:46.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.280 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:46.280 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128867+0000", 2024-09-15T03:47:46.280 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025759749999999999 2024-09-15T03:47:46.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.280 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.280 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:46.280 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128881+0000", 2024-09-15T03:47:46.280 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3823e-05 2024-09-15T03:47:46.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.280 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.280 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:46.280 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128888+0000", 2024-09-15T03:47:46.281 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.605e-06 2024-09-15T03:47:46.281 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.281 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.281 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:46.281 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128949+0000", 2024-09-15T03:47:46.281 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0949999999999998e-05 2024-09-15T03:47:46.281 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.281 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.281 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:46.281 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128957+0000", 2024-09-15T03:47:46.281 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4760000000000001e-06 2024-09-15T03:47:46.281 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.281 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.282 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:46.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.129323+0000", 2024-09-15T03:47:46.282 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036638900000000001 2024-09-15T03:47:46.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.282 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.282 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:46.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.129331+0000", 2024-09-15T03:47:46.282 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1319999999999994e-06 2024-09-15T03:47:46.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.282 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.282 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:46.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.129567+0000", 2024-09-15T03:47:46.282 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000235393 2024-09-15T03:47:46.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.283 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.283 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:46.283 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.129586+0000", 2024-09-15T03:47:46.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9517999999999998e-05 2024-09-15T03:47:46.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:46.283 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:46.283 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:46.283 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.140519+0000", 2024-09-15T03:47:46.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010932855 2024-09-15T03:47:46.283 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.283 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:46.283 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.283 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:46.284 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:46.284 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:46.285 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:46.285 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:46.286 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:46.286 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:46.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:47:46.287 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:46.284+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:46.287 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:46.287 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:46.288 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:46.288 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:46.288 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-15T03:47:46.288 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:46.289 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T03:47:46.289 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:46.289 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T03:47:46.386 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:46.385+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:46.423 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:46.424 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:46.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:47:46.486 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:46.485+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:46.528 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:46.528 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:46.537 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:47:46.587 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:46.585+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:46.633 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:46.633 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:46.642 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:47:46.688 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:46.687+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:46.738 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:46.738 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:46.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:47:46.788 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:46.788+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:46.865 INFO:tasks.rados.rados.0.smithi001.stdout:342: done (2 left) 2024-09-15T03:47:46.865 INFO:tasks.rados.rados.0.smithi001.stdout:343: done (1 left) 2024-09-15T03:47:46.865 INFO:tasks.rados.rados.0.smithi001.stdout:344: done (0 left) 2024-09-15T03:47:46.865 INFO:tasks.rados.rados.0.smithi001.stdout:345: rmattr oid 9 current snap is 42 2024-09-15T03:47:46.865 INFO:tasks.rados.rados.0.smithi001.stdout:345: done (0 left) 2024-09-15T03:47:46.865 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 7 current snap is 42 2024-09-15T03:47:46.866 INFO:tasks.rados.rados.0.smithi001.stdout:346: seq_num 133 ranges {1900544=688128,2588672=720896,3309568=475136,3784704=360448} 2024-09-15T03:47:46.877 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T03:47:46.878 INFO:tasks.rados.rados.0.smithi001.stdout:346: writing smithi00132172-7 from 1900544 to 2588672 tid 1 2024-09-15T03:47:46.883 INFO:tasks.rados.rados.0.smithi001.stdout:346: writing smithi00132172-7 from 2588672 to 3309568 tid 2 2024-09-15T03:47:46.887 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:46.887 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:46.888 INFO:tasks.rados.rados.0.smithi001.stdout:346: writing smithi00132172-7 from 3309568 to 3784704 tid 3 2024-09-15T03:47:46.889 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:46.888+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:46.890 INFO:tasks.rados.rados.0.smithi001.stdout:346: writing smithi00132172-7 from 3784704 to 4145152 tid 4 2024-09-15T03:47:46.891 INFO:tasks.rados.rados.0.smithi001.stdout:347: snap_create 2024-09-15T03:47:46.891 INFO:tasks.rados.rados.0.smithi001.stdout:346: finishing write tid 1 to smithi00132172-7 2024-09-15T03:47:46.891 INFO:tasks.rados.rados.0.smithi001.stdout:346: finishing write tid 2 to smithi00132172-7 2024-09-15T03:47:46.891 INFO:tasks.rados.rados.0.smithi001.stdout:346: finishing write tid 3 to smithi00132172-7 2024-09-15T03:47:46.894 INFO:tasks.rados.rados.0.smithi001.stdout:346: finishing write tid 4 to smithi00132172-7 2024-09-15T03:47:46.894 INFO:tasks.rados.rados.0.smithi001.stdout:346: finishing write tid 5 to smithi00132172-7 2024-09-15T03:47:46.896 INFO:tasks.rados.rados.0.smithi001.stdout:346: finishing write tid 6 to smithi00132172-7 2024-09-15T03:47:46.896 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 7 v 124 (ObjNum 133 snap 42 seq_num 133) dirty exists 2024-09-15T03:47:46.897 INFO:tasks.rados.rados.0.smithi001.stdout:346: left oid 7 (ObjNum 133 snap 42 seq_num 133) 2024-09-15T03:47:46.898 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:47:46.990 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:46.988+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:47.035 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:47.035 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:47.044 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:47:47.090 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:47.090+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:47.146 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:47.146 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:47.154 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:47:47.191 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:47.191+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:47.293 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:47.291+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:47.294 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:47.294 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:47.303 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:47:47.394 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:47.394+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:47.403 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:47.403 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:47.411 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:47:47.496 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:47.494+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:47.512 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:47.512 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:47.521 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:47:47.596 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:47.596+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:47.622 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:47.622 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:47.633 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:47:47.718 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:47.717+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:47.774 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:47.775 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:47.783 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:47:47.824 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:47.823+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:47.871 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-15T03:47:47.872 INFO:tasks.rados.rados.0.smithi001.stdout:346: done (1 left) 2024-09-15T03:47:47.873 INFO:tasks.rados.rados.0.smithi001.stdout:347: done (0 left) 2024-09-15T03:47:47.873 INFO:tasks.rados.rados.0.smithi001.stdout:348: read oid 5 snap -1 2024-09-15T03:47:47.873 INFO:tasks.rados.rados.0.smithi001.stdout:348: expect (ObjNum 105 snap 29 seq_num 105) 2024-09-15T03:47:47.873 INFO:tasks.rados.rados.0.smithi001.stdout:349: snap_create 2024-09-15T03:47:47.887 INFO:tasks.rados.rados.0.smithi001.stdout:348: expect (ObjNum 105 snap 29 seq_num 105) 2024-09-15T03:47:47.900 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:47.899+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:47.927 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:47.928 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:47.936 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:47:48.000 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:47.999+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:48.076 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:48.076 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:48.084 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:47:48.129 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:48.128+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:48.186 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:48.186 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:48.195 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:47:48.236 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:48.235+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:48.290 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:47:48.334 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:48.333+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:48.337 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:48.337 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:48.347 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:47:48.405 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:48.404+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:48.409 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:48.409 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:48.425 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:48.425 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:48.439 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:48.453 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:48.454 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:48.469 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:48.469 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:48.484 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:48.484 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:48.492 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-15T03:47:48.492 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-15T03:47:48.493 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:48.499 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:48.500 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:48.501 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:47:48.515 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:48.515 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:48.515 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:48.514+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:48.529 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:48.530 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:48.545 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:48.545 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:48.545 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-15T03:47:48.607 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:48.607+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:48.618 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:48.618 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-15T03:47:48.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-15T03:47:48.619 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-15T03:47:48.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.619 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:333 3.9s0 3:94e9e019:::smithi00132172-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:48.619 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.856848+0000", 2024-09-15T03:47:48.621 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 52.758323247, 2024-09-15T03:47:48.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012151039000000001, 2024-09-15T03:47:48.621 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:48.621 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:48.621 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:48.621 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:48.621 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:48.621 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 333 2024-09-15T03:47:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856848+0000", 2024-09-15T03:47:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856848+0000", 2024-09-15T03:47:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856849+0000", 2024-09-15T03:47:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5499999999999999e-07 2024-09-15T03:47:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.378e-06 2024-09-15T03:47:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:48.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856854+0000", 2024-09-15T03:47:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.653e-06 2024-09-15T03:47:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856856+0000", 2024-09-15T03:47:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4059999999999999e-06 2024-09-15T03:47:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.862404+0000", 2024-09-15T03:47:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0055481310000000004 2024-09-15T03:47:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.862519+0000", 2024-09-15T03:47:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011449400000000001 2024-09-15T03:47:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864592+0000", 2024-09-15T03:47:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020730200000000001 2024-09-15T03:47:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864687+0000", 2024-09-15T03:47:48.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5531000000000004e-05 2024-09-15T03:47:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.868999+0000", 2024-09-15T03:47:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0043121710000000001 2024-09-15T03:47:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:458 3.9s0 3:97177042:::smithi00132172-18:head [append 0~753664] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.841113+0000", 2024-09-15T03:47:48.626 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 37.774058842000002, 2024-09-15T03:47:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0334256719999999, 2024-09-15T03:47:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 458 2024-09-15T03:47:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841113+0000", 2024-09-15T03:47:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:48.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841113+0000", 2024-09-15T03:47:48.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:48.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841114+0000", 2024-09-15T03:47:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.491e-06 2024-09-15T03:47:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841577+0000", 2024-09-15T03:47:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000462571 2024-09-15T03:47:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841589+0000", 2024-09-15T03:47:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2119999999999999e-05 2024-09-15T03:47:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841596+0000", 2024-09-15T03:47:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0740000000000004e-06 2024-09-15T03:47:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841708+0000", 2024-09-15T03:47:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112126 2024-09-15T03:47:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:47:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841712+0000", 2024-09-15T03:47:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3639999999999999e-06 2024-09-15T03:47:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.856485+0000", 2024-09-15T03:47:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0147736949999999 2024-09-15T03:47:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.856699+0000", 2024-09-15T03:47:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000213372 2024-09-15T03:47:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:48.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.862261+0000", 2024-09-15T03:47:48.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0055625450000000003 2024-09-15T03:47:48.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:48.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.862781+0000", 2024-09-15T03:47:48.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00052013500000000002 2024-09-15T03:47:48.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:48.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.862885+0000", 2024-09-15T03:47:48.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103455 2024-09-15T03:47:48.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:48.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.863028+0000", 2024-09-15T03:47:48.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014313999999999999 2024-09-15T03:47:48.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:48.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.864079+0000", 2024-09-15T03:47:48.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010511229999999999 2024-09-15T03:47:48.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:48.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874458+0000", 2024-09-15T03:47:48.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010379019999999999 2024-09-15T03:47:48.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:48.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874538+0000", 2024-09-15T03:47:48.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0440999999999998e-05 2024-09-15T03:47:48.634 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.634 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:48.634 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.634 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:459 3.9s0 3:97177042:::smithi00132172-18:head [append 0~720896] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:48.634 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.846766+0000", 2024-09-15T03:47:48.634 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 37.768405219000002, 2024-09-15T03:47:48.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0281560110000001, 2024-09-15T03:47:48.634 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:48.635 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:48.635 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:48.635 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:48.635 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:48.635 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 459 2024-09-15T03:47:48.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.635 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:48.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:48.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.846766+0000", 2024-09-15T03:47:48.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:48.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.846766+0000", 2024-09-15T03:47:48.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:48.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.846769+0000", 2024-09-15T03:47:48.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.323e-06 2024-09-15T03:47:48.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:48.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847323+0000", 2024-09-15T03:47:48.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000554267 2024-09-15T03:47:48.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:48.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847338+0000", 2024-09-15T03:47:48.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.465e-05 2024-09-15T03:47:48.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:48.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847345+0000", 2024-09-15T03:47:48.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6029999999999999e-06 2024-09-15T03:47:48.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847494+0000", 2024-09-15T03:47:48.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000148541 2024-09-15T03:47:48.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:47:48.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847498+0000", 2024-09-15T03:47:48.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7629999999999998e-06 2024-09-15T03:47:48.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.864944+0000", 2024-09-15T03:47:48.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0174461050000001 2024-09-15T03:47:48.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:48.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.865122+0000", 2024-09-15T03:47:48.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000178123 2024-09-15T03:47:48.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:48.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867632+0000", 2024-09-15T03:47:48.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002510249 2024-09-15T03:47:48.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:48.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867775+0000", 2024-09-15T03:47:48.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014326000000000001 2024-09-15T03:47:48.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:48.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868663+0000", 2024-09-15T03:47:48.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00088783799999999995 2024-09-15T03:47:48.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:48.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868802+0000", 2024-09-15T03:47:48.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000139173 2024-09-15T03:47:48.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:48.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874167+0000", 2024-09-15T03:47:48.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.005365117 2024-09-15T03:47:48.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:48.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874842+0000", 2024-09-15T03:47:48.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00067414199999999997 2024-09-15T03:47:48.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:48.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874922+0000", 2024-09-15T03:47:48.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0857000000000002e-05 2024-09-15T03:47:48.642 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.642 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:48.642 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.642 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:460 3.9s0 3:97177042:::smithi00132172-18:head [append 0~49152] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:48.642 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.847359+0000", 2024-09-15T03:47:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 37.767812845000002, 2024-09-15T03:47:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0278644889999999, 2024-09-15T03:47:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 460 2024-09-15T03:47:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:48.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847359+0000", 2024-09-15T03:47:48.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:48.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847359+0000", 2024-09-15T03:47:48.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:48.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847360+0000", 2024-09-15T03:47:48.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.26e-06 2024-09-15T03:47:48.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:48.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847390+0000", 2024-09-15T03:47:48.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9533999999999999e-05 2024-09-15T03:47:48.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:48.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847406+0000", 2024-09-15T03:47:48.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6725999999999999e-05 2024-09-15T03:47:48.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:48.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847412+0000", 2024-09-15T03:47:48.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4149999999999998e-06 2024-09-15T03:47:48.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847522+0000", 2024-09-15T03:47:48.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011041500000000001 2024-09-15T03:47:48.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:47:48.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847523+0000", 2024-09-15T03:47:48.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.09e-07 2024-09-15T03:47:48.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868928+0000", 2024-09-15T03:47:48.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0214053089999999 2024-09-15T03:47:48.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:48.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869043+0000", 2024-09-15T03:47:48.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000115021 2024-09-15T03:47:48.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:48.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869635+0000", 2024-09-15T03:47:48.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00059164199999999997 2024-09-15T03:47:48.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:48.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869755+0000", 2024-09-15T03:47:48.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0001205 2024-09-15T03:47:48.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:48.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870274+0000", 2024-09-15T03:47:48.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00051842900000000005 2024-09-15T03:47:48.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:48.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870322+0000", 2024-09-15T03:47:48.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8164000000000002e-05 2024-09-15T03:47:48.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:48.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874941+0000", 2024-09-15T03:47:48.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0046186140000000001 2024-09-15T03:47:48.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:48.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875140+0000", 2024-09-15T03:47:48.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019890700000000001 2024-09-15T03:47:48.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:48.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875223+0000", 2024-09-15T03:47:48.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3644000000000001e-05 2024-09-15T03:47:48.650 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.650 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:48.650 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.650 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:461 3.9s0 3:97177042:::smithi00132172-18:head [setxattr _header (57) in=7b] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:48.650 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.847417+0000", 2024-09-15T03:47:48.650 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 37.767754740999997, 2024-09-15T03:47:48.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0332682499999999, 2024-09-15T03:47:48.650 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:48.650 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:48.651 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:48.651 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:48.651 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:48.651 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 461 2024-09-15T03:47:48.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.651 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:48.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:48.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847417+0000", 2024-09-15T03:47:48.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:48.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847417+0000", 2024-09-15T03:47:48.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:48.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847418+0000", 2024-09-15T03:47:48.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8999999999999995e-07 2024-09-15T03:47:48.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:48.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847421+0000", 2024-09-15T03:47:48.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1769999999999998e-06 2024-09-15T03:47:48.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:48.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847428+0000", 2024-09-15T03:47:48.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6969999999999996e-06 2024-09-15T03:47:48.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:48.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847432+0000", 2024-09-15T03:47:48.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3209999999999998e-06 2024-09-15T03:47:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847542+0000", 2024-09-15T03:47:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011035999999999999 2024-09-15T03:47:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:47:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847543+0000", 2024-09-15T03:47:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1200000000000002e-07 2024-09-15T03:47:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875612+0000", 2024-09-15T03:47:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0280692330000001 2024-09-15T03:47:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875735+0000", 2024-09-15T03:47:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012276600000000001 2024-09-15T03:47:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876307+0000", 2024-09-15T03:47:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00057198600000000002 2024-09-15T03:47:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876408+0000", 2024-09-15T03:47:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010148400000000001 2024-09-15T03:47:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876508+0000", 2024-09-15T03:47:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9328999999999997e-05 2024-09-15T03:47:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:48.660 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876546+0000", 2024-09-15T03:47:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8361000000000001e-05 2024-09-15T03:47:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876602+0000", 2024-09-15T03:47:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6412999999999997e-05 2024-09-15T03:47:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.880589+0000", 2024-09-15T03:47:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0039866190000000003 2024-09-15T03:47:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.880685+0000", 2024-09-15T03:47:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6001999999999993e-05 2024-09-15T03:47:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:462 3.9s0 3:97177042:::smithi00132172-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.847436+0000", 2024-09-15T03:47:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 37.767735782999999, 2024-09-15T03:47:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.03734576, 2024-09-15T03:47:48.662 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:48.663 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:48.663 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:48.663 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:48.663 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:48.664 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 462 2024-09-15T03:47:48.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.664 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:48.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.664 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:48.664 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847436+0000", 2024-09-15T03:47:48.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.664 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:48.664 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847436+0000", 2024-09-15T03:47:48.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:48.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847436+0000", 2024-09-15T03:47:48.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3099999999999998e-07 2024-09-15T03:47:48.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:48.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847438+0000", 2024-09-15T03:47:48.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9570000000000001e-06 2024-09-15T03:47:48.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:48.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847445+0000", 2024-09-15T03:47:48.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.238e-06 2024-09-15T03:47:48.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:48.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847454+0000", 2024-09-15T03:47:48.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5930000000000006e-06 2024-09-15T03:47:48.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847562+0000", 2024-09-15T03:47:48.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108448 2024-09-15T03:47:48.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:47:48.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847563+0000", 2024-09-15T03:47:48.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0200000000000002e-07 2024-09-15T03:47:48.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876482+0000", 2024-09-15T03:47:48.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.028918658 2024-09-15T03:47:48.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:48.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876593+0000", 2024-09-15T03:47:48.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011145300000000001 2024-09-15T03:47:48.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.668 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.668 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.881189+0000", 2024-09-15T03:47:48.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0045956139999999996 2024-09-15T03:47:48.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.668 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:48.668 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.881293+0000", 2024-09-15T03:47:48.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103875 2024-09-15T03:47:48.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.669 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:48.669 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.884782+0000", 2024-09-15T03:47:48.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003488891 2024-09-15T03:47:48.669 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.669 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:48.669 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.669 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.5s6 65/64 ECSubWrite(tid=100, reqid=osd.1.0:100, at_version=65'35, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:48.669 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.865131+0000", 2024-09-15T03:47:48.669 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 36.750040382999998, 2024-09-15T03:47:48.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0096540099999999993, 2024-09-15T03:47:48.670 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:48.670 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:48.670 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:48.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:48.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.865131+0000", 2024-09-15T03:47:48.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:48.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.865131+0000", 2024-09-15T03:47:48.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:48.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.865132+0000", 2024-09-15T03:47:48.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1399999999999996e-07 2024-09-15T03:47:48.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:48.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.865151+0000", 2024-09-15T03:47:48.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9151e-05 2024-09-15T03:47:48.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:48.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.865166+0000", 2024-09-15T03:47:48.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4922999999999999e-05 2024-09-15T03:47:48.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:48.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.865173+0000", 2024-09-15T03:47:48.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0500000000000003e-06 2024-09-15T03:47:48.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.873827+0000", 2024-09-15T03:47:48.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0086539719999999994 2024-09-15T03:47:48.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.673 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:48.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.873837+0000", 2024-09-15T03:47:48.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5969999999999998e-06 2024-09-15T03:47:48.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.673 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:48.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874178+0000", 2024-09-15T03:47:48.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00034180199999999999 2024-09-15T03:47:48.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.673 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.673 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:48.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874191+0000", 2024-09-15T03:47:48.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2476999999999999e-05 2024-09-15T03:47:48.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:48.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874240+0000", 2024-09-15T03:47:48.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8615999999999997e-05 2024-09-15T03:47:48.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:48.674 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874252+0000", 2024-09-15T03:47:48.674 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2480000000000001e-05 2024-09-15T03:47:48.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.674 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:48.675 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874785+0000", 2024-09-15T03:47:48.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00053322799999999996 2024-09-15T03:47:48.675 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.675 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:48.675 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.675 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:561 3.9s0 3:9cd57211:::smithi00132172-16:head [append 0~557056] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:48.675 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:20.935324+0000", 2024-09-15T03:47:48.675 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.679848081999999, 2024-09-15T03:47:48.675 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012459909999999999, 2024-09-15T03:47:48.675 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:48.675 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:48.676 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:48.676 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:48.676 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:48.676 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 561 2024-09-15T03:47:48.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.676 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:48.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:48.676 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935324+0000", 2024-09-15T03:47:48.676 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.676 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:48.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935324+0000", 2024-09-15T03:47:48.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:48.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935326+0000", 2024-09-15T03:47:48.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8909999999999999e-06 2024-09-15T03:47:48.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.677 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:48.677 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935587+0000", 2024-09-15T03:47:48.677 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026019699999999999 2024-09-15T03:47:48.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.678 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:48.678 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935610+0000", 2024-09-15T03:47:48.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2949e-05 2024-09-15T03:47:48.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.678 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:48.678 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935618+0000", 2024-09-15T03:47:48.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6289999999999999e-06 2024-09-15T03:47:48.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.678 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.678 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935688+0000", 2024-09-15T03:47:48.678 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9801999999999993e-05 2024-09-15T03:47:48.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-15T03:47:48.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935698+0000", 2024-09-15T03:47:48.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0103e-05 2024-09-15T03:47:48.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.679 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939626+0000", 2024-09-15T03:47:48.679 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0039276759999999997 2024-09-15T03:47:48.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.679 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:48.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939844+0000", 2024-09-15T03:47:48.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000217741 2024-09-15T03:47:48.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:48.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941127+0000", 2024-09-15T03:47:48.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001283719 2024-09-15T03:47:48.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.680 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:48.680 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941395+0000", 2024-09-15T03:47:48.680 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026823800000000002 2024-09-15T03:47:48.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:48.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941448+0000", 2024-09-15T03:47:48.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2751999999999999e-05 2024-09-15T03:47:48.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:48.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941646+0000", 2024-09-15T03:47:48.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019804100000000001 2024-09-15T03:47:48.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.681 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.681 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:48.681 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943664+0000", 2024-09-15T03:47:48.681 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020181090000000001 2024-09-15T03:47:48.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:48.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.947709+0000", 2024-09-15T03:47:48.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0040445059999999998 2024-09-15T03:47:48.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.682 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:48.682 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.947783+0000", 2024-09-15T03:47:48.682 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4556999999999998e-05 2024-09-15T03:47:48.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.682 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:48.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.683 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:562 3.9s0 3:9cd57211:::smithi00132172-16:head [append 0~638976] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:48.683 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:20.938590+0000", 2024-09-15T03:47:48.683 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.676581297999999, 2024-09-15T03:47:48.683 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011368382, 2024-09-15T03:47:48.683 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:48.683 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:48.683 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:48.683 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:48.683 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:48.683 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 562 2024-09-15T03:47:48.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.683 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:48.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:48.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.938590+0000", 2024-09-15T03:47:48.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:48.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.938590+0000", 2024-09-15T03:47:48.684 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.684 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:48.684 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.938592+0000", 2024-09-15T03:47:48.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.623e-06 2024-09-15T03:47:48.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:48.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939253+0000", 2024-09-15T03:47:48.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00066056899999999996 2024-09-15T03:47:48.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.685 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:48.685 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939271+0000", 2024-09-15T03:47:48.685 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8482e-05 2024-09-15T03:47:48.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.685 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:48.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939280+0000", 2024-09-15T03:47:48.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5569999999999996e-06 2024-09-15T03:47:48.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941736+0000", 2024-09-15T03:47:48.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024565640000000001 2024-09-15T03:47:48.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.686 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.686 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:48.686 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941861+0000", 2024-09-15T03:47:48.686 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012484 2024-09-15T03:47:48.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.687 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:48.687 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943223+0000", 2024-09-15T03:47:48.687 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0013619260000000001 2024-09-15T03:47:48.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.687 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:48.687 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943354+0000", 2024-09-15T03:47:48.687 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013100399999999999 2024-09-15T03:47:48.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.687 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.687 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:48.687 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943376+0000", 2024-09-15T03:47:48.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1829999999999999e-05 2024-09-15T03:47:48.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.688 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:48.688 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943579+0000", 2024-09-15T03:47:48.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020374400000000001 2024-09-15T03:47:48.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.688 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:48.688 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943904+0000", 2024-09-15T03:47:48.688 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000324846 2024-09-15T03:47:48.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:48.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949897+0000", 2024-09-15T03:47:48.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0059927879999999998 2024-09-15T03:47:48.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.689 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:48.689 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949959+0000", 2024-09-15T03:47:48.689 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1608999999999999e-05 2024-09-15T03:47:48.689 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.689 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:48.689 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.689 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:565 3.9s0 3:9cd57211:::smithi00132172-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:48.690 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:20.943038+0000", 2024-09-15T03:47:48.690 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.672133954, 2024-09-15T03:47:48.690 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011190904, 2024-09-15T03:47:48.690 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:48.690 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:48.690 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:48.690 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:48.690 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:48.690 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 565 2024-09-15T03:47:48.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.690 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:48.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.690 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:48.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943038+0000", 2024-09-15T03:47:48.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:48.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943038+0000", 2024-09-15T03:47:48.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.691 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:48.691 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943038+0000", 2024-09-15T03:47:48.691 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6800000000000001e-07 2024-09-15T03:47:48.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943040+0000", 2024-09-15T03:47:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3700000000000002e-06 2024-09-15T03:47:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943047+0000", 2024-09-15T03:47:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1299999999999998e-06 2024-09-15T03:47:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943050+0000", 2024-09-15T03:47:48.692 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6229999999999999e-06 2024-09-15T03:47:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.947990+0000", 2024-09-15T03:47:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0049398690000000004 2024-09-15T03:47:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.948080+0000", 2024-09-15T03:47:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9721000000000004e-05 2024-09-15T03:47:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.693 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.952263+0000", 2024-09-15T03:47:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.004183417 2024-09-15T03:47:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.952374+0000", 2024-09-15T03:47:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110435 2024-09-15T03:47:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.954229+0000", 2024-09-15T03:47:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018548709999999999 2024-09-15T03:47:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.694 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:569 3.9s0 3:97177042:::smithi00132172-18:head [rmxattr Z in=1b,rmxattr ZVpala^a[hcaro^RXmp_SYlliaqih_SpraepZfQX in=40b,rmxattr aNhZ\\ePhXcfSNf\\hQ^orWZst^jcpVp[skbdkms]j in=40b,rmxattr ehO_catUjgbjTPQOspNgZXioh\\UidVPlqcarXe]r in=40b,rmxattr hTUN^iUTTOa\\lp\\napoiZNkQnN_jfWcgjVeQRahm in=40b] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:20.943054+0000", 2024-09-15T03:47:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.672117768, 2024-09-15T03:47:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010701584, 2024-09-15T03:47:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:48.695 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 569 2024-09-15T03:47:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943054+0000", 2024-09-15T03:47:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943054+0000", 2024-09-15T03:47:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943054+0000", 2024-09-15T03:47:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:47:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943058+0000", 2024-09-15T03:47:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.631e-06 2024-09-15T03:47:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943063+0000", 2024-09-15T03:47:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3800000000000002e-06 2024-09-15T03:47:48.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943067+0000", 2024-09-15T03:47:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4829999999999999e-06 2024-09-15T03:47:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.948103+0000", 2024-09-15T03:47:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0050365490000000004 2024-09-15T03:47:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.948602+0000", 2024-09-15T03:47:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00049903600000000001 2024-09-15T03:47:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949373+0000", 2024-09-15T03:47:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00077054200000000004 2024-09-15T03:47:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949630+0000", 2024-09-15T03:47:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00025681499999999997 2024-09-15T03:47:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949645+0000", 2024-09-15T03:47:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5704000000000001e-05 2024-09-15T03:47:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949690+0000", 2024-09-15T03:47:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4413999999999999e-05 2024-09-15T03:47:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949979+0000", 2024-09-15T03:47:48.700 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00028949100000000001 2024-09-15T03:47:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.951859+0000", 2024-09-15T03:47:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018798179999999999 2024-09-15T03:47:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.953755+0000", 2024-09-15T03:47:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018963109999999999 2024-09-15T03:47:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:48.701 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:667 3.4s0 3:2d1c787c:::smithi00132172-45:head [read 0~1703936,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:29.978831+0000", 2024-09-15T03:47:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 18.636340353000001, 2024-09-15T03:47:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0098251169999999995, 2024-09-15T03:47:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:48.702 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 667 2024-09-15T03:47:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978831+0000", 2024-09-15T03:47:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:48.703 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978831+0000", 2024-09-15T03:47:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978832+0000", 2024-09-15T03:47:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9299999999999999e-07 2024-09-15T03:47:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978834+0000", 2024-09-15T03:47:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3429999999999998e-06 2024-09-15T03:47:48.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978838+0000", 2024-09-15T03:47:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8809999999999998e-06 2024-09-15T03:47:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978840+0000", 2024-09-15T03:47:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.216e-06 2024-09-15T03:47:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.705 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978980+0000", 2024-09-15T03:47:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000140214 2024-09-15T03:47:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978989+0000", 2024-09-15T03:47:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4079999999999998e-06 2024-09-15T03:47:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.981199+0000", 2024-09-15T03:47:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022103539999999999 2024-09-15T03:47:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.981288+0000", 2024-09-15T03:47:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8819000000000005e-05 2024-09-15T03:47:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.988656+0000", 2024-09-15T03:47:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0073684889999999998 2024-09-15T03:47:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.707 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:668 3.4s0 3:2d1c787c:::smithi00132172-45:head [read 0~1703936,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:29.978875+0000", 2024-09-15T03:47:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 18.636296237, 2024-09-15T03:47:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011314753, 2024-09-15T03:47:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 668 2024-09-15T03:47:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.708 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978875+0000", 2024-09-15T03:47:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978875+0000", 2024-09-15T03:47:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978876+0000", 2024-09-15T03:47:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.46e-07 2024-09-15T03:47:48.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978878+0000", 2024-09-15T03:47:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4399999999999999e-06 2024-09-15T03:47:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978884+0000", 2024-09-15T03:47:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8930000000000002e-06 2024-09-15T03:47:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978887+0000", 2024-09-15T03:47:48.710 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8040000000000002e-06 2024-09-15T03:47:48.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.711 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.711 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.979013+0000", 2024-09-15T03:47:48.711 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012650899999999999 2024-09-15T03:47:48.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.711 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:48.711 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.979020+0000", 2024-09-15T03:47:48.711 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8159999999999996e-06 2024-09-15T03:47:48.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.711 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.712 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.981562+0000", 2024-09-15T03:47:48.712 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002542267 2024-09-15T03:47:48.712 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.712 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:48.712 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.981653+0000", 2024-09-15T03:47:48.712 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0509999999999999e-05 2024-09-15T03:47:48.712 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.712 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:48.712 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.990190+0000", 2024-09-15T03:47:48.712 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0085371679999999995 2024-09-15T03:47:48.712 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.712 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:48.713 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.713 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:692 3.9s0 3:97177042:::smithi00132172-18:head [copy-from ver 4] snapc 2f=[2d,2b,29,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:48.713 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.088835+0000", 2024-09-15T03:47:48.713 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 18.526336930999999, 2024-09-15T03:47:48.713 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.016647229999999999, 2024-09-15T03:47:48.713 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:48.713 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:48.713 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:48.713 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:48.713 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:48.714 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 692 2024-09-15T03:47:48.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.714 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:48.714 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.714 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:48.714 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088835+0000", 2024-09-15T03:47:48.714 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.714 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.714 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:48.714 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088835+0000", 2024-09-15T03:47:48.714 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.714 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.715 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:48.715 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088837+0000", 2024-09-15T03:47:48.715 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8670000000000001e-06 2024-09-15T03:47:48.715 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.715 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.715 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:48.715 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088843+0000", 2024-09-15T03:47:48.715 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8179999999999998e-06 2024-09-15T03:47:48.715 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.715 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.715 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:48.715 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088860+0000", 2024-09-15T03:47:48.715 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6566999999999999e-05 2024-09-15T03:47:48.716 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.716 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.716 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:48.716 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088881+0000", 2024-09-15T03:47:48.716 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0619999999999999e-05 2024-09-15T03:47:48.716 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.716 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.716 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.716 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088963+0000", 2024-09-15T03:47:48.716 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2650000000000003e-05 2024-09-15T03:47:48.716 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.716 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.716 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:48.717 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089325+0000", 2024-09-15T03:47:48.717 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00036168700000000002 2024-09-15T03:47:48.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.717 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.717 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:48.717 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102185+0000", 2024-09-15T03:47:48.717 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012859786 2024-09-15T03:47:48.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.717 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.717 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:48.717 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102633+0000", 2024-09-15T03:47:48.717 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00044844699999999998 2024-09-15T03:47:48.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.718 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.718 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:48.718 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102713+0000", 2024-09-15T03:47:48.718 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9616999999999993e-05 2024-09-15T03:47:48.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.718 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.718 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:48.718 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102771+0000", 2024-09-15T03:47:48.718 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7797999999999998e-05 2024-09-15T03:47:48.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.718 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.718 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:48.718 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102793+0000", 2024-09-15T03:47:48.718 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2827999999999999e-05 2024-09-15T03:47:48.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.719 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.719 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:48.719 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.104072+0000", 2024-09-15T03:47:48.719 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012787759999999999 2024-09-15T03:47:48.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.719 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.719 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:48.719 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.105482+0000", 2024-09-15T03:47:48.719 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001409769 2024-09-15T03:47:48.719 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.719 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:48.719 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.720 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.720 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:693 3.9s0 3:97177042:::smithi00132172-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:48.720 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.089498+0000", 2024-09-15T03:47:48.720 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 18.525673995999998, 2024-09-15T03:47:48.720 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013753056, 2024-09-15T03:47:48.720 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:48.720 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:48.720 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:48.720 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:48.720 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:48.720 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 693 2024-09-15T03:47:48.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.721 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:48.721 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.721 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:48.721 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089498+0000", 2024-09-15T03:47:48.721 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.721 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.721 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.721 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:48.721 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089498+0000", 2024-09-15T03:47:48.721 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.721 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.721 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.721 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:48.722 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089499+0000", 2024-09-15T03:47:48.722 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.074e-06 2024-09-15T03:47:48.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.722 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:48.722 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089503+0000", 2024-09-15T03:47:48.722 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1470000000000003e-06 2024-09-15T03:47:48.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.722 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:48.722 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089513+0000", 2024-09-15T03:47:48.722 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0248000000000001e-05 2024-09-15T03:47:48.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.723 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:48.723 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089519+0000", 2024-09-15T03:47:48.723 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2149999999999998e-06 2024-09-15T03:47:48.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.723 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.723 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089719+0000", 2024-09-15T03:47:48.723 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019960400000000001 2024-09-15T03:47:48.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.723 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:48.723 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089776+0000", 2024-09-15T03:47:48.723 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7123000000000002e-05 2024-09-15T03:47:48.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.724 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.724 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102827+0000", 2024-09-15T03:47:48.724 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013050858 2024-09-15T03:47:48.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.724 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:48.724 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102976+0000", 2024-09-15T03:47:48.724 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014900800000000001 2024-09-15T03:47:48.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.724 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:48.725 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.103251+0000", 2024-09-15T03:47:48.725 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00027477900000000001 2024-09-15T03:47:48.725 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.725 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:48.725 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.725 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.725 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:704 3.8s0 3:1af4eaf3:::smithi00132172-20:head [rollback 2d] snapc 2f=[2d,2b,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-15T03:47:48.725 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.982834+0000", 2024-09-15T03:47:48.725 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.632337889999999, 2024-09-15T03:47:48.725 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011049209000000001, 2024-09-15T03:47:48.725 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:48.725 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:48.726 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:48.726 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:48.726 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:48.726 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 704 2024-09-15T03:47:48.726 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.726 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:48.726 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.726 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:48.726 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982834+0000", 2024-09-15T03:47:48.726 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.726 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.726 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.726 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:48.726 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982834+0000", 2024-09-15T03:47:48.727 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.727 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:48.727 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982834+0000", 2024-09-15T03:47:48.727 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3000000000000001e-07 2024-09-15T03:47:48.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.727 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:48.727 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982836+0000", 2024-09-15T03:47:48.727 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0870000000000001e-06 2024-09-15T03:47:48.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.728 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:48.728 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982842+0000", 2024-09-15T03:47:48.728 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.879e-06 2024-09-15T03:47:48.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.728 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:48.728 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982846+0000", 2024-09-15T03:47:48.728 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8500000000000004e-06 2024-09-15T03:47:48.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.728 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.728 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982967+0000", 2024-09-15T03:47:48.728 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012074999999999999 2024-09-15T03:47:48.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.729 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:48.729 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982977+0000", 2024-09-15T03:47:48.729 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0397999999999999e-05 2024-09-15T03:47:48.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.729 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.729 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.985406+0000", 2024-09-15T03:47:48.729 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024286619999999998 2024-09-15T03:47:48.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.729 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:48.729 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.985533+0000", 2024-09-15T03:47:48.730 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012677399999999999 2024-09-15T03:47:48.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.730 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:48.730 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986275+0000", 2024-09-15T03:47:48.730 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0007427 2024-09-15T03:47:48.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.730 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:48.730 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986403+0000", 2024-09-15T03:47:48.730 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012725799999999999 2024-09-15T03:47:48.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.731 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:48.731 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986644+0000", 2024-09-15T03:47:48.731 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000241876 2024-09-15T03:47:48.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.731 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:48.731 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986705+0000", 2024-09-15T03:47:48.731 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0350999999999999e-05 2024-09-15T03:47:48.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.731 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:48.731 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987544+0000", 2024-09-15T03:47:48.731 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00083908500000000005 2024-09-15T03:47:48.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.732 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:48.732 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.990656+0000", 2024-09-15T03:47:48.732 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0031123750000000001 2024-09-15T03:47:48.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.732 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:48.732 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.993883+0000", 2024-09-15T03:47:48.732 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032266339999999999 2024-09-15T03:47:48.732 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.732 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:48.732 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.733 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:705 3.8s0 3:1af4eaf3:::smithi00132172-20:head [append 0~0] snapc 2f=[2d,2b,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-15T03:47:48.733 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.982870+0000", 2024-09-15T03:47:48.733 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 17.632301585, 2024-09-15T03:47:48.733 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.01140421, 2024-09-15T03:47:48.733 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:48.733 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:48.733 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:48.733 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:48.733 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:48.733 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 705 2024-09-15T03:47:48.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.733 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:48.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.734 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:48.734 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982870+0000", 2024-09-15T03:47:48.734 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.734 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.734 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:48.734 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982870+0000", 2024-09-15T03:47:48.734 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.734 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.734 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:48.734 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982871+0000", 2024-09-15T03:47:48.735 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2900000000000004e-07 2024-09-15T03:47:48.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.735 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:48.735 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982874+0000", 2024-09-15T03:47:48.735 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9610000000000001e-06 2024-09-15T03:47:48.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.735 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:48.735 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982878+0000", 2024-09-15T03:47:48.735 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7620000000000003e-06 2024-09-15T03:47:48.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.736 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:48.736 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982881+0000", 2024-09-15T03:47:48.736 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.112e-06 2024-09-15T03:47:48.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.736 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.736 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982995+0000", 2024-09-15T03:47:48.736 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000113544 2024-09-15T03:47:48.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.736 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:48.736 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982999+0000", 2024-09-15T03:47:48.736 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9339999999999999e-06 2024-09-15T03:47:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986479+0000", 2024-09-15T03:47:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0034799399999999999 2024-09-15T03:47:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986599+0000", 2024-09-15T03:47:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000120364 2024-09-15T03:47:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:48.737 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987178+0000", 2024-09-15T03:47:48.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00057913799999999998 2024-09-15T03:47:48.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.738 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:48.738 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987321+0000", 2024-09-15T03:47:48.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000142208 2024-09-15T03:47:48.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.738 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:48.738 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987335+0000", 2024-09-15T03:47:48.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4578e-05 2024-09-15T03:47:48.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.739 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:48.739 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987690+0000", 2024-09-15T03:47:48.739 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00035494600000000003 2024-09-15T03:47:48.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.739 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:48.739 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987799+0000", 2024-09-15T03:47:48.739 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000109379 2024-09-15T03:47:48.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.739 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:48.739 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.991063+0000", 2024-09-15T03:47:48.739 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032640249999999998 2024-09-15T03:47:48.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.740 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:48.740 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.994274+0000", 2024-09-15T03:47:48.740 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032107899999999998 2024-09-15T03:47:48.740 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.740 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:48.740 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.740 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:708 3.4s0 3:2d1c787c:::smithi00132172-45:head [copy-from ver 28] snapc 32=[32,2d,2b,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-15T03:47:48.740 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:31.983317+0000", 2024-09-15T03:47:48.740 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.631855051999999, 2024-09-15T03:47:48.740 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.021360130000000001, 2024-09-15T03:47:48.741 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:48.741 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:48.741 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:48.741 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:48.741 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:48.741 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 708 2024-09-15T03:47:48.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.741 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:48.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.741 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:48.741 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983317+0000", 2024-09-15T03:47:48.741 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.742 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:48.742 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983317+0000", 2024-09-15T03:47:48.742 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.742 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:48.742 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983318+0000", 2024-09-15T03:47:48.742 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2780000000000001e-06 2024-09-15T03:47:48.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.743 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:48.743 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983331+0000", 2024-09-15T03:47:48.743 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2821e-05 2024-09-15T03:47:48.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.743 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:48.743 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983349+0000", 2024-09-15T03:47:48.743 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8544999999999999e-05 2024-09-15T03:47:48.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.743 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:48.743 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983359+0000", 2024-09-15T03:47:48.743 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8570000000000007e-06 2024-09-15T03:47:48.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.744 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.744 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983434+0000", 2024-09-15T03:47:48.744 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4556999999999998e-05 2024-09-15T03:47:48.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.744 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-15T03:47:48.744 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983452+0000", 2024-09-15T03:47:48.744 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8439000000000001e-05 2024-09-15T03:47:48.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.745 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.745 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.985636+0000", 2024-09-15T03:47:48.745 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021841320000000001 2024-09-15T03:47:48.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.745 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:48.745 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.985755+0000", 2024-09-15T03:47:48.745 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118363 2024-09-15T03:47:48.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.745 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:48.745 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.998959+0000", 2024-09-15T03:47:48.745 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013204053 2024-09-15T03:47:48.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.746 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:48.746 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.999917+0000", 2024-09-15T03:47:48.746 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095851300000000003 2024-09-15T03:47:48.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.746 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:48.746 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.999931+0000", 2024-09-15T03:47:48.746 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3631999999999999e-05 2024-09-15T03:47:48.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.746 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:48.746 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000553+0000", 2024-09-15T03:47:48.747 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00062246600000000001 2024-09-15T03:47:48.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.747 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:48.747 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000695+0000", 2024-09-15T03:47:48.747 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014162499999999999 2024-09-15T03:47:48.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.747 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:48.747 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.004291+0000", 2024-09-15T03:47:48.747 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0035959289999999999 2024-09-15T03:47:48.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.748 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:48.748 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.004677+0000", 2024-09-15T03:47:48.748 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00038591999999999999 2024-09-15T03:47:48.748 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.748 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:48.748 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.748 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:709 3.4s0 3:2d1c787c:::smithi00132172-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-15T03:47:48.748 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:31.983373+0000", 2024-09-15T03:47:48.748 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 16.631798211, 2024-09-15T03:47:48.748 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.017317583000000001, 2024-09-15T03:47:48.748 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:48.749 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:48.749 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:48.749 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:48.749 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:48.749 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 709 2024-09-15T03:47:48.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.749 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:48.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.749 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:48.749 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983373+0000", 2024-09-15T03:47:48.749 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.750 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:48.750 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983373+0000", 2024-09-15T03:47:48.750 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.750 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:48.750 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983374+0000", 2024-09-15T03:47:48.750 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8599999999999998e-07 2024-09-15T03:47:48.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.750 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:48.750 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983377+0000", 2024-09-15T03:47:48.750 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0139999999999999e-06 2024-09-15T03:47:48.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.751 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:48.751 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983388+0000", 2024-09-15T03:47:48.751 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0988999999999999e-05 2024-09-15T03:47:48.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.751 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:48.751 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983392+0000", 2024-09-15T03:47:48.751 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5059999999999997e-06 2024-09-15T03:47:48.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.751 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.751 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983542+0000", 2024-09-15T03:47:48.752 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000149887 2024-09-15T03:47:48.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.752 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.752 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:48.752 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983550+0000", 2024-09-15T03:47:48.752 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9589999999999995e-06 2024-09-15T03:47:48.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.752 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.752 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.752 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.985938+0000", 2024-09-15T03:47:48.752 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023879930000000001 2024-09-15T03:47:48.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.752 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.753 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:48.753 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.986005+0000", 2024-09-15T03:47:48.753 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6926000000000003e-05 2024-09-15T03:47:48.753 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.753 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.753 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.753 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000370+0000", 2024-09-15T03:47:48.753 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.014364656999999999 2024-09-15T03:47:48.753 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.753 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.753 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:48.753 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000485+0000", 2024-09-15T03:47:48.753 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011524899999999999 2024-09-15T03:47:48.753 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.754 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.754 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:48.754 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000691+0000", 2024-09-15T03:47:48.754 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020581700000000001 2024-09-15T03:47:48.754 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.754 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:48.754 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.754 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.754 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.754 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:761 3.9s0 3:9cd57211:::smithi00132172-16:head [append 0~507904] snapc 32=[32,2d,2b,26,25,24,23,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T03:47:48.754 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:34.955885+0000", 2024-09-15T03:47:48.754 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 13.659286268000001, 2024-09-15T03:47:48.754 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011160715, 2024-09-15T03:47:48.754 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:48.755 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:48.755 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:48.755 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:48.755 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:48.755 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 761 2024-09-15T03:47:48.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.755 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:48.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.755 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:48.755 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.955885+0000", 2024-09-15T03:47:48.755 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.756 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:48.756 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.955885+0000", 2024-09-15T03:47:48.756 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:48.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.756 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:48.756 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.955887+0000", 2024-09-15T03:47:48.756 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8169999999999999e-06 2024-09-15T03:47:48.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.756 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:48.756 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956332+0000", 2024-09-15T03:47:48.756 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00044459600000000001 2024-09-15T03:47:48.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.757 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.757 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:48.757 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956347+0000", 2024-09-15T03:47:48.757 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5685999999999999e-05 2024-09-15T03:47:48.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.757 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.757 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:48.757 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956354+0000", 2024-09-15T03:47:48.757 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6810000000000001e-06 2024-09-15T03:47:48.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.757 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.758 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:48.758 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956805+0000", 2024-09-15T03:47:48.758 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00045055300000000002 2024-09-15T03:47:48.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.758 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:48.758 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956981+0000", 2024-09-15T03:47:48.758 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017590100000000001 2024-09-15T03:47:48.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.758 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:48.758 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.958362+0000", 2024-09-15T03:47:48.758 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001381415 2024-09-15T03:47:48.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.759 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:48.759 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.958524+0000", 2024-09-15T03:47:48.759 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016167800000000001 2024-09-15T03:47:48.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.759 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:48.759 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.958797+0000", 2024-09-15T03:47:48.759 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000273274 2024-09-15T03:47:48.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.759 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:48.760 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.959021+0000", 2024-09-15T03:47:48.760 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022373099999999999 2024-09-15T03:47:48.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.760 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.760 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:48.760 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.961002+0000", 2024-09-15T03:47:48.760 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019813069999999999 2024-09-15T03:47:48.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.760 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.760 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:48.760 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.966647+0000", 2024-09-15T03:47:48.760 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00564541 2024-09-15T03:47:48.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:48.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:48.761 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:48.761 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.967046+0000", 2024-09-15T03:47:48.761 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00039866599999999999 2024-09-15T03:47:48.761 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.761 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:48.761 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.761 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:48.761 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:48.761 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:48.763 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:47:48.763 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:48.762+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:48.863 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:48.863+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:48.875 INFO:tasks.rados.rados.0.smithi001.stdout:348: done (1 left) 2024-09-15T03:47:48.876 INFO:tasks.rados.rados.0.smithi001.stdout:349: done (0 left) 2024-09-15T03:47:48.876 INFO:tasks.rados.rados.0.smithi001.stdout:350: copy_from oid 4 from oid 49 current snap is 44 2024-09-15T03:47:48.876 INFO:tasks.rados.rados.0.smithi001.stdout:351: snap_create 2024-09-15T03:47:48.878 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:48.879 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:48.879 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:48.879 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:48.879 INFO:tasks.rados.rados.0.smithi001.stdout:350: finishing copy_from to smithi00132172-4 2024-09-15T03:47:48.879 INFO:tasks.rados.rados.0.smithi001.stdout:350: got expected ENOENT (src dne) 2024-09-15T03:47:48.880 INFO:tasks.rados.rados.0.smithi001.stdout:350: finishing copy_from racing read to smithi00132172-4 2024-09-15T03:47:48.888 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:47:48.949 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: filestore_inject_stall = '' (not observed, change may require restart) osd_enable_op_tracker = '' 2024-09-15T03:47:48.952 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T03:47:48.952 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T03:47:48.964 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:48.963+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:48.990 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:48.990 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:48.990 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:48.990 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:48.990 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:48.998 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:47:49.065 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:49.063+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:49.101 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:49.101 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:49.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:49.101 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:49.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.101 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs6 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:49.101 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108711+0000", 2024-09-15T03:47:49.101 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.990054064000006, 2024-09-15T03:47:49.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01472483, 2024-09-15T03:47:49.102 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.102 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.102 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108711+0000", 2024-09-15T03:47:49.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108711+0000", 2024-09-15T03:47:49.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108712+0000", 2024-09-15T03:47:49.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.85e-07 2024-09-15T03:47:49.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108745+0000", 2024-09-15T03:47:49.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2762000000000002e-05 2024-09-15T03:47:49.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108760+0000", 2024-09-15T03:47:49.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4567e-05 2024-09-15T03:47:49.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108780+0000", 2024-09-15T03:47:49.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0122000000000001e-05 2024-09-15T03:47:49.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108842+0000", 2024-09-15T03:47:49.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2297000000000003e-05 2024-09-15T03:47:49.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108851+0000", 2024-09-15T03:47:49.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0129999999999999e-06 2024-09-15T03:47:49.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108942+0000", 2024-09-15T03:47:49.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0713999999999999e-05 2024-09-15T03:47:49.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108949+0000", 2024-09-15T03:47:49.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1600000000000001e-06 2024-09-15T03:47:49.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109975+0000", 2024-09-15T03:47:49.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010264429999999999 2024-09-15T03:47:49.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109994+0000", 2024-09-15T03:47:49.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8374000000000001e-05 2024-09-15T03:47:49.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123436+0000", 2024-09-15T03:47:49.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013442592999999999 2024-09-15T03:47:49.107 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.107 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.107 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.107 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs6 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:49.107 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108809+0000", 2024-09-15T03:47:49.107 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.989956336000006, 2024-09-15T03:47:49.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014757497, 2024-09-15T03:47:49.108 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.108 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.108 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108809+0000", 2024-09-15T03:47:49.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108809+0000", 2024-09-15T03:47:49.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108810+0000", 2024-09-15T03:47:49.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.15e-07 2024-09-15T03:47:49.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108825+0000", 2024-09-15T03:47:49.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5092999999999999e-05 2024-09-15T03:47:49.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108835+0000", 2024-09-15T03:47:49.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0251000000000001e-05 2024-09-15T03:47:49.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108841+0000", 2024-09-15T03:47:49.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0510000000000002e-06 2024-09-15T03:47:49.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108971+0000", 2024-09-15T03:47:49.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012976599999999999 2024-09-15T03:47:49.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108976+0000", 2024-09-15T03:47:49.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8450000000000002e-06 2024-09-15T03:47:49.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109032+0000", 2024-09-15T03:47:49.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6119999999999998e-05 2024-09-15T03:47:49.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109978+0000", 2024-09-15T03:47:49.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00094603899999999997 2024-09-15T03:47:49.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110007+0000", 2024-09-15T03:47:49.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9062999999999999e-05 2024-09-15T03:47:49.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110025+0000", 2024-09-15T03:47:49.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7717999999999999e-05 2024-09-15T03:47:49.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123567+0000", 2024-09-15T03:47:49.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013542136 2024-09-15T03:47:49.113 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.113 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.113 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.114 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:324 3.7s0 3:e40d1694:::smithi00132172-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:49.114 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:49.114 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.254642902000001, 2024-09-15T03:47:49.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0031668960000002, 2024-09-15T03:47:49.114 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.114 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:49.114 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:49.114 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:49.114 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:49.114 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 324 2024-09-15T03:47:49.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.115 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:49.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:49.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:49.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:47:49.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844125+0000", 2024-09-15T03:47:49.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.142e-06 2024-09-15T03:47:49.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844132+0000", 2024-09-15T03:47:49.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0439999999999998e-06 2024-09-15T03:47:49.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844136+0000", 2024-09-15T03:47:49.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.126e-06 2024-09-15T03:47:49.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.847618+0000", 2024-09-15T03:47:49.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0034813790000000002 2024-09-15T03:47:49.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:49.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.847709+0000", 2024-09-15T03:47:49.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0835999999999996e-05 2024-09-15T03:47:49.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.849483+0000", 2024-09-15T03:47:49.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001774929 2024-09-15T03:47:49.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:49.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.849574+0000", 2024-09-15T03:47:49.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0847999999999995e-05 2024-09-15T03:47:49.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.847290+0000", 2024-09-15T03:47:49.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9977151819999999 2024-09-15T03:47:49.119 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.119 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.119 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.119 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:334 3.7s0 3:ece1472c:::smithi00132172-43:head [append 0~0] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:49.119 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:49.119 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.241915814000002, 2024-09-15T03:47:49.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.997592021, 2024-09-15T03:47:49.119 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.119 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:49.119 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:49.120 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:49.120 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:49.120 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 334 2024-09-15T03:47:49.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.120 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:49.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:49.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856851+0000", 2024-09-15T03:47:49.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.238e-06 2024-09-15T03:47:49.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856855+0000", 2024-09-15T03:47:49.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3640000000000001e-06 2024-09-15T03:47:49.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856868+0000", 2024-09-15T03:47:49.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2588999999999999e-05 2024-09-15T03:47:49.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856873+0000", 2024-09-15T03:47:49.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5350000000000004e-06 2024-09-15T03:47:49.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856916+0000", 2024-09-15T03:47:49.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2889000000000002e-05 2024-09-15T03:47:49.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:49.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856983+0000", 2024-09-15T03:47:49.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6746000000000001e-05 2024-09-15T03:47:49.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857448+0000", 2024-09-15T03:47:49.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046498500000000001 2024-09-15T03:47:49.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857720+0000", 2024-09-15T03:47:49.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027227600000000001 2024-09-15T03:47:49.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857741+0000", 2024-09-15T03:47:49.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1149999999999999e-05 2024-09-15T03:47:49.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857795+0000", 2024-09-15T03:47:49.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3919999999999999e-05 2024-09-15T03:47:49.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859582+0000", 2024-09-15T03:47:49.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017862640000000001 2024-09-15T03:47:49.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:49.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854347+0000", 2024-09-15T03:47:49.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9947654780000001 2024-09-15T03:47:49.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854442+0000", 2024-09-15T03:47:49.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4586999999999994e-05 2024-09-15T03:47:49.126 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.126 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.126 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.126 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:335 3.7s0 3:ece1472c:::smithi00132172-43:head [rollback 12] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:49.126 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:49.126 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.241885797000002, 2024-09-15T03:47:49.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9977859200000001, 2024-09-15T03:47:49.126 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.126 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:49.126 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:49.127 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:49.127 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:49.127 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 335 2024-09-15T03:47:49.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.127 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:49.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:49.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:49.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8099999999999999e-07 2024-09-15T03:47:49.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856882+0000", 2024-09-15T03:47:49.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.649e-06 2024-09-15T03:47:49.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856885+0000", 2024-09-15T03:47:49.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.664e-06 2024-09-15T03:47:49.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856888+0000", 2024-09-15T03:47:49.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3910000000000001e-06 2024-09-15T03:47:49.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857826+0000", 2024-09-15T03:47:49.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00093817900000000003 2024-09-15T03:47:49.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:49.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857940+0000", 2024-09-15T03:47:49.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114424 2024-09-15T03:47:49.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858504+0000", 2024-09-15T03:47:49.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056400899999999997 2024-09-15T03:47:49.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858593+0000", 2024-09-15T03:47:49.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8463999999999999e-05 2024-09-15T03:47:49.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858604+0000", 2024-09-15T03:47:49.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1567e-05 2024-09-15T03:47:49.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858642+0000", 2024-09-15T03:47:49.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7369000000000001e-05 2024-09-15T03:47:49.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859588+0000", 2024-09-15T03:47:49.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00094661799999999998 2024-09-15T03:47:49.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:49.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854594+0000", 2024-09-15T03:47:49.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9950055550000001 2024-09-15T03:47:49.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854666+0000", 2024-09-15T03:47:49.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1749999999999996e-05 2024-09-15T03:47:49.132 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.132 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.133 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.133 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:336 3.7s0 3:ece1472c:::smithi00132172-43:head [append 0~0] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:49.133 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:49.133 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.241854746999998, 2024-09-15T03:47:49.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9980017820000002, 2024-09-15T03:47:49.133 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.133 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:49.133 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:49.133 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:49.133 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:49.133 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 336 2024-09-15T03:47:49.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.134 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:49.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:49.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:49.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8000000000000002e-07 2024-09-15T03:47:49.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856913+0000", 2024-09-15T03:47:49.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5400000000000001e-06 2024-09-15T03:47:49.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856918+0000", 2024-09-15T03:47:49.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1540000000000001e-06 2024-09-15T03:47:49.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856921+0000", 2024-09-15T03:47:49.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2550000000000002e-06 2024-09-15T03:47:49.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858655+0000", 2024-09-15T03:47:49.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017336070000000001 2024-09-15T03:47:49.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:49.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858741+0000", 2024-09-15T03:47:49.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6305999999999997e-05 2024-09-15T03:47:49.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859193+0000", 2024-09-15T03:47:49.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045221200000000001 2024-09-15T03:47:49.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859319+0000", 2024-09-15T03:47:49.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012542900000000001 2024-09-15T03:47:49.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859331+0000", 2024-09-15T03:47:49.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2028000000000001e-05 2024-09-15T03:47:49.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859369+0000", 2024-09-15T03:47:49.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8522999999999997e-05 2024-09-15T03:47:49.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859649+0000", 2024-09-15T03:47:49.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028008200000000002 2024-09-15T03:47:49.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:49.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854800+0000", 2024-09-15T03:47:49.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9951506540000001 2024-09-15T03:47:49.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854913+0000", 2024-09-15T03:47:49.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112712 2024-09-15T03:47:49.139 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.139 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.139 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.140 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:346 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:49.140 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:47:49.140 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.260358377000003, 2024-09-15T03:47:49.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005631948, 2024-09-15T03:47:49.140 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.140 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:49.140 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:49.140 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:49.140 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:49.140 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 346 2024-09-15T03:47:49.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.140 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:47:49.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:47:49.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838410+0000", 2024-09-15T03:47:49.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4430000000000002e-06 2024-09-15T03:47:49.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838419+0000", 2024-09-15T03:47:49.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1730000000000001e-06 2024-09-15T03:47:49.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838444+0000", 2024-09-15T03:47:49.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5111000000000001e-05 2024-09-15T03:47:49.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838455+0000", 2024-09-15T03:47:49.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0839000000000001e-05 2024-09-15T03:47:49.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:49.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3415000000000002e-05 2024-09-15T03:47:49.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:49.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838527+0000", 2024-09-15T03:47:49.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0550000000000005e-06 2024-09-15T03:47:49.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840638+0000", 2024-09-15T03:47:49.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021105379999999999 2024-09-15T03:47:49.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:49.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840729+0000", 2024-09-15T03:47:49.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0977999999999995e-05 2024-09-15T03:47:49.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.841834+0000", 2024-09-15T03:47:49.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011055640000000001 2024-09-15T03:47:49.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842223+0000", 2024-09-15T03:47:49.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038899900000000001 2024-09-15T03:47:49.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842241+0000", 2024-09-15T03:47:49.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7317e-05 2024-09-15T03:47:49.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842298+0000", 2024-09-15T03:47:49.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7825000000000003e-05 2024-09-15T03:47:49.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844866+0000", 2024-09-15T03:47:49.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025675289999999998 2024-09-15T03:47:49.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:49.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843973+0000", 2024-09-15T03:47:49.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99910710599999997 2024-09-15T03:47:49.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844039+0000", 2024-09-15T03:47:49.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6056000000000006e-05 2024-09-15T03:47:49.147 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.147 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.147 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.148 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:347 3.2s0 3:48ceb02f:::smithi00132172-5:head [rollback c] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:49.148 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:47:49.148 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.260273581, 2024-09-15T03:47:49.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005723841, 2024-09-15T03:47:49.148 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.148 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:49.148 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:49.148 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:49.148 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:49.148 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 347 2024-09-15T03:47:49.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.148 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:47:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:47:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838493+0000", 2024-09-15T03:47:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7300000000000005e-07 2024-09-15T03:47:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838498+0000", 2024-09-15T03:47:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9300000000000002e-06 2024-09-15T03:47:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838508+0000", 2024-09-15T03:47:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9790000000000005e-06 2024-09-15T03:47:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838513+0000", 2024-09-15T03:47:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8890000000000001e-06 2024-09-15T03:47:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838583+0000", 2024-09-15T03:47:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0696000000000002e-05 2024-09-15T03:47:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838590+0000", 2024-09-15T03:47:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2990000000000002e-06 2024-09-15T03:47:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842313+0000", 2024-09-15T03:47:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037238810000000001 2024-09-15T03:47:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842432+0000", 2024-09-15T03:47:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118474 2024-09-15T03:47:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843108+0000", 2024-09-15T03:47:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00067639299999999996 2024-09-15T03:47:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843227+0000", 2024-09-15T03:47:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119092 2024-09-15T03:47:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843242+0000", 2024-09-15T03:47:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4216000000000001e-05 2024-09-15T03:47:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843292+0000", 2024-09-15T03:47:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0800999999999997e-05 2024-09-15T03:47:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844878+0000", 2024-09-15T03:47:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015852749999999999 2024-09-15T03:47:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844147+0000", 2024-09-15T03:47:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99926949200000004 2024-09-15T03:47:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844216+0000", 2024-09-15T03:47:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8551000000000001e-05 2024-09-15T03:47:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:348 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.260247638000003, 2024-09-15T03:47:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.006019768, 2024-09-15T03:47:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 348 2024-09-15T03:47:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8700000000000001e-07 2024-09-15T03:47:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838521+0000", 2024-09-15T03:47:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3740000000000001e-06 2024-09-15T03:47:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838527+0000", 2024-09-15T03:47:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1999999999999999e-06 2024-09-15T03:47:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838532+0000", 2024-09-15T03:47:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2059999999999998e-06 2024-09-15T03:47:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838609+0000", 2024-09-15T03:47:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7193000000000004e-05 2024-09-15T03:47:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838615+0000", 2024-09-15T03:47:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7980000000000002e-06 2024-09-15T03:47:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843986+0000", 2024-09-15T03:47:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053706450000000003 2024-09-15T03:47:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844106+0000", 2024-09-15T03:47:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120079 2024-09-15T03:47:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844669+0000", 2024-09-15T03:47:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056290300000000001 2024-09-15T03:47:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844804+0000", 2024-09-15T03:47:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013575600000000001 2024-09-15T03:47:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844820+0000", 2024-09-15T03:47:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5145e-05 2024-09-15T03:47:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844893+0000", 2024-09-15T03:47:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3270999999999997e-05 2024-09-15T03:47:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845043+0000", 2024-09-15T03:47:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150144 2024-09-15T03:47:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844473+0000", 2024-09-15T03:47:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99943048700000003 2024-09-15T03:47:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844538+0000", 2024-09-15T03:47:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4179999999999999e-05 2024-09-15T03:47:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:352 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~557056] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:47:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.254049993999999, 2024-09-15T03:47:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.016265065, 2024-09-15T03:47:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 352 2024-09-15T03:47:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:47:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:47:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844718+0000", 2024-09-15T03:47:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2340000000000001e-06 2024-09-15T03:47:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845237+0000", 2024-09-15T03:47:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051953399999999999 2024-09-15T03:47:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845273+0000", 2024-09-15T03:47:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5683999999999999e-05 2024-09-15T03:47:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845281+0000", 2024-09-15T03:47:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8439999999999998e-06 2024-09-15T03:47:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845365+0000", 2024-09-15T03:47:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4651e-05 2024-09-15T03:47:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845486+0000", 2024-09-15T03:47:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012009499999999999 2024-09-15T03:47:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847101+0000", 2024-09-15T03:47:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016157350000000001 2024-09-15T03:47:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847602+0000", 2024-09-15T03:47:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050066200000000003 2024-09-15T03:47:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847622+0000", 2024-09-15T03:47:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9749999999999999e-05 2024-09-15T03:47:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848616+0000", 2024-09-15T03:47:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099390799999999994 2024-09-15T03:47:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848709+0000", 2024-09-15T03:47:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2874000000000006e-05 2024-09-15T03:47:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.860888+0000", 2024-09-15T03:47:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.012179707 2024-09-15T03:47:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.860981+0000", 2024-09-15T03:47:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2386999999999995e-05 2024-09-15T03:47:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:353 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~671744] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:47:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.250487800999998, 2024-09-15T03:47:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.012912206, 2024-09-15T03:47:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 353 2024-09-15T03:47:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:47:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:47:49.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848279+0000", 2024-09-15T03:47:49.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.362e-06 2024-09-15T03:47:49.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849024+0000", 2024-09-15T03:47:49.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00074495500000000005 2024-09-15T03:47:49.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849039+0000", 2024-09-15T03:47:49.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5119e-05 2024-09-15T03:47:49.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849046+0000", 2024-09-15T03:47:49.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6359999999999997e-06 2024-09-15T03:47:49.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849115+0000", 2024-09-15T03:47:49.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8998999999999994e-05 2024-09-15T03:47:49.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:49.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849251+0000", 2024-09-15T03:47:49.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013589799999999999 2024-09-15T03:47:49.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850605+0000", 2024-09-15T03:47:49.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013544659999999999 2024-09-15T03:47:49.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850703+0000", 2024-09-15T03:47:49.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7576000000000002e-05 2024-09-15T03:47:49.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850719+0000", 2024-09-15T03:47:49.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5767e-05 2024-09-15T03:47:49.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851562+0000", 2024-09-15T03:47:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084361899999999999 2024-09-15T03:47:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851705+0000", 2024-09-15T03:47:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000142893 2024-09-15T03:47:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:49.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861112+0000", 2024-09-15T03:47:49.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.009407349 2024-09-15T03:47:49.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861190+0000", 2024-09-15T03:47:49.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7566999999999998e-05 2024-09-15T03:47:49.185 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.185 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.185 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.186 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:354 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~98304] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:49.186 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:47:49.186 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.249685364999998, 2024-09-15T03:47:49.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0177595669999999, 2024-09-15T03:47:49.187 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.187 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:49.187 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:49.187 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:49.187 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:49.187 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 354 2024-09-15T03:47:49.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.188 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:47:49.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:47:49.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849081+0000", 2024-09-15T03:47:49.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0610000000000001e-06 2024-09-15T03:47:49.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849157+0000", 2024-09-15T03:47:49.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5804999999999997e-05 2024-09-15T03:47:49.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849168+0000", 2024-09-15T03:47:49.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0730000000000001e-05 2024-09-15T03:47:49.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849174+0000", 2024-09-15T03:47:49.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3609999999999997e-06 2024-09-15T03:47:49.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850814+0000", 2024-09-15T03:47:49.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001640124 2024-09-15T03:47:49.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:49.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850894+0000", 2024-09-15T03:47:49.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0167999999999995e-05 2024-09-15T03:47:49.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851406+0000", 2024-09-15T03:47:49.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051203200000000005 2024-09-15T03:47:49.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851506+0000", 2024-09-15T03:47:49.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9457999999999995e-05 2024-09-15T03:47:49.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851574+0000", 2024-09-15T03:47:49.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8145000000000006e-05 2024-09-15T03:47:49.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851632+0000", 2024-09-15T03:47:49.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8108e-05 2024-09-15T03:47:49.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851712+0000", 2024-09-15T03:47:49.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9380000000000002e-05 2024-09-15T03:47:49.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:49.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.866753+0000", 2024-09-15T03:47:49.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0150417979999999 2024-09-15T03:47:49.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.866840+0000", 2024-09-15T03:47:49.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6397000000000006e-05 2024-09-15T03:47:49.201 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.202 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.202 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.202 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:355 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [setxattr _header (57) in=7b] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:49.202 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:47:49.203 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.249556052000003, 2024-09-15T03:47:49.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.018024966, 2024-09-15T03:47:49.203 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.203 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:49.203 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:49.203 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:49.204 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:49.204 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 355 2024-09-15T03:47:49.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.204 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:47:49.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:47:49.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849210+0000", 2024-09-15T03:47:49.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.33e-07 2024-09-15T03:47:49.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849215+0000", 2024-09-15T03:47:49.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4379999999999997e-06 2024-09-15T03:47:49.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849234+0000", 2024-09-15T03:47:49.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9409999999999999e-05 2024-09-15T03:47:49.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849241+0000", 2024-09-15T03:47:49.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6370000000000002e-06 2024-09-15T03:47:49.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851823+0000", 2024-09-15T03:47:49.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025817689999999998 2024-09-15T03:47:49.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:49.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851934+0000", 2024-09-15T03:47:49.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111357 2024-09-15T03:47:49.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852466+0000", 2024-09-15T03:47:49.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053248100000000001 2024-09-15T03:47:49.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852580+0000", 2024-09-15T03:47:49.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011379699999999999 2024-09-15T03:47:49.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852607+0000", 2024-09-15T03:47:49.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279e-05 2024-09-15T03:47:49.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852693+0000", 2024-09-15T03:47:49.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6792999999999994e-05 2024-09-15T03:47:49.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852869+0000", 2024-09-15T03:47:49.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017537499999999999 2024-09-15T03:47:49.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:49.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867006+0000", 2024-09-15T03:47:49.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.014136996 2024-09-15T03:47:49.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867234+0000", 2024-09-15T03:47:49.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022870099999999999 2024-09-15T03:47:49.218 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.218 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.219 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.219 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:356 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:49.219 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:47:49.219 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.249519288999998, 2024-09-15T03:47:49.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.020403521, 2024-09-15T03:47:49.220 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.220 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:49.220 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:49.220 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:49.221 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:49.221 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 356 2024-09-15T03:47:49.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.221 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:47:49.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:47:49.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849247+0000", 2024-09-15T03:47:49.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2699999999999999e-07 2024-09-15T03:47:49.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849250+0000", 2024-09-15T03:47:49.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2610000000000002e-06 2024-09-15T03:47:49.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849257+0000", 2024-09-15T03:47:49.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.703e-06 2024-09-15T03:47:49.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849261+0000", 2024-09-15T03:47:49.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7910000000000002e-06 2024-09-15T03:47:49.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852672+0000", 2024-09-15T03:47:49.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003411627 2024-09-15T03:47:49.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:49.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852801+0000", 2024-09-15T03:47:49.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000128918 2024-09-15T03:47:49.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867347+0000", 2024-09-15T03:47:49.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0145458380000001 2024-09-15T03:47:49.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:49.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867476+0000", 2024-09-15T03:47:49.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012915100000000001 2024-09-15T03:47:49.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.869650+0000", 2024-09-15T03:47:49.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021737050000000002 2024-09-15T03:47:49.231 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.231 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.232 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.232 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/64 ECSubWrite(tid=108, reqid=client.4521.0:466, at_version=65'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:49.232 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.856519+0000", 2024-09-15T03:47:49.232 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 37.242246942000001, 2024-09-15T03:47:49.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015639281000000001, 2024-09-15T03:47:49.233 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.233 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.233 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856519+0000", 2024-09-15T03:47:49.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856519+0000", 2024-09-15T03:47:49.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856519+0000", 2024-09-15T03:47:49.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4200000000000005e-07 2024-09-15T03:47:49.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857060+0000", 2024-09-15T03:47:49.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054053200000000004 2024-09-15T03:47:49.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857069+0000", 2024-09-15T03:47:49.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2979999999999997e-06 2024-09-15T03:47:49.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857074+0000", 2024-09-15T03:47:49.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7570000000000004e-06 2024-09-15T03:47:49.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857122+0000", 2024-09-15T03:47:49.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7828999999999999e-05 2024-09-15T03:47:49.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857128+0000", 2024-09-15T03:47:49.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6409999999999996e-06 2024-09-15T03:47:49.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857491+0000", 2024-09-15T03:47:49.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036239300000000003 2024-09-15T03:47:49.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857503+0000", 2024-09-15T03:47:49.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2468e-05 2024-09-15T03:47:49.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857718+0000", 2024-09-15T03:47:49.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021492200000000001 2024-09-15T03:47:49.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857745+0000", 2024-09-15T03:47:49.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7189e-05 2024-09-15T03:47:49.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.872158+0000", 2024-09-15T03:47:49.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01441241 2024-09-15T03:47:49.245 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.245 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.245 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.246 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:527 3.7s0 3:e4b8e19d:::smithi00132172-41:head [copy-from ver 51] snapc 23=[23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:49.246 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:47:49.246 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.149950326999999, 2024-09-15T03:47:49.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030362388000000001, 2024-09-15T03:47:49.246 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.247 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:49.247 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:49.247 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:49.247 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:49.247 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 527 2024-09-15T03:47:49.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.248 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:47:49.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:47:49.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948817+0000", 2024-09-15T03:47:49.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3230000000000001e-06 2024-09-15T03:47:49.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948825+0000", 2024-09-15T03:47:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5900000000000008e-06 2024-09-15T03:47:49.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948838+0000", 2024-09-15T03:47:49.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2712e-05 2024-09-15T03:47:49.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948842+0000", 2024-09-15T03:47:49.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4170000000000002e-06 2024-09-15T03:47:49.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948923+0000", 2024-09-15T03:47:49.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0686000000000006e-05 2024-09-15T03:47:49.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:47:49.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949020+0000", 2024-09-15T03:47:49.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6984999999999994e-05 2024-09-15T03:47:49.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954390+0000", 2024-09-15T03:47:49.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053704790000000001 2024-09-15T03:47:49.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:49.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954558+0000", 2024-09-15T03:47:49.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000167202 2024-09-15T03:47:49.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.973998+0000", 2024-09-15T03:47:49.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019440316999999999 2024-09-15T03:47:49.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974527+0000", 2024-09-15T03:47:49.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052943800000000002 2024-09-15T03:47:49.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974541+0000", 2024-09-15T03:47:49.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3774999999999999e-05 2024-09-15T03:47:49.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974841+0000", 2024-09-15T03:47:49.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029942899999999998 2024-09-15T03:47:49.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.976067+0000", 2024-09-15T03:47:49.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012265399999999999 2024-09-15T03:47:49.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:49.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.978952+0000", 2024-09-15T03:47:49.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028848839999999999 2024-09-15T03:47:49.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.979178+0000", 2024-09-15T03:47:49.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022561099999999999 2024-09-15T03:47:49.257 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.257 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.257 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.258 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:528 3.7s0 3:e4b8e19d:::smithi00132172-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:49.258 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:49.258 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.149915917999998, 2024-09-15T03:47:49.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026927036000000001, 2024-09-15T03:47:49.258 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.258 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:49.258 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:49.258 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:49.258 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:49.258 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 528 2024-09-15T03:47:49.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.258 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:49.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:49.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:49.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9700000000000003e-07 2024-09-15T03:47:49.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948852+0000", 2024-09-15T03:47:49.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.604e-06 2024-09-15T03:47:49.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948855+0000", 2024-09-15T03:47:49.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9369999999999997e-06 2024-09-15T03:47:49.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948858+0000", 2024-09-15T03:47:49.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2709999999999999e-06 2024-09-15T03:47:49.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949080+0000", 2024-09-15T03:47:49.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000222148 2024-09-15T03:47:49.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:47:49.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949156+0000", 2024-09-15T03:47:49.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6215000000000001e-05 2024-09-15T03:47:49.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954754+0000", 2024-09-15T03:47:49.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055978060000000003 2024-09-15T03:47:49.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:49.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954858+0000", 2024-09-15T03:47:49.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010419199999999999 2024-09-15T03:47:49.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974861+0000", 2024-09-15T03:47:49.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020002671999999999 2024-09-15T03:47:49.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:49.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974987+0000", 2024-09-15T03:47:49.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012638699999999999 2024-09-15T03:47:49.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.975777+0000", 2024-09-15T03:47:49.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00078950699999999995 2024-09-15T03:47:49.263 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.263 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.263 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.264 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:833 3.7s0 3:ece1472c:::smithi00132172-43:head [read 0~4816896,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e97)", 2024-09-15T03:47:49.264 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:43.913147+0000", 2024-09-15T03:47:49.264 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.1856188029999997, 2024-09-15T03:47:49.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014828559999999999, 2024-09-15T03:47:49.264 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.264 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:49.264 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:49.264 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:49.264 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:49.264 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 833 2024-09-15T03:47:49.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.264 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913147+0000", 2024-09-15T03:47:49.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913147+0000", 2024-09-15T03:47:49.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913147+0000", 2024-09-15T03:47:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7000000000000001e-07 2024-09-15T03:47:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913149+0000", 2024-09-15T03:47:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8929999999999999e-06 2024-09-15T03:47:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913152+0000", 2024-09-15T03:47:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.472e-06 2024-09-15T03:47:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913154+0000", 2024-09-15T03:47:49.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0940000000000002e-06 2024-09-15T03:47:49.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913477+0000", 2024-09-15T03:47:49.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032283000000000002 2024-09-15T03:47:49.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:49.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913585+0000", 2024-09-15T03:47:49.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010744899999999999 2024-09-15T03:47:49.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.927975+0000", 2024-09-15T03:47:49.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014390551999999999 2024-09-15T03:47:49.270 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.270 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.270 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.270 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:834 3.7s0 3:ece1472c:::smithi00132172-43:head [read 0~4816896,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e97)", 2024-09-15T03:47:49.270 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:43.913171+0000", 2024-09-15T03:47:49.270 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.1855950010000003, 2024-09-15T03:47:49.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017820687000000002, 2024-09-15T03:47:49.270 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.270 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:49.270 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:49.271 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:49.271 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:49.271 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 834 2024-09-15T03:47:49.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.271 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913171+0000", 2024-09-15T03:47:49.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913171+0000", 2024-09-15T03:47:49.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913171+0000", 2024-09-15T03:47:49.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6e-07 2024-09-15T03:47:49.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913173+0000", 2024-09-15T03:47:49.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7719999999999999e-06 2024-09-15T03:47:49.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913176+0000", 2024-09-15T03:47:49.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8020000000000002e-06 2024-09-15T03:47:49.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913180+0000", 2024-09-15T03:47:49.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.692e-06 2024-09-15T03:47:49.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.914761+0000", 2024-09-15T03:47:49.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001580936 2024-09-15T03:47:49.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:49.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.914879+0000", 2024-09-15T03:47:49.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117518 2024-09-15T03:47:49.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.930991+0000", 2024-09-15T03:47:49.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016112707 2024-09-15T03:47:49.275 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.275 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.275 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.275 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:853 3.7s0 3:e40d1694:::smithi00132172-50:head [copy-from ver 33] snapc 3e=[3e,3d,3b,3a,38,37,35,34,32,2d,2b,25,24,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-15T03:47:49.275 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:44.879131+0000", 2024-09-15T03:47:49.275 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.2196341459999998, 2024-09-15T03:47:49.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019602471, 2024-09-15T03:47:49.275 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.275 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:49.275 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:49.275 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:49.276 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:49.276 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 853 2024-09-15T03:47:49.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.276 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.879131+0000", 2024-09-15T03:47:49.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.879131+0000", 2024-09-15T03:47:49.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.879132+0000", 2024-09-15T03:47:49.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4400000000000001e-07 2024-09-15T03:47:49.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.879134+0000", 2024-09-15T03:47:49.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0190000000000001e-06 2024-09-15T03:47:49.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.879137+0000", 2024-09-15T03:47:49.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4929999999999998e-06 2024-09-15T03:47:49.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.879139+0000", 2024-09-15T03:47:49.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1280000000000002e-06 2024-09-15T03:47:49.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.879180+0000", 2024-09-15T03:47:49.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0750000000000001e-05 2024-09-15T03:47:49.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:49.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.879764+0000", 2024-09-15T03:47:49.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00058396700000000001 2024-09-15T03:47:49.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.888816+0000", 2024-09-15T03:47:49.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090521460000000005 2024-09-15T03:47:49.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.889281+0000", 2024-09-15T03:47:49.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046523499999999999 2024-09-15T03:47:49.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.889296+0000", 2024-09-15T03:47:49.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4965e-05 2024-09-15T03:47:49.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.889576+0000", 2024-09-15T03:47:49.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027968999999999997 2024-09-15T03:47:49.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.889890+0000", 2024-09-15T03:47:49.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031355400000000001 2024-09-15T03:47:49.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:49.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.891927+0000", 2024-09-15T03:47:49.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020376259999999998 2024-09-15T03:47:49.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.898734+0000", 2024-09-15T03:47:49.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068065540000000002 2024-09-15T03:47:49.282 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.282 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.282 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.282 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.282 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.282 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:49.284 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:47:49.285 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:49.282+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:49.385 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:49.383+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:49.426 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:49.426 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:49.426 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:49.426 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:49.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:47:49.485 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:49.485+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:49.576 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:49.576 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:49.576 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:49.576 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:49.576 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:49.584 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:47:49.587 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:49.586+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:49.688 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 38/28 ECSubWrite(tid=33, reqid=client.4521.0:222, at_version=38'14, trim_to=0'0, roll_forward_to=35'12))", 2024-09-15T03:47:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:47:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.55502104, 2024-09-15T03:47:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9949187450000001, 2024-09-15T03:47:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:47:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:47:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130996+0000", 2024-09-15T03:47:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8809999999999998e-06 2024-09-15T03:47:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131020+0000", 2024-09-15T03:47:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4190999999999999e-05 2024-09-15T03:47:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131039+0000", 2024-09-15T03:47:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9228999999999998e-05 2024-09-15T03:47:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131048+0000", 2024-09-15T03:47:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2380000000000003e-06 2024-09-15T03:47:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131122+0000", 2024-09-15T03:47:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3146999999999996e-05 2024-09-15T03:47:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131132+0000", 2024-09-15T03:47:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.08e-05 2024-09-15T03:47:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131381+0000", 2024-09-15T03:47:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000248936 2024-09-15T03:47:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131392+0000", 2024-09-15T03:47:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0987e-05 2024-09-15T03:47:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131444+0000", 2024-09-15T03:47:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1585000000000002e-05 2024-09-15T03:47:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131465+0000", 2024-09-15T03:47:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1365999999999998e-05 2024-09-15T03:47:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.125911+0000", 2024-09-15T03:47:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9944453850000001 2024-09-15T03:47:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 38/28 ECSubWrite(tid=34, reqid=client.4521.0:222, at_version=0'0, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T03:47:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:47:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.553621941000003, 2024-09-15T03:47:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.002752697, 2024-09-15T03:47:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:47:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:47:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132392+0000", 2024-09-15T03:47:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5899999999999995e-07 2024-09-15T03:47:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132415+0000", 2024-09-15T03:47:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3561e-05 2024-09-15T03:47:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132436+0000", 2024-09-15T03:47:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0879000000000001e-05 2024-09-15T03:47:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132444+0000", 2024-09-15T03:47:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9200000000000004e-06 2024-09-15T03:47:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132497+0000", 2024-09-15T03:47:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2299000000000002e-05 2024-09-15T03:47:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132508+0000", 2024-09-15T03:47:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1100999999999999e-05 2024-09-15T03:47:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132615+0000", 2024-09-15T03:47:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107546 2024-09-15T03:47:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132635+0000", 2024-09-15T03:47:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9374000000000001e-05 2024-09-15T03:47:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132677+0000", 2024-09-15T03:47:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2456999999999997e-05 2024-09-15T03:47:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132711+0000", 2024-09-15T03:47:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4118000000000001e-05 2024-09-15T03:47:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135144+0000", 2024-09-15T03:47:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.002432583 2024-09-15T03:47:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=45, reqid=client.4521.0:227, at_version=38'24, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:47:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.530695398999995, 2024-09-15T03:47:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.974765339, 2024-09-15T03:47:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.43e-07 2024-09-15T03:47:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156110+0000", 2024-09-15T03:47:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000792409 2024-09-15T03:47:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156139+0000", 2024-09-15T03:47:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9207e-05 2024-09-15T03:47:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156148+0000", 2024-09-15T03:47:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1269999999999992e-06 2024-09-15T03:47:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156248+0000", 2024-09-15T03:47:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9421999999999998e-05 2024-09-15T03:47:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156258+0000", 2024-09-15T03:47:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6989999999999998e-06 2024-09-15T03:47:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156674+0000", 2024-09-15T03:47:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041682600000000002 2024-09-15T03:47:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156705+0000", 2024-09-15T03:47:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0525000000000003e-05 2024-09-15T03:47:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157742+0000", 2024-09-15T03:47:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010374 2024-09-15T03:47:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157777+0000", 2024-09-15T03:47:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4878e-05 2024-09-15T03:47:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.130083+0000", 2024-09-15T03:47:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972305703 2024-09-15T03:47:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=37, reqid=client.4521.0:236, at_version=38'5, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.529081942999994, 2024-09-15T03:47:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9740836640000001, 2024-09-15T03:47:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1300000000000001e-07 2024-09-15T03:47:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156953+0000", 2024-09-15T03:47:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1585000000000001e-05 2024-09-15T03:47:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156974+0000", 2024-09-15T03:47:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0764e-05 2024-09-15T03:47:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156981+0000", 2024-09-15T03:47:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2760000000000003e-06 2024-09-15T03:47:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157061+0000", 2024-09-15T03:47:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9583e-05 2024-09-15T03:47:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157075+0000", 2024-09-15T03:47:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4019000000000001e-05 2024-09-15T03:47:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157351+0000", 2024-09-15T03:47:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027641100000000001 2024-09-15T03:47:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157746+0000", 2024-09-15T03:47:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039553199999999998 2024-09-15T03:47:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157838+0000", 2024-09-15T03:47:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1632000000000006e-05 2024-09-15T03:47:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157856+0000", 2024-09-15T03:47:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7873999999999999e-05 2024-09-15T03:47:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.131015+0000", 2024-09-15T03:47:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9731586750000001 2024-09-15T03:47:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=46, reqid=client.4521.0:228, at_version=38'25, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:47:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.529023160999998, 2024-09-15T03:47:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9732838510000001, 2024-09-15T03:47:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0999999999999998e-07 2024-09-15T03:47:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157149+0000", 2024-09-15T03:47:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000158763 2024-09-15T03:47:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157165+0000", 2024-09-15T03:47:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5883999999999999e-05 2024-09-15T03:47:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157175+0000", 2024-09-15T03:47:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4949999999999997e-06 2024-09-15T03:47:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157219+0000", 2024-09-15T03:47:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4069000000000001e-05 2024-09-15T03:47:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157228+0000", 2024-09-15T03:47:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7829999999999993e-06 2024-09-15T03:47:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157345+0000", 2024-09-15T03:47:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116655 2024-09-15T03:47:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157746+0000", 2024-09-15T03:47:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040045100000000002 2024-09-15T03:47:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157838+0000", 2024-09-15T03:47:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1955999999999998e-05 2024-09-15T03:47:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157888+0000", 2024-09-15T03:47:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0943000000000003e-05 2024-09-15T03:47:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.130274+0000", 2024-09-15T03:47:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972385142 2024-09-15T03:47:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=38, reqid=client.4521.0:237, at_version=38'6, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:47:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.527680591000006, 2024-09-15T03:47:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9764011529999999, 2024-09-15T03:47:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:47:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:47:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158333+0000", 2024-09-15T03:47:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8599999999999997e-07 2024-09-15T03:47:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158353+0000", 2024-09-15T03:47:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9504000000000002e-05 2024-09-15T03:47:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158368+0000", 2024-09-15T03:47:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4966999999999999e-05 2024-09-15T03:47:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158374+0000", 2024-09-15T03:47:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.798e-06 2024-09-15T03:47:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158430+0000", 2024-09-15T03:47:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5744e-05 2024-09-15T03:47:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158440+0000", 2024-09-15T03:47:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4560000000000006e-06 2024-09-15T03:47:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158529+0000", 2024-09-15T03:47:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9172999999999995e-05 2024-09-15T03:47:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158543+0000", 2024-09-15T03:47:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.395e-05 2024-09-15T03:47:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158654+0000", 2024-09-15T03:47:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011110700000000001 2024-09-15T03:47:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158671+0000", 2024-09-15T03:47:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7068999999999999e-05 2024-09-15T03:47:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134733+0000", 2024-09-15T03:47:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.976062599 2024-09-15T03:47:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=47, reqid=client.4521.0:229, at_version=38'26, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:47:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:47:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.527676628999998, 2024-09-15T03:47:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9727810969999999, 2024-09-15T03:47:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:47:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:47:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158337+0000", 2024-09-15T03:47:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.15e-07 2024-09-15T03:47:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158351+0000", 2024-09-15T03:47:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3871e-05 2024-09-15T03:47:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158372+0000", 2024-09-15T03:47:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1058999999999999e-05 2024-09-15T03:47:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158376+0000", 2024-09-15T03:47:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5249999999999999e-06 2024-09-15T03:47:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158420+0000", 2024-09-15T03:47:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3918000000000002e-05 2024-09-15T03:47:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158429+0000", 2024-09-15T03:47:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5150000000000006e-06 2024-09-15T03:47:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158514+0000", 2024-09-15T03:47:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5252000000000003e-05 2024-09-15T03:47:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158542+0000", 2024-09-15T03:47:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.798e-05 2024-09-15T03:47:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158653+0000", 2024-09-15T03:47:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111315 2024-09-15T03:47:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158701+0000", 2024-09-15T03:47:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8241000000000001e-05 2024-09-15T03:47:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.131117+0000", 2024-09-15T03:47:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972416006 2024-09-15T03:47:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=39, reqid=client.4521.0:238, at_version=38'7, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:47:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.526989473, 2024-09-15T03:47:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9758214509999998, 2024-09-15T03:47:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:47:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:47:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159024+0000", 2024-09-15T03:47:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6700000000000003e-07 2024-09-15T03:47:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159096+0000", 2024-09-15T03:47:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1386999999999999e-05 2024-09-15T03:47:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159113+0000", 2024-09-15T03:47:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7020999999999998e-05 2024-09-15T03:47:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159122+0000", 2024-09-15T03:47:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4720000000000001e-06 2024-09-15T03:47:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159168+0000", 2024-09-15T03:47:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5766000000000002e-05 2024-09-15T03:47:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159176+0000", 2024-09-15T03:47:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3070000000000003e-06 2024-09-15T03:47:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159258+0000", 2024-09-15T03:47:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1398999999999998e-05 2024-09-15T03:47:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159292+0000", 2024-09-15T03:47:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4480000000000002e-05 2024-09-15T03:47:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159342+0000", 2024-09-15T03:47:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9450999999999998e-05 2024-09-15T03:47:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159362+0000", 2024-09-15T03:47:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0633e-05 2024-09-15T03:47:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134845+0000", 2024-09-15T03:47:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.975482768 2024-09-15T03:47:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=48, reqid=client.4521.0:229, at_version=0'0, trim_to=0'0, roll_forward_to=38'26))", 2024-09-15T03:47:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:47:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.525622677000001, 2024-09-15T03:47:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.974485472, 2024-09-15T03:47:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:47:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:47:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160391+0000", 2024-09-15T03:47:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.136e-06 2024-09-15T03:47:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160405+0000", 2024-09-15T03:47:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3613e-05 2024-09-15T03:47:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160426+0000", 2024-09-15T03:47:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1234999999999999e-05 2024-09-15T03:47:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160434+0000", 2024-09-15T03:47:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.61e-06 2024-09-15T03:47:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160496+0000", 2024-09-15T03:47:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2384000000000004e-05 2024-09-15T03:47:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160507+0000", 2024-09-15T03:47:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1005000000000001e-05 2024-09-15T03:47:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160632+0000", 2024-09-15T03:47:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124566 2024-09-15T03:47:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160653+0000", 2024-09-15T03:47:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0707999999999999e-05 2024-09-15T03:47:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160701+0000", 2024-09-15T03:47:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8751999999999997e-05 2024-09-15T03:47:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160739+0000", 2024-09-15T03:47:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7486e-05 2024-09-15T03:47:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134876+0000", 2024-09-15T03:47:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9741369770000001 2024-09-15T03:47:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=40, reqid=client.4521.0:238, at_version=0'0, trim_to=0'0, roll_forward_to=38'7))", 2024-09-15T03:47:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:47:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.523776256000005, 2024-09-15T03:47:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9727862730000001, 2024-09-15T03:47:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:47:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:47:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162238+0000", 2024-09-15T03:47:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.672e-06 2024-09-15T03:47:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162253+0000", 2024-09-15T03:47:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4932e-05 2024-09-15T03:47:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162284+0000", 2024-09-15T03:47:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1211000000000003e-05 2024-09-15T03:47:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162296+0000", 2024-09-15T03:47:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1229999999999999e-05 2024-09-15T03:47:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162373+0000", 2024-09-15T03:47:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7146999999999998e-05 2024-09-15T03:47:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162387+0000", 2024-09-15T03:47:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4290999999999999e-05 2024-09-15T03:47:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162502+0000", 2024-09-15T03:47:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115054 2024-09-15T03:47:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162516+0000", 2024-09-15T03:47:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3597999999999999e-05 2024-09-15T03:47:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162558+0000", 2024-09-15T03:47:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2562999999999999e-05 2024-09-15T03:47:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162584+0000", 2024-09-15T03:47:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5738e-05 2024-09-15T03:47:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135023+0000", 2024-09-15T03:47:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9724388369999999 2024-09-15T03:47:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 39/28 ECSubWrite(tid=38, reqid=osd.11.0:38, at_version=39'16, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T03:47:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.542393955999998, 2024-09-15T03:47:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99210653800000004, 2024-09-15T03:47:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1800000000000002e-07 2024-09-15T03:47:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143648+0000", 2024-09-15T03:47:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.849e-05 2024-09-15T03:47:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143670+0000", 2024-09-15T03:47:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2036e-05 2024-09-15T03:47:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143680+0000", 2024-09-15T03:47:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8320000000000001e-06 2024-09-15T03:47:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143783+0000", 2024-09-15T03:47:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103569 2024-09-15T03:47:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143801+0000", 2024-09-15T03:47:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8077999999999999e-05 2024-09-15T03:47:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144077+0000", 2024-09-15T03:47:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027569500000000001 2024-09-15T03:47:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144459+0000", 2024-09-15T03:47:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038227299999999998 2024-09-15T03:47:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144519+0000", 2024-09-15T03:47:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9907000000000001e-05 2024-09-15T03:47:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144628+0000", 2024-09-15T03:47:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108958 2024-09-15T03:47:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135726+0000", 2024-09-15T03:47:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99109738199999997 2024-09-15T03:47:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 39/31 ECSubWrite(tid=41, reqid=osd.6.0:41, at_version=39'9, trim_to=0'0, roll_forward_to=38'7))", 2024-09-15T03:47:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.541743701000001, 2024-09-15T03:47:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99106653099999997, 2024-09-15T03:47:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.97e-07 2024-09-15T03:47:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144297+0000", 2024-09-15T03:47:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7815000000000001e-05 2024-09-15T03:47:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144314+0000", 2024-09-15T03:47:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6269e-05 2024-09-15T03:47:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144322+0000", 2024-09-15T03:47:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1880000000000002e-06 2024-09-15T03:47:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144767+0000", 2024-09-15T03:47:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044555300000000001 2024-09-15T03:47:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144777+0000", 2024-09-15T03:47:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5950000000000005e-06 2024-09-15T03:47:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144992+0000", 2024-09-15T03:47:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000215479 2024-09-15T03:47:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145142+0000", 2024-09-15T03:47:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149302 2024-09-15T03:47:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145249+0000", 2024-09-15T03:47:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106971 2024-09-15T03:47:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145299+0000", 2024-09-15T03:47:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0413e-05 2024-09-15T03:47:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135336+0000", 2024-09-15T03:47:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99003674900000005 2024-09-15T03:47:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 39/28 ECSubWrite(tid=50, reqid=osd.2.0:50, at_version=39'28, trim_to=0'0, roll_forward_to=38'26))", 2024-09-15T03:47:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.541446219000001, 2024-09-15T03:47:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99095576299999999, 2024-09-15T03:47:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.03e-07 2024-09-15T03:47:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144593+0000", 2024-09-15T03:47:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6520999999999999e-05 2024-09-15T03:47:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144611+0000", 2024-09-15T03:47:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7963000000000001e-05 2024-09-15T03:47:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144619+0000", 2024-09-15T03:47:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7409999999999992e-06 2024-09-15T03:47:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144821+0000", 2024-09-15T03:47:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000201652 2024-09-15T03:47:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144832+0000", 2024-09-15T03:47:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0827e-05 2024-09-15T03:47:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145099+0000", 2024-09-15T03:47:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026688299999999999 2024-09-15T03:47:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145143+0000", 2024-09-15T03:47:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.481e-05 2024-09-15T03:47:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145249+0000", 2024-09-15T03:47:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010618799999999999 2024-09-15T03:47:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145340+0000", 2024-09-15T03:47:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0340000000000006e-05 2024-09-15T03:47:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135522+0000", 2024-09-15T03:47:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99018263500000003 2024-09-15T03:47:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=43, reqid=client.4521.0:240, at_version=39'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:47:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.538105498999997, 2024-09-15T03:47:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99104975500000003, 2024-09-15T03:47:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:47:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:47:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147908+0000", 2024-09-15T03:47:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.05e-07 2024-09-15T03:47:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147931+0000", 2024-09-15T03:47:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3621e-05 2024-09-15T03:47:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147945+0000", 2024-09-15T03:47:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3648999999999999e-05 2024-09-15T03:47:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147951+0000", 2024-09-15T03:47:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0510000000000002e-06 2024-09-15T03:47:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148055+0000", 2024-09-15T03:47:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104421 2024-09-15T03:47:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148068+0000", 2024-09-15T03:47:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3019e-05 2024-09-15T03:47:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148375+0000", 2024-09-15T03:47:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030683400000000002 2024-09-15T03:47:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148399+0000", 2024-09-15T03:47:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3760999999999998e-05 2024-09-15T03:47:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148453+0000", 2024-09-15T03:47:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3851999999999999e-05 2024-09-15T03:47:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148481+0000", 2024-09-15T03:47:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8637999999999999e-05 2024-09-15T03:47:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.138957+0000", 2024-09-15T03:47:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99047580400000002 2024-09-15T03:47:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 39/28 ECSubWrite(tid=39, reqid=osd.11.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:47:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:47:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.537973010000002, 2024-09-15T03:47:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99452509, 2024-09-15T03:47:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:47:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:47:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148041+0000", 2024-09-15T03:47:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0999999999999998e-07 2024-09-15T03:47:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148052+0000", 2024-09-15T03:47:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.129e-05 2024-09-15T03:47:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148064+0000", 2024-09-15T03:47:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173e-05 2024-09-15T03:47:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148068+0000", 2024-09-15T03:47:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5680000000000001e-06 2024-09-15T03:47:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148136+0000", 2024-09-15T03:47:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7508999999999993e-05 2024-09-15T03:47:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148148+0000", 2024-09-15T03:47:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2002e-05 2024-09-15T03:47:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148265+0000", 2024-09-15T03:47:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117669 2024-09-15T03:47:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148289+0000", 2024-09-15T03:47:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3672e-05 2024-09-15T03:47:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148348+0000", 2024-09-15T03:47:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8838000000000001e-05 2024-09-15T03:47:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148368+0000", 2024-09-15T03:47:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9942999999999999e-05 2024-09-15T03:47:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142565+0000", 2024-09-15T03:47:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99419715900000005 2024-09-15T03:47:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=45, reqid=client.4521.0:241, at_version=39'17, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:47:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.534969986999997, 2024-09-15T03:47:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99118202200000005, 2024-09-15T03:47:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:47:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:47:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151044+0000", 2024-09-15T03:47:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1340000000000001e-06 2024-09-15T03:47:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151073+0000", 2024-09-15T03:47:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8894000000000002e-05 2024-09-15T03:47:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151095+0000", 2024-09-15T03:47:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1889e-05 2024-09-15T03:47:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151107+0000", 2024-09-15T03:47:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2136999999999999e-05 2024-09-15T03:47:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151189+0000", 2024-09-15T03:47:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1883000000000002e-05 2024-09-15T03:47:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151203+0000", 2024-09-15T03:47:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3825999999999999e-05 2024-09-15T03:47:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151305+0000", 2024-09-15T03:47:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101816 2024-09-15T03:47:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151337+0000", 2024-09-15T03:47:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2703000000000002e-05 2024-09-15T03:47:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151387+0000", 2024-09-15T03:47:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9366000000000001e-05 2024-09-15T03:47:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151405+0000", 2024-09-15T03:47:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8748000000000001e-05 2024-09-15T03:47:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142225+0000", 2024-09-15T03:47:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99081962599999995 2024-09-15T03:47:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 39/31 ECSubWrite(tid=42, reqid=osd.6.0:41, at_version=0'0, trim_to=0'0, roll_forward_to=39'9))", 2024-09-15T03:47:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:47:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.534673945999998, 2024-09-15T03:47:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99092639999999999, 2024-09-15T03:47:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:47:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:47:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151340+0000", 2024-09-15T03:47:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8299999999999996e-07 2024-09-15T03:47:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151350+0000", 2024-09-15T03:47:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0675e-05 2024-09-15T03:47:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151362+0000", 2024-09-15T03:47:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2048999999999999e-05 2024-09-15T03:47:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151369+0000", 2024-09-15T03:47:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3230000000000003e-06 2024-09-15T03:47:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151425+0000", 2024-09-15T03:47:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5964000000000002e-05 2024-09-15T03:47:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151435+0000", 2024-09-15T03:47:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9240000000000002e-06 2024-09-15T03:47:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151530+0000", 2024-09-15T03:47:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5649000000000005e-05 2024-09-15T03:47:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151558+0000", 2024-09-15T03:47:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7976000000000001e-05 2024-09-15T03:47:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151599+0000", 2024-09-15T03:47:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1152000000000002e-05 2024-09-15T03:47:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151617+0000", 2024-09-15T03:47:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7676000000000002e-05 2024-09-15T03:47:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142265+0000", 2024-09-15T03:47:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99064822900000005 2024-09-15T03:47:49.796 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.796 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.796 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.796 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=46, reqid=client.4521.0:242, at_version=39'18, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:49.796 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:49.796 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.534648310000001, 2024-09-15T03:47:49.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.991000204, 2024-09-15T03:47:49.796 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.796 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.796 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:49.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:49.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:49.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3399999999999999e-07 2024-09-15T03:47:49.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151393+0000", 2024-09-15T03:47:49.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8149000000000001e-05 2024-09-15T03:47:49.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151405+0000", 2024-09-15T03:47:49.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2063e-05 2024-09-15T03:47:49.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151415+0000", 2024-09-15T03:47:49.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4579999999999999e-06 2024-09-15T03:47:49.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151472+0000", 2024-09-15T03:47:49.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7018999999999997e-05 2024-09-15T03:47:49.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151480+0000", 2024-09-15T03:47:49.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1410000000000005e-06 2024-09-15T03:47:49.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151571+0000", 2024-09-15T03:47:49.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1243999999999996e-05 2024-09-15T03:47:49.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151602+0000", 2024-09-15T03:47:49.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0834000000000003e-05 2024-09-15T03:47:49.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151632+0000", 2024-09-15T03:47:49.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9527e-05 2024-09-15T03:47:49.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151685+0000", 2024-09-15T03:47:49.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3599e-05 2024-09-15T03:47:49.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142365+0000", 2024-09-15T03:47:49.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99067953600000003 2024-09-15T03:47:49.802 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.802 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.802 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.803 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 39/28 ECSubWrite(tid=51, reqid=osd.2.0:50, at_version=0'0, trim_to=0'0, roll_forward_to=39'28))", 2024-09-15T03:47:49.803 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:47:49.803 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.531284872000001, 2024-09-15T03:47:49.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98766307399999997, 2024-09-15T03:47:49.803 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.803 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.803 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:47:49.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:47:49.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154729+0000", 2024-09-15T03:47:49.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3799999999999996e-07 2024-09-15T03:47:49.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154742+0000", 2024-09-15T03:47:49.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2909e-05 2024-09-15T03:47:49.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154755+0000", 2024-09-15T03:47:49.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3251999999999999e-05 2024-09-15T03:47:49.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154761+0000", 2024-09-15T03:47:49.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6559999999999997e-06 2024-09-15T03:47:49.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154829+0000", 2024-09-15T03:47:49.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8549999999999999e-05 2024-09-15T03:47:49.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154840+0000", 2024-09-15T03:47:49.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0159999999999999e-05 2024-09-15T03:47:49.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154943+0000", 2024-09-15T03:47:49.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010347400000000001 2024-09-15T03:47:49.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154955+0000", 2024-09-15T03:47:49.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1935e-05 2024-09-15T03:47:49.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156442+0000", 2024-09-15T03:47:49.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014867859999999999 2024-09-15T03:47:49.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156466+0000", 2024-09-15T03:47:49.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4077999999999999e-05 2024-09-15T03:47:49.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142391+0000", 2024-09-15T03:47:49.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98592533599999999 2024-09-15T03:47:49.810 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.810 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.810 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.810 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=48, reqid=client.4521.0:242, at_version=0'0, trim_to=0'0, roll_forward_to=39'18))", 2024-09-15T03:47:49.810 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:47:49.810 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.529397852999999, 2024-09-15T03:47:49.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98592495099999999, 2024-09-15T03:47:49.810 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:49.810 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:49.811 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:49.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:49.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:47:49.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:49.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:47:49.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:49.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:49.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156616+0000", 2024-09-15T03:47:49.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1999999999999998e-07 2024-09-15T03:47:49.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:49.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156629+0000", 2024-09-15T03:47:49.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2870999999999999e-05 2024-09-15T03:47:49.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:49.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156649+0000", 2024-09-15T03:47:49.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9786e-05 2024-09-15T03:47:49.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:49.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156657+0000", 2024-09-15T03:47:49.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5110000000000003e-06 2024-09-15T03:47:49.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:49.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156723+0000", 2024-09-15T03:47:49.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6209000000000003e-05 2024-09-15T03:47:49.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:49.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156734+0000", 2024-09-15T03:47:49.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0475999999999999e-05 2024-09-15T03:47:49.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:49.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156868+0000", 2024-09-15T03:47:49.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013402999999999999 2024-09-15T03:47:49.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:49.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156889+0000", 2024-09-15T03:47:49.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1146e-05 2024-09-15T03:47:49.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:49.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156937+0000", 2024-09-15T03:47:49.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8031999999999997e-05 2024-09-15T03:47:49.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:49.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156970+0000", 2024-09-15T03:47:49.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2462999999999997e-05 2024-09-15T03:47:49.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:49.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:49.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:49.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142540+0000", 2024-09-15T03:47:49.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98557050700000004 2024-09-15T03:47:49.816 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.816 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.816 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.816 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:49.817 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:49.817 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:49.818 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:49.687+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:49.818 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:47:49.819 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:49.818+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:49.881 INFO:tasks.rados.rados.0.smithi001.stdout:350: done (1 left) 2024-09-15T03:47:49.881 INFO:tasks.rados.rados.0.smithi001.stdout:351: done (0 left) 2024-09-15T03:47:49.881 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 29 current snap is 45 2024-09-15T03:47:49.881 INFO:tasks.rados.rados.0.smithi001.stdout:352: seq_num 134 ranges {376832=704512,1081344=475136,1556480=573440,2129920=507904} 2024-09-15T03:47:49.884 INFO:tasks.rados.rados.0.smithi001.stdout:352: writing smithi00132172-29 from 376832 to 1081344 tid 1 2024-09-15T03:47:49.887 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T03:47:49.888 INFO:tasks.rados.rados.0.smithi001.stdout:352: writing smithi00132172-29 from 1081344 to 1556480 tid 2 2024-09-15T03:47:49.891 INFO:tasks.rados.rados.0.smithi001.stdout:352: writing smithi00132172-29 from 1556480 to 2129920 tid 3 2024-09-15T03:47:49.894 INFO:tasks.rados.rados.0.smithi001.stdout:352: writing smithi00132172-29 from 2129920 to 2637824 tid 4 2024-09-15T03:47:49.895 INFO:tasks.rados.rados.0.smithi001.stdout:353: snap_create 2024-09-15T03:47:49.895 INFO:tasks.rados.rados.0.smithi001.stdout:352: finishing write tid 1 to smithi00132172-29 2024-09-15T03:47:49.895 INFO:tasks.rados.rados.0.smithi001.stdout:352: finishing write tid 2 to smithi00132172-29 2024-09-15T03:47:49.895 INFO:tasks.rados.rados.0.smithi001.stdout:352: finishing write tid 3 to smithi00132172-29 2024-09-15T03:47:49.898 INFO:tasks.rados.rados.0.smithi001.stdout:352: finishing write tid 4 to smithi00132172-29 2024-09-15T03:47:49.898 INFO:tasks.rados.rados.0.smithi001.stdout:352: finishing write tid 5 to smithi00132172-29 2024-09-15T03:47:49.899 INFO:tasks.rados.rados.0.smithi001.stdout:352: finishing write tid 6 to smithi00132172-29 2024-09-15T03:47:49.899 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 29 v 124 (ObjNum 134 snap 45 seq_num 134) dirty exists 2024-09-15T03:47:49.899 INFO:tasks.rados.rados.0.smithi001.stdout:352: left oid 29 (ObjNum 134 snap 45 seq_num 134) 2024-09-15T03:47:49.919 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:49.918+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:49.926 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:49.926 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:49.926 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:49.926 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:49.934 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:47:49.984 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [10] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] 2024-09-15T03:47:49.984 INFO:tasks.thrashosds.thrasher:Reviving osd 10 2024-09-15T03:47:49.984 INFO:tasks.ceph.osd.10:Restarting daemon 2024-09-15T03:47:49.984 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-09-15T03:47:49.987 INFO:tasks.ceph.osd.10:Started 2024-09-15T03:47:49.988 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:47:50.019 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:50.018+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:50.035 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:50.035 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:50.036 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:50.036 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:50.036 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:50.044 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:47:50.066 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:50.065+0000 7fd62ec09440 -1 Falling back to public interface 2024-09-15T03:47:50.079 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:50.078+0000 7fd62ec09440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:47:50.079 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:50.079+0000 7fd62ec09440 -1 journal do_read_entry(33468416): bad header magic 2024-09-15T03:47:50.080 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:50.079+0000 7fd62ec09440 -1 journal do_read_entry(33468416): bad header magic 2024-09-15T03:47:50.093 INFO:teuthology.orchestra.run.smithi190.stderr:no valid command found; 10 closest matches: 2024-09-15T03:47:50.093 INFO:teuthology.orchestra.run.smithi190.stderr:0 2024-09-15T03:47:50.093 INFO:teuthology.orchestra.run.smithi190.stderr:1 2024-09-15T03:47:50.093 INFO:teuthology.orchestra.run.smithi190.stderr:2 2024-09-15T03:47:50.094 INFO:teuthology.orchestra.run.smithi190.stderr:abort 2024-09-15T03:47:50.094 INFO:teuthology.orchestra.run.smithi190.stderr:assert 2024-09-15T03:47:50.094 INFO:teuthology.orchestra.run.smithi190.stderr:config diff 2024-09-15T03:47:50.094 INFO:teuthology.orchestra.run.smithi190.stderr:config diff get 2024-09-15T03:47:50.094 INFO:teuthology.orchestra.run.smithi190.stderr:config get 2024-09-15T03:47:50.094 INFO:teuthology.orchestra.run.smithi190.stderr:config help [] 2024-09-15T03:47:50.094 INFO:teuthology.orchestra.run.smithi190.stderr:config set ... 2024-09-15T03:47:50.094 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: invalid command 2024-09-15T03:47:50.096 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-15T03:47:50.096 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-10, ['dump_ops_in_flight'] 2024-09-15T03:47:50.117 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:50.116+0000 7fd62ec09440 -1 osd.10 78 log_to_monitors true 2024-09-15T03:47:50.121 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:50.120+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:50.187 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=21, reqid=client.4521.0:184, at_version=33'6, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.079082237999998, 2024-09-15T03:47:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010901256999999999, 2024-09-15T03:47:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0200000000000001e-07 2024-09-15T03:47:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106529+0000", 2024-09-15T03:47:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9539e-05 2024-09-15T03:47:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106542+0000", 2024-09-15T03:47:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2731999999999999e-05 2024-09-15T03:47:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106549+0000", 2024-09-15T03:47:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3339999999999996e-06 2024-09-15T03:47:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106613+0000", 2024-09-15T03:47:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4527000000000001e-05 2024-09-15T03:47:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106622+0000", 2024-09-15T03:47:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.755e-06 2024-09-15T03:47:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106841+0000", 2024-09-15T03:47:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021906200000000001 2024-09-15T03:47:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106869+0000", 2024-09-15T03:47:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.739e-05 2024-09-15T03:47:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106948+0000", 2024-09-15T03:47:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9422e-05 2024-09-15T03:47:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106965+0000", 2024-09-15T03:47:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7010000000000001e-05 2024-09-15T03:47:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.117410+0000", 2024-09-15T03:47:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010445284000000001 2024-09-15T03:47:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.077168751000002, 2024-09-15T03:47:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015296944, 2024-09-15T03:47:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5499999999999998e-07 2024-09-15T03:47:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108447+0000", 2024-09-15T03:47:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3314000000000001e-05 2024-09-15T03:47:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108460+0000", 2024-09-15T03:47:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3444e-05 2024-09-15T03:47:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108464+0000", 2024-09-15T03:47:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3610000000000003e-06 2024-09-15T03:47:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108494+0000", 2024-09-15T03:47:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9688000000000001e-05 2024-09-15T03:47:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108502+0000", 2024-09-15T03:47:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.678e-06 2024-09-15T03:47:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108601+0000", 2024-09-15T03:47:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9748999999999997e-05 2024-09-15T03:47:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108610+0000", 2024-09-15T03:47:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2490000000000001e-06 2024-09-15T03:47:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109977+0000", 2024-09-15T03:47:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013669450000000001 2024-09-15T03:47:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109993+0000", 2024-09-15T03:47:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6688e-05 2024-09-15T03:47:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123720+0000", 2024-09-15T03:47:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013726172999999999 2024-09-15T03:47:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.077119089999997, 2024-09-15T03:47:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015105210000000001, 2024-09-15T03:47:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108473+0000", 2024-09-15T03:47:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.84e-07 2024-09-15T03:47:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108486+0000", 2024-09-15T03:47:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3845999999999999e-05 2024-09-15T03:47:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108491+0000", 2024-09-15T03:47:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4719999999999997e-06 2024-09-15T03:47:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108495+0000", 2024-09-15T03:47:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.822e-06 2024-09-15T03:47:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108546+0000", 2024-09-15T03:47:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0853000000000003e-05 2024-09-15T03:47:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108555+0000", 2024-09-15T03:47:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3319999999999998e-06 2024-09-15T03:47:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108638+0000", 2024-09-15T03:47:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3580999999999998e-05 2024-09-15T03:47:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109983+0000", 2024-09-15T03:47:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001344392 2024-09-15T03:47:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110023+0000", 2024-09-15T03:47:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9895999999999999e-05 2024-09-15T03:47:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110040+0000", 2024-09-15T03:47:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.747e-05 2024-09-15T03:47:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123577+0000", 2024-09-15T03:47:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013537262 2024-09-15T03:47:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.077085210999996, 2024-09-15T03:47:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015311313999999999, 2024-09-15T03:47:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8500000000000002e-07 2024-09-15T03:47:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108517+0000", 2024-09-15T03:47:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0543000000000001e-05 2024-09-15T03:47:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108524+0000", 2024-09-15T03:47:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7170000000000002e-06 2024-09-15T03:47:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108528+0000", 2024-09-15T03:47:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3970000000000004e-06 2024-09-15T03:47:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108632+0000", 2024-09-15T03:47:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103687 2024-09-15T03:47:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108637+0000", 2024-09-15T03:47:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4389999999999999e-06 2024-09-15T03:47:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108713+0000", 2024-09-15T03:47:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6223000000000005e-05 2024-09-15T03:47:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109984+0000", 2024-09-15T03:47:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012700719999999999 2024-09-15T03:47:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110023+0000", 2024-09-15T03:47:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9864999999999999e-05 2024-09-15T03:47:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110041+0000", 2024-09-15T03:47:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8040999999999999e-05 2024-09-15T03:47:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123817+0000", 2024-09-15T03:47:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013776045000000001 2024-09-15T03:47:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.076591602999997, 2024-09-15T03:47:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014864502, 2024-09-15T03:47:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:47:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109016+0000", 2024-09-15T03:47:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5889999999999999e-05 2024-09-15T03:47:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109029+0000", 2024-09-15T03:47:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2812e-05 2024-09-15T03:47:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109035+0000", 2024-09-15T03:47:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0109999999999997e-06 2024-09-15T03:47:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109073+0000", 2024-09-15T03:47:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8260000000000003e-05 2024-09-15T03:47:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109080+0000", 2024-09-15T03:47:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0190000000000001e-06 2024-09-15T03:47:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109165+0000", 2024-09-15T03:47:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4957000000000007e-05 2024-09-15T03:47:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109984+0000", 2024-09-15T03:47:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081866399999999998 2024-09-15T03:47:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110024+0000", 2024-09-15T03:47:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9644999999999997e-05 2024-09-15T03:47:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110072+0000", 2024-09-15T03:47:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8418999999999998e-05 2024-09-15T03:47:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123864+0000", 2024-09-15T03:47:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013792234 2024-09-15T03:47:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=34, reqid=client.4521.0:197, at_version=34'21, trim_to=0'0, roll_forward_to=34'18))", 2024-09-15T03:47:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.061579722999994, 2024-09-15T03:47:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012992399, 2024-09-15T03:47:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124013+0000", 2024-09-15T03:47:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6299999999999993e-07 2024-09-15T03:47:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124044+0000", 2024-09-15T03:47:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0995999999999999e-05 2024-09-15T03:47:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124062+0000", 2024-09-15T03:47:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8413000000000001e-05 2024-09-15T03:47:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124069+0000", 2024-09-15T03:47:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3930000000000002e-06 2024-09-15T03:47:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124150+0000", 2024-09-15T03:47:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1097e-05 2024-09-15T03:47:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124161+0000", 2024-09-15T03:47:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0720000000000001e-05 2024-09-15T03:47:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124402+0000", 2024-09-15T03:47:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000240442 2024-09-15T03:47:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125141+0000", 2024-09-15T03:47:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00073915500000000002 2024-09-15T03:47:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125191+0000", 2024-09-15T03:47:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9951999999999999e-05 2024-09-15T03:47:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125251+0000", 2024-09-15T03:47:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9938000000000001e-05 2024-09-15T03:47:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.137004+0000", 2024-09-15T03:47:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011753329999999999 2024-09-15T03:47:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=35, reqid=client.4521.0:197, at_version=0'0, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:47:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.059639679, 2024-09-15T03:47:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011202456, 2024-09-15T03:47:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125953+0000", 2024-09-15T03:47:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9100000000000002e-07 2024-09-15T03:47:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125964+0000", 2024-09-15T03:47:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1399999999999999e-05 2024-09-15T03:47:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125979+0000", 2024-09-15T03:47:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4810000000000001e-05 2024-09-15T03:47:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126004+0000", 2024-09-15T03:47:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4913e-05 2024-09-15T03:47:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126059+0000", 2024-09-15T03:47:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5668999999999998e-05 2024-09-15T03:47:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126071+0000", 2024-09-15T03:47:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1219999999999999e-05 2024-09-15T03:47:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126151+0000", 2024-09-15T03:47:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0572e-05 2024-09-15T03:47:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126163+0000", 2024-09-15T03:47:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2101e-05 2024-09-15T03:47:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.127945+0000", 2024-09-15T03:47:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001781803 2024-09-15T03:47:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.127982+0000", 2024-09-15T03:47:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7129999999999999e-05 2024-09-15T03:47:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.137154+0000", 2024-09-15T03:47:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0091719469999999997 2024-09-15T03:47:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:243 3.3s0 3:c7c3ef16:::smithi00132172-25:head [create,setxattr iZnWfebab`ptjURR\\f`psW]tpS`qpUaeWkY[Oqep (1330) in=40b,setxattr f[mZ_qQmcZejQ`fbdeic_tkPUaTYcnm[WprjVmc] (12488) in=40b,setxattr tkZgQ]S_pdlhZnqrVaZRa`oQQr_USmUfhado^dXT (1121) in=40b,setxattr srWjTqXhcth`hfZmQY_fg]UqNjWofa^dfgOV`sYP (455) in=40b,setxattr ^_mdiRroj_TtQ^gcNj`XtkW[ooqY`PWXqYmcOk_n (138) in=40b,setxattr jQlajdtcalhkcfYaakZfNb[^_TjkfcS`Vrk]q_d[ (806) in=40b,setxattr ZfYgkVcrsXcnpchgfcbldNrUTasnQNmfZW]Nbh\\o (561) in=40b,setxattr _hkYOqjPNhmhUlSNktfeQpsqSWjO\\fbf[fpTlZiO (1227) in=40b,setxattr ]mdV^Xdc\\sViXdkXNU[js_Qhh[b[sbfnZVXkdids (11788) in=40b,setxattr tb^mbOrNhfrhtce^g (455) in=17b,setxattr koTkVgrfU\\m`StnZ`Rirjhbhg`PhVqm^a^Xgaaqn (35) in=40b,setxattr aUtZPUPXPnQmT]\\\\n\\lZpnXakdNkTldVQoQhPUf] (559) in=40b,setxattr mrhR]^U`Tteo[Topmql`SNma\\pqfk`if]OppnYt` (770) in=40b,setxattr Oetk]RrnWeXbVctVtVrhXf_RfTdelqX^lWp[sTkp (1226) in=40b,setxattr WtedXoOd`VqnbksOqQkatk`gdTnW^lXbUtrlYgkl (3576) in=40b,setxattr dddOWN\\j\\XXg^fr[rZcskeVX]jOQ^ZmoWepk[]\\a (17824) in=40b,setxattr TeRO^djPgNZiemQ_aRXgP[]jgUZa\\^\\iUgesaqaN (208) in=40b,setxattr teqm]mUobZbWjcdQ_jTa[f[k[Sd\\lh]n\\osigqkN (7436) in=40b,setxattr dphaol[XkeitbkomWYaQctWfataUhbSiNntkWo`P (1893) in=40b,setxattr oQsnhO]ZklSVYQ[WUpaelY[p[Qie^UepOt`Z[q (455) in=38b] snapc 7=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:47:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.016675790999997, 2024-09-15T03:47:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012365956000000001, 2024-09-15T03:47:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 243 2024-09-15T03:47:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168917+0000", 2024-09-15T03:47:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.725e-06 2024-09-15T03:47:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169030+0000", 2024-09-15T03:47:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113035 2024-09-15T03:47:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169047+0000", 2024-09-15T03:47:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6701000000000002e-05 2024-09-15T03:47:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169054+0000", 2024-09-15T03:47:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1369999999999996e-06 2024-09-15T03:47:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169140+0000", 2024-09-15T03:47:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6069999999999994e-05 2024-09-15T03:47:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169270+0000", 2024-09-15T03:47:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130071 2024-09-15T03:47:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.170661+0000", 2024-09-15T03:47:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001390666 2024-09-15T03:47:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171751+0000", 2024-09-15T03:47:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010899079999999999 2024-09-15T03:47:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171812+0000", 2024-09-15T03:47:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0797000000000001e-05 2024-09-15T03:47:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172279+0000", 2024-09-15T03:47:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046704500000000003 2024-09-15T03:47:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172299+0000", 2024-09-15T03:47:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0387999999999999e-05 2024-09-15T03:47:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.176110+0000", 2024-09-15T03:47:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038112340000000001 2024-09-15T03:47:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.181282+0000", 2024-09-15T03:47:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051711789999999997 2024-09-15T03:47:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/28 ECSubWrite(tid=15, reqid=client.4521.0:337, at_version=49'10, trim_to=0'0, roll_forward_to=43'8))", 2024-09-15T03:47:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.327685881999997, 2024-09-15T03:47:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010500893000000001, 2024-09-15T03:47:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857906+0000", 2024-09-15T03:47:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.63e-07 2024-09-15T03:47:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857928+0000", 2024-09-15T03:47:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2155999999999999e-05 2024-09-15T03:47:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857937+0000", 2024-09-15T03:47:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2359999999999993e-06 2024-09-15T03:47:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857941+0000", 2024-09-15T03:47:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0960000000000003e-06 2024-09-15T03:47:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857999+0000", 2024-09-15T03:47:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7407000000000001e-05 2024-09-15T03:47:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858011+0000", 2024-09-15T03:47:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1759999999999999e-05 2024-09-15T03:47:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858240+0000", 2024-09-15T03:47:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022949299999999999 2024-09-15T03:47:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858251+0000", 2024-09-15T03:47:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1168999999999999e-05 2024-09-15T03:47:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858332+0000", 2024-09-15T03:47:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1204000000000004e-05 2024-09-15T03:47:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858360+0000", 2024-09-15T03:47:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7767999999999999e-05 2024-09-15T03:47:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.868406+0000", 2024-09-15T03:47:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010046141 2024-09-15T03:47:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:349 3.3s0 3:c7c3ef16:::smithi00132172-25:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.347109856000003, 2024-09-15T03:47:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005145406, 2024-09-15T03:47:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 349 2024-09-15T03:47:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838485+0000", 2024-09-15T03:47:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0620000000000001e-06 2024-09-15T03:47:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838494+0000", 2024-09-15T03:47:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3179999999999996e-06 2024-09-15T03:47:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3689999999999998e-05 2024-09-15T03:47:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838812+0000", 2024-09-15T03:47:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029437300000000001 2024-09-15T03:47:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838905+0000", 2024-09-15T03:47:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3196999999999996e-05 2024-09-15T03:47:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838914+0000", 2024-09-15T03:47:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0890000000000006e-06 2024-09-15T03:47:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840773+0000", 2024-09-15T03:47:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018584820000000001 2024-09-15T03:47:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840889+0000", 2024-09-15T03:47:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115986 2024-09-15T03:47:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842016+0000", 2024-09-15T03:47:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001126878 2024-09-15T03:47:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842320+0000", 2024-09-15T03:47:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030409799999999999 2024-09-15T03:47:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842360+0000", 2024-09-15T03:47:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0750000000000001e-05 2024-09-15T03:47:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842448+0000", 2024-09-15T03:47:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7842000000000004e-05 2024-09-15T03:47:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844580+0000", 2024-09-15T03:47:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021313719999999999 2024-09-15T03:47:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843574+0000", 2024-09-15T03:47:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99899468199999997 2024-09-15T03:47:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843627+0000", 2024-09-15T03:47:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2587000000000003e-05 2024-09-15T03:47:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:350 3.3s0 3:c7c3ef16:::smithi00132172-25:head [rollback 14] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.346758195, 2024-09-15T03:47:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0050383839999999, 2024-09-15T03:47:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 350 2024-09-15T03:47:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838834+0000", 2024-09-15T03:47:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0200000000000001e-07 2024-09-15T03:47:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838839+0000", 2024-09-15T03:47:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9790000000000001e-06 2024-09-15T03:47:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838852+0000", 2024-09-15T03:47:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3138e-05 2024-09-15T03:47:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838858+0000", 2024-09-15T03:47:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8320000000000002e-06 2024-09-15T03:47:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839003+0000", 2024-09-15T03:47:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014513500000000001 2024-09-15T03:47:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839011+0000", 2024-09-15T03:47:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7719999999999994e-06 2024-09-15T03:47:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842395+0000", 2024-09-15T03:47:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033840110000000001 2024-09-15T03:47:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842580+0000", 2024-09-15T03:47:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000185067 2024-09-15T03:47:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843394+0000", 2024-09-15T03:47:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081377299999999995 2024-09-15T03:47:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843510+0000", 2024-09-15T03:47:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0001165 2024-09-15T03:47:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843524+0000", 2024-09-15T03:47:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4022999999999999e-05 2024-09-15T03:47:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843579+0000", 2024-09-15T03:47:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4654999999999998e-05 2024-09-15T03:47:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844583+0000", 2024-09-15T03:47:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010042429999999999 2024-09-15T03:47:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843772+0000", 2024-09-15T03:47:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99918911200000005 2024-09-15T03:47:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843872+0000", 2024-09-15T03:47:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9341999999999999e-05 2024-09-15T03:47:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:351 3.3s0 3:c7c3ef16:::smithi00132172-25:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.346700562000002, 2024-09-15T03:47:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0051167679999999, 2024-09-15T03:47:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 351 2024-09-15T03:47:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0800000000000004e-07 2024-09-15T03:47:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838895+0000", 2024-09-15T03:47:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.241e-06 2024-09-15T03:47:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838904+0000", 2024-09-15T03:47:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9549999999999998e-06 2024-09-15T03:47:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838909+0000", 2024-09-15T03:47:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7869999999999998e-06 2024-09-15T03:47:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839045+0000", 2024-09-15T03:47:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013556399999999999 2024-09-15T03:47:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839052+0000", 2024-09-15T03:47:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8199999999999999e-06 2024-09-15T03:47:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843587+0000", 2024-09-15T03:47:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045349739999999998 2024-09-15T03:47:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843708+0000", 2024-09-15T03:47:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120776 2024-09-15T03:47:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844246+0000", 2024-09-15T03:47:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053838200000000001 2024-09-15T03:47:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844335+0000", 2024-09-15T03:47:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9240999999999995e-05 2024-09-15T03:47:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844347+0000", 2024-09-15T03:47:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2187e-05 2024-09-15T03:47:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844460+0000", 2024-09-15T03:47:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112571 2024-09-15T03:47:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844584+0000", 2024-09-15T03:47:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012403800000000001 2024-09-15T03:47:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843972+0000", 2024-09-15T03:47:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99938787299999998 2024-09-15T03:47:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844008+0000", 2024-09-15T03:47:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5750999999999997e-05 2024-09-15T03:47:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.339566611000002, 2024-09-15T03:47:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013010173, 2024-09-15T03:47:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846026+0000", 2024-09-15T03:47:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1230000000000001e-06 2024-09-15T03:47:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846047+0000", 2024-09-15T03:47:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0774000000000001e-05 2024-09-15T03:47:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846064+0000", 2024-09-15T03:47:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7560000000000001e-05 2024-09-15T03:47:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846072+0000", 2024-09-15T03:47:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5020000000000003e-06 2024-09-15T03:47:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846187+0000", 2024-09-15T03:47:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115717 2024-09-15T03:47:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846201+0000", 2024-09-15T03:47:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3329999999999999e-05 2024-09-15T03:47:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846303+0000", 2024-09-15T03:47:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101807 2024-09-15T03:47:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846312+0000", 2024-09-15T03:47:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5529999999999998e-06 2024-09-15T03:47:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846382+0000", 2024-09-15T03:47:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0146000000000002e-05 2024-09-15T03:47:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846520+0000", 2024-09-15T03:47:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013814099999999999 2024-09-15T03:47:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.859035+0000", 2024-09-15T03:47:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012514519999999999 2024-09-15T03:47:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.339469864000002, 2024-09-15T03:47:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013044694000000001, 2024-09-15T03:47:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846122+0000", 2024-09-15T03:47:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0399999999999996e-07 2024-09-15T03:47:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846174+0000", 2024-09-15T03:47:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1672999999999998e-05 2024-09-15T03:47:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846184+0000", 2024-09-15T03:47:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0514e-05 2024-09-15T03:47:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846190+0000", 2024-09-15T03:47:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2820000000000002e-06 2024-09-15T03:47:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846341+0000", 2024-09-15T03:47:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150803 2024-09-15T03:47:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846351+0000", 2024-09-15T03:47:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3899999999999999e-06 2024-09-15T03:47:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846450+0000", 2024-09-15T03:47:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9839000000000004e-05 2024-09-15T03:47:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846482+0000", 2024-09-15T03:47:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1089e-05 2024-09-15T03:47:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846615+0000", 2024-09-15T03:47:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013305899999999999 2024-09-15T03:47:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846634+0000", 2024-09-15T03:47:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8899999999999999e-05 2024-09-15T03:47:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.859166+0000", 2024-09-15T03:47:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012532641000000001 2024-09-15T03:47:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 60/28 ECSubWrite(tid=73, reqid=client.4521.0:424, at_version=60'41, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.285161017, 2024-09-15T03:47:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011124821, 2024-09-15T03:47:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900431+0000", 2024-09-15T03:47:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2900000000000003e-07 2024-09-15T03:47:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900451+0000", 2024-09-15T03:47:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0066999999999999e-05 2024-09-15T03:47:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900466+0000", 2024-09-15T03:47:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5062000000000001e-05 2024-09-15T03:47:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900474+0000", 2024-09-15T03:47:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6310000000000003e-06 2024-09-15T03:47:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900524+0000", 2024-09-15T03:47:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0037000000000002e-05 2024-09-15T03:47:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900535+0000", 2024-09-15T03:47:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1209000000000001e-05 2024-09-15T03:47:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900659+0000", 2024-09-15T03:47:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012441000000000001 2024-09-15T03:47:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900971+0000", 2024-09-15T03:47:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031183799999999999 2024-09-15T03:47:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901144+0000", 2024-09-15T03:47:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172261 2024-09-15T03:47:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901734+0000", 2024-09-15T03:47:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059069700000000005 2024-09-15T03:47:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.911555+0000", 2024-09-15T03:47:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0098208800000000006 2024-09-15T03:47:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPush(2.2 88/87 [PushOp(2:5cea7035:::rbd_info:head, version: 24'2, data_included: [0~19], data_size: 19, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:5cea7035:::rbd_info:head@24'2, size: 19, copy_subset: [0~19], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:19, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-15T03:47:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.846903+0000", 2024-09-15T03:47:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.338687890999999, 2024-09-15T03:47:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013354107, 2024-09-15T03:47:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846903+0000", 2024-09-15T03:47:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846903+0000", 2024-09-15T03:47:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846904+0000", 2024-09-15T03:47:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4099999999999998e-07 2024-09-15T03:47:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846920+0000", 2024-09-15T03:47:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6061000000000001e-05 2024-09-15T03:47:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846945+0000", 2024-09-15T03:47:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4519e-05 2024-09-15T03:47:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846958+0000", 2024-09-15T03:47:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3546999999999999e-05 2024-09-15T03:47:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.847077+0000", 2024-09-15T03:47:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011864 2024-09-15T03:47:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for flush", 2024-09-15T03:47:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.847094+0000", 2024-09-15T03:47:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7017e-05 2024-09-15T03:47:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.859991+0000", 2024-09-15T03:47:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012897136 2024-09-15T03:47:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.860001+0000", 2024-09-15T03:47:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5089999999999999e-06 2024-09-15T03:47:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.860258+0000", 2024-09-15T03:47:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000256937 2024-09-15T03:47:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 88/87 ECSubWrite(tid=210, reqid=client.4521.0:740, at_version=88'113, trim_to=0'0, roll_forward_to=88'111))", 2024-09-15T03:47:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.915188+0000", 2024-09-15T03:47:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.270403765999999, 2024-09-15T03:47:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012691618999999999, 2024-09-15T03:47:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:50.286 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915188+0000", 2024-09-15T03:47:50.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915188+0000", 2024-09-15T03:47:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915188+0000", 2024-09-15T03:47:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3399999999999999e-07 2024-09-15T03:47:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915216+0000", 2024-09-15T03:47:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8067000000000001e-05 2024-09-15T03:47:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915233+0000", 2024-09-15T03:47:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.683e-05 2024-09-15T03:47:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915239+0000", 2024-09-15T03:47:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5720000000000002e-06 2024-09-15T03:47:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915313+0000", 2024-09-15T03:47:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4066000000000005e-05 2024-09-15T03:47:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915325+0000", 2024-09-15T03:47:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.252e-05 2024-09-15T03:47:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915603+0000", 2024-09-15T03:47:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027800699999999998 2024-09-15T03:47:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915645+0000", 2024-09-15T03:47:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1777000000000003e-05 2024-09-15T03:47:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915687+0000", 2024-09-15T03:47:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2181000000000002e-05 2024-09-15T03:47:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.917623+0000", 2024-09-15T03:47:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019357210000000001 2024-09-15T03:47:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.927879+0000", 2024-09-15T03:47:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010256243999999999 2024-09-15T03:47:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:866 3.5s1 3:af1dd10e:::smithi00132172-27:head [append 0~0] snapc 3e=[3e,3d,3b,3a,38,37,35,34,2d,2b,25,24,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-15T03:47:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:45.860999+0000", 2024-09-15T03:47:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.3245919639999997, 2024-09-15T03:47:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016187865999999999, 2024-09-15T03:47:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 866 2024-09-15T03:47:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.860999+0000", 2024-09-15T03:47:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.860999+0000", 2024-09-15T03:47:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861001+0000", 2024-09-15T03:47:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5519999999999999e-06 2024-09-15T03:47:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861008+0000", 2024-09-15T03:47:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2710000000000003e-06 2024-09-15T03:47:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861027+0000", 2024-09-15T03:47:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8416000000000001e-05 2024-09-15T03:47:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861036+0000", 2024-09-15T03:47:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8899999999999996e-06 2024-09-15T03:47:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861092+0000", 2024-09-15T03:47:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6746000000000002e-05 2024-09-15T03:47:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861102+0000", 2024-09-15T03:47:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8130000000000007e-06 2024-09-15T03:47:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861811+0000", 2024-09-15T03:47:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070912899999999997 2024-09-15T03:47:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861936+0000", 2024-09-15T03:47:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012470999999999999 2024-09-15T03:47:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.862842+0000", 2024-09-15T03:47:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00090626499999999996 2024-09-15T03:47:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.863195+0000", 2024-09-15T03:47:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035236600000000001 2024-09-15T03:47:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.863208+0000", 2024-09-15T03:47:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3848e-05 2024-09-15T03:47:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.863243+0000", 2024-09-15T03:47:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4307000000000002e-05 2024-09-15T03:47:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.866214+0000", 2024-09-15T03:47:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002970838 2024-09-15T03:47:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.866515+0000", 2024-09-15T03:47:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000301682 2024-09-15T03:47:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.877187+0000", 2024-09-15T03:47:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010672032999999999 2024-09-15T03:47:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:867 3.5s1 3:af1dd10e:::smithi00132172-27:head [rollback 2d] snapc 3e=[3e,3d,3b,3a,38,37,35,34,2d,2b,25,24,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-15T03:47:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:45.861150+0000", 2024-09-15T03:47:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.3244411859999996, 2024-09-15T03:47:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018882498000000001, 2024-09-15T03:47:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 867 2024-09-15T03:47:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861150+0000", 2024-09-15T03:47:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861150+0000", 2024-09-15T03:47:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861151+0000", 2024-09-15T03:47:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.92e-07 2024-09-15T03:47:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861155+0000", 2024-09-15T03:47:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7189999999999999e-06 2024-09-15T03:47:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861163+0000", 2024-09-15T03:47:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8769999999999999e-06 2024-09-15T03:47:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861169+0000", 2024-09-15T03:47:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0579999999999997e-06 2024-09-15T03:47:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861213+0000", 2024-09-15T03:47:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4249000000000003e-05 2024-09-15T03:47:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861221+0000", 2024-09-15T03:47:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4950000000000008e-06 2024-09-15T03:47:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.863319+0000", 2024-09-15T03:47:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020973709999999998 2024-09-15T03:47:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.863432+0000", 2024-09-15T03:47:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113465 2024-09-15T03:47:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865365+0000", 2024-09-15T03:47:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001932546 2024-09-15T03:47:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865488+0000", 2024-09-15T03:47:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122929 2024-09-15T03:47:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865517+0000", 2024-09-15T03:47:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9091999999999999e-05 2024-09-15T03:47:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.870930+0000", 2024-09-15T03:47:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054136760000000001 2024-09-15T03:47:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.870964+0000", 2024-09-15T03:47:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3377999999999998e-05 2024-09-15T03:47:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.871243+0000", 2024-09-15T03:47:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027943199999999998 2024-09-15T03:47:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.880033+0000", 2024-09-15T03:47:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087895190000000008 2024-09-15T03:47:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:868 3.5s1 3:af1dd10e:::smithi00132172-27:head [append 0~0] snapc 3e=[3e,3d,3b,3a,38,37,35,34,2d,2b,25,24,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-15T03:47:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:45.861231+0000", 2024-09-15T03:47:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.3243601969999999, 2024-09-15T03:47:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018901188999999999, 2024-09-15T03:47:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 868 2024-09-15T03:47:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861231+0000", 2024-09-15T03:47:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861231+0000", 2024-09-15T03:47:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861232+0000", 2024-09-15T03:47:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3799999999999997e-07 2024-09-15T03:47:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861234+0000", 2024-09-15T03:47:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7640000000000001e-06 2024-09-15T03:47:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861242+0000", 2024-09-15T03:47:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6830000000000008e-06 2024-09-15T03:47:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861247+0000", 2024-09-15T03:47:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3719999999999998e-06 2024-09-15T03:47:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865555+0000", 2024-09-15T03:47:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004308976 2024-09-15T03:47:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865641+0000", 2024-09-15T03:47:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5801000000000001e-05 2024-09-15T03:47:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.866070+0000", 2024-09-15T03:47:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042870300000000001 2024-09-15T03:47:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.866174+0000", 2024-09-15T03:47:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104422 2024-09-15T03:47:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.870933+0000", 2024-09-15T03:47:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047587000000000003 2024-09-15T03:47:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.871013+0000", 2024-09-15T03:47:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0059e-05 2024-09-15T03:47:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.871046+0000", 2024-09-15T03:47:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3281999999999997e-05 2024-09-15T03:47:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.871558+0000", 2024-09-15T03:47:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051141000000000001 2024-09-15T03:47:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.880132+0000", 2024-09-15T03:47:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085744789999999994 2024-09-15T03:47:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.315 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.315 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:50.316 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:47:50.317 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:50.315+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:50.417 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:50.415+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:50.418 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:50.418 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:50.418 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:50.418 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:50.426 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:47:50.518 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:50.516+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:50.567 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:50.567 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:50.567 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:50.568 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:50.568 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:50.575 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:47:50.619 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:50.618+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:50.719 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:50.719 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:50.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:50.719 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:50.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.719 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:197 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [delete] snapc 3=[3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e34)", 2024-09-15T03:47:50.719 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:50.719 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.604513893999993, 2024-09-15T03:47:50.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021934704999999999, 2024-09-15T03:47:50.719 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.719 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:50.720 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.720 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.720 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.720 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197 2024-09-15T03:47:50.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.720 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:50.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:50.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112266+0000", 2024-09-15T03:47:50.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5000000000000002e-07 2024-09-15T03:47:50.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112269+0000", 2024-09-15T03:47:50.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.066e-06 2024-09-15T03:47:50.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112276+0000", 2024-09-15T03:47:50.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-09-15T03:47:50.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112281+0000", 2024-09-15T03:47:50.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3970000000000004e-06 2024-09-15T03:47:50.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123281+0000", 2024-09-15T03:47:50.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011000575 2024-09-15T03:47:50.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123395+0000", 2024-09-15T03:47:50.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113398 2024-09-15T03:47:50.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123963+0000", 2024-09-15T03:47:50.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000568101 2024-09-15T03:47:50.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124220+0000", 2024-09-15T03:47:50.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025704900000000003 2024-09-15T03:47:50.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124229+0000", 2024-09-15T03:47:50.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4860000000000003e-06 2024-09-15T03:47:50.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124277+0000", 2024-09-15T03:47:50.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7574000000000002e-05 2024-09-15T03:47:50.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124312+0000", 2024-09-15T03:47:50.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5256000000000003e-05 2024-09-15T03:47:50.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:50.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125716+0000", 2024-09-15T03:47:50.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014043720000000001 2024-09-15T03:47:50.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.134200+0000", 2024-09-15T03:47:50.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084835120000000003 2024-09-15T03:47:50.726 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.726 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.726 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.726 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:220 3.bs0 3:d99ddedc:::smithi00132172-44:head [copy-from ver 22] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:50.726 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:50.726 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 65.589134224000006, 2024-09-15T03:47:50.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024273429999999999, 2024-09-15T03:47:50.727 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.727 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:50.727 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.727 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.727 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.727 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 220 2024-09-15T03:47:50.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.727 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:50.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:50.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127647+0000", 2024-09-15T03:47:50.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1799999999999999e-06 2024-09-15T03:47:50.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127656+0000", 2024-09-15T03:47:50.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3759999999999997e-06 2024-09-15T03:47:50.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127674+0000", 2024-09-15T03:47:50.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7374e-05 2024-09-15T03:47:50.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127680+0000", 2024-09-15T03:47:50.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6949999999999997e-06 2024-09-15T03:47:50.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127736+0000", 2024-09-15T03:47:50.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6011999999999998e-05 2024-09-15T03:47:50.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:50.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127744+0000", 2024-09-15T03:47:50.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2870000000000004e-06 2024-09-15T03:47:50.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130338+0000", 2024-09-15T03:47:50.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025940249999999998 2024-09-15T03:47:50.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130423+0000", 2024-09-15T03:47:50.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5303000000000006e-05 2024-09-15T03:47:50.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.147840+0000", 2024-09-15T03:47:50.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017416580000000001 2024-09-15T03:47:50.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148318+0000", 2024-09-15T03:47:50.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047822899999999999 2024-09-15T03:47:50.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148329+0000", 2024-09-15T03:47:50.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0611000000000001e-05 2024-09-15T03:47:50.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148488+0000", 2024-09-15T03:47:50.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015938399999999999 2024-09-15T03:47:50.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148870+0000", 2024-09-15T03:47:50.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000382405 2024-09-15T03:47:50.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:50.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.151092+0000", 2024-09-15T03:47:50.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022212529999999999 2024-09-15T03:47:50.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.151918+0000", 2024-09-15T03:47:50.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082671600000000002 2024-09-15T03:47:50.735 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.735 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.735 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.735 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:221 3.bs0 3:d99ddedc:::smithi00132172-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:50.735 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:50.735 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 65.589090068999994, 2024-09-15T03:47:50.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021194474000000001, 2024-09-15T03:47:50.735 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.736 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:50.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.736 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 221 2024-09-15T03:47:50.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.736 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:50.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:50.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127690+0000", 2024-09-15T03:47:50.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:47:50.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127692+0000", 2024-09-15T03:47:50.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6589999999999999e-06 2024-09-15T03:47:50.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127700+0000", 2024-09-15T03:47:50.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4769999999999997e-06 2024-09-15T03:47:50.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127704+0000", 2024-09-15T03:47:50.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3819999999999997e-06 2024-09-15T03:47:50.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127805+0000", 2024-09-15T03:47:50.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100639 2024-09-15T03:47:50.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:50.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127809+0000", 2024-09-15T03:47:50.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8870000000000002e-06 2024-09-15T03:47:50.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130527+0000", 2024-09-15T03:47:50.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002718579 2024-09-15T03:47:50.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:50.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130558+0000", 2024-09-15T03:47:50.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0827999999999997e-05 2024-09-15T03:47:50.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148546+0000", 2024-09-15T03:47:50.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017988356 2024-09-15T03:47:50.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148683+0000", 2024-09-15T03:47:50.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013650899999999999 2024-09-15T03:47:50.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148884+0000", 2024-09-15T03:47:50.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020061499999999999 2024-09-15T03:47:50.741 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.741 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.741 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.741 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:357 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:50.742 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:50.742 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.867738535000001, 2024-09-15T03:47:50.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.02017917, 2024-09-15T03:47:50.742 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.742 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:50.742 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.742 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.742 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.742 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 357 2024-09-15T03:47:50.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.742 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:50.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:50.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849043+0000", 2024-09-15T03:47:50.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.295e-06 2024-09-15T03:47:50.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849051+0000", 2024-09-15T03:47:50.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4079999999999998e-06 2024-09-15T03:47:50.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849069+0000", 2024-09-15T03:47:50.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7462999999999999e-05 2024-09-15T03:47:50.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849077+0000", 2024-09-15T03:47:50.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8869999999999992e-06 2024-09-15T03:47:50.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849194+0000", 2024-09-15T03:47:50.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011723899999999999 2024-09-15T03:47:50.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:50.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849296+0000", 2024-09-15T03:47:50.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102011 2024-09-15T03:47:50.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861106+0000", 2024-09-15T03:47:50.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0118097610000001 2024-09-15T03:47:50.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861222+0000", 2024-09-15T03:47:50.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116561 2024-09-15T03:47:50.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.869220+0000", 2024-09-15T03:47:50.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007997545 2024-09-15T03:47:50.747 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.747 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.747 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.747 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:358 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:50.748 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:50.748 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.867683589000002, 2024-09-15T03:47:50.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0228638050000001, 2024-09-15T03:47:50.748 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.748 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:50.748 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.748 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.748 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.748 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 358 2024-09-15T03:47:50.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.748 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:50.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:50.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:50.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4099999999999997e-07 2024-09-15T03:47:50.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849101+0000", 2024-09-15T03:47:50.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8180000000000003e-06 2024-09-15T03:47:50.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849110+0000", 2024-09-15T03:47:50.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1409999999999994e-06 2024-09-15T03:47:50.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849116+0000", 2024-09-15T03:47:50.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2390000000000002e-06 2024-09-15T03:47:50.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849326+0000", 2024-09-15T03:47:50.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000210029 2024-09-15T03:47:50.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:50.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849421+0000", 2024-09-15T03:47:50.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5775999999999999e-05 2024-09-15T03:47:50.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861647+0000", 2024-09-15T03:47:50.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0122257139999999 2024-09-15T03:47:50.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861777+0000", 2024-09-15T03:47:50.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130087 2024-09-15T03:47:50.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.871959+0000", 2024-09-15T03:47:50.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010182160000000001 2024-09-15T03:47:50.752 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.753 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.753 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.753 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:359 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:50.753 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:50.753 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.867658437000003, 2024-09-15T03:47:50.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0240468149999999, 2024-09-15T03:47:50.753 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.753 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:50.753 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.753 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.753 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.754 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 359 2024-09-15T03:47:50.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.754 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:50.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:50.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:50.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:47:50.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849124+0000", 2024-09-15T03:47:50.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2679999999999999e-06 2024-09-15T03:47:50.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849132+0000", 2024-09-15T03:47:50.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4030000000000001e-06 2024-09-15T03:47:50.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849136+0000", 2024-09-15T03:47:50.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2289999999999997e-06 2024-09-15T03:47:50.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849446+0000", 2024-09-15T03:47:50.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031043299999999999 2024-09-15T03:47:50.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:50.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849553+0000", 2024-09-15T03:47:50.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106415 2024-09-15T03:47:50.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.863404+0000", 2024-09-15T03:47:50.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0138511029999999 2024-09-15T03:47:50.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.863475+0000", 2024-09-15T03:47:50.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0814000000000003e-05 2024-09-15T03:47:50.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.873168+0000", 2024-09-15T03:47:50.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00969274 2024-09-15T03:47:50.758 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.758 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.758 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.758 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:474 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~786432] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:50.758 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:50.758 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.841733761999997, 2024-09-15T03:47:50.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98604943700000003, 2024-09-15T03:47:50.758 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.759 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:50.759 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.759 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.759 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.759 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 474 2024-09-15T03:47:50.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.759 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:50.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:50.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875048+0000", 2024-09-15T03:47:50.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1600000000000001e-06 2024-09-15T03:47:50.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875844+0000", 2024-09-15T03:47:50.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079686299999999998 2024-09-15T03:47:50.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875857+0000", 2024-09-15T03:47:50.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2889e-05 2024-09-15T03:47:50.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875864+0000", 2024-09-15T03:47:50.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6139999999999997e-06 2024-09-15T03:47:50.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875916+0000", 2024-09-15T03:47:50.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1992e-05 2024-09-15T03:47:50.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:50.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875918+0000", 2024-09-15T03:47:50.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3300000000000001e-06 2024-09-15T03:47:50.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.845918+0000", 2024-09-15T03:47:50.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96999959199999997 2024-09-15T03:47:50.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.846229+0000", 2024-09-15T03:47:50.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000311601 2024-09-15T03:47:50.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848445+0000", 2024-09-15T03:47:50.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022160499999999998 2024-09-15T03:47:50.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848880+0000", 2024-09-15T03:47:50.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043432600000000002 2024-09-15T03:47:50.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848952+0000", 2024-09-15T03:47:50.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2509000000000007e-05 2024-09-15T03:47:50.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.849311+0000", 2024-09-15T03:47:50.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000358611 2024-09-15T03:47:50.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851131+0000", 2024-09-15T03:47:50.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001820115 2024-09-15T03:47:50.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:50.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861048+0000", 2024-09-15T03:47:50.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099164830000000002 2024-09-15T03:47:50.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861095+0000", 2024-09-15T03:47:50.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7302000000000002e-05 2024-09-15T03:47:50.765 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.765 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.765 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.766 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:475 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~491520] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:50.766 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:50.766 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.838967291000003, 2024-09-15T03:47:50.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98407319000000004, 2024-09-15T03:47:50.766 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.766 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:50.766 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.766 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.766 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.766 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 475 2024-09-15T03:47:50.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.766 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:50.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:50.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877813+0000", 2024-09-15T03:47:50.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8800000000000002e-07 2024-09-15T03:47:50.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878260+0000", 2024-09-15T03:47:50.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044732099999999998 2024-09-15T03:47:50.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878270+0000", 2024-09-15T03:47:50.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9799999999999993e-06 2024-09-15T03:47:50.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878276+0000", 2024-09-15T03:47:50.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2269999999999998e-06 2024-09-15T03:47:50.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878313+0000", 2024-09-15T03:47:50.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6570999999999999e-05 2024-09-15T03:47:50.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:50.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878315+0000", 2024-09-15T03:47:50.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8729999999999999e-06 2024-09-15T03:47:50.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851316+0000", 2024-09-15T03:47:50.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97300154800000005 2024-09-15T03:47:50.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851440+0000", 2024-09-15T03:47:50.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124287 2024-09-15T03:47:50.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852732+0000", 2024-09-15T03:47:50.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001291174 2024-09-15T03:47:50.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852991+0000", 2024-09-15T03:47:50.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025905299999999997 2024-09-15T03:47:50.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853003+0000", 2024-09-15T03:47:50.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2908000000000001e-05 2024-09-15T03:47:50.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853104+0000", 2024-09-15T03:47:50.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010011800000000001 2024-09-15T03:47:50.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854342+0000", 2024-09-15T03:47:50.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012380189999999999 2024-09-15T03:47:50.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:50.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861206+0000", 2024-09-15T03:47:50.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068648499999999996 2024-09-15T03:47:50.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861885+0000", 2024-09-15T03:47:50.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00067857300000000002 2024-09-15T03:47:50.773 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.773 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.773 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.773 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:476 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~278528] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:50.773 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:50.773 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.837340277999999, 2024-09-15T03:47:50.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98268067299999995, 2024-09-15T03:47:50.774 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.774 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:50.774 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.774 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.774 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.774 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 476 2024-09-15T03:47:50.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.774 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:50.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:50.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:50.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.13e-07 2024-09-15T03:47:50.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879706+0000", 2024-09-15T03:47:50.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026656100000000001 2024-09-15T03:47:50.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879715+0000", 2024-09-15T03:47:50.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9330000000000007e-06 2024-09-15T03:47:50.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879720+0000", 2024-09-15T03:47:50.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0880000000000002e-06 2024-09-15T03:47:50.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879743+0000", 2024-09-15T03:47:50.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2654e-05 2024-09-15T03:47:50.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:50.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879743+0000", 2024-09-15T03:47:50.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.99e-07 2024-09-15T03:47:50.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853170+0000", 2024-09-15T03:47:50.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97342687900000002 2024-09-15T03:47:50.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853297+0000", 2024-09-15T03:47:50.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127157 2024-09-15T03:47:50.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854150+0000", 2024-09-15T03:47:50.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085288900000000001 2024-09-15T03:47:50.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854292+0000", 2024-09-15T03:47:50.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014199300000000001 2024-09-15T03:47:50.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854559+0000", 2024-09-15T03:47:50.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026668999999999998 2024-09-15T03:47:50.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855998+0000", 2024-09-15T03:47:50.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001438714 2024-09-15T03:47:50.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856547+0000", 2024-09-15T03:47:50.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054880399999999996 2024-09-15T03:47:50.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:50.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861858+0000", 2024-09-15T03:47:50.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053110320000000003 2024-09-15T03:47:50.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.862120+0000", 2024-09-15T03:47:50.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026196699999999999 2024-09-15T03:47:50.780 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.781 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.781 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.781 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:477 3.bs0 3:d52faf55:::smithi00132172-40:head [setxattr _header (57) in=7b] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:50.781 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:50.781 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.837052341000003, 2024-09-15T03:47:50.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98562198000000001, 2024-09-15T03:47:50.781 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.781 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:50.781 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.781 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.781 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.782 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 477 2024-09-15T03:47:50.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.782 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:50.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:50.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:50.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2800000000000002e-07 2024-09-15T03:47:50.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879730+0000", 2024-09-15T03:47:50.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0589999999999998e-06 2024-09-15T03:47:50.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879737+0000", 2024-09-15T03:47:50.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0419999999999997e-06 2024-09-15T03:47:50.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879741+0000", 2024-09-15T03:47:50.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9960000000000004e-06 2024-09-15T03:47:50.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879756+0000", 2024-09-15T03:47:50.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4681999999999999e-05 2024-09-15T03:47:50.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:50.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879756+0000", 2024-09-15T03:47:50.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.22e-07 2024-09-15T03:47:50.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854590+0000", 2024-09-15T03:47:50.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97483319000000002 2024-09-15T03:47:50.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854718+0000", 2024-09-15T03:47:50.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012842100000000001 2024-09-15T03:47:50.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855416+0000", 2024-09-15T03:47:50.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069759200000000003 2024-09-15T03:47:50.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855551+0000", 2024-09-15T03:47:50.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013517600000000001 2024-09-15T03:47:50.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856002+0000", 2024-09-15T03:47:50.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045169399999999999 2024-09-15T03:47:50.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856065+0000", 2024-09-15T03:47:50.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3028999999999996e-05 2024-09-15T03:47:50.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856554+0000", 2024-09-15T03:47:50.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000488486 2024-09-15T03:47:50.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:50.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865278+0000", 2024-09-15T03:47:50.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087241929999999999 2024-09-15T03:47:50.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865349+0000", 2024-09-15T03:47:50.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0669999999999999e-05 2024-09-15T03:47:50.788 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.788 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.788 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.788 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:478 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:50.789 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:50.789 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.837033546000001, 2024-09-15T03:47:50.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99471090500000003, 2024-09-15T03:47:50.789 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.789 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:50.789 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.789 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.789 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.789 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 478 2024-09-15T03:47:50.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.789 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:50.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:50.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:50.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2599999999999998e-07 2024-09-15T03:47:50.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879748+0000", 2024-09-15T03:47:50.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9609999999999999e-06 2024-09-15T03:47:50.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879755+0000", 2024-09-15T03:47:50.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1879999999999996e-06 2024-09-15T03:47:50.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879760+0000", 2024-09-15T03:47:50.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2560000000000002e-06 2024-09-15T03:47:50.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879797+0000", 2024-09-15T03:47:50.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6452999999999998e-05 2024-09-15T03:47:50.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:50.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879799+0000", 2024-09-15T03:47:50.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6619999999999999e-06 2024-09-15T03:47:50.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855637+0000", 2024-09-15T03:47:50.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97583850999999999 2024-09-15T03:47:50.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:50.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855763+0000", 2024-09-15T03:47:50.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012612699999999999 2024-09-15T03:47:50.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865707+0000", 2024-09-15T03:47:50.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099439660000000003 2024-09-15T03:47:50.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865803+0000", 2024-09-15T03:47:50.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5946999999999995e-05 2024-09-15T03:47:50.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.874456+0000", 2024-09-15T03:47:50.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0086534090000000008 2024-09-15T03:47:50.794 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.794 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.794 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.794 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:486 3.bs0 3:df8b18a3:::smithi00132172-17:head [delete] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e64)", 2024-09-15T03:47:50.795 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:50.795 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.834221067000001, 2024-09-15T03:47:50.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98362617100000005, 2024-09-15T03:47:50.795 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.795 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:50.795 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.795 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.795 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.795 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 486 2024-09-15T03:47:50.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.795 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:50.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:50.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882559+0000", 2024-09-15T03:47:50.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0359999999999999e-06 2024-09-15T03:47:50.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882564+0000", 2024-09-15T03:47:50.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6530000000000002e-06 2024-09-15T03:47:50.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882574+0000", 2024-09-15T03:47:50.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0464e-05 2024-09-15T03:47:50.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882578+0000", 2024-09-15T03:47:50.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2459999999999997e-06 2024-09-15T03:47:50.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882624+0000", 2024-09-15T03:47:50.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5698000000000002e-05 2024-09-15T03:47:50.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:50.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882627+0000", 2024-09-15T03:47:50.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3350000000000001e-06 2024-09-15T03:47:50.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856574+0000", 2024-09-15T03:47:50.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97394739399999997 2024-09-15T03:47:50.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856853+0000", 2024-09-15T03:47:50.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027918399999999998 2024-09-15T03:47:50.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.866184+0000", 2024-09-15T03:47:50.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093311609999999993 2024-09-15T03:47:50.800 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.800 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.800 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.800 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:495 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [copy-from ver 51] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-15T03:47:50.800 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:50.800 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.828320017999999, 2024-09-15T03:47:50.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023555178, 2024-09-15T03:47:50.800 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.800 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:50.801 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.801 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.801 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.801 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 495 2024-09-15T03:47:50.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.801 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:50.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:50.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888461+0000", 2024-09-15T03:47:50.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.719e-06 2024-09-15T03:47:50.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888471+0000", 2024-09-15T03:47:50.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0161e-05 2024-09-15T03:47:50.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888500+0000", 2024-09-15T03:47:50.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8989e-05 2024-09-15T03:47:50.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888508+0000", 2024-09-15T03:47:50.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5299999999999999e-06 2024-09-15T03:47:50.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888554+0000", 2024-09-15T03:47:50.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6489000000000002e-05 2024-09-15T03:47:50.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888809+0000", 2024-09-15T03:47:50.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000255467 2024-09-15T03:47:50.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.905724+0000", 2024-09-15T03:47:50.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016914348999999999 2024-09-15T03:47:50.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906244+0000", 2024-09-15T03:47:50.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052054900000000003 2024-09-15T03:47:50.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906262+0000", 2024-09-15T03:47:50.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7499999999999998e-05 2024-09-15T03:47:50.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.907367+0000", 2024-09-15T03:47:50.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011049460000000001 2024-09-15T03:47:50.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.907384+0000", 2024-09-15T03:47:50.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7000999999999999e-05 2024-09-15T03:47:50.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:50.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.911965+0000", 2024-09-15T03:47:50.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045809470000000001 2024-09-15T03:47:50.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.912014+0000", 2024-09-15T03:47:50.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9530999999999997e-05 2024-09-15T03:47:50.807 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.807 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.807 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.807 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:689 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~5144576,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:50.807 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.088846+0000", 2024-09-15T03:47:50.807 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.627933597999998, 2024-09-15T03:47:50.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020311780000000002, 2024-09-15T03:47:50.808 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.808 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:50.808 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.808 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.808 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.808 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 689 2024-09-15T03:47:50.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.808 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088846+0000", 2024-09-15T03:47:50.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088846+0000", 2024-09-15T03:47:50.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088847+0000", 2024-09-15T03:47:50.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9300000000000006e-07 2024-09-15T03:47:50.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088852+0000", 2024-09-15T03:47:50.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1850000000000003e-06 2024-09-15T03:47:50.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088863+0000", 2024-09-15T03:47:50.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1418e-05 2024-09-15T03:47:50.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088873+0000", 2024-09-15T03:47:50.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4020000000000008e-06 2024-09-15T03:47:50.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.089560+0000", 2024-09-15T03:47:50.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068774899999999998 2024-09-15T03:47:50.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090073+0000", 2024-09-15T03:47:50.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051222799999999999 2024-09-15T03:47:50.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.109157+0000", 2024-09-15T03:47:50.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019084805 2024-09-15T03:47:50.811 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.812 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.812 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.812 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:690 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~5144576,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:50.812 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:50.812 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.627900459999999, 2024-09-15T03:47:50.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025340499999999998, 2024-09-15T03:47:50.812 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.812 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:50.812 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.812 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.812 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.813 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 690 2024-09-15T03:47:50.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.813 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:50.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:50.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:50.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7999999999999996e-07 2024-09-15T03:47:50.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088882+0000", 2024-09-15T03:47:50.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1839999999999999e-06 2024-09-15T03:47:50.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088892+0000", 2024-09-15T03:47:50.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3540000000000006e-06 2024-09-15T03:47:50.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088896+0000", 2024-09-15T03:47:50.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5210000000000004e-06 2024-09-15T03:47:50.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.093968+0000", 2024-09-15T03:47:50.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050715170000000002 2024-09-15T03:47:50.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.094098+0000", 2024-09-15T03:47:50.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013070000000000001 2024-09-15T03:47:50.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.114219+0000", 2024-09-15T03:47:50.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020120743999999999 2024-09-15T03:47:50.816 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.816 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.816 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.816 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:691 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~5144576,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:50.817 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.088900+0000", 2024-09-15T03:47:50.817 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.627878717000002, 2024-09-15T03:47:50.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031407274999999998, 2024-09-15T03:47:50.817 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.817 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:50.817 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.817 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.817 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.817 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 691 2024-09-15T03:47:50.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.817 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088900+0000", 2024-09-15T03:47:50.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088900+0000", 2024-09-15T03:47:50.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088901+0000", 2024-09-15T03:47:50.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9099999999999999e-07 2024-09-15T03:47:50.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088904+0000", 2024-09-15T03:47:50.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0299999999999998e-06 2024-09-15T03:47:50.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088911+0000", 2024-09-15T03:47:50.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.708e-06 2024-09-15T03:47:50.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088915+0000", 2024-09-15T03:47:50.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0579999999999999e-06 2024-09-15T03:47:50.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.096143+0000", 2024-09-15T03:47:50.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007228099 2024-09-15T03:47:50.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.096202+0000", 2024-09-15T03:47:50.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9667999999999999e-05 2024-09-15T03:47:50.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.120308+0000", 2024-09-15T03:47:50.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024105320999999999 2024-09-15T03:47:50.821 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.821 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.821 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.821 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(osd.0.17:2 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [assert-version v51,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:50.821 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.090507+0000", 2024-09-15T03:47:50.821 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.62627225, 2024-09-15T03:47:50.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025010186, 2024-09-15T03:47:50.821 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.821 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:50.821 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090507+0000", 2024-09-15T03:47:50.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090507+0000", 2024-09-15T03:47:50.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090508+0000", 2024-09-15T03:47:50.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.176e-06 2024-09-15T03:47:50.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090512+0000", 2024-09-15T03:47:50.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2989999999999998e-06 2024-09-15T03:47:50.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090522+0000", 2024-09-15T03:47:50.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6549999999999999e-06 2024-09-15T03:47:50.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090526+0000", 2024-09-15T03:47:50.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8500000000000004e-06 2024-09-15T03:47:50.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.094385+0000", 2024-09-15T03:47:50.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038591670000000002 2024-09-15T03:47:50.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.094541+0000", 2024-09-15T03:47:50.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015590999999999999 2024-09-15T03:47:50.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.115517+0000", 2024-09-15T03:47:50.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020976129 2024-09-15T03:47:50.825 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.825 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.825 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.825 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:749 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~737280,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-15T03:47:50.825 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.935286+0000", 2024-09-15T03:47:50.825 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.78149271, 2024-09-15T03:47:50.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021703044000000001, 2024-09-15T03:47:50.825 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.826 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:50.826 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.826 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.826 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.826 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 749 2024-09-15T03:47:50.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.826 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935286+0000", 2024-09-15T03:47:50.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935286+0000", 2024-09-15T03:47:50.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935287+0000", 2024-09-15T03:47:50.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.51e-07 2024-09-15T03:47:50.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935290+0000", 2024-09-15T03:47:50.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8710000000000001e-06 2024-09-15T03:47:50.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935297+0000", 2024-09-15T03:47:50.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3390000000000004e-06 2024-09-15T03:47:50.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935302+0000", 2024-09-15T03:47:50.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5560000000000001e-06 2024-09-15T03:47:50.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936361+0000", 2024-09-15T03:47:50.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010593200000000001 2024-09-15T03:47:50.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:47:50.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936486+0000", 2024-09-15T03:47:50.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124657 2024-09-15T03:47:50.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.952979+0000", 2024-09-15T03:47:50.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016493154999999999 2024-09-15T03:47:50.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.953132+0000", 2024-09-15T03:47:50.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015346500000000001 2024-09-15T03:47:50.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.956989+0000", 2024-09-15T03:47:50.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038571299999999998 2024-09-15T03:47:50.830 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.830 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.830 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.830 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:750 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [create,setxattr \\^tkqj^qnPPfYPq\\WZoebWOkaebalTko_OhaYsrZ (1787) in=40b,setxattr cg]RoaNaVsOQTrQfp[VeRj_tmeegdoc`PtRYpskN (1998) in=40b,setxattr \\_N_s\\f^o_Ojh]gqWPkfsZ\\WTSbY]SOlX`mlV[^q (6944) in=40b,setxattr nsdiO]arkaqtYt (33) in=14b,setxattr ti`p[nirg_rWRRSe[oZhhn]b]ZsTimedg]]p[sQY (1049) in=40b,setxattr i[YNharOrm`bbP]UtYhjlqVWP\\XnccYlfhV_deTf (383) in=40b,setxattr ka_q[S]a]naRePWgcgUWmZSaW^RVV\\_rNXdZefal (875) in=40b,setxattr gaoaanUtdfqUrimS^_NfUtarRjNSVRf^_PbUpdtf (1645) in=40b,setxattr qNb^mqtq^Vsg]m\\aWdRTbbgOlZYptsW]O[_R`Vaa (1155) in=40b,setxattr gUPP_ttp`PdrngYsWrVdfZZVgff^Q`YRSjgOXTao (417) in=40b,setxattr mi`p\\f[PUQf^[[skf^\\Za_iXO_oTscOWNnQqcR[[ (4560) in=40b,setxattr N^_ka_abmi]seatXYWSQtibSdeOsfV\\OXsmmXlYm (1893) in=40b,setxattr aoqZmYaUpNW (33) in=11b,setxattr Z[bbsdRZTfkrP`]akaXifm`oV[YTOPrPaeeXV[fl (1611) in=40b,setxattr oZ`SalVqYt^T_[b_a]r[rlaaWOa_WZQ^^opf\\aN_ (243) in=40b,setxattr k\\V^enpkstj^VpoopX\\PcoTY]eogbs`pQQW]ofrm (347) in=40b,setxattr cfXfpnr`SogqjNa^cTVYtpgdciPp_eTUg]Ujn\\at (1926) in=40b,setxattr t`ojSZgRjbb[aaUXSepgQf_nc`kaOetle^n^`Vfd (1087) in=40b,setxattr ONYdltaQhe\\a\\da\\ehQRpsS^Ugdgmf\\WWigfUarV (631) in=40b,setxattr [Tj`Rbkt]qjatpSf_fPsanNr^ipYnOQlPnUZlmRU (4068) in=40b] snapc 32=[32,2d,2b,26,25,24,23,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T03:47:50.831 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.935912+0000", 2024-09-15T03:47:50.831 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.780867386000001, 2024-09-15T03:47:50.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03915623, 2024-09-15T03:47:50.831 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.831 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:50.831 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:50.831 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:50.831 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:50.831 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 750 2024-09-15T03:47:50.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.831 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935912+0000", 2024-09-15T03:47:50.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935912+0000", 2024-09-15T03:47:50.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935913+0000", 2024-09-15T03:47:50.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7599999999999996e-07 2024-09-15T03:47:50.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935984+0000", 2024-09-15T03:47:50.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1136000000000005e-05 2024-09-15T03:47:50.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935997+0000", 2024-09-15T03:47:50.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3482000000000001e-05 2024-09-15T03:47:50.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936004+0000", 2024-09-15T03:47:50.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3260000000000001e-06 2024-09-15T03:47:50.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936878+0000", 2024-09-15T03:47:50.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087483999999999999 2024-09-15T03:47:50.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:47:50.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.937895+0000", 2024-09-15T03:47:50.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010163209999999999 2024-09-15T03:47:50.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.965388+0000", 2024-09-15T03:47:50.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027492986000000001 2024-09-15T03:47:50.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:50.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.965630+0000", 2024-09-15T03:47:50.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000242012 2024-09-15T03:47:50.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.966983+0000", 2024-09-15T03:47:50.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013535349999999999 2024-09-15T03:47:50.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.967360+0000", 2024-09-15T03:47:50.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037704399999999999 2024-09-15T03:47:50.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.967371+0000", 2024-09-15T03:47:50.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0988999999999999e-05 2024-09-15T03:47:50.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.967436+0000", 2024-09-15T03:47:50.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4679999999999997e-05 2024-09-15T03:47:50.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.969080+0000", 2024-09-15T03:47:50.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001644017 2024-09-15T03:47:50.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:50.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.969532+0000", 2024-09-15T03:47:50.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045157699999999999 2024-09-15T03:47:50.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.975068+0000", 2024-09-15T03:47:50.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055364089999999999 2024-09-15T03:47:50.838 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.838 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.838 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.838 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.5s5 99/89 ECSubWrite(tid=134, reqid=client.4521.0:867, at_version=99'50, trim_to=0'0, roll_forward_to=96'47))", 2024-09-15T03:47:50.838 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:45.865389+0000", 2024-09-15T03:47:50.838 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.8513900860000003, 2024-09-15T03:47:50.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019027228, 2024-09-15T03:47:50.838 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:50.839 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:50.839 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:50.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:50.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865389+0000", 2024-09-15T03:47:50.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:50.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865389+0000", 2024-09-15T03:47:50.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:50.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:50.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865390+0000", 2024-09-15T03:47:50.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:47:50.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:50.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865411+0000", 2024-09-15T03:47:50.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1005e-05 2024-09-15T03:47:50.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:50.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865424+0000", 2024-09-15T03:47:50.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3671e-05 2024-09-15T03:47:50.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:50.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865431+0000", 2024-09-15T03:47:50.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5989999999999998e-06 2024-09-15T03:47:50.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:50.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865487+0000", 2024-09-15T03:47:50.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5933999999999997e-05 2024-09-15T03:47:50.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:50.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865497+0000", 2024-09-15T03:47:50.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0079e-05 2024-09-15T03:47:50.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:50.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865620+0000", 2024-09-15T03:47:50.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012282499999999999 2024-09-15T03:47:50.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:50.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865632+0000", 2024-09-15T03:47:50.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1841999999999999e-05 2024-09-15T03:47:50.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:50.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.870932+0000", 2024-09-15T03:47:50.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005300478 2024-09-15T03:47:50.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:50.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.870953+0000", 2024-09-15T03:47:50.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0797000000000001e-05 2024-09-15T03:47:50.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:50.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:50.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:50.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.884416+0000", 2024-09-15T03:47:50.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013463454999999999 2024-09-15T03:47:50.843 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.843 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.844 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.844 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:50.844 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:50.844 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:50.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:47:50.846 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:50.844+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:50.885 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-15T03:47:50.885 INFO:tasks.rados.rados.0.smithi001.stdout:352: done (1 left) 2024-09-15T03:47:50.886 INFO:tasks.rados.rados.0.smithi001.stdout:353: done (0 left) 2024-09-15T03:47:50.886 INFO:tasks.rados.rados.0.smithi001.stdout:354: snap_create 2024-09-15T03:47:50.890 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:50.889+0000 7fd62628c640 -1 osd.10 78 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:47:50.946 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:50.945+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:50.954 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:47:50.955 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:50.955 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:50.955 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:50.956 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:50.964 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:47:51.047 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:51.046+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:51.061 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:51.061 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:51.061 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-09-15T03:47:51.062 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-09-15T03:47:51.062 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:51.070 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:47:51.082 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:51.083 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:51.097 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:51.097 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:51.112 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:51.113 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:51.126 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:51.141 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:51.141 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:51.148 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:51.147+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:51.155 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:51.155 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:51.170 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:51.171 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:51.174 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:51.174 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-09-15T03:47:51.174 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-09-15T03:47:51.174 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-09-15T03:47:51.174 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.174 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4255.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-15T03:47:51.174 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:51.174 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 93.269229359999997, 2024-09-15T03:47:51.174 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10988437500000001, 2024-09-15T03:47:51.174 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:51.175 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:51.175 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:51.175 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4255", 2024-09-15T03:47:51.175 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.67:0/632876599", 2024-09-15T03:47:51.175 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1 2024-09-15T03:47:51.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.175 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:51.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:51.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:51.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:51.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902354+0000", 2024-09-15T03:47:51.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:51.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902355+0000", 2024-09-15T03:47:51.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.429e-06 2024-09-15T03:47:51.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:51.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902362+0000", 2024-09-15T03:47:51.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7290000000000003e-06 2024-09-15T03:47:51.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.177 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:51.177 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902373+0000", 2024-09-15T03:47:51.177 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0533999999999999e-05 2024-09-15T03:47:51.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.177 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:51.177 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902379+0000", 2024-09-15T03:47:51.177 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4350000000000002e-06 2024-09-15T03:47:51.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.177 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.177 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902428+0000", 2024-09-15T03:47:51.177 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8571e-05 2024-09-15T03:47:51.178 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.178 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.178 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:47:51.178 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:17.902434+0000", 2024-09-15T03:47:51.178 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5610000000000004e-06 2024-09-15T03:47:51.178 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.178 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.178 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.178 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011307+0000", 2024-09-15T03:47:51.178 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.108872306 2024-09-15T03:47:51.178 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.178 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.178 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:51.179 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011448+0000", 2024-09-15T03:47:51.179 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014137199999999999 2024-09-15T03:47:51.179 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.179 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.180 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-09-15T03:47:51.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011569+0000", 2024-09-15T03:47:51.180 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120925 2024-09-15T03:47:51.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.180 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.180 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:51.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011695+0000", 2024-09-15T03:47:51.180 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012586800000000001 2024-09-15T03:47:51.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.181 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:51.181 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011703+0000", 2024-09-15T03:47:51.181 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4190000000000002e-06 2024-09-15T03:47:51.181 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.181 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:51.181 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011739+0000", 2024-09-15T03:47:51.181 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5474e-05 2024-09-15T03:47:51.181 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.181 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-09-15T03:47:51.181 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.011940+0000", 2024-09-15T03:47:51.182 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020108699999999999 2024-09-15T03:47:51.182 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.182 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.182 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:47:51.182 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012179+0000", 2024-09-15T03:47:51.182 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023950299999999999 2024-09-15T03:47:51.182 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.182 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.182 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:51.182 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012215+0000", 2024-09-15T03:47:51.182 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5332e-05 2024-09-15T03:47:51.182 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.182 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.183 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:51.183 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:18.012238+0000", 2024-09-15T03:47:51.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3830000000000001e-05 2024-09-15T03:47:51.183 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.183 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:51.183 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.183 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.183 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4263.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22)", 2024-09-15T03:47:51.183 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:51.183 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 87.127832674000004, 2024-09-15T03:47:51.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019495788999999999, 2024-09-15T03:47:51.183 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:51.183 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:51.184 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:51.184 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4263", 2024-09-15T03:47:51.184 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/646873808", 2024-09-15T03:47:51.184 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2 2024-09-15T03:47:51.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.184 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:51.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.184 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:51.184 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:51.184 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.184 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:51.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043751+0000", 2024-09-15T03:47:51.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.185 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.185 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:51.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043752+0000", 2024-09-15T03:47:51.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.815e-06 2024-09-15T03:47:51.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.185 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.185 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:51.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043757+0000", 2024-09-15T03:47:51.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4240000000000003e-06 2024-09-15T03:47:51.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.186 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.186 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:51.186 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043766+0000", 2024-09-15T03:47:51.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8629999999999997e-06 2024-09-15T03:47:51.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.186 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.186 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:51.186 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043773+0000", 2024-09-15T03:47:51.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0099999999999998e-06 2024-09-15T03:47:51.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.186 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043830+0000", 2024-09-15T03:47:51.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7089999999999997e-05 2024-09-15T03:47:51.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-09-15T03:47:51.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.043837+0000", 2024-09-15T03:47:51.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.08e-06 2024-09-15T03:47:51.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063103+0000", 2024-09-15T03:47:51.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019265619000000001 2024-09-15T03:47:51.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:51.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:24.063246+0000", 2024-09-15T03:47:51.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000143888 2024-09-15T03:47:51.188 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.188 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:51.188 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.189 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.189 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.189 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 39/31 ECSubWrite(tid=43, reqid=client.4521.0:243, at_version=39'17, trim_to=0'0, roll_forward_to=32'15))", 2024-09-15T03:47:51.189 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:51.189 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 65.000759285000001, 2024-09-15T03:47:51.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011878227, 2024-09-15T03:47:51.189 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:51.189 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:51.189 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:51.189 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.190 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:51.190 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:51.190 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.190 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.190 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:51.190 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170824+0000", 2024-09-15T03:47:51.190 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.190 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.190 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:51.190 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.170825+0000", 2024-09-15T03:47:51.190 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9599999999999999e-07 2024-09-15T03:47:51.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:51.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171100+0000", 2024-09-15T03:47:51.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027548500000000001 2024-09-15T03:47:51.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:51.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171112+0000", 2024-09-15T03:47:51.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2152e-05 2024-09-15T03:47:51.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:51.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171119+0000", 2024-09-15T03:47:51.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5050000000000004e-06 2024-09-15T03:47:51.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171162+0000", 2024-09-15T03:47:51.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2908000000000003e-05 2024-09-15T03:47:51.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:51.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171169+0000", 2024-09-15T03:47:51.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9739999999999996e-06 2024-09-15T03:47:51.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:51.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.171398+0000", 2024-09-15T03:47:51.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022954799999999999 2024-09-15T03:47:51.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:51.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173640+0000", 2024-09-15T03:47:51.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022417000000000001 2024-09-15T03:47:51.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:51.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173705+0000", 2024-09-15T03:47:51.194 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4907999999999996e-05 2024-09-15T03:47:51.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.194 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:51.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.173742+0000", 2024-09-15T03:47:51.194 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6940999999999997e-05 2024-09-15T03:47:51.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.195 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:51.195 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.182702+0000", 2024-09-15T03:47:51.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0089604100000000002 2024-09-15T03:47:51.195 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.195 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:51.195 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.195 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.195 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.bs2 39/31 ECSubWrite(tid=43, reqid=client.4521.0:252, at_version=39'27, trim_to=0'0, roll_forward_to=39'25))", 2024-09-15T03:47:51.195 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:51.195 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 64.993041966999996, 2024-09-15T03:47:51.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011286757, 2024-09-15T03:47:51.196 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:51.196 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:51.196 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:51.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.196 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:51.196 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:51.196 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.196 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:51.196 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178541+0000", 2024-09-15T03:47:51.196 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:51.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178542+0000", 2024-09-15T03:47:51.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9400000000000004e-07 2024-09-15T03:47:51.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:51.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178754+0000", 2024-09-15T03:47:51.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021228800000000001 2024-09-15T03:47:51.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:51.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178765+0000", 2024-09-15T03:47:51.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0176e-05 2024-09-15T03:47:51.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:51.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178769+0000", 2024-09-15T03:47:51.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1359999999999999e-06 2024-09-15T03:47:51.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178810+0000", 2024-09-15T03:47:51.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0825999999999998e-05 2024-09-15T03:47:51.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:51.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.178820+0000", 2024-09-15T03:47:51.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0237999999999999e-05 2024-09-15T03:47:51.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.199 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:51.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179033+0000", 2024-09-15T03:47:51.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021361500000000001 2024-09-15T03:47:51.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.199 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:51.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179040+0000", 2024-09-15T03:47:51.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0509999999999999e-06 2024-09-15T03:47:51.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.200 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:51.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179102+0000", 2024-09-15T03:47:51.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1977000000000006e-05 2024-09-15T03:47:51.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.200 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:51.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.179114+0000", 2024-09-15T03:47:51.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1223000000000001e-05 2024-09-15T03:47:51.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.200 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:51.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:46.189828+0000", 2024-09-15T03:47:51.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010714433000000001 2024-09-15T03:47:51.201 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.201 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:51.201 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.201 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.201 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.201 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 51/31 ECSubWrite(tid=67, reqid=client.4521.0:350, at_version=51'33, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:51.201 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:51.201 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 53.327741926000002, 2024-09-15T03:47:51.201 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.015569239, 2024-09-15T03:47:51.202 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:51.202 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:51.202 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:51.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:51.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:51.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:51.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843841+0000", 2024-09-15T03:47:51.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:51.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843843+0000", 2024-09-15T03:47:51.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.195e-06 2024-09-15T03:47:51.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:51.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843880+0000", 2024-09-15T03:47:51.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7527999999999997e-05 2024-09-15T03:47:51.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:51.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843898+0000", 2024-09-15T03:47:51.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7696000000000001e-05 2024-09-15T03:47:51.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:51.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843907+0000", 2024-09-15T03:47:51.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8820000000000007e-06 2024-09-15T03:47:51.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843978+0000", 2024-09-15T03:47:51.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1537000000000003e-05 2024-09-15T03:47:51.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:51.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.843989+0000", 2024-09-15T03:47:51.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0720000000000001e-05 2024-09-15T03:47:51.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:51.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844102+0000", 2024-09-15T03:47:51.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000113406 2024-09-15T03:47:51.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:51.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844131+0000", 2024-09-15T03:47:51.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8762e-05 2024-09-15T03:47:51.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:51.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844180+0000", 2024-09-15T03:47:51.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9329999999999997e-05 2024-09-15T03:47:51.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:51.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844201+0000", 2024-09-15T03:47:51.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0772999999999999e-05 2024-09-15T03:47:51.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:51.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.859411+0000", 2024-09-15T03:47:51.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01520941 2024-09-15T03:47:51.207 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.207 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:51.207 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.207 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:51.207 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:51.207 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 53.326863015000001, 2024-09-15T03:47:51.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012156452, 2024-09-15T03:47:51.207 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:51.207 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:51.207 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:51.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.207 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:51.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:51.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.208 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:51.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844720+0000", 2024-09-15T03:47:51.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.208 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:51.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844721+0000", 2024-09-15T03:47:51.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1100000000000005e-07 2024-09-15T03:47:51.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:51.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844743+0000", 2024-09-15T03:47:51.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1999000000000001e-05 2024-09-15T03:47:51.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:51.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844758+0000", 2024-09-15T03:47:51.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4718999999999999e-05 2024-09-15T03:47:51.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:51.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844772+0000", 2024-09-15T03:47:51.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4637000000000001e-05 2024-09-15T03:47:51.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844842+0000", 2024-09-15T03:47:51.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9596000000000002e-05 2024-09-15T03:47:51.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:51.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844853+0000", 2024-09-15T03:47:51.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0912e-05 2024-09-15T03:47:51.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:51.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844966+0000", 2024-09-15T03:47:51.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011276 2024-09-15T03:47:51.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:51.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844977+0000", 2024-09-15T03:47:51.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1676e-05 2024-09-15T03:47:51.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:51.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847414+0000", 2024-09-15T03:47:51.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024363760000000001 2024-09-15T03:47:51.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.212 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.212 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:51.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847438+0000", 2024-09-15T03:47:51.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3737e-05 2024-09-15T03:47:51.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.212 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.212 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:51.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.856877+0000", 2024-09-15T03:47:51.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094392290000000004 2024-09-15T03:47:51.212 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.212 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:51.213 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.213 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.213 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:51.213 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:51.213 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 53.326793619999997, 2024-09-15T03:47:51.213 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012258211, 2024-09-15T03:47:51.213 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:51.213 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:51.213 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:51.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.213 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:51.213 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:51.213 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:51.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:51.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:51.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844790+0000", 2024-09-15T03:47:51.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.58e-07 2024-09-15T03:47:51.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.215 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.215 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:51.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844818+0000", 2024-09-15T03:47:51.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.796e-05 2024-09-15T03:47:51.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.215 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.215 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:51.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844829+0000", 2024-09-15T03:47:51.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0895e-05 2024-09-15T03:47:51.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.215 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.215 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:51.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.844836+0000", 2024-09-15T03:47:51.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.618e-06 2024-09-15T03:47:51.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845023+0000", 2024-09-15T03:47:51.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018743500000000001 2024-09-15T03:47:51.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:51.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845035+0000", 2024-09-15T03:47:51.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1958e-05 2024-09-15T03:47:51.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:51.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845147+0000", 2024-09-15T03:47:51.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112281 2024-09-15T03:47:51.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:51.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847417+0000", 2024-09-15T03:47:51.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022693969999999998 2024-09-15T03:47:51.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.218 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:51.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847462+0000", 2024-09-15T03:47:51.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5466999999999997e-05 2024-09-15T03:47:51.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.218 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.218 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:51.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847503+0000", 2024-09-15T03:47:51.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1022999999999997e-05 2024-09-15T03:47:51.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.218 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.218 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:51.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.857048+0000", 2024-09-15T03:47:51.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095447190000000001 2024-09-15T03:47:51.218 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.219 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:51.219 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.219 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.219 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.3s1 51/31 ECSubWrite(tid=68, reqid=client.4521.0:351, at_version=51'34, trim_to=0'0, roll_forward_to=47'30))", 2024-09-15T03:47:51.219 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:51.219 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 53.326375501999998, 2024-09-15T03:47:51.219 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.014544793, 2024-09-15T03:47:51.219 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:51.219 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:51.219 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:51.219 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.219 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:51.219 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:51.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:51.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845208+0000", 2024-09-15T03:47:51.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:51.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845209+0000", 2024-09-15T03:47:51.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5300000000000003e-07 2024-09-15T03:47:51.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.221 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.221 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:51.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845227+0000", 2024-09-15T03:47:51.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8759000000000001e-05 2024-09-15T03:47:51.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.221 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.221 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:51.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845241+0000", 2024-09-15T03:47:51.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3740000000000001e-05 2024-09-15T03:47:51.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.221 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.221 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:51.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845247+0000", 2024-09-15T03:47:51.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2779999999999999e-06 2024-09-15T03:47:51.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.222 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.222 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845308+0000", 2024-09-15T03:47:51.222 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0702000000000003e-05 2024-09-15T03:47:51.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.222 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:51.222 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845317+0000", 2024-09-15T03:47:51.222 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3770000000000002e-06 2024-09-15T03:47:51.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.222 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:51.222 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.845413+0000", 2024-09-15T03:47:51.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5356e-05 2024-09-15T03:47:51.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:51.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847421+0000", 2024-09-15T03:47:51.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020077200000000002 2024-09-15T03:47:51.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:51.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847466+0000", 2024-09-15T03:47:51.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5754999999999998e-05 2024-09-15T03:47:51.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:51.224 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.847572+0000", 2024-09-15T03:47:51.224 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010594000000000001 2024-09-15T03:47:51.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:51.224 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:46:57.859753+0000", 2024-09-15T03:47:51.224 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012180412999999999 2024-09-15T03:47:51.224 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.224 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:51.225 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.225 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as1 60/28 ECSubWrite(tid=73, reqid=client.4521.0:424, at_version=60'41, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:51.225 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:51.225 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 44.271172247999999, 2024-09-15T03:47:51.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01137208, 2024-09-15T03:47:51.225 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:51.225 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:51.226 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:51.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.226 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:51.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:51.226 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.226 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:51.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900411+0000", 2024-09-15T03:47:51.226 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:51.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900412+0000", 2024-09-15T03:47:51.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7599999999999996e-07 2024-09-15T03:47:51.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:51.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:51.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7986999999999999e-05 2024-09-15T03:47:51.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:51.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900445+0000", 2024-09-15T03:47:51.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.468e-05 2024-09-15T03:47:51.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:51.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900451+0000", 2024-09-15T03:47:51.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5799999999999997e-06 2024-09-15T03:47:51.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900507+0000", 2024-09-15T03:47:51.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5855e-05 2024-09-15T03:47:51.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:51.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900530+0000", 2024-09-15T03:47:51.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2929e-05 2024-09-15T03:47:51.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:51.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900640+0000", 2024-09-15T03:47:51.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000110536 2024-09-15T03:47:51.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:51.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900648+0000", 2024-09-15T03:47:51.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9470000000000003e-06 2024-09-15T03:47:51.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:51.230 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900764+0000", 2024-09-15T03:47:51.230 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011518099999999999 2024-09-15T03:47:51.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.230 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:51.230 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.900803+0000", 2024-09-15T03:47:51.230 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9941999999999998e-05 2024-09-15T03:47:51.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.230 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:51.230 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:06.911783+0000", 2024-09-15T03:47:51.230 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010979667 2024-09-15T03:47:51.230 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.230 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:51.231 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.231 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.231 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.231 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:517 3.3s1 3:caaee79c:::smithi00132172-10:head [read 0~3145728,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:51.231 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.947050+0000", 2024-09-15T03:47:51.231 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 35.224533315000002, 2024-09-15T03:47:51.231 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012110134, 2024-09-15T03:47:51.231 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:51.231 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:51.231 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:51.231 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:51.231 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:51.231 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 517 2024-09-15T03:47:51.231 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.232 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:51.232 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.232 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:51.232 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947050+0000", 2024-09-15T03:47:51.232 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.232 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.232 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.232 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:51.232 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947050+0000", 2024-09-15T03:47:51.232 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.232 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.232 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.232 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:51.232 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947052+0000", 2024-09-15T03:47:51.233 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9099999999999999e-06 2024-09-15T03:47:51.233 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.233 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.233 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:51.233 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947060+0000", 2024-09-15T03:47:51.233 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8529999999999992e-06 2024-09-15T03:47:51.233 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.233 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.233 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:51.233 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947074+0000", 2024-09-15T03:47:51.233 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4344e-05 2024-09-15T03:47:51.233 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.233 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.233 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:51.234 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947082+0000", 2024-09-15T03:47:51.234 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9249999999999995e-06 2024-09-15T03:47:51.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.234 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.234 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.234 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947138+0000", 2024-09-15T03:47:51.234 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6054000000000002e-05 2024-09-15T03:47:51.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.234 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.234 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:51.234 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947339+0000", 2024-09-15T03:47:51.235 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000200829 2024-09-15T03:47:51.235 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.235 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.235 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:51.235 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.959160+0000", 2024-09-15T03:47:51.235 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011821219000000001 2024-09-15T03:47:51.235 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.235 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:51.235 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.235 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.235 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.235 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:518 3.3s1 3:caaee79c:::smithi00132172-10:head [read 0~3145728,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:51.235 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.947091+0000", 2024-09-15T03:47:51.235 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 35.224492173000002, 2024-09-15T03:47:51.236 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017608233000000001, 2024-09-15T03:47:51.236 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:51.236 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:51.236 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:51.236 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:51.236 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:51.236 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 518 2024-09-15T03:47:51.236 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.236 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:51.236 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.236 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:51.236 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947091+0000", 2024-09-15T03:47:51.236 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.237 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.237 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:51.237 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947091+0000", 2024-09-15T03:47:51.237 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.237 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.237 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:51.237 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947092+0000", 2024-09-15T03:47:51.237 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7800000000000002e-07 2024-09-15T03:47:51.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.238 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.238 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:51.238 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947095+0000", 2024-09-15T03:47:51.238 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2959999999999999e-06 2024-09-15T03:47:51.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.238 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.238 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:51.238 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947102+0000", 2024-09-15T03:47:51.238 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7469999999999999e-06 2024-09-15T03:47:51.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.238 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.238 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:51.239 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947106+0000", 2024-09-15T03:47:51.239 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6550000000000003e-06 2024-09-15T03:47:51.239 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.239 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.239 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.239 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947622+0000", 2024-09-15T03:47:51.239 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000515727 2024-09-15T03:47:51.239 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.239 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.239 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:51.239 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947751+0000", 2024-09-15T03:47:51.239 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012920600000000001 2024-09-15T03:47:51.239 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.240 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.240 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:51.240 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.964699+0000", 2024-09-15T03:47:51.240 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.016948123999999998 2024-09-15T03:47:51.240 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.240 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:51.240 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.240 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.240 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.240 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:519 3.3s1 3:caaee79c:::smithi00132172-10:head [read 0~3145728,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:51.240 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:51.240 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 35.224458466999998, 2024-09-15T03:47:51.241 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.021743243999999998, 2024-09-15T03:47:51.241 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:51.241 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:51.241 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:51.241 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:51.241 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:51.241 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 519 2024-09-15T03:47:51.241 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.241 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:51.241 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.241 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:51.241 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:51.241 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.242 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.242 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:51.242 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:51.242 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.242 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.242 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:51.242 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947125+0000", 2024-09-15T03:47:51.242 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5999999999999999e-07 2024-09-15T03:47:51.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.242 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.242 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:51.243 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947130+0000", 2024-09-15T03:47:51.243 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7720000000000002e-06 2024-09-15T03:47:51.243 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.243 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.243 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:51.243 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947137+0000", 2024-09-15T03:47:51.243 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2370000000000003e-06 2024-09-15T03:47:51.243 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.243 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.243 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:51.243 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.947141+0000", 2024-09-15T03:47:51.243 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8809999999999998e-06 2024-09-15T03:47:51.243 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.244 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.244 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.244 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.948008+0000", 2024-09-15T03:47:51.244 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00086656700000000005 2024-09-15T03:47:51.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.244 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.244 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:51.244 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.948113+0000", 2024-09-15T03:47:51.244 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104904 2024-09-15T03:47:51.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.244 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.244 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:51.244 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.968868+0000", 2024-09-15T03:47:51.244 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.020755422999999999 2024-09-15T03:47:51.245 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.245 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:51.245 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.245 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.245 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.3s1 69/68 ECSubReadReply(tid=163, attrs_read=0))", 2024-09-15T03:47:51.245 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:51.245 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 35.217882723000002, 2024-09-15T03:47:51.245 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011102266, 2024-09-15T03:47:51.245 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:51.245 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:51.245 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:51.245 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.246 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:51.246 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:51.246 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.246 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.246 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:51.246 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:51.246 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.246 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.246 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:51.246 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.953701+0000", 2024-09-15T03:47:51.246 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1199999999999998e-07 2024-09-15T03:47:51.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.247 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.247 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:51.247 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956300+0000", 2024-09-15T03:47:51.247 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025987459999999999 2024-09-15T03:47:51.247 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.247 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.247 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:51.247 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956314+0000", 2024-09-15T03:47:51.247 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4469e-05 2024-09-15T03:47:51.247 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.247 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.247 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:51.247 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956319+0000", 2024-09-15T03:47:51.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.746e-06 2024-09-15T03:47:51.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.248 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.248 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.248 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.959591+0000", 2024-09-15T03:47:51.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032721510000000001 2024-09-15T03:47:51.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.248 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.248 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:51.248 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.964803+0000", 2024-09-15T03:47:51.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0052117420000000001 2024-09-15T03:47:51.248 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.248 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:51.249 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.249 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.249 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.3s1 69/68 ECSubReadReply(tid=164, attrs_read=0))", 2024-09-15T03:47:51.249 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:51.249 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 35.215258290000001, 2024-09-15T03:47:51.249 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012633276000000001, 2024-09-15T03:47:51.249 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:51.249 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:51.249 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:51.249 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.249 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:51.249 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:51.249 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.250 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.250 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.250 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:51.250 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:51.250 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.250 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.250 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.250 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:51.250 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.956325+0000", 2024-09-15T03:47:51.250 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5900000000000003e-07 2024-09-15T03:47:51.253 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.253 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.253 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:51.253 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957786+0000", 2024-09-15T03:47:51.253 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001460755 2024-09-15T03:47:51.253 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.253 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.253 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:51.253 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957807+0000", 2024-09-15T03:47:51.254 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0937e-05 2024-09-15T03:47:51.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.254 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.254 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:51.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.957814+0000", 2024-09-15T03:47:51.254 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5989999999999998e-06 2024-09-15T03:47:51.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.254 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.254 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.964911+0000", 2024-09-15T03:47:51.254 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0070976370000000004 2024-09-15T03:47:51.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.254 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.255 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:51.255 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:15.968958+0000", 2024-09-15T03:47:51.255 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.004046989 2024-09-15T03:47:51.255 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.255 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:51.255 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.255 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:567 3.ds0 3:b166213b:::smithi00132172-21:head [rollback 3] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:51.255 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.947576+0000", 2024-09-15T03:47:51.255 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 30.224007361999998, 2024-09-15T03:47:51.255 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013871319999999999, 2024-09-15T03:47:51.255 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:51.255 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:51.256 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:51.256 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:51.256 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:51.256 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 567 2024-09-15T03:47:51.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.256 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:51.256 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.256 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:51.256 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947576+0000", 2024-09-15T03:47:51.256 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.256 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.256 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:51.257 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947576+0000", 2024-09-15T03:47:51.257 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.257 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.257 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.257 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:51.257 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947577+0000", 2024-09-15T03:47:51.257 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.37e-07 2024-09-15T03:47:51.257 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.257 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.257 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:51.257 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947580+0000", 2024-09-15T03:47:51.257 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5710000000000002e-06 2024-09-15T03:47:51.257 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.258 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.258 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:51.258 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947588+0000", 2024-09-15T03:47:51.258 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8939999999999993e-06 2024-09-15T03:47:51.258 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.258 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.258 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:51.258 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.947593+0000", 2024-09-15T03:47:51.258 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7049999999999998e-06 2024-09-15T03:47:51.258 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.258 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.258 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.258 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948070+0000", 2024-09-15T03:47:51.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047685099999999998 2024-09-15T03:47:51.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:51.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948177+0000", 2024-09-15T03:47:51.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010765299999999999 2024-09-15T03:47:51.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:51.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948986+0000", 2024-09-15T03:47:51.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000808977 2024-09-15T03:47:51.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:51.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949117+0000", 2024-09-15T03:47:51.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000131032 2024-09-15T03:47:51.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:51.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949133+0000", 2024-09-15T03:47:51.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5798999999999999e-05 2024-09-15T03:47:51.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:51.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950936+0000", 2024-09-15T03:47:51.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00180316 2024-09-15T03:47:51.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.261 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.261 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:51.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950994+0000", 2024-09-15T03:47:51.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7884999999999999e-05 2024-09-15T03:47:51.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.261 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.261 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:51.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951532+0000", 2024-09-15T03:47:51.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00053787699999999997 2024-09-15T03:47:51.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.261 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.261 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:51.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961447+0000", 2024-09-15T03:47:51.262 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0099152790000000008 2024-09-15T03:47:51.262 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.262 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:51.262 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.262 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.262 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:568 3.ds0 3:b166213b:::smithi00132172-21:head [append 0~0] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:51.262 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:20.948048+0000", 2024-09-15T03:47:51.262 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 30.223535436999999, 2024-09-15T03:47:51.262 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013579269, 2024-09-15T03:47:51.262 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:51.262 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:51.263 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:51.263 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:51.263 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:51.263 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 568 2024-09-15T03:47:51.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.263 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:51.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.263 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:51.263 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948048+0000", 2024-09-15T03:47:51.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.263 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:51.263 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948048+0000", 2024-09-15T03:47:51.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.264 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.264 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:51.264 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948049+0000", 2024-09-15T03:47:51.264 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7199999999999998e-07 2024-09-15T03:47:51.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.264 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.264 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:51.264 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948056+0000", 2024-09-15T03:47:51.264 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0280000000000003e-06 2024-09-15T03:47:51.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.264 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:51.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948066+0000", 2024-09-15T03:47:51.265 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0141e-05 2024-09-15T03:47:51.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:51.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.948072+0000", 2024-09-15T03:47:51.265 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1519999999999997e-06 2024-09-15T03:47:51.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949221+0000", 2024-09-15T03:47:51.265 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001149125 2024-09-15T03:47:51.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.266 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.266 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:51.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949313+0000", 2024-09-15T03:47:51.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1469000000000001e-05 2024-09-15T03:47:51.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.266 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.266 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:51.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.949818+0000", 2024-09-15T03:47:51.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00050549700000000004 2024-09-15T03:47:51.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.266 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.266 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:51.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950002+0000", 2024-09-15T03:47:51.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018414700000000001 2024-09-15T03:47:51.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.267 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:51.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.950940+0000", 2024-09-15T03:47:51.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00093786100000000001 2024-09-15T03:47:51.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.267 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:51.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951004+0000", 2024-09-15T03:47:51.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3859e-05 2024-09-15T03:47:51.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:51.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951020+0000", 2024-09-15T03:47:51.268 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.628e-05 2024-09-15T03:47:51.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:51.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.951753+0000", 2024-09-15T03:47:51.268 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00073276999999999999 2024-09-15T03:47:51.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:51.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:20.961627+0000", 2024-09-15T03:47:51.269 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0098741679999999991 2024-09-15T03:47:51.269 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.269 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:51.269 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.269 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.269 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.269 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:677 3.3s1 3:caaee79c:::smithi00132172-10:head [copy-from ver 31] snapc 2f=[2d,2b,29,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:51.269 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:30.087209+0000", 2024-09-15T03:47:51.269 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.084374245999999, 2024-09-15T03:47:51.269 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.025257667000000001, 2024-09-15T03:47:51.269 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:51.269 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:51.269 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:51.269 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:51.270 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:51.270 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 677 2024-09-15T03:47:51.270 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.270 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:51.270 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.270 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:51.270 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087209+0000", 2024-09-15T03:47:51.270 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.270 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.270 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.270 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:51.270 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087209+0000", 2024-09-15T03:47:51.270 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.271 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:51.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087211+0000", 2024-09-15T03:47:51.271 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2500000000000001e-06 2024-09-15T03:47:51.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.271 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:51.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087222+0000", 2024-09-15T03:47:51.271 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.113e-05 2024-09-15T03:47:51.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.271 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:51.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087252+0000", 2024-09-15T03:47:51.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9488999999999998e-05 2024-09-15T03:47:51.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:51.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087264+0000", 2024-09-15T03:47:51.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1875999999999999e-05 2024-09-15T03:47:51.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087368+0000", 2024-09-15T03:47:51.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104035 2024-09-15T03:47:51.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:51.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087560+0000", 2024-09-15T03:47:51.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019211400000000001 2024-09-15T03:47:51.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:51.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.104984+0000", 2024-09-15T03:47:51.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017424489000000001 2024-09-15T03:47:51.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:51.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105388+0000", 2024-09-15T03:47:51.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000403686 2024-09-15T03:47:51.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.274 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.274 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:51.274 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105404+0000", 2024-09-15T03:47:51.274 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6268000000000001e-05 2024-09-15T03:47:51.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.274 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.274 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:51.274 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105641+0000", 2024-09-15T03:47:51.274 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023648399999999999 2024-09-15T03:47:51.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.274 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.274 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:51.274 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105662+0000", 2024-09-15T03:47:51.275 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0602999999999999e-05 2024-09-15T03:47:51.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.275 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.275 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:51.275 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.110231+0000", 2024-09-15T03:47:51.275 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0045697089999999999 2024-09-15T03:47:51.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.275 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.275 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:51.275 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.112467+0000", 2024-09-15T03:47:51.275 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002235534 2024-09-15T03:47:51.275 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.275 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:51.275 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.276 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.276 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:678 3.3s1 3:caaee79c:::smithi00132172-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:51.276 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:30.087279+0000", 2024-09-15T03:47:51.276 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.084304791000001, 2024-09-15T03:47:51.276 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.018991965, 2024-09-15T03:47:51.276 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:51.276 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-09-15T03:47:51.276 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:51.276 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:51.276 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:51.276 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 678 2024-09-15T03:47:51.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.277 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:51.277 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.277 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:51.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087279+0000", 2024-09-15T03:47:51.277 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.277 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.277 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:51.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087279+0000", 2024-09-15T03:47:51.277 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.277 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.277 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:51.278 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087280+0000", 2024-09-15T03:47:51.278 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.147e-06 2024-09-15T03:47:51.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.278 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.278 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:51.278 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087285+0000", 2024-09-15T03:47:51.278 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2059999999999998e-06 2024-09-15T03:47:51.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.278 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.278 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:51.278 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087303+0000", 2024-09-15T03:47:51.278 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7646e-05 2024-09-15T03:47:51.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.279 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:51.279 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087313+0000", 2024-09-15T03:47:51.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0862999999999999e-05 2024-09-15T03:47:51.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.279 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.279 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087765+0000", 2024-09-15T03:47:51.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045120400000000003 2024-09-15T03:47:51.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.279 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:51.279 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.087962+0000", 2024-09-15T03:47:51.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000197095 2024-09-15T03:47:51.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.280 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.280 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.280 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.105788+0000", 2024-09-15T03:47:51.280 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017826143999999999 2024-09-15T03:47:51.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.280 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.280 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:51.280 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.106034+0000", 2024-09-15T03:47:51.280 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024586999999999998 2024-09-15T03:47:51.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.280 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.280 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:51.281 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.106270+0000", 2024-09-15T03:47:51.281 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023678999999999999 2024-09-15T03:47:51.281 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.281 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:51.281 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.281 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.281 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.281 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4521.0:688 3.3s1 3:c7c3ef16:::smithi00132172-25:head [create,setxattr og^`]cfbTbdPrNph_pkYgeiTrgatpl] (805) in=31b,setxattr `Trb[\\aaaksnX\\dQOYP^TPXpYtmQ_maX\\[hbePkT (981) in=40b,setxattr UkQ^W[[sWh]fSrWg]^\\oZpWaamVkVUnZYbR[Vql\\ (663) in=40b,setxattr ofr[Os^]tjr^jicnNpPaUeXQTfldVQPdSPjcgWcT (7228) in=40b,setxattr Vl`NPsTVpnj_gO_srrbrZ\\eq (805) in=24b,setxattr ^_mcpljcT[qX_UeeYZY\\_ZTSsSsojP\\esUkVphT] (1963) in=40b,setxattr [oY^bSn[ealaPaOhprWW_SNaa]VhaXN[jmo`YYgt (138) in=40b,setxattr tT[SSseaPa\\UPS`\\g\\pRqsd`^lo__arabQlQmob` (663) in=40b,setxattr o^\\]jXsesZhoknr\\dagnjidgsPOhkfg[ffVe[PQW (1259) in=40b,setxattr c^VX_^n^gsh^P`aVbXpaS\\cgkpZO]OmXmchTtWZd (1612) in=40b,setxattr _^aeiTb[Uhh`rNNUOZOPt_Z_Qmeps`XXQ^qidetf (629) in=40b,setxattr ms[glaQ`N\\pqbicVTmgp^\\Rg]h`fefatkfg\\js_[ (139) in=40b,setxattr O\\gsPtYmrReq]if[pkOgPtqS`]Nrq_eYpsefkSgl (700) in=40b,setxattr RgefRtSsOT`jTQb^h\\igU`_lbscW`fgkXroNdmea (1751) in=40b,setxattr sgaV_cSaelbWQYill^k`RsVajcXpUSfW]eNd_nat (807) in=40b,setxattr P\\lm\\r`\\jpssZmkPqcQSbje_]]`YOSZTeRUUSSY[ (1365) in=40b,setxattr aa^ePena\\opOdeWXTj[XU]N^jQS]bens\\Tc_g]ea (16140) in=40b,setxattr R[qckQQO`n[[egbgcZsdtUbdPdnPNlOmsbP_OXOQ (1227) in=40b,setxattr SpRUpSaQZ]nlQ]kpXZnOWSUS]tsmf`qZiPTkq]Yi (209) in=40b,setxattr Y[Y[X`O_XaVOb]NZ_PgmhRX]Vds_dfbp]h[SPkYb (417) in=40b] snapc 2f=[2d,2b,29,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:51.281 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:30.088531+0000", 2024-09-15T03:47:51.281 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.083052056, 2024-09-15T03:47:51.281 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013293503999999999, 2024-09-15T03:47:51.281 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:51.281 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:51.282 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-09-15T03:47:51.282 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4521", 2024-09-15T03:47:51.282 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:51.282 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 688 2024-09-15T03:47:51.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.282 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:51.282 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.282 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:51.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088531+0000", 2024-09-15T03:47:51.282 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.282 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.282 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:51.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088531+0000", 2024-09-15T03:47:51.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.283 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.283 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:51.283 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088533+0000", 2024-09-15T03:47:51.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5379999999999999e-06 2024-09-15T03:47:51.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.283 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.283 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:51.283 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088662+0000", 2024-09-15T03:47:51.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000129002 2024-09-15T03:47:51.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.283 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.283 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:51.284 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088688+0000", 2024-09-15T03:47:51.284 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6084999999999998e-05 2024-09-15T03:47:51.284 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.284 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.284 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:51.284 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088696+0000", 2024-09-15T03:47:51.284 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0490000000000004e-06 2024-09-15T03:47:51.284 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.284 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.284 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.284 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.088746+0000", 2024-09-15T03:47:51.284 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0074000000000002e-05 2024-09-15T03:47:51.284 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.285 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.285 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-09-15T03:47:51.285 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.089084+0000", 2024-09-15T03:47:51.285 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033810200000000001 2024-09-15T03:47:51.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.285 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.285 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:51.285 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090498+0000", 2024-09-15T03:47:51.285 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001414205 2024-09-15T03:47:51.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.285 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.285 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:51.285 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090936+0000", 2024-09-15T03:47:51.286 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043802799999999997 2024-09-15T03:47:51.286 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.286 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.286 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:51.286 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.090951+0000", 2024-09-15T03:47:51.286 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4953999999999999e-05 2024-09-15T03:47:51.286 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.286 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.286 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:51.286 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.091069+0000", 2024-09-15T03:47:51.286 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000117236 2024-09-15T03:47:51.286 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.286 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.286 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:51.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.091128+0000", 2024-09-15T03:47:51.287 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9580999999999998e-05 2024-09-15T03:47:51.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.287 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.287 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-09-15T03:47:51.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.097312+0000", 2024-09-15T03:47:51.287 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0061838129999999998 2024-09-15T03:47:51.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.287 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.287 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:51.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.101825+0000", 2024-09-15T03:47:51.287 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0045128369999999996 2024-09-15T03:47:51.287 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.287 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:51.288 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.288 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.288 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.288 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.5s4 83/80 ECSubWrite(tid=24, reqid=client.4521.0:675, at_version=83'37, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:51.288 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-15T03:47:30.126290+0000", 2024-09-15T03:47:51.288 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.045293213000001, 2024-09-15T03:47:51.288 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.014228881000000001, 2024-09-15T03:47:51.288 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-09-15T03:47:51.288 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-09-15T03:47:51.288 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-09-15T03:47:51.288 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.288 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-09-15T03:47:51.288 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.126290+0000", 2024-09-15T03:47:51.289 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.289 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.289 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-09-15T03:47:51.289 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.126290+0000", 2024-09-15T03:47:51.289 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-09-15T03:47:51.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.289 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.289 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-09-15T03:47:51.289 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.126291+0000", 2024-09-15T03:47:51.289 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6499999999999998e-07 2024-09-15T03:47:51.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.289 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.290 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-09-15T03:47:51.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128867+0000", 2024-09-15T03:47:51.290 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025759749999999999 2024-09-15T03:47:51.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.290 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.290 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-09-15T03:47:51.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128881+0000", 2024-09-15T03:47:51.290 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3823e-05 2024-09-15T03:47:51.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.290 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.290 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-09-15T03:47:51.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128888+0000", 2024-09-15T03:47:51.290 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.605e-06 2024-09-15T03:47:51.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.291 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.291 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-09-15T03:47:51.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128949+0000", 2024-09-15T03:47:51.291 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0949999999999998e-05 2024-09-15T03:47:51.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.291 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.291 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-09-15T03:47:51.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.128957+0000", 2024-09-15T03:47:51.291 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4760000000000001e-06 2024-09-15T03:47:51.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.291 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.291 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:51.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.129323+0000", 2024-09-15T03:47:51.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036638900000000001 2024-09-15T03:47:51.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.292 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.292 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:51.292 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.129331+0000", 2024-09-15T03:47:51.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1319999999999994e-06 2024-09-15T03:47:51.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.292 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.292 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:51.292 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.129567+0000", 2024-09-15T03:47:51.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000235393 2024-09-15T03:47:51.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.292 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.293 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-09-15T03:47:51.293 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.129586+0000", 2024-09-15T03:47:51.293 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9517999999999998e-05 2024-09-15T03:47:51.293 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-15T03:47:51.293 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-15T03:47:51.293 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-09-15T03:47:51.293 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-09-15T03:47:30.140519+0000", 2024-09-15T03:47:51.293 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010932855 2024-09-15T03:47:51.293 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.293 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:51.293 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.293 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-15T03:47:51.293 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-15T03:47:51.293 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:51.297 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:51.297 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:51.298 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:51.298 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:51.298 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:47:51.299 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:51.299 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:51.299 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-15T03:47:51.299 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:51.300 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T03:47:51.300 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:51.300 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T03:47:51.301 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:51.296+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:51.398 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:51.398 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:51.399 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:51.398+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:51.407 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:47:51.500 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:51.498+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:51.504 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:51.504 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:51.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:47:51.600 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:51.598+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:51.609 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:51.609 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:51.618 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:47:51.701 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:51.701+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:51.714 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:51.714 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:51.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:47:51.803 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:51.801+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:51.818 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:51.818 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:51.826 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:47:51.888 INFO:tasks.rados.rados.0.smithi001.stdout:354: done (0 left) 2024-09-15T03:47:51.888 INFO:tasks.rados.rados.0.smithi001.stdout:355: copy_from oid 45 from oid 2 current snap is 47 2024-09-15T03:47:51.888 INFO:tasks.rados.rados.0.smithi001.stdout:356: read oid 19 snap 45 2024-09-15T03:47:51.888 INFO:tasks.rados.rados.0.smithi001.stdout:356: expect (ObjNum 66 snap 6 seq_num 66) 2024-09-15T03:47:51.888 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 38 current snap is 47 2024-09-15T03:47:51.888 INFO:tasks.rados.rados.0.smithi001.stdout:357: seq_num 135 ranges {3260416=720896,3981312=622592,4603904=688128,5292032=245760} 2024-09-15T03:47:51.911 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:51.910+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:51.912 INFO:tasks.rados.rados.0.smithi001.stdout:357: writing smithi00132172-38 from 3260416 to 3981312 tid 1 2024-09-15T03:47:51.915 INFO:tasks.rados.rados.0.smithi001.stdout:357: writing smithi00132172-38 from 3981312 to 4603904 tid 2 2024-09-15T03:47:51.919 INFO:tasks.rados.rados.0.smithi001.stdout:357: writing smithi00132172-38 from 4603904 to 5292032 tid 3 2024-09-15T03:47:51.920 INFO:tasks.rados.rados.0.smithi001.stdout:357: writing smithi00132172-38 from 5292032 to 5537792 tid 4 2024-09-15T03:47:51.920 INFO:tasks.rados.rados.0.smithi001.stdout:358: rollback oid 32 current snap is 47 2024-09-15T03:47:51.921 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 32 to 20 2024-09-15T03:47:51.921 INFO:tasks.rados.rados.0.smithi001.stdout:356: expect (ObjNum 66 snap 6 seq_num 66) 2024-09-15T03:47:51.962 INFO:tasks.rados.rados.0.smithi001.stdout:356: done (3 left) 2024-09-15T03:47:51.962 INFO:tasks.rados.rados.0.smithi001.stdout:359: snap_remove snap 0 2024-09-15T03:47:51.970 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:51.970 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:51.979 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:47:52.005 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:52.004+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:52.085 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:52.085 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:52.093 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:47:52.106 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:52.105+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:52.193 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:52.193 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:52.202 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:47:52.207 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:52.206+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:52.303 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:52.303 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:52.308 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:52.306+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:52.312 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:47:52.408 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:52.408+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:52.413 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:52.413 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:52.421 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:47:52.509 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:52.509+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:52.562 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:52.563 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:52.571 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:47:52.610 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:52.610+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:52.671 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:52.671 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:52.680 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:47:52.725 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:52.724+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:52.781 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:52.781 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:52.789 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:47:52.831 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:52.830+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:52.902 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T03:47:52.907 INFO:tasks.rados.rados.0.smithi001.stdout:359: done (3 left) 2024-09-15T03:47:52.907 INFO:tasks.rados.rados.0.smithi001.stdout:360: snap_remove snap 16 2024-09-15T03:47:52.914 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:52.913+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:52.946 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:52.946 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:52.957 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:47:53.013 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:53.012+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:53.100 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:53.100 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:53.109 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:47:53.152 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:53.151+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:53.209 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:53.210 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:53.216 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:53.215+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:53.218 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:47:53.301 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:47:53.345 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:53.344+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:53.360 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:53.360 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:53.369 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:47:53.418 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:53.416+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:53.418 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:53.418 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:53.433 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:53.433 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:53.449 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:53.449 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:53.463 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:53.464 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:53.479 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:53.479 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:53.494 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:53.494 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:53.509 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:53.509 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:53.515 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:53.515 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-15T03:47:53.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-15T03:47:53.515 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-15T03:47:53.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.515 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:333 3.9s0 3:94e9e019:::smithi00132172-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:53.516 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.856848+0000", 2024-09-15T03:47:53.516 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 57.656160937000003, 2024-09-15T03:47:53.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012151039000000001, 2024-09-15T03:47:53.516 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:53.516 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:53.516 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:53.516 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:53.516 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:53.516 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 333 2024-09-15T03:47:53.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.516 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:53.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:53.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856848+0000", 2024-09-15T03:47:53.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:53.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856848+0000", 2024-09-15T03:47:53.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:53.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856849+0000", 2024-09-15T03:47:53.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5499999999999999e-07 2024-09-15T03:47:53.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:53.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:53.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.378e-06 2024-09-15T03:47:53.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:53.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856854+0000", 2024-09-15T03:47:53.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.653e-06 2024-09-15T03:47:53.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:53.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856856+0000", 2024-09-15T03:47:53.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4059999999999999e-06 2024-09-15T03:47:53.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.862404+0000", 2024-09-15T03:47:53.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0055481310000000004 2024-09-15T03:47:53.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:53.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.862519+0000", 2024-09-15T03:47:53.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011449400000000001 2024-09-15T03:47:53.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864592+0000", 2024-09-15T03:47:53.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020730200000000001 2024-09-15T03:47:53.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:53.520 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864687+0000", 2024-09-15T03:47:53.520 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5531000000000004e-05 2024-09-15T03:47:53.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.520 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:53.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.868999+0000", 2024-09-15T03:47:53.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0043121710000000001 2024-09-15T03:47:53.521 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.521 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:53.521 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.521 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:458 3.9s0 3:97177042:::smithi00132172-18:head [append 0~753664] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:53.521 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.841113+0000", 2024-09-15T03:47:53.522 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 42.671896531999998, 2024-09-15T03:47:53.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0334256719999999, 2024-09-15T03:47:53.522 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:53.522 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:53.522 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:53.522 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:53.522 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:53.522 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 458 2024-09-15T03:47:53.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.522 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:53.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:53.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841113+0000", 2024-09-15T03:47:53.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:53.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841113+0000", 2024-09-15T03:47:53.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:53.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841114+0000", 2024-09-15T03:47:53.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.491e-06 2024-09-15T03:47:53.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:53.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841577+0000", 2024-09-15T03:47:53.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000462571 2024-09-15T03:47:53.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:53.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841589+0000", 2024-09-15T03:47:53.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2119999999999999e-05 2024-09-15T03:47:53.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:53.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841596+0000", 2024-09-15T03:47:53.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0740000000000004e-06 2024-09-15T03:47:53.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841708+0000", 2024-09-15T03:47:53.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112126 2024-09-15T03:47:53.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:47:53.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841712+0000", 2024-09-15T03:47:53.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3639999999999999e-06 2024-09-15T03:47:53.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.856485+0000", 2024-09-15T03:47:53.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0147736949999999 2024-09-15T03:47:53.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:53.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.856699+0000", 2024-09-15T03:47:53.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000213372 2024-09-15T03:47:53.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:53.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.862261+0000", 2024-09-15T03:47:53.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0055625450000000003 2024-09-15T03:47:53.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:53.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.862781+0000", 2024-09-15T03:47:53.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00052013500000000002 2024-09-15T03:47:53.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.527 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:53.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.862885+0000", 2024-09-15T03:47:53.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103455 2024-09-15T03:47:53.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:53.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.863028+0000", 2024-09-15T03:47:53.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014313999999999999 2024-09-15T03:47:53.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:53.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.864079+0000", 2024-09-15T03:47:53.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010511229999999999 2024-09-15T03:47:53.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:53.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874458+0000", 2024-09-15T03:47:53.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010379019999999999 2024-09-15T03:47:53.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:53.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874538+0000", 2024-09-15T03:47:53.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0440999999999998e-05 2024-09-15T03:47:53.529 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.529 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:53.529 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.529 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:459 3.9s0 3:97177042:::smithi00132172-18:head [append 0~720896] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:53.530 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.846766+0000", 2024-09-15T03:47:53.530 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 42.666242908999997, 2024-09-15T03:47:53.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0281560110000001, 2024-09-15T03:47:53.530 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:53.530 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:53.530 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:53.530 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:53.530 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:53.530 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 459 2024-09-15T03:47:53.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.530 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:53.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:53.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.846766+0000", 2024-09-15T03:47:53.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:53.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.846766+0000", 2024-09-15T03:47:53.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:53.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.846769+0000", 2024-09-15T03:47:53.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.323e-06 2024-09-15T03:47:53.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:53.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847323+0000", 2024-09-15T03:47:53.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000554267 2024-09-15T03:47:53.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:53.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847338+0000", 2024-09-15T03:47:53.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.465e-05 2024-09-15T03:47:53.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:53.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847345+0000", 2024-09-15T03:47:53.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6029999999999999e-06 2024-09-15T03:47:53.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847494+0000", 2024-09-15T03:47:53.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000148541 2024-09-15T03:47:53.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:47:53.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847498+0000", 2024-09-15T03:47:53.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7629999999999998e-06 2024-09-15T03:47:53.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.864944+0000", 2024-09-15T03:47:53.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0174461050000001 2024-09-15T03:47:53.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:53.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.865122+0000", 2024-09-15T03:47:53.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000178123 2024-09-15T03:47:53.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:53.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867632+0000", 2024-09-15T03:47:53.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002510249 2024-09-15T03:47:53.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:53.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867775+0000", 2024-09-15T03:47:53.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014326000000000001 2024-09-15T03:47:53.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:53.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868663+0000", 2024-09-15T03:47:53.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00088783799999999995 2024-09-15T03:47:53.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:53.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868802+0000", 2024-09-15T03:47:53.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000139173 2024-09-15T03:47:53.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:53.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874167+0000", 2024-09-15T03:47:53.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.005365117 2024-09-15T03:47:53.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:53.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874842+0000", 2024-09-15T03:47:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00067414199999999997 2024-09-15T03:47:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874922+0000", 2024-09-15T03:47:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0857000000000002e-05 2024-09-15T03:47:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:460 3.9s0 3:97177042:::smithi00132172-18:head [append 0~49152] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:53.537 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.847359+0000", 2024-09-15T03:47:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 42.665650534999997, 2024-09-15T03:47:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0278644889999999, 2024-09-15T03:47:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 460 2024-09-15T03:47:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:53.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847359+0000", 2024-09-15T03:47:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847359+0000", 2024-09-15T03:47:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847360+0000", 2024-09-15T03:47:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.26e-06 2024-09-15T03:47:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847390+0000", 2024-09-15T03:47:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9533999999999999e-05 2024-09-15T03:47:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847406+0000", 2024-09-15T03:47:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6725999999999999e-05 2024-09-15T03:47:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847412+0000", 2024-09-15T03:47:53.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4149999999999998e-06 2024-09-15T03:47:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847522+0000", 2024-09-15T03:47:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011041500000000001 2024-09-15T03:47:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:47:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847523+0000", 2024-09-15T03:47:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.09e-07 2024-09-15T03:47:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868928+0000", 2024-09-15T03:47:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0214053089999999 2024-09-15T03:47:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869043+0000", 2024-09-15T03:47:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000115021 2024-09-15T03:47:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869635+0000", 2024-09-15T03:47:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00059164199999999997 2024-09-15T03:47:53.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869755+0000", 2024-09-15T03:47:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0001205 2024-09-15T03:47:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870274+0000", 2024-09-15T03:47:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00051842900000000005 2024-09-15T03:47:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:53.543 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870322+0000", 2024-09-15T03:47:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8164000000000002e-05 2024-09-15T03:47:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874941+0000", 2024-09-15T03:47:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0046186140000000001 2024-09-15T03:47:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875140+0000", 2024-09-15T03:47:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019890700000000001 2024-09-15T03:47:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875223+0000", 2024-09-15T03:47:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3644000000000001e-05 2024-09-15T03:47:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:461 3.9s0 3:97177042:::smithi00132172-18:head [setxattr _header (57) in=7b] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.847417+0000", 2024-09-15T03:47:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 42.665592431, 2024-09-15T03:47:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0332682499999999, 2024-09-15T03:47:53.545 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 461 2024-09-15T03:47:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847417+0000", 2024-09-15T03:47:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847417+0000", 2024-09-15T03:47:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847418+0000", 2024-09-15T03:47:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8999999999999995e-07 2024-09-15T03:47:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:53.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847421+0000", 2024-09-15T03:47:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1769999999999998e-06 2024-09-15T03:47:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847428+0000", 2024-09-15T03:47:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6969999999999996e-06 2024-09-15T03:47:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847432+0000", 2024-09-15T03:47:53.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3209999999999998e-06 2024-09-15T03:47:53.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847542+0000", 2024-09-15T03:47:53.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011035999999999999 2024-09-15T03:47:53.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:47:53.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847543+0000", 2024-09-15T03:47:53.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1200000000000002e-07 2024-09-15T03:47:53.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875612+0000", 2024-09-15T03:47:53.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0280692330000001 2024-09-15T03:47:53.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:53.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875735+0000", 2024-09-15T03:47:53.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012276600000000001 2024-09-15T03:47:53.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:53.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876307+0000", 2024-09-15T03:47:53.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00057198600000000002 2024-09-15T03:47:53.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:53.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876408+0000", 2024-09-15T03:47:53.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010148400000000001 2024-09-15T03:47:53.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:53.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876508+0000", 2024-09-15T03:47:53.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9328999999999997e-05 2024-09-15T03:47:53.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876546+0000", 2024-09-15T03:47:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8361000000000001e-05 2024-09-15T03:47:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876602+0000", 2024-09-15T03:47:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6412999999999997e-05 2024-09-15T03:47:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.880589+0000", 2024-09-15T03:47:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0039866190000000003 2024-09-15T03:47:53.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.880685+0000", 2024-09-15T03:47:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6001999999999993e-05 2024-09-15T03:47:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:462 3.9s0 3:97177042:::smithi00132172-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.847436+0000", 2024-09-15T03:47:53.553 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 42.665573473000002, 2024-09-15T03:47:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.03734576, 2024-09-15T03:47:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 462 2024-09-15T03:47:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847436+0000", 2024-09-15T03:47:53.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847436+0000", 2024-09-15T03:47:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847436+0000", 2024-09-15T03:47:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3099999999999998e-07 2024-09-15T03:47:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:53.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847438+0000", 2024-09-15T03:47:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9570000000000001e-06 2024-09-15T03:47:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847445+0000", 2024-09-15T03:47:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.238e-06 2024-09-15T03:47:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847454+0000", 2024-09-15T03:47:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5930000000000006e-06 2024-09-15T03:47:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847562+0000", 2024-09-15T03:47:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108448 2024-09-15T03:47:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:47:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847563+0000", 2024-09-15T03:47:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0200000000000002e-07 2024-09-15T03:47:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876482+0000", 2024-09-15T03:47:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.028918658 2024-09-15T03:47:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876593+0000", 2024-09-15T03:47:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011145300000000001 2024-09-15T03:47:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.881189+0000", 2024-09-15T03:47:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0045956139999999996 2024-09-15T03:47:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.881293+0000", 2024-09-15T03:47:53.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103875 2024-09-15T03:47:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.884782+0000", 2024-09-15T03:47:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003488891 2024-09-15T03:47:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.5s6 65/64 ECSubWrite(tid=100, reqid=osd.1.0:100, at_version=65'35, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.865131+0000", 2024-09-15T03:47:53.559 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 41.647878073000001, 2024-09-15T03:47:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0096540099999999993, 2024-09-15T03:47:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.865131+0000", 2024-09-15T03:47:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:53.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.865131+0000", 2024-09-15T03:47:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.865132+0000", 2024-09-15T03:47:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1399999999999996e-07 2024-09-15T03:47:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.865151+0000", 2024-09-15T03:47:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9151e-05 2024-09-15T03:47:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.865166+0000", 2024-09-15T03:47:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4922999999999999e-05 2024-09-15T03:47:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.865173+0000", 2024-09-15T03:47:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0500000000000003e-06 2024-09-15T03:47:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.873827+0000", 2024-09-15T03:47:53.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0086539719999999994 2024-09-15T03:47:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.873837+0000", 2024-09-15T03:47:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5969999999999998e-06 2024-09-15T03:47:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874178+0000", 2024-09-15T03:47:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00034180199999999999 2024-09-15T03:47:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874191+0000", 2024-09-15T03:47:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2476999999999999e-05 2024-09-15T03:47:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874240+0000", 2024-09-15T03:47:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8615999999999997e-05 2024-09-15T03:47:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874252+0000", 2024-09-15T03:47:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2480000000000001e-05 2024-09-15T03:47:53.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:53.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874785+0000", 2024-09-15T03:47:53.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00053322799999999996 2024-09-15T03:47:53.565 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.565 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:53.565 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.565 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:561 3.9s0 3:9cd57211:::smithi00132172-16:head [append 0~557056] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:53.565 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:20.935324+0000", 2024-09-15T03:47:53.565 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 32.577685772000002, 2024-09-15T03:47:53.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012459909999999999, 2024-09-15T03:47:53.566 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:53.566 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:53.566 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:53.566 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:53.566 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:53.566 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 561 2024-09-15T03:47:53.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.566 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:53.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:53.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935324+0000", 2024-09-15T03:47:53.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:53.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935324+0000", 2024-09-15T03:47:53.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:53.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935326+0000", 2024-09-15T03:47:53.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8909999999999999e-06 2024-09-15T03:47:53.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:53.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935587+0000", 2024-09-15T03:47:53.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026019699999999999 2024-09-15T03:47:53.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:53.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935610+0000", 2024-09-15T03:47:53.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2949e-05 2024-09-15T03:47:53.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:53.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935618+0000", 2024-09-15T03:47:53.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6289999999999999e-06 2024-09-15T03:47:53.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935688+0000", 2024-09-15T03:47:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9801999999999993e-05 2024-09-15T03:47:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-15T03:47:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935698+0000", 2024-09-15T03:47:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0103e-05 2024-09-15T03:47:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939626+0000", 2024-09-15T03:47:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0039276759999999997 2024-09-15T03:47:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:53.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939844+0000", 2024-09-15T03:47:53.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000217741 2024-09-15T03:47:53.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:53.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941127+0000", 2024-09-15T03:47:53.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001283719 2024-09-15T03:47:53.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941395+0000", 2024-09-15T03:47:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026823800000000002 2024-09-15T03:47:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941448+0000", 2024-09-15T03:47:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2751999999999999e-05 2024-09-15T03:47:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:53.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941646+0000", 2024-09-15T03:47:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019804100000000001 2024-09-15T03:47:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943664+0000", 2024-09-15T03:47:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020181090000000001 2024-09-15T03:47:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.947709+0000", 2024-09-15T03:47:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0040445059999999998 2024-09-15T03:47:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.947783+0000", 2024-09-15T03:47:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4556999999999998e-05 2024-09-15T03:47:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:562 3.9s0 3:9cd57211:::smithi00132172-16:head [append 0~638976] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:20.938590+0000", 2024-09-15T03:47:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 32.574418987999998, 2024-09-15T03:47:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011368382, 2024-09-15T03:47:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:53.575 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 562 2024-09-15T03:47:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.938590+0000", 2024-09-15T03:47:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:53.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.938590+0000", 2024-09-15T03:47:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.938592+0000", 2024-09-15T03:47:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.623e-06 2024-09-15T03:47:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939253+0000", 2024-09-15T03:47:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00066056899999999996 2024-09-15T03:47:53.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939271+0000", 2024-09-15T03:47:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8482e-05 2024-09-15T03:47:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939280+0000", 2024-09-15T03:47:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5569999999999996e-06 2024-09-15T03:47:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941736+0000", 2024-09-15T03:47:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024565640000000001 2024-09-15T03:47:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941861+0000", 2024-09-15T03:47:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012484 2024-09-15T03:47:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943223+0000", 2024-09-15T03:47:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0013619260000000001 2024-09-15T03:47:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943354+0000", 2024-09-15T03:47:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013100399999999999 2024-09-15T03:47:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943376+0000", 2024-09-15T03:47:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1829999999999999e-05 2024-09-15T03:47:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943579+0000", 2024-09-15T03:47:53.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020374400000000001 2024-09-15T03:47:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943904+0000", 2024-09-15T03:47:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000324846 2024-09-15T03:47:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949897+0000", 2024-09-15T03:47:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0059927879999999998 2024-09-15T03:47:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:53.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949959+0000", 2024-09-15T03:47:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1608999999999999e-05 2024-09-15T03:47:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:565 3.9s0 3:9cd57211:::smithi00132172-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:20.943038+0000", 2024-09-15T03:47:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 32.569971643999999, 2024-09-15T03:47:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011190904, 2024-09-15T03:47:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:53.582 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 565 2024-09-15T03:47:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943038+0000", 2024-09-15T03:47:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943038+0000", 2024-09-15T03:47:53.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943038+0000", 2024-09-15T03:47:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6800000000000001e-07 2024-09-15T03:47:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943040+0000", 2024-09-15T03:47:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3700000000000002e-06 2024-09-15T03:47:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943047+0000", 2024-09-15T03:47:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1299999999999998e-06 2024-09-15T03:47:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943050+0000", 2024-09-15T03:47:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6229999999999999e-06 2024-09-15T03:47:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.947990+0000", 2024-09-15T03:47:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0049398690000000004 2024-09-15T03:47:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.948080+0000", 2024-09-15T03:47:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9721000000000004e-05 2024-09-15T03:47:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.952263+0000", 2024-09-15T03:47:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.004183417 2024-09-15T03:47:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.952374+0000", 2024-09-15T03:47:53.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110435 2024-09-15T03:47:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.954229+0000", 2024-09-15T03:47:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018548709999999999 2024-09-15T03:47:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:569 3.9s0 3:97177042:::smithi00132172-18:head [rmxattr Z in=1b,rmxattr ZVpala^a[hcaro^RXmp_SYlliaqih_SpraepZfQX in=40b,rmxattr aNhZ\\ePhXcfSNf\\hQ^orWZst^jcpVp[skbdkms]j in=40b,rmxattr ehO_catUjgbjTPQOspNgZXioh\\UidVPlqcarXe]r in=40b,rmxattr hTUN^iUTTOa\\lp\\napoiZNkQnN_jfWcgjVeQRahm in=40b] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:20.943054+0000", 2024-09-15T03:47:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 32.569955458000003, 2024-09-15T03:47:53.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010701584, 2024-09-15T03:47:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 569 2024-09-15T03:47:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943054+0000", 2024-09-15T03:47:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943054+0000", 2024-09-15T03:47:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943054+0000", 2024-09-15T03:47:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:47:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943058+0000", 2024-09-15T03:47:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.631e-06 2024-09-15T03:47:53.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943063+0000", 2024-09-15T03:47:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3800000000000002e-06 2024-09-15T03:47:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943067+0000", 2024-09-15T03:47:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4829999999999999e-06 2024-09-15T03:47:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.948103+0000", 2024-09-15T03:47:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0050365490000000004 2024-09-15T03:47:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.948602+0000", 2024-09-15T03:47:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00049903600000000001 2024-09-15T03:47:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949373+0000", 2024-09-15T03:47:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00077054200000000004 2024-09-15T03:47:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949630+0000", 2024-09-15T03:47:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00025681499999999997 2024-09-15T03:47:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949645+0000", 2024-09-15T03:47:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5704000000000001e-05 2024-09-15T03:47:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949690+0000", 2024-09-15T03:47:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4413999999999999e-05 2024-09-15T03:47:53.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949979+0000", 2024-09-15T03:47:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00028949100000000001 2024-09-15T03:47:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.951859+0000", 2024-09-15T03:47:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018798179999999999 2024-09-15T03:47:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:53.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.953755+0000", 2024-09-15T03:47:53.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018963109999999999 2024-09-15T03:47:53.594 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.594 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:53.594 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.594 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:667 3.4s0 3:2d1c787c:::smithi00132172-45:head [read 0~1703936,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:53.594 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:29.978831+0000", 2024-09-15T03:47:53.594 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 23.534178043000001, 2024-09-15T03:47:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0098251169999999995, 2024-09-15T03:47:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 667 2024-09-15T03:47:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978831+0000", 2024-09-15T03:47:53.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978831+0000", 2024-09-15T03:47:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978832+0000", 2024-09-15T03:47:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9299999999999999e-07 2024-09-15T03:47:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978834+0000", 2024-09-15T03:47:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3429999999999998e-06 2024-09-15T03:47:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978838+0000", 2024-09-15T03:47:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8809999999999998e-06 2024-09-15T03:47:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978840+0000", 2024-09-15T03:47:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.216e-06 2024-09-15T03:47:53.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978980+0000", 2024-09-15T03:47:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000140214 2024-09-15T03:47:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978989+0000", 2024-09-15T03:47:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4079999999999998e-06 2024-09-15T03:47:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.981199+0000", 2024-09-15T03:47:53.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022103539999999999 2024-09-15T03:47:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.981288+0000", 2024-09-15T03:47:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8819000000000005e-05 2024-09-15T03:47:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.988656+0000", 2024-09-15T03:47:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0073684889999999998 2024-09-15T03:47:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:53.599 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:668 3.4s0 3:2d1c787c:::smithi00132172-45:head [read 0~1703936,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:29.978875+0000", 2024-09-15T03:47:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 23.534133926999999, 2024-09-15T03:47:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011314753, 2024-09-15T03:47:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 668 2024-09-15T03:47:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978875+0000", 2024-09-15T03:47:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978875+0000", 2024-09-15T03:47:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:53.601 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978876+0000", 2024-09-15T03:47:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.46e-07 2024-09-15T03:47:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978878+0000", 2024-09-15T03:47:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4399999999999999e-06 2024-09-15T03:47:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978884+0000", 2024-09-15T03:47:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8930000000000002e-06 2024-09-15T03:47:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.602 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978887+0000", 2024-09-15T03:47:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8040000000000002e-06 2024-09-15T03:47:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.979013+0000", 2024-09-15T03:47:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012650899999999999 2024-09-15T03:47:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.979020+0000", 2024-09-15T03:47:53.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8159999999999996e-06 2024-09-15T03:47:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.981562+0000", 2024-09-15T03:47:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002542267 2024-09-15T03:47:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.981653+0000", 2024-09-15T03:47:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0509999999999999e-05 2024-09-15T03:47:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:53.604 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.990190+0000", 2024-09-15T03:47:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0085371679999999995 2024-09-15T03:47:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:692 3.9s0 3:97177042:::smithi00132172-18:head [copy-from ver 4] snapc 2f=[2d,2b,29,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.088835+0000", 2024-09-15T03:47:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 23.424174620999999, 2024-09-15T03:47:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.016647229999999999, 2024-09-15T03:47:53.605 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 692 2024-09-15T03:47:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088835+0000", 2024-09-15T03:47:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088835+0000", 2024-09-15T03:47:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088837+0000", 2024-09-15T03:47:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8670000000000001e-06 2024-09-15T03:47:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088843+0000", 2024-09-15T03:47:53.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8179999999999998e-06 2024-09-15T03:47:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088860+0000", 2024-09-15T03:47:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6566999999999999e-05 2024-09-15T03:47:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088881+0000", 2024-09-15T03:47:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0619999999999999e-05 2024-09-15T03:47:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088963+0000", 2024-09-15T03:47:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2650000000000003e-05 2024-09-15T03:47:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089325+0000", 2024-09-15T03:47:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00036168700000000002 2024-09-15T03:47:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102185+0000", 2024-09-15T03:47:53.609 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012859786 2024-09-15T03:47:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102633+0000", 2024-09-15T03:47:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00044844699999999998 2024-09-15T03:47:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102713+0000", 2024-09-15T03:47:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9616999999999993e-05 2024-09-15T03:47:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.610 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102771+0000", 2024-09-15T03:47:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7797999999999998e-05 2024-09-15T03:47:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102793+0000", 2024-09-15T03:47:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2827999999999999e-05 2024-09-15T03:47:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.104072+0000", 2024-09-15T03:47:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012787759999999999 2024-09-15T03:47:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.105482+0000", 2024-09-15T03:47:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001409769 2024-09-15T03:47:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:693 3.9s0 3:97177042:::smithi00132172-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.089498+0000", 2024-09-15T03:47:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 23.423511686000001, 2024-09-15T03:47:53.612 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013753056, 2024-09-15T03:47:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 693 2024-09-15T03:47:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089498+0000", 2024-09-15T03:47:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089498+0000", 2024-09-15T03:47:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089499+0000", 2024-09-15T03:47:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.074e-06 2024-09-15T03:47:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089503+0000", 2024-09-15T03:47:53.614 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1470000000000003e-06 2024-09-15T03:47:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089513+0000", 2024-09-15T03:47:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0248000000000001e-05 2024-09-15T03:47:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089519+0000", 2024-09-15T03:47:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2149999999999998e-06 2024-09-15T03:47:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.615 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089719+0000", 2024-09-15T03:47:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019960400000000001 2024-09-15T03:47:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089776+0000", 2024-09-15T03:47:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7123000000000002e-05 2024-09-15T03:47:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102827+0000", 2024-09-15T03:47:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013050858 2024-09-15T03:47:53.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102976+0000", 2024-09-15T03:47:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014900800000000001 2024-09-15T03:47:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.103251+0000", 2024-09-15T03:47:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00027477900000000001 2024-09-15T03:47:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:704 3.8s0 3:1af4eaf3:::smithi00132172-20:head [rollback 2d] snapc 2f=[2d,2b,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-15T03:47:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.982834+0000", 2024-09-15T03:47:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.530175580000002, 2024-09-15T03:47:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011049209000000001, 2024-09-15T03:47:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 704 2024-09-15T03:47:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:53.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982834+0000", 2024-09-15T03:47:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982834+0000", 2024-09-15T03:47:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982834+0000", 2024-09-15T03:47:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3000000000000001e-07 2024-09-15T03:47:53.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982836+0000", 2024-09-15T03:47:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0870000000000001e-06 2024-09-15T03:47:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982842+0000", 2024-09-15T03:47:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.879e-06 2024-09-15T03:47:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:53.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982846+0000", 2024-09-15T03:47:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8500000000000004e-06 2024-09-15T03:47:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982967+0000", 2024-09-15T03:47:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012074999999999999 2024-09-15T03:47:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982977+0000", 2024-09-15T03:47:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0397999999999999e-05 2024-09-15T03:47:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.985406+0000", 2024-09-15T03:47:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024286619999999998 2024-09-15T03:47:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.985533+0000", 2024-09-15T03:47:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012677399999999999 2024-09-15T03:47:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986275+0000", 2024-09-15T03:47:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0007427 2024-09-15T03:47:53.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986403+0000", 2024-09-15T03:47:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012725799999999999 2024-09-15T03:47:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986644+0000", 2024-09-15T03:47:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000241876 2024-09-15T03:47:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:53.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986705+0000", 2024-09-15T03:47:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0350999999999999e-05 2024-09-15T03:47:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987544+0000", 2024-09-15T03:47:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00083908500000000005 2024-09-15T03:47:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.990656+0000", 2024-09-15T03:47:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0031123750000000001 2024-09-15T03:47:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.993883+0000", 2024-09-15T03:47:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032266339999999999 2024-09-15T03:47:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:705 3.8s0 3:1af4eaf3:::smithi00132172-20:head [append 0~0] snapc 2f=[2d,2b,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-15T03:47:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.982870+0000", 2024-09-15T03:47:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 22.530139275, 2024-09-15T03:47:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.01140421, 2024-09-15T03:47:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:53.625 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 705 2024-09-15T03:47:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982870+0000", 2024-09-15T03:47:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982870+0000", 2024-09-15T03:47:53.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982871+0000", 2024-09-15T03:47:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2900000000000004e-07 2024-09-15T03:47:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982874+0000", 2024-09-15T03:47:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9610000000000001e-06 2024-09-15T03:47:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982878+0000", 2024-09-15T03:47:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7620000000000003e-06 2024-09-15T03:47:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982881+0000", 2024-09-15T03:47:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.112e-06 2024-09-15T03:47:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982995+0000", 2024-09-15T03:47:53.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000113544 2024-09-15T03:47:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982999+0000", 2024-09-15T03:47:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9339999999999999e-06 2024-09-15T03:47:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986479+0000", 2024-09-15T03:47:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0034799399999999999 2024-09-15T03:47:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986599+0000", 2024-09-15T03:47:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000120364 2024-09-15T03:47:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987178+0000", 2024-09-15T03:47:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00057913799999999998 2024-09-15T03:47:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987321+0000", 2024-09-15T03:47:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000142208 2024-09-15T03:47:53.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987335+0000", 2024-09-15T03:47:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4578e-05 2024-09-15T03:47:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987690+0000", 2024-09-15T03:47:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00035494600000000003 2024-09-15T03:47:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987799+0000", 2024-09-15T03:47:53.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000109379 2024-09-15T03:47:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.991063+0000", 2024-09-15T03:47:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032640249999999998 2024-09-15T03:47:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.994274+0000", 2024-09-15T03:47:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032107899999999998 2024-09-15T03:47:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:708 3.4s0 3:2d1c787c:::smithi00132172-45:head [copy-from ver 28] snapc 32=[32,2d,2b,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-15T03:47:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:31.983317+0000", 2024-09-15T03:47:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.529692742000002, 2024-09-15T03:47:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.021360130000000001, 2024-09-15T03:47:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 708 2024-09-15T03:47:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983317+0000", 2024-09-15T03:47:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983317+0000", 2024-09-15T03:47:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983318+0000", 2024-09-15T03:47:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2780000000000001e-06 2024-09-15T03:47:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983331+0000", 2024-09-15T03:47:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2821e-05 2024-09-15T03:47:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983349+0000", 2024-09-15T03:47:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8544999999999999e-05 2024-09-15T03:47:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983359+0000", 2024-09-15T03:47:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8570000000000007e-06 2024-09-15T03:47:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983434+0000", 2024-09-15T03:47:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4556999999999998e-05 2024-09-15T03:47:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-15T03:47:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983452+0000", 2024-09-15T03:47:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8439000000000001e-05 2024-09-15T03:47:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.985636+0000", 2024-09-15T03:47:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021841320000000001 2024-09-15T03:47:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.985755+0000", 2024-09-15T03:47:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118363 2024-09-15T03:47:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.998959+0000", 2024-09-15T03:47:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013204053 2024-09-15T03:47:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.999917+0000", 2024-09-15T03:47:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095851300000000003 2024-09-15T03:47:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.999931+0000", 2024-09-15T03:47:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3631999999999999e-05 2024-09-15T03:47:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000553+0000", 2024-09-15T03:47:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00062246600000000001 2024-09-15T03:47:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000695+0000", 2024-09-15T03:47:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014162499999999999 2024-09-15T03:47:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.004291+0000", 2024-09-15T03:47:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0035959289999999999 2024-09-15T03:47:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.004677+0000", 2024-09-15T03:47:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00038591999999999999 2024-09-15T03:47:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:709 3.4s0 3:2d1c787c:::smithi00132172-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-15T03:47:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:31.983373+0000", 2024-09-15T03:47:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.529635900999999, 2024-09-15T03:47:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.017317583000000001, 2024-09-15T03:47:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 709 2024-09-15T03:47:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983373+0000", 2024-09-15T03:47:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983373+0000", 2024-09-15T03:47:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983374+0000", 2024-09-15T03:47:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8599999999999998e-07 2024-09-15T03:47:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983377+0000", 2024-09-15T03:47:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0139999999999999e-06 2024-09-15T03:47:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983388+0000", 2024-09-15T03:47:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0988999999999999e-05 2024-09-15T03:47:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983392+0000", 2024-09-15T03:47:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5059999999999997e-06 2024-09-15T03:47:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983542+0000", 2024-09-15T03:47:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000149887 2024-09-15T03:47:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983550+0000", 2024-09-15T03:47:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9589999999999995e-06 2024-09-15T03:47:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.985938+0000", 2024-09-15T03:47:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023879930000000001 2024-09-15T03:47:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.986005+0000", 2024-09-15T03:47:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6926000000000003e-05 2024-09-15T03:47:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000370+0000", 2024-09-15T03:47:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.014364656999999999 2024-09-15T03:47:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000485+0000", 2024-09-15T03:47:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011524899999999999 2024-09-15T03:47:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000691+0000", 2024-09-15T03:47:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020581700000000001 2024-09-15T03:47:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:761 3.9s0 3:9cd57211:::smithi00132172-16:head [append 0~507904] snapc 32=[32,2d,2b,26,25,24,23,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T03:47:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:34.955885+0000", 2024-09-15T03:47:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 18.557123957999998, 2024-09-15T03:47:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011160715, 2024-09-15T03:47:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 761 2024-09-15T03:47:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.955885+0000", 2024-09-15T03:47:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:53.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.955885+0000", 2024-09-15T03:47:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.955887+0000", 2024-09-15T03:47:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8169999999999999e-06 2024-09-15T03:47:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956332+0000", 2024-09-15T03:47:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00044459600000000001 2024-09-15T03:47:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956347+0000", 2024-09-15T03:47:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5685999999999999e-05 2024-09-15T03:47:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956354+0000", 2024-09-15T03:47:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6810000000000001e-06 2024-09-15T03:47:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956805+0000", 2024-09-15T03:47:53.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00045055300000000002 2024-09-15T03:47:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956981+0000", 2024-09-15T03:47:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017590100000000001 2024-09-15T03:47:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.958362+0000", 2024-09-15T03:47:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001381415 2024-09-15T03:47:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.958524+0000", 2024-09-15T03:47:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016167800000000001 2024-09-15T03:47:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.958797+0000", 2024-09-15T03:47:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000273274 2024-09-15T03:47:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.959021+0000", 2024-09-15T03:47:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022373099999999999 2024-09-15T03:47:53.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.961002+0000", 2024-09-15T03:47:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019813069999999999 2024-09-15T03:47:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.966647+0000", 2024-09-15T03:47:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00564541 2024-09-15T03:47:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:53.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.967046+0000", 2024-09-15T03:47:53.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00039866599999999999 2024-09-15T03:47:53.653 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.653 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:53.653 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.653 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:53.653 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:53.653 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:53.655 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:47:53.655 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:53.656 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:53.656 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:53.656 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:53.656 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:53.657 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:53.657 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:47:53.657 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:47:53.657 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:53.658 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:53.659 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:53.653+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:53.755 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:53.755+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:53.772 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:53.772 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:53.781 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:47:53.856 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:53.854+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:53.908 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-15T03:47:53.909 INFO:tasks.rados.rados.0.smithi001.stdout:360: done (3 left) 2024-09-15T03:47:53.909 INFO:tasks.rados.rados.0.smithi001.stdout:361: copy_from oid 34 from oid 30 current snap is 47 2024-09-15T03:47:53.909 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 5 current snap is 47 2024-09-15T03:47:53.909 INFO:tasks.rados.rados.0.smithi001.stdout:362: seq_num 136 ranges {1769472=737280,2506752=770048,3276800=540672,3817472=245760} 2024-09-15T03:47:53.924 INFO:tasks.rados.rados.0.smithi001.stdout:362: writing smithi00132172-5 from 1769472 to 2506752 tid 1 2024-09-15T03:47:53.929 INFO:tasks.rados.rados.0.smithi001.stdout:362: writing smithi00132172-5 from 2506752 to 3276800 tid 2 2024-09-15T03:47:53.933 INFO:tasks.rados.rados.0.smithi001.stdout:362: writing smithi00132172-5 from 3276800 to 3817472 tid 3 2024-09-15T03:47:53.934 INFO:tasks.rados.rados.0.smithi001.stdout:362: writing smithi00132172-5 from 3817472 to 4063232 tid 4 2024-09-15T03:47:53.934 INFO:tasks.rados.rados.0.smithi001.stdout:363: snap_create 2024-09-15T03:47:53.935 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:53.935 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:53.935 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:53.935 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:53.935 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:53.944 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:47:53.989 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:53.988+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:54.047 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:54.047 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:54.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:54.047 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:54.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.048 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs6 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:54.048 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108711+0000", 2024-09-15T03:47:54.048 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.936435670999998, 2024-09-15T03:47:54.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01472483, 2024-09-15T03:47:54.048 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.048 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.048 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108711+0000", 2024-09-15T03:47:54.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108711+0000", 2024-09-15T03:47:54.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108712+0000", 2024-09-15T03:47:54.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.85e-07 2024-09-15T03:47:54.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108745+0000", 2024-09-15T03:47:54.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2762000000000002e-05 2024-09-15T03:47:54.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108760+0000", 2024-09-15T03:47:54.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4567e-05 2024-09-15T03:47:54.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108780+0000", 2024-09-15T03:47:54.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0122000000000001e-05 2024-09-15T03:47:54.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108842+0000", 2024-09-15T03:47:54.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2297000000000003e-05 2024-09-15T03:47:54.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108851+0000", 2024-09-15T03:47:54.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0129999999999999e-06 2024-09-15T03:47:54.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108942+0000", 2024-09-15T03:47:54.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0713999999999999e-05 2024-09-15T03:47:54.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108949+0000", 2024-09-15T03:47:54.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1600000000000001e-06 2024-09-15T03:47:54.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109975+0000", 2024-09-15T03:47:54.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010264429999999999 2024-09-15T03:47:54.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109994+0000", 2024-09-15T03:47:54.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8374000000000001e-05 2024-09-15T03:47:54.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123436+0000", 2024-09-15T03:47:54.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013442592999999999 2024-09-15T03:47:54.053 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.053 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.053 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.053 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs6 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:54.054 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108809+0000", 2024-09-15T03:47:54.054 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.936337942999998, 2024-09-15T03:47:54.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014757497, 2024-09-15T03:47:54.054 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.054 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.054 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108809+0000", 2024-09-15T03:47:54.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108809+0000", 2024-09-15T03:47:54.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108810+0000", 2024-09-15T03:47:54.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.15e-07 2024-09-15T03:47:54.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108825+0000", 2024-09-15T03:47:54.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5092999999999999e-05 2024-09-15T03:47:54.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108835+0000", 2024-09-15T03:47:54.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0251000000000001e-05 2024-09-15T03:47:54.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108841+0000", 2024-09-15T03:47:54.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0510000000000002e-06 2024-09-15T03:47:54.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108971+0000", 2024-09-15T03:47:54.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012976599999999999 2024-09-15T03:47:54.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108976+0000", 2024-09-15T03:47:54.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8450000000000002e-06 2024-09-15T03:47:54.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109032+0000", 2024-09-15T03:47:54.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6119999999999998e-05 2024-09-15T03:47:54.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109978+0000", 2024-09-15T03:47:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00094603899999999997 2024-09-15T03:47:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110007+0000", 2024-09-15T03:47:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9062999999999999e-05 2024-09-15T03:47:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110025+0000", 2024-09-15T03:47:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7717999999999999e-05 2024-09-15T03:47:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123567+0000", 2024-09-15T03:47:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013542136 2024-09-15T03:47:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:324 3.7s0 3:e40d1694:::smithi00132172-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.201024509, 2024-09-15T03:47:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0031668960000002, 2024-09-15T03:47:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:54.061 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:54.061 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:54.061 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:54.061 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 324 2024-09-15T03:47:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:47:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844125+0000", 2024-09-15T03:47:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.142e-06 2024-09-15T03:47:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844132+0000", 2024-09-15T03:47:54.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0439999999999998e-06 2024-09-15T03:47:54.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844136+0000", 2024-09-15T03:47:54.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.126e-06 2024-09-15T03:47:54.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.847618+0000", 2024-09-15T03:47:54.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0034813790000000002 2024-09-15T03:47:54.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:54.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.847709+0000", 2024-09-15T03:47:54.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0835999999999996e-05 2024-09-15T03:47:54.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.849483+0000", 2024-09-15T03:47:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001774929 2024-09-15T03:47:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.849574+0000", 2024-09-15T03:47:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0847999999999995e-05 2024-09-15T03:47:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.847290+0000", 2024-09-15T03:47:54.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9977151819999999 2024-09-15T03:47:54.067 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.067 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.067 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.067 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:334 3.7s0 3:ece1472c:::smithi00132172-43:head [append 0~0] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:54.067 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:54.067 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.188297421000001, 2024-09-15T03:47:54.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.997592021, 2024-09-15T03:47:54.067 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.067 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 334 2024-09-15T03:47:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856851+0000", 2024-09-15T03:47:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.238e-06 2024-09-15T03:47:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856855+0000", 2024-09-15T03:47:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3640000000000001e-06 2024-09-15T03:47:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856868+0000", 2024-09-15T03:47:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2588999999999999e-05 2024-09-15T03:47:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856873+0000", 2024-09-15T03:47:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5350000000000004e-06 2024-09-15T03:47:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856916+0000", 2024-09-15T03:47:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2889000000000002e-05 2024-09-15T03:47:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856983+0000", 2024-09-15T03:47:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6746000000000001e-05 2024-09-15T03:47:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857448+0000", 2024-09-15T03:47:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046498500000000001 2024-09-15T03:47:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857720+0000", 2024-09-15T03:47:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027227600000000001 2024-09-15T03:47:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857741+0000", 2024-09-15T03:47:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1149999999999999e-05 2024-09-15T03:47:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857795+0000", 2024-09-15T03:47:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3919999999999999e-05 2024-09-15T03:47:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859582+0000", 2024-09-15T03:47:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017862640000000001 2024-09-15T03:47:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854347+0000", 2024-09-15T03:47:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9947654780000001 2024-09-15T03:47:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854442+0000", 2024-09-15T03:47:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4586999999999994e-05 2024-09-15T03:47:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:335 3.7s0 3:ece1472c:::smithi00132172-43:head [rollback 12] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.188267404000001, 2024-09-15T03:47:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9977859200000001, 2024-09-15T03:47:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 335 2024-09-15T03:47:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8099999999999999e-07 2024-09-15T03:47:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856882+0000", 2024-09-15T03:47:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.649e-06 2024-09-15T03:47:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856885+0000", 2024-09-15T03:47:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.664e-06 2024-09-15T03:47:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856888+0000", 2024-09-15T03:47:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3910000000000001e-06 2024-09-15T03:47:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857826+0000", 2024-09-15T03:47:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00093817900000000003 2024-09-15T03:47:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857940+0000", 2024-09-15T03:47:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114424 2024-09-15T03:47:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858504+0000", 2024-09-15T03:47:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056400899999999997 2024-09-15T03:47:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858593+0000", 2024-09-15T03:47:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8463999999999999e-05 2024-09-15T03:47:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858604+0000", 2024-09-15T03:47:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1567e-05 2024-09-15T03:47:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858642+0000", 2024-09-15T03:47:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7369000000000001e-05 2024-09-15T03:47:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859588+0000", 2024-09-15T03:47:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00094661799999999998 2024-09-15T03:47:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854594+0000", 2024-09-15T03:47:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9950055550000001 2024-09-15T03:47:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854666+0000", 2024-09-15T03:47:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1749999999999996e-05 2024-09-15T03:47:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:336 3.7s0 3:ece1472c:::smithi00132172-43:head [append 0~0] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.188236353999997, 2024-09-15T03:47:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9980017820000002, 2024-09-15T03:47:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 336 2024-09-15T03:47:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8000000000000002e-07 2024-09-15T03:47:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856913+0000", 2024-09-15T03:47:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5400000000000001e-06 2024-09-15T03:47:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856918+0000", 2024-09-15T03:47:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1540000000000001e-06 2024-09-15T03:47:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856921+0000", 2024-09-15T03:47:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2550000000000002e-06 2024-09-15T03:47:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858655+0000", 2024-09-15T03:47:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017336070000000001 2024-09-15T03:47:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858741+0000", 2024-09-15T03:47:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6305999999999997e-05 2024-09-15T03:47:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859193+0000", 2024-09-15T03:47:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045221200000000001 2024-09-15T03:47:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859319+0000", 2024-09-15T03:47:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012542900000000001 2024-09-15T03:47:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859331+0000", 2024-09-15T03:47:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2028000000000001e-05 2024-09-15T03:47:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859369+0000", 2024-09-15T03:47:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8522999999999997e-05 2024-09-15T03:47:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859649+0000", 2024-09-15T03:47:54.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028008200000000002 2024-09-15T03:47:54.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:54.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854800+0000", 2024-09-15T03:47:54.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9951506540000001 2024-09-15T03:47:54.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854913+0000", 2024-09-15T03:47:54.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112712 2024-09-15T03:47:54.088 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.088 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.088 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.088 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:346 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:54.088 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:47:54.088 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.206739984000002, 2024-09-15T03:47:54.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005631948, 2024-09-15T03:47:54.088 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.088 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:54.088 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:54.088 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:54.088 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:54.089 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 346 2024-09-15T03:47:54.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.089 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:47:54.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:47:54.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838410+0000", 2024-09-15T03:47:54.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4430000000000002e-06 2024-09-15T03:47:54.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838419+0000", 2024-09-15T03:47:54.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1730000000000001e-06 2024-09-15T03:47:54.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838444+0000", 2024-09-15T03:47:54.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5111000000000001e-05 2024-09-15T03:47:54.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838455+0000", 2024-09-15T03:47:54.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0839000000000001e-05 2024-09-15T03:47:54.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:54.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3415000000000002e-05 2024-09-15T03:47:54.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:54.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838527+0000", 2024-09-15T03:47:54.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0550000000000005e-06 2024-09-15T03:47:54.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840638+0000", 2024-09-15T03:47:54.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021105379999999999 2024-09-15T03:47:54.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:54.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840729+0000", 2024-09-15T03:47:54.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0977999999999995e-05 2024-09-15T03:47:54.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.841834+0000", 2024-09-15T03:47:54.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011055640000000001 2024-09-15T03:47:54.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842223+0000", 2024-09-15T03:47:54.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038899900000000001 2024-09-15T03:47:54.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842241+0000", 2024-09-15T03:47:54.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7317e-05 2024-09-15T03:47:54.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842298+0000", 2024-09-15T03:47:54.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7825000000000003e-05 2024-09-15T03:47:54.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844866+0000", 2024-09-15T03:47:54.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025675289999999998 2024-09-15T03:47:54.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:54.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843973+0000", 2024-09-15T03:47:54.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99910710599999997 2024-09-15T03:47:54.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844039+0000", 2024-09-15T03:47:54.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6056000000000006e-05 2024-09-15T03:47:54.095 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.095 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.095 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.096 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:347 3.2s0 3:48ceb02f:::smithi00132172-5:head [rollback c] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:54.096 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:47:54.096 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.206655187999999, 2024-09-15T03:47:54.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005723841, 2024-09-15T03:47:54.096 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.096 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:54.096 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:54.096 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:54.096 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:54.096 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 347 2024-09-15T03:47:54.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.097 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:47:54.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:47:54.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838493+0000", 2024-09-15T03:47:54.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7300000000000005e-07 2024-09-15T03:47:54.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838498+0000", 2024-09-15T03:47:54.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9300000000000002e-06 2024-09-15T03:47:54.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838508+0000", 2024-09-15T03:47:54.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9790000000000005e-06 2024-09-15T03:47:54.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838513+0000", 2024-09-15T03:47:54.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8890000000000001e-06 2024-09-15T03:47:54.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838583+0000", 2024-09-15T03:47:54.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0696000000000002e-05 2024-09-15T03:47:54.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:54.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838590+0000", 2024-09-15T03:47:54.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2990000000000002e-06 2024-09-15T03:47:54.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842313+0000", 2024-09-15T03:47:54.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037238810000000001 2024-09-15T03:47:54.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:54.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842432+0000", 2024-09-15T03:47:54.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118474 2024-09-15T03:47:54.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843108+0000", 2024-09-15T03:47:54.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00067639299999999996 2024-09-15T03:47:54.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843227+0000", 2024-09-15T03:47:54.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119092 2024-09-15T03:47:54.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843242+0000", 2024-09-15T03:47:54.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4216000000000001e-05 2024-09-15T03:47:54.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843292+0000", 2024-09-15T03:47:54.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0800999999999997e-05 2024-09-15T03:47:54.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844878+0000", 2024-09-15T03:47:54.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015852749999999999 2024-09-15T03:47:54.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:54.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844147+0000", 2024-09-15T03:47:54.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99926949200000004 2024-09-15T03:47:54.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844216+0000", 2024-09-15T03:47:54.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8551000000000001e-05 2024-09-15T03:47:54.103 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.103 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.104 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.104 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:348 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:54.104 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:54.104 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.206629245000002, 2024-09-15T03:47:54.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.006019768, 2024-09-15T03:47:54.104 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.104 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:54.104 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:54.104 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:54.104 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:54.104 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 348 2024-09-15T03:47:54.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8700000000000001e-07 2024-09-15T03:47:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838521+0000", 2024-09-15T03:47:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3740000000000001e-06 2024-09-15T03:47:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838527+0000", 2024-09-15T03:47:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1999999999999999e-06 2024-09-15T03:47:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838532+0000", 2024-09-15T03:47:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2059999999999998e-06 2024-09-15T03:47:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838609+0000", 2024-09-15T03:47:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7193000000000004e-05 2024-09-15T03:47:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838615+0000", 2024-09-15T03:47:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7980000000000002e-06 2024-09-15T03:47:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843986+0000", 2024-09-15T03:47:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053706450000000003 2024-09-15T03:47:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844106+0000", 2024-09-15T03:47:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120079 2024-09-15T03:47:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844669+0000", 2024-09-15T03:47:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056290300000000001 2024-09-15T03:47:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844804+0000", 2024-09-15T03:47:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013575600000000001 2024-09-15T03:47:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844820+0000", 2024-09-15T03:47:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5145e-05 2024-09-15T03:47:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844893+0000", 2024-09-15T03:47:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3270999999999997e-05 2024-09-15T03:47:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845043+0000", 2024-09-15T03:47:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150144 2024-09-15T03:47:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844473+0000", 2024-09-15T03:47:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99943048700000003 2024-09-15T03:47:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844538+0000", 2024-09-15T03:47:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4179999999999999e-05 2024-09-15T03:47:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:352 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~557056] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:47:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.200431600999998, 2024-09-15T03:47:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.016265065, 2024-09-15T03:47:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 352 2024-09-15T03:47:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:47:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:47:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844718+0000", 2024-09-15T03:47:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2340000000000001e-06 2024-09-15T03:47:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845237+0000", 2024-09-15T03:47:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051953399999999999 2024-09-15T03:47:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845273+0000", 2024-09-15T03:47:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5683999999999999e-05 2024-09-15T03:47:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845281+0000", 2024-09-15T03:47:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8439999999999998e-06 2024-09-15T03:47:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845365+0000", 2024-09-15T03:47:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4651e-05 2024-09-15T03:47:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845486+0000", 2024-09-15T03:47:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012009499999999999 2024-09-15T03:47:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847101+0000", 2024-09-15T03:47:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016157350000000001 2024-09-15T03:47:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847602+0000", 2024-09-15T03:47:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050066200000000003 2024-09-15T03:47:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847622+0000", 2024-09-15T03:47:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9749999999999999e-05 2024-09-15T03:47:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848616+0000", 2024-09-15T03:47:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099390799999999994 2024-09-15T03:47:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848709+0000", 2024-09-15T03:47:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2874000000000006e-05 2024-09-15T03:47:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.860888+0000", 2024-09-15T03:47:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.012179707 2024-09-15T03:47:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.860981+0000", 2024-09-15T03:47:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2386999999999995e-05 2024-09-15T03:47:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:353 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~671744] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:47:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.196869407999998, 2024-09-15T03:47:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.012912206, 2024-09-15T03:47:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 353 2024-09-15T03:47:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:47:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:47:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848279+0000", 2024-09-15T03:47:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.362e-06 2024-09-15T03:47:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849024+0000", 2024-09-15T03:47:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00074495500000000005 2024-09-15T03:47:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849039+0000", 2024-09-15T03:47:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5119e-05 2024-09-15T03:47:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849046+0000", 2024-09-15T03:47:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6359999999999997e-06 2024-09-15T03:47:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849115+0000", 2024-09-15T03:47:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8998999999999994e-05 2024-09-15T03:47:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849251+0000", 2024-09-15T03:47:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013589799999999999 2024-09-15T03:47:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850605+0000", 2024-09-15T03:47:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013544659999999999 2024-09-15T03:47:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850703+0000", 2024-09-15T03:47:54.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7576000000000002e-05 2024-09-15T03:47:54.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850719+0000", 2024-09-15T03:47:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5767e-05 2024-09-15T03:47:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851562+0000", 2024-09-15T03:47:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084361899999999999 2024-09-15T03:47:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851705+0000", 2024-09-15T03:47:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000142893 2024-09-15T03:47:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861112+0000", 2024-09-15T03:47:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.009407349 2024-09-15T03:47:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861190+0000", 2024-09-15T03:47:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7566999999999998e-05 2024-09-15T03:47:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:354 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~98304] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:47:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.196066971999997, 2024-09-15T03:47:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0177595669999999, 2024-09-15T03:47:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 354 2024-09-15T03:47:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:47:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:47:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849081+0000", 2024-09-15T03:47:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0610000000000001e-06 2024-09-15T03:47:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849157+0000", 2024-09-15T03:47:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5804999999999997e-05 2024-09-15T03:47:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849168+0000", 2024-09-15T03:47:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0730000000000001e-05 2024-09-15T03:47:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849174+0000", 2024-09-15T03:47:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3609999999999997e-06 2024-09-15T03:47:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850814+0000", 2024-09-15T03:47:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001640124 2024-09-15T03:47:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850894+0000", 2024-09-15T03:47:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0167999999999995e-05 2024-09-15T03:47:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851406+0000", 2024-09-15T03:47:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051203200000000005 2024-09-15T03:47:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851506+0000", 2024-09-15T03:47:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9457999999999995e-05 2024-09-15T03:47:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851574+0000", 2024-09-15T03:47:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8145000000000006e-05 2024-09-15T03:47:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851632+0000", 2024-09-15T03:47:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8108e-05 2024-09-15T03:47:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851712+0000", 2024-09-15T03:47:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9380000000000002e-05 2024-09-15T03:47:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.866753+0000", 2024-09-15T03:47:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0150417979999999 2024-09-15T03:47:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.866840+0000", 2024-09-15T03:47:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6397000000000006e-05 2024-09-15T03:47:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:355 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [setxattr _header (57) in=7b] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:47:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.195937659000002, 2024-09-15T03:47:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.018024966, 2024-09-15T03:47:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 355 2024-09-15T03:47:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:47:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:47:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849210+0000", 2024-09-15T03:47:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.33e-07 2024-09-15T03:47:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849215+0000", 2024-09-15T03:47:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4379999999999997e-06 2024-09-15T03:47:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849234+0000", 2024-09-15T03:47:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9409999999999999e-05 2024-09-15T03:47:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849241+0000", 2024-09-15T03:47:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6370000000000002e-06 2024-09-15T03:47:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851823+0000", 2024-09-15T03:47:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025817689999999998 2024-09-15T03:47:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851934+0000", 2024-09-15T03:47:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111357 2024-09-15T03:47:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852466+0000", 2024-09-15T03:47:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053248100000000001 2024-09-15T03:47:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852580+0000", 2024-09-15T03:47:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011379699999999999 2024-09-15T03:47:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852607+0000", 2024-09-15T03:47:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279e-05 2024-09-15T03:47:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852693+0000", 2024-09-15T03:47:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6792999999999994e-05 2024-09-15T03:47:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852869+0000", 2024-09-15T03:47:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017537499999999999 2024-09-15T03:47:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867006+0000", 2024-09-15T03:47:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.014136996 2024-09-15T03:47:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867234+0000", 2024-09-15T03:47:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022870099999999999 2024-09-15T03:47:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:356 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:47:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.195900895999998, 2024-09-15T03:47:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.020403521, 2024-09-15T03:47:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 356 2024-09-15T03:47:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:47:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:47:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849247+0000", 2024-09-15T03:47:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2699999999999999e-07 2024-09-15T03:47:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849250+0000", 2024-09-15T03:47:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2610000000000002e-06 2024-09-15T03:47:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849257+0000", 2024-09-15T03:47:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.703e-06 2024-09-15T03:47:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849261+0000", 2024-09-15T03:47:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7910000000000002e-06 2024-09-15T03:47:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852672+0000", 2024-09-15T03:47:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003411627 2024-09-15T03:47:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852801+0000", 2024-09-15T03:47:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000128918 2024-09-15T03:47:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867347+0000", 2024-09-15T03:47:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0145458380000001 2024-09-15T03:47:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867476+0000", 2024-09-15T03:47:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012915100000000001 2024-09-15T03:47:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.869650+0000", 2024-09-15T03:47:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021737050000000002 2024-09-15T03:47:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs6 65/64 ECSubWrite(tid=108, reqid=client.4521.0:466, at_version=65'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.856519+0000", 2024-09-15T03:47:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 42.188628549000001, 2024-09-15T03:47:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015639281000000001, 2024-09-15T03:47:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856519+0000", 2024-09-15T03:47:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856519+0000", 2024-09-15T03:47:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856519+0000", 2024-09-15T03:47:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4200000000000005e-07 2024-09-15T03:47:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857060+0000", 2024-09-15T03:47:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054053200000000004 2024-09-15T03:47:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857069+0000", 2024-09-15T03:47:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2979999999999997e-06 2024-09-15T03:47:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857074+0000", 2024-09-15T03:47:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7570000000000004e-06 2024-09-15T03:47:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857122+0000", 2024-09-15T03:47:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7828999999999999e-05 2024-09-15T03:47:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857128+0000", 2024-09-15T03:47:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6409999999999996e-06 2024-09-15T03:47:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857491+0000", 2024-09-15T03:47:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036239300000000003 2024-09-15T03:47:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857503+0000", 2024-09-15T03:47:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2468e-05 2024-09-15T03:47:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857718+0000", 2024-09-15T03:47:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021492200000000001 2024-09-15T03:47:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.857745+0000", 2024-09-15T03:47:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7189e-05 2024-09-15T03:47:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.872158+0000", 2024-09-15T03:47:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01441241 2024-09-15T03:47:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:527 3.7s0 3:e4b8e19d:::smithi00132172-41:head [copy-from ver 51] snapc 23=[23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:47:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.096331933999998, 2024-09-15T03:47:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030362388000000001, 2024-09-15T03:47:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 527 2024-09-15T03:47:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:47:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:47:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948817+0000", 2024-09-15T03:47:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3230000000000001e-06 2024-09-15T03:47:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948825+0000", 2024-09-15T03:47:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5900000000000008e-06 2024-09-15T03:47:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948838+0000", 2024-09-15T03:47:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2712e-05 2024-09-15T03:47:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948842+0000", 2024-09-15T03:47:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4170000000000002e-06 2024-09-15T03:47:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948923+0000", 2024-09-15T03:47:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0686000000000006e-05 2024-09-15T03:47:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:47:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949020+0000", 2024-09-15T03:47:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6984999999999994e-05 2024-09-15T03:47:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954390+0000", 2024-09-15T03:47:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053704790000000001 2024-09-15T03:47:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954558+0000", 2024-09-15T03:47:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000167202 2024-09-15T03:47:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.973998+0000", 2024-09-15T03:47:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019440316999999999 2024-09-15T03:47:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974527+0000", 2024-09-15T03:47:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052943800000000002 2024-09-15T03:47:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974541+0000", 2024-09-15T03:47:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3774999999999999e-05 2024-09-15T03:47:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974841+0000", 2024-09-15T03:47:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029942899999999998 2024-09-15T03:47:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.976067+0000", 2024-09-15T03:47:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012265399999999999 2024-09-15T03:47:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.978952+0000", 2024-09-15T03:47:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028848839999999999 2024-09-15T03:47:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.979178+0000", 2024-09-15T03:47:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022561099999999999 2024-09-15T03:47:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:528 3.7s0 3:e4b8e19d:::smithi00132172-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.096297524999997, 2024-09-15T03:47:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026927036000000001, 2024-09-15T03:47:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 528 2024-09-15T03:47:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9700000000000003e-07 2024-09-15T03:47:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948852+0000", 2024-09-15T03:47:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.604e-06 2024-09-15T03:47:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948855+0000", 2024-09-15T03:47:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9369999999999997e-06 2024-09-15T03:47:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948858+0000", 2024-09-15T03:47:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2709999999999999e-06 2024-09-15T03:47:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949080+0000", 2024-09-15T03:47:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000222148 2024-09-15T03:47:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:47:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949156+0000", 2024-09-15T03:47:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6215000000000001e-05 2024-09-15T03:47:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954754+0000", 2024-09-15T03:47:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055978060000000003 2024-09-15T03:47:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954858+0000", 2024-09-15T03:47:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010419199999999999 2024-09-15T03:47:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974861+0000", 2024-09-15T03:47:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020002671999999999 2024-09-15T03:47:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974987+0000", 2024-09-15T03:47:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012638699999999999 2024-09-15T03:47:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.975777+0000", 2024-09-15T03:47:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00078950699999999995 2024-09-15T03:47:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:833 3.7s0 3:ece1472c:::smithi00132172-43:head [read 0~4816896,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e97)", 2024-09-15T03:47:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:43.913147+0000", 2024-09-15T03:47:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.13200041, 2024-09-15T03:47:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014828559999999999, 2024-09-15T03:47:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 833 2024-09-15T03:47:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913147+0000", 2024-09-15T03:47:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913147+0000", 2024-09-15T03:47:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913147+0000", 2024-09-15T03:47:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7000000000000001e-07 2024-09-15T03:47:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913149+0000", 2024-09-15T03:47:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8929999999999999e-06 2024-09-15T03:47:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913152+0000", 2024-09-15T03:47:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.472e-06 2024-09-15T03:47:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913154+0000", 2024-09-15T03:47:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0940000000000002e-06 2024-09-15T03:47:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913477+0000", 2024-09-15T03:47:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032283000000000002 2024-09-15T03:47:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913585+0000", 2024-09-15T03:47:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010744899999999999 2024-09-15T03:47:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.927975+0000", 2024-09-15T03:47:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014390551999999999 2024-09-15T03:47:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:834 3.7s0 3:ece1472c:::smithi00132172-43:head [read 0~4816896,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e97)", 2024-09-15T03:47:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:43.913171+0000", 2024-09-15T03:47:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.131976608, 2024-09-15T03:47:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017820687000000002, 2024-09-15T03:47:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 834 2024-09-15T03:47:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913171+0000", 2024-09-15T03:47:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913171+0000", 2024-09-15T03:47:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913171+0000", 2024-09-15T03:47:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6e-07 2024-09-15T03:47:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913173+0000", 2024-09-15T03:47:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7719999999999999e-06 2024-09-15T03:47:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913176+0000", 2024-09-15T03:47:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8020000000000002e-06 2024-09-15T03:47:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.913180+0000", 2024-09-15T03:47:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.692e-06 2024-09-15T03:47:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.914761+0000", 2024-09-15T03:47:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001580936 2024-09-15T03:47:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.914879+0000", 2024-09-15T03:47:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117518 2024-09-15T03:47:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:43.930991+0000", 2024-09-15T03:47:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016112707 2024-09-15T03:47:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:853 3.7s0 3:e40d1694:::smithi00132172-50:head [copy-from ver 33] snapc 3e=[3e,3d,3b,3a,38,37,35,34,32,2d,2b,25,24,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-15T03:47:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:44.879131+0000", 2024-09-15T03:47:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.1660157529999999, 2024-09-15T03:47:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019602471, 2024-09-15T03:47:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 853 2024-09-15T03:47:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.879131+0000", 2024-09-15T03:47:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.879131+0000", 2024-09-15T03:47:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.879132+0000", 2024-09-15T03:47:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4400000000000001e-07 2024-09-15T03:47:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.879134+0000", 2024-09-15T03:47:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0190000000000001e-06 2024-09-15T03:47:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.879137+0000", 2024-09-15T03:47:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4929999999999998e-06 2024-09-15T03:47:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.879139+0000", 2024-09-15T03:47:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1280000000000002e-06 2024-09-15T03:47:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.879180+0000", 2024-09-15T03:47:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0750000000000001e-05 2024-09-15T03:47:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.879764+0000", 2024-09-15T03:47:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00058396700000000001 2024-09-15T03:47:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.888816+0000", 2024-09-15T03:47:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090521460000000005 2024-09-15T03:47:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.889281+0000", 2024-09-15T03:47:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046523499999999999 2024-09-15T03:47:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.889296+0000", 2024-09-15T03:47:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4965e-05 2024-09-15T03:47:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.889576+0000", 2024-09-15T03:47:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027968999999999997 2024-09-15T03:47:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.889890+0000", 2024-09-15T03:47:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031355400000000001 2024-09-15T03:47:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.891927+0000", 2024-09-15T03:47:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020376259999999998 2024-09-15T03:47:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:44.898734+0000", 2024-09-15T03:47:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068065540000000002 2024-09-15T03:47:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.180 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:54.182 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:47:54.182 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:54.181+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:54.190 INFO:tasks.rados.rados.0.smithi001.stdout:358: finishing rollback tid 0 to smithi00132172-32 2024-09-15T03:47:54.191 INFO:tasks.rados.rados.0.smithi001.stdout:358: finishing rollback tid 1 to smithi00132172-32 2024-09-15T03:47:54.192 INFO:tasks.rados.rados.0.smithi001.stdout:358: finishing rollback tid 2 to smithi00132172-32 2024-09-15T03:47:54.192 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 32 v 58 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-15T03:47:54.199 INFO:tasks.rados.rados.0.smithi001.stdout:357: finishing write tid 1 to smithi00132172-38 2024-09-15T03:47:54.200 INFO:tasks.rados.rados.0.smithi001.stdout:357: finishing write tid 2 to smithi00132172-38 2024-09-15T03:47:54.201 INFO:tasks.rados.rados.0.smithi001.stdout:357: finishing write tid 3 to smithi00132172-38 2024-09-15T03:47:54.202 INFO:tasks.rados.rados.0.smithi001.stdout:361: finishing copy_from racing read to smithi00132172-34 2024-09-15T03:47:54.206 INFO:tasks.rados.rados.0.smithi001.stdout:355: finishing copy_from racing read to smithi00132172-45 2024-09-15T03:47:54.206 INFO:tasks.rados.rados.0.smithi001.stdout:361: finishing copy_from to smithi00132172-34 2024-09-15T03:47:54.206 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 34 v 16 (ObjNum 74 snap 12 seq_num 74) dirty exists 2024-09-15T03:47:54.208 INFO:tasks.rados.rados.0.smithi001.stdout:357: finishing write tid 4 to smithi00132172-38 2024-09-15T03:47:54.208 INFO:tasks.rados.rados.0.smithi001.stdout:355: finishing copy_from to smithi00132172-45 2024-09-15T03:47:54.208 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 45 v 110 (ObjNum 128 snap 41 seq_num 128) dirty exists 2024-09-15T03:47:54.208 INFO:tasks.rados.rados.0.smithi001.stdout:357: finishing write tid 5 to smithi00132172-38 2024-09-15T03:47:54.215 INFO:tasks.rados.rados.0.smithi001.stdout:362: finishing write tid 1 to smithi00132172-5 2024-09-15T03:47:54.216 INFO:tasks.rados.rados.0.smithi001.stdout:362: finishing write tid 2 to smithi00132172-5 2024-09-15T03:47:54.216 INFO:tasks.rados.rados.0.smithi001.stdout:357: finishing write tid 6 to smithi00132172-38 2024-09-15T03:47:54.216 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 38 v 65 (ObjNum 135 snap 47 seq_num 135) dirty exists 2024-09-15T03:47:54.216 INFO:tasks.rados.rados.0.smithi001.stdout:357: left oid 38 (ObjNum 135 snap 47 seq_num 135) 2024-09-15T03:47:54.228 INFO:tasks.rados.rados.0.smithi001.stdout:362: finishing write tid 3 to smithi00132172-5 2024-09-15T03:47:54.229 INFO:tasks.rados.rados.0.smithi001.stdout:362: finishing write tid 4 to smithi00132172-5 2024-09-15T03:47:54.232 INFO:tasks.rados.rados.0.smithi001.stdout:362: finishing write tid 5 to smithi00132172-5 2024-09-15T03:47:54.236 INFO:tasks.rados.rados.0.smithi001.stdout:362: finishing write tid 6 to smithi00132172-5 2024-09-15T03:47:54.236 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 5 v 52 (ObjNum 136 snap 47 seq_num 136) dirty exists 2024-09-15T03:47:54.236 INFO:tasks.rados.rados.0.smithi001.stdout:362: left oid 5 (ObjNum 136 snap 47 seq_num 136) 2024-09-15T03:47:54.282 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:54.281+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:54.366 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:54.367 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:54.367 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:54.367 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:54.377 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:47:54.421 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:54.421+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:54.480 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:54.480 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:54.480 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:54.480 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:54.480 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:54.484 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:54.483+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:54.488 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:47:54.584 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:54.584+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:54.630 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 38/28 ECSubWrite(tid=33, reqid=client.4521.0:222, at_version=38'14, trim_to=0'0, roll_forward_to=35'12))", 2024-09-15T03:47:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:47:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.498047655999997, 2024-09-15T03:47:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9949187450000001, 2024-09-15T03:47:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:47:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:47:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130996+0000", 2024-09-15T03:47:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8809999999999998e-06 2024-09-15T03:47:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131020+0000", 2024-09-15T03:47:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4190999999999999e-05 2024-09-15T03:47:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131039+0000", 2024-09-15T03:47:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9228999999999998e-05 2024-09-15T03:47:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131048+0000", 2024-09-15T03:47:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2380000000000003e-06 2024-09-15T03:47:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131122+0000", 2024-09-15T03:47:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3146999999999996e-05 2024-09-15T03:47:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131132+0000", 2024-09-15T03:47:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.08e-05 2024-09-15T03:47:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131381+0000", 2024-09-15T03:47:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000248936 2024-09-15T03:47:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131392+0000", 2024-09-15T03:47:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0987e-05 2024-09-15T03:47:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131444+0000", 2024-09-15T03:47:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1585000000000002e-05 2024-09-15T03:47:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131465+0000", 2024-09-15T03:47:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1365999999999998e-05 2024-09-15T03:47:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.125911+0000", 2024-09-15T03:47:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9944453850000001 2024-09-15T03:47:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 38/28 ECSubWrite(tid=34, reqid=client.4521.0:222, at_version=0'0, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T03:47:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:47:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.496648557, 2024-09-15T03:47:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.002752697, 2024-09-15T03:47:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:47:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:47:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132392+0000", 2024-09-15T03:47:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5899999999999995e-07 2024-09-15T03:47:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132415+0000", 2024-09-15T03:47:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3561e-05 2024-09-15T03:47:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132436+0000", 2024-09-15T03:47:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0879000000000001e-05 2024-09-15T03:47:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132444+0000", 2024-09-15T03:47:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9200000000000004e-06 2024-09-15T03:47:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132497+0000", 2024-09-15T03:47:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2299000000000002e-05 2024-09-15T03:47:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132508+0000", 2024-09-15T03:47:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1100999999999999e-05 2024-09-15T03:47:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132615+0000", 2024-09-15T03:47:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107546 2024-09-15T03:47:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132635+0000", 2024-09-15T03:47:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9374000000000001e-05 2024-09-15T03:47:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132677+0000", 2024-09-15T03:47:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2456999999999997e-05 2024-09-15T03:47:54.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132711+0000", 2024-09-15T03:47:54.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4118000000000001e-05 2024-09-15T03:47:54.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135144+0000", 2024-09-15T03:47:54.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.002432583 2024-09-15T03:47:54.642 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.642 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.643 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.643 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=45, reqid=client.4521.0:227, at_version=38'24, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:47:54.643 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:54.643 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.473722015000007, 2024-09-15T03:47:54.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.974765339, 2024-09-15T03:47:54.643 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.643 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.643 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:54.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:54.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:47:54.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.43e-07 2024-09-15T03:47:54.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156110+0000", 2024-09-15T03:47:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000792409 2024-09-15T03:47:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156139+0000", 2024-09-15T03:47:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9207e-05 2024-09-15T03:47:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156148+0000", 2024-09-15T03:47:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1269999999999992e-06 2024-09-15T03:47:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156248+0000", 2024-09-15T03:47:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9421999999999998e-05 2024-09-15T03:47:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156258+0000", 2024-09-15T03:47:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6989999999999998e-06 2024-09-15T03:47:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156674+0000", 2024-09-15T03:47:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041682600000000002 2024-09-15T03:47:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156705+0000", 2024-09-15T03:47:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0525000000000003e-05 2024-09-15T03:47:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157742+0000", 2024-09-15T03:47:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010374 2024-09-15T03:47:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157777+0000", 2024-09-15T03:47:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4878e-05 2024-09-15T03:47:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.130083+0000", 2024-09-15T03:47:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972305703 2024-09-15T03:47:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=37, reqid=client.4521.0:236, at_version=38'5, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.472108559000006, 2024-09-15T03:47:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9740836640000001, 2024-09-15T03:47:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:47:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1300000000000001e-07 2024-09-15T03:47:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156953+0000", 2024-09-15T03:47:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1585000000000001e-05 2024-09-15T03:47:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156974+0000", 2024-09-15T03:47:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0764e-05 2024-09-15T03:47:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156981+0000", 2024-09-15T03:47:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2760000000000003e-06 2024-09-15T03:47:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157061+0000", 2024-09-15T03:47:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9583e-05 2024-09-15T03:47:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157075+0000", 2024-09-15T03:47:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4019000000000001e-05 2024-09-15T03:47:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157351+0000", 2024-09-15T03:47:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027641100000000001 2024-09-15T03:47:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157746+0000", 2024-09-15T03:47:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039553199999999998 2024-09-15T03:47:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157838+0000", 2024-09-15T03:47:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1632000000000006e-05 2024-09-15T03:47:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157856+0000", 2024-09-15T03:47:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7873999999999999e-05 2024-09-15T03:47:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.131015+0000", 2024-09-15T03:47:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9731586750000001 2024-09-15T03:47:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=46, reqid=client.4521.0:228, at_version=38'25, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:47:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.472049776999995, 2024-09-15T03:47:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9732838510000001, 2024-09-15T03:47:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:54.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:47:54.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0999999999999998e-07 2024-09-15T03:47:54.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157149+0000", 2024-09-15T03:47:54.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000158763 2024-09-15T03:47:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157165+0000", 2024-09-15T03:47:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5883999999999999e-05 2024-09-15T03:47:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157175+0000", 2024-09-15T03:47:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4949999999999997e-06 2024-09-15T03:47:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157219+0000", 2024-09-15T03:47:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4069000000000001e-05 2024-09-15T03:47:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157228+0000", 2024-09-15T03:47:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7829999999999993e-06 2024-09-15T03:47:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157345+0000", 2024-09-15T03:47:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116655 2024-09-15T03:47:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157746+0000", 2024-09-15T03:47:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040045100000000002 2024-09-15T03:47:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157838+0000", 2024-09-15T03:47:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1955999999999998e-05 2024-09-15T03:47:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157888+0000", 2024-09-15T03:47:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0943000000000003e-05 2024-09-15T03:47:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.130274+0000", 2024-09-15T03:47:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972385142 2024-09-15T03:47:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=38, reqid=client.4521.0:237, at_version=38'6, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:47:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.470707207000004, 2024-09-15T03:47:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9764011529999999, 2024-09-15T03:47:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:47:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:47:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158333+0000", 2024-09-15T03:47:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8599999999999997e-07 2024-09-15T03:47:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158353+0000", 2024-09-15T03:47:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9504000000000002e-05 2024-09-15T03:47:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158368+0000", 2024-09-15T03:47:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4966999999999999e-05 2024-09-15T03:47:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158374+0000", 2024-09-15T03:47:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.798e-06 2024-09-15T03:47:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158430+0000", 2024-09-15T03:47:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5744e-05 2024-09-15T03:47:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158440+0000", 2024-09-15T03:47:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4560000000000006e-06 2024-09-15T03:47:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158529+0000", 2024-09-15T03:47:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9172999999999995e-05 2024-09-15T03:47:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158543+0000", 2024-09-15T03:47:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.395e-05 2024-09-15T03:47:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158654+0000", 2024-09-15T03:47:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011110700000000001 2024-09-15T03:47:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158671+0000", 2024-09-15T03:47:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7068999999999999e-05 2024-09-15T03:47:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134733+0000", 2024-09-15T03:47:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.976062599 2024-09-15T03:47:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=47, reqid=client.4521.0:229, at_version=38'26, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:47:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:47:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.470703244999996, 2024-09-15T03:47:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9727810969999999, 2024-09-15T03:47:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:47:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:47:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158337+0000", 2024-09-15T03:47:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.15e-07 2024-09-15T03:47:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158351+0000", 2024-09-15T03:47:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3871e-05 2024-09-15T03:47:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158372+0000", 2024-09-15T03:47:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1058999999999999e-05 2024-09-15T03:47:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158376+0000", 2024-09-15T03:47:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5249999999999999e-06 2024-09-15T03:47:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158420+0000", 2024-09-15T03:47:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3918000000000002e-05 2024-09-15T03:47:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158429+0000", 2024-09-15T03:47:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5150000000000006e-06 2024-09-15T03:47:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158514+0000", 2024-09-15T03:47:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5252000000000003e-05 2024-09-15T03:47:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158542+0000", 2024-09-15T03:47:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.798e-05 2024-09-15T03:47:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158653+0000", 2024-09-15T03:47:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111315 2024-09-15T03:47:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158701+0000", 2024-09-15T03:47:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8241000000000001e-05 2024-09-15T03:47:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.131117+0000", 2024-09-15T03:47:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972416006 2024-09-15T03:47:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=39, reqid=client.4521.0:238, at_version=38'7, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:47:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:47:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.470016088999998, 2024-09-15T03:47:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9758214509999998, 2024-09-15T03:47:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:47:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:47:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159024+0000", 2024-09-15T03:47:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6700000000000003e-07 2024-09-15T03:47:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159096+0000", 2024-09-15T03:47:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1386999999999999e-05 2024-09-15T03:47:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159113+0000", 2024-09-15T03:47:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7020999999999998e-05 2024-09-15T03:47:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159122+0000", 2024-09-15T03:47:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4720000000000001e-06 2024-09-15T03:47:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159168+0000", 2024-09-15T03:47:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5766000000000002e-05 2024-09-15T03:47:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159176+0000", 2024-09-15T03:47:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3070000000000003e-06 2024-09-15T03:47:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159258+0000", 2024-09-15T03:47:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1398999999999998e-05 2024-09-15T03:47:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159292+0000", 2024-09-15T03:47:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4480000000000002e-05 2024-09-15T03:47:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159342+0000", 2024-09-15T03:47:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9450999999999998e-05 2024-09-15T03:47:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159362+0000", 2024-09-15T03:47:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0633e-05 2024-09-15T03:47:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134845+0000", 2024-09-15T03:47:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.975482768 2024-09-15T03:47:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=48, reqid=client.4521.0:229, at_version=0'0, trim_to=0'0, roll_forward_to=38'26))", 2024-09-15T03:47:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:47:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.468649292999999, 2024-09-15T03:47:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.974485472, 2024-09-15T03:47:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:47:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:47:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160391+0000", 2024-09-15T03:47:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.136e-06 2024-09-15T03:47:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160405+0000", 2024-09-15T03:47:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3613e-05 2024-09-15T03:47:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160426+0000", 2024-09-15T03:47:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1234999999999999e-05 2024-09-15T03:47:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160434+0000", 2024-09-15T03:47:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.61e-06 2024-09-15T03:47:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160496+0000", 2024-09-15T03:47:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2384000000000004e-05 2024-09-15T03:47:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160507+0000", 2024-09-15T03:47:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1005000000000001e-05 2024-09-15T03:47:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160632+0000", 2024-09-15T03:47:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124566 2024-09-15T03:47:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160653+0000", 2024-09-15T03:47:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0707999999999999e-05 2024-09-15T03:47:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160701+0000", 2024-09-15T03:47:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8751999999999997e-05 2024-09-15T03:47:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160739+0000", 2024-09-15T03:47:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7486e-05 2024-09-15T03:47:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134876+0000", 2024-09-15T03:47:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9741369770000001 2024-09-15T03:47:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=40, reqid=client.4521.0:238, at_version=0'0, trim_to=0'0, roll_forward_to=38'7))", 2024-09-15T03:47:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:47:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.466802872000002, 2024-09-15T03:47:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9727862730000001, 2024-09-15T03:47:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:47:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:47:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162238+0000", 2024-09-15T03:47:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.672e-06 2024-09-15T03:47:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162253+0000", 2024-09-15T03:47:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4932e-05 2024-09-15T03:47:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162284+0000", 2024-09-15T03:47:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1211000000000003e-05 2024-09-15T03:47:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162296+0000", 2024-09-15T03:47:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1229999999999999e-05 2024-09-15T03:47:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162373+0000", 2024-09-15T03:47:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7146999999999998e-05 2024-09-15T03:47:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162387+0000", 2024-09-15T03:47:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4290999999999999e-05 2024-09-15T03:47:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162502+0000", 2024-09-15T03:47:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115054 2024-09-15T03:47:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162516+0000", 2024-09-15T03:47:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3597999999999999e-05 2024-09-15T03:47:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162558+0000", 2024-09-15T03:47:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2562999999999999e-05 2024-09-15T03:47:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162584+0000", 2024-09-15T03:47:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5738e-05 2024-09-15T03:47:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135023+0000", 2024-09-15T03:47:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9724388369999999 2024-09-15T03:47:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 39/28 ECSubWrite(tid=38, reqid=osd.11.0:38, at_version=39'16, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T03:47:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.485420571999995, 2024-09-15T03:47:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99210653800000004, 2024-09-15T03:47:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:47:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1800000000000002e-07 2024-09-15T03:47:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143648+0000", 2024-09-15T03:47:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.849e-05 2024-09-15T03:47:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143670+0000", 2024-09-15T03:47:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2036e-05 2024-09-15T03:47:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143680+0000", 2024-09-15T03:47:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8320000000000001e-06 2024-09-15T03:47:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143783+0000", 2024-09-15T03:47:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103569 2024-09-15T03:47:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143801+0000", 2024-09-15T03:47:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8077999999999999e-05 2024-09-15T03:47:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144077+0000", 2024-09-15T03:47:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027569500000000001 2024-09-15T03:47:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144459+0000", 2024-09-15T03:47:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038227299999999998 2024-09-15T03:47:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144519+0000", 2024-09-15T03:47:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9907000000000001e-05 2024-09-15T03:47:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144628+0000", 2024-09-15T03:47:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108958 2024-09-15T03:47:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135726+0000", 2024-09-15T03:47:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99109738199999997 2024-09-15T03:47:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 39/31 ECSubWrite(tid=41, reqid=osd.6.0:41, at_version=39'9, trim_to=0'0, roll_forward_to=38'7))", 2024-09-15T03:47:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.484770316999999, 2024-09-15T03:47:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99106653099999997, 2024-09-15T03:47:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:47:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.97e-07 2024-09-15T03:47:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144297+0000", 2024-09-15T03:47:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7815000000000001e-05 2024-09-15T03:47:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144314+0000", 2024-09-15T03:47:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6269e-05 2024-09-15T03:47:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144322+0000", 2024-09-15T03:47:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1880000000000002e-06 2024-09-15T03:47:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144767+0000", 2024-09-15T03:47:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044555300000000001 2024-09-15T03:47:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144777+0000", 2024-09-15T03:47:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5950000000000005e-06 2024-09-15T03:47:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144992+0000", 2024-09-15T03:47:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000215479 2024-09-15T03:47:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145142+0000", 2024-09-15T03:47:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149302 2024-09-15T03:47:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145249+0000", 2024-09-15T03:47:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106971 2024-09-15T03:47:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145299+0000", 2024-09-15T03:47:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0413e-05 2024-09-15T03:47:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135336+0000", 2024-09-15T03:47:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99003674900000005 2024-09-15T03:47:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 39/28 ECSubWrite(tid=50, reqid=osd.2.0:50, at_version=39'28, trim_to=0'0, roll_forward_to=38'26))", 2024-09-15T03:47:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.484472835000005, 2024-09-15T03:47:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99095576299999999, 2024-09-15T03:47:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:47:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.03e-07 2024-09-15T03:47:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144593+0000", 2024-09-15T03:47:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6520999999999999e-05 2024-09-15T03:47:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144611+0000", 2024-09-15T03:47:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7963000000000001e-05 2024-09-15T03:47:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144619+0000", 2024-09-15T03:47:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7409999999999992e-06 2024-09-15T03:47:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144821+0000", 2024-09-15T03:47:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000201652 2024-09-15T03:47:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144832+0000", 2024-09-15T03:47:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0827e-05 2024-09-15T03:47:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145099+0000", 2024-09-15T03:47:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026688299999999999 2024-09-15T03:47:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145143+0000", 2024-09-15T03:47:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.481e-05 2024-09-15T03:47:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145249+0000", 2024-09-15T03:47:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010618799999999999 2024-09-15T03:47:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145340+0000", 2024-09-15T03:47:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0340000000000006e-05 2024-09-15T03:47:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135522+0000", 2024-09-15T03:47:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99018263500000003 2024-09-15T03:47:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=43, reqid=client.4521.0:240, at_version=39'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:47:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.481132114999994, 2024-09-15T03:47:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99104975500000003, 2024-09-15T03:47:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:47:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:47:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147908+0000", 2024-09-15T03:47:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.05e-07 2024-09-15T03:47:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147931+0000", 2024-09-15T03:47:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3621e-05 2024-09-15T03:47:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147945+0000", 2024-09-15T03:47:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3648999999999999e-05 2024-09-15T03:47:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147951+0000", 2024-09-15T03:47:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0510000000000002e-06 2024-09-15T03:47:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148055+0000", 2024-09-15T03:47:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104421 2024-09-15T03:47:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148068+0000", 2024-09-15T03:47:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3019e-05 2024-09-15T03:47:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148375+0000", 2024-09-15T03:47:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030683400000000002 2024-09-15T03:47:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148399+0000", 2024-09-15T03:47:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3760999999999998e-05 2024-09-15T03:47:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148453+0000", 2024-09-15T03:47:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3851999999999999e-05 2024-09-15T03:47:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148481+0000", 2024-09-15T03:47:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8637999999999999e-05 2024-09-15T03:47:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.138957+0000", 2024-09-15T03:47:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99047580400000002 2024-09-15T03:47:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 39/28 ECSubWrite(tid=39, reqid=osd.11.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:47:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:47:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.480999625999999, 2024-09-15T03:47:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99452509, 2024-09-15T03:47:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:47:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:47:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148041+0000", 2024-09-15T03:47:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0999999999999998e-07 2024-09-15T03:47:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148052+0000", 2024-09-15T03:47:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.129e-05 2024-09-15T03:47:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148064+0000", 2024-09-15T03:47:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173e-05 2024-09-15T03:47:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148068+0000", 2024-09-15T03:47:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5680000000000001e-06 2024-09-15T03:47:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148136+0000", 2024-09-15T03:47:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7508999999999993e-05 2024-09-15T03:47:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148148+0000", 2024-09-15T03:47:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2002e-05 2024-09-15T03:47:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148265+0000", 2024-09-15T03:47:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117669 2024-09-15T03:47:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148289+0000", 2024-09-15T03:47:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3672e-05 2024-09-15T03:47:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148348+0000", 2024-09-15T03:47:54.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8838000000000001e-05 2024-09-15T03:47:54.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148368+0000", 2024-09-15T03:47:54.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9942999999999999e-05 2024-09-15T03:47:54.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142565+0000", 2024-09-15T03:47:54.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99419715900000005 2024-09-15T03:47:54.719 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.719 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.720 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.720 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=45, reqid=client.4521.0:241, at_version=39'17, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:54.720 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:47:54.720 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.477996602999994, 2024-09-15T03:47:54.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99118202200000005, 2024-09-15T03:47:54.720 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.720 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.720 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:47:54.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:47:54.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151044+0000", 2024-09-15T03:47:54.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1340000000000001e-06 2024-09-15T03:47:54.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151073+0000", 2024-09-15T03:47:54.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8894000000000002e-05 2024-09-15T03:47:54.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151095+0000", 2024-09-15T03:47:54.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1889e-05 2024-09-15T03:47:54.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151107+0000", 2024-09-15T03:47:54.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2136999999999999e-05 2024-09-15T03:47:54.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151189+0000", 2024-09-15T03:47:54.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1883000000000002e-05 2024-09-15T03:47:54.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151203+0000", 2024-09-15T03:47:54.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3825999999999999e-05 2024-09-15T03:47:54.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151305+0000", 2024-09-15T03:47:54.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101816 2024-09-15T03:47:54.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151337+0000", 2024-09-15T03:47:54.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2703000000000002e-05 2024-09-15T03:47:54.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151387+0000", 2024-09-15T03:47:54.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9366000000000001e-05 2024-09-15T03:47:54.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151405+0000", 2024-09-15T03:47:54.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8748000000000001e-05 2024-09-15T03:47:54.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142225+0000", 2024-09-15T03:47:54.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99081962599999995 2024-09-15T03:47:54.725 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.725 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.725 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.726 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 39/31 ECSubWrite(tid=42, reqid=osd.6.0:41, at_version=0'0, trim_to=0'0, roll_forward_to=39'9))", 2024-09-15T03:47:54.726 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:47:54.726 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.477700561999995, 2024-09-15T03:47:54.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99092639999999999, 2024-09-15T03:47:54.726 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.726 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.726 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:47:54.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:47:54.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151340+0000", 2024-09-15T03:47:54.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8299999999999996e-07 2024-09-15T03:47:54.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151350+0000", 2024-09-15T03:47:54.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0675e-05 2024-09-15T03:47:54.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151362+0000", 2024-09-15T03:47:54.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2048999999999999e-05 2024-09-15T03:47:54.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151369+0000", 2024-09-15T03:47:54.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3230000000000003e-06 2024-09-15T03:47:54.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151425+0000", 2024-09-15T03:47:54.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5964000000000002e-05 2024-09-15T03:47:54.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151435+0000", 2024-09-15T03:47:54.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9240000000000002e-06 2024-09-15T03:47:54.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151530+0000", 2024-09-15T03:47:54.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5649000000000005e-05 2024-09-15T03:47:54.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151558+0000", 2024-09-15T03:47:54.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7976000000000001e-05 2024-09-15T03:47:54.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151599+0000", 2024-09-15T03:47:54.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1152000000000002e-05 2024-09-15T03:47:54.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151617+0000", 2024-09-15T03:47:54.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7676000000000002e-05 2024-09-15T03:47:54.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142265+0000", 2024-09-15T03:47:54.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99064822900000005 2024-09-15T03:47:54.731 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.731 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.731 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.731 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=46, reqid=client.4521.0:242, at_version=39'18, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:47:54.731 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:54.732 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.477674926000006, 2024-09-15T03:47:54.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.991000204, 2024-09-15T03:47:54.732 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.732 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.732 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:54.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:54.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:47:54.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3399999999999999e-07 2024-09-15T03:47:54.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151393+0000", 2024-09-15T03:47:54.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8149000000000001e-05 2024-09-15T03:47:54.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151405+0000", 2024-09-15T03:47:54.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2063e-05 2024-09-15T03:47:54.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151415+0000", 2024-09-15T03:47:54.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4579999999999999e-06 2024-09-15T03:47:54.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151472+0000", 2024-09-15T03:47:54.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7018999999999997e-05 2024-09-15T03:47:54.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151480+0000", 2024-09-15T03:47:54.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1410000000000005e-06 2024-09-15T03:47:54.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151571+0000", 2024-09-15T03:47:54.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1243999999999996e-05 2024-09-15T03:47:54.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151602+0000", 2024-09-15T03:47:54.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0834000000000003e-05 2024-09-15T03:47:54.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151632+0000", 2024-09-15T03:47:54.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9527e-05 2024-09-15T03:47:54.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151685+0000", 2024-09-15T03:47:54.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3599e-05 2024-09-15T03:47:54.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142365+0000", 2024-09-15T03:47:54.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99067953600000003 2024-09-15T03:47:54.737 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.737 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.737 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.738 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 39/28 ECSubWrite(tid=51, reqid=osd.2.0:50, at_version=0'0, trim_to=0'0, roll_forward_to=39'28))", 2024-09-15T03:47:54.738 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:47:54.738 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.474311487999998, 2024-09-15T03:47:54.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98766307399999997, 2024-09-15T03:47:54.738 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.738 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.738 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:47:54.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:47:54.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154729+0000", 2024-09-15T03:47:54.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3799999999999996e-07 2024-09-15T03:47:54.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154742+0000", 2024-09-15T03:47:54.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2909e-05 2024-09-15T03:47:54.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154755+0000", 2024-09-15T03:47:54.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3251999999999999e-05 2024-09-15T03:47:54.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154761+0000", 2024-09-15T03:47:54.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6559999999999997e-06 2024-09-15T03:47:54.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154829+0000", 2024-09-15T03:47:54.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8549999999999999e-05 2024-09-15T03:47:54.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154840+0000", 2024-09-15T03:47:54.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0159999999999999e-05 2024-09-15T03:47:54.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154943+0000", 2024-09-15T03:47:54.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010347400000000001 2024-09-15T03:47:54.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154955+0000", 2024-09-15T03:47:54.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1935e-05 2024-09-15T03:47:54.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156442+0000", 2024-09-15T03:47:54.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014867859999999999 2024-09-15T03:47:54.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156466+0000", 2024-09-15T03:47:54.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4077999999999999e-05 2024-09-15T03:47:54.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142391+0000", 2024-09-15T03:47:54.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98592533599999999 2024-09-15T03:47:54.743 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.743 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.743 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.744 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=48, reqid=client.4521.0:242, at_version=0'0, trim_to=0'0, roll_forward_to=39'18))", 2024-09-15T03:47:54.744 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:47:54.744 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.472424469000003, 2024-09-15T03:47:54.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98592495099999999, 2024-09-15T03:47:54.744 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:54.744 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:54.744 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:54.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:54.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:47:54.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:54.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:47:54.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:54.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:54.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156616+0000", 2024-09-15T03:47:54.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1999999999999998e-07 2024-09-15T03:47:54.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:54.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156629+0000", 2024-09-15T03:47:54.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2870999999999999e-05 2024-09-15T03:47:54.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:54.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156649+0000", 2024-09-15T03:47:54.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9786e-05 2024-09-15T03:47:54.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:54.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156657+0000", 2024-09-15T03:47:54.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5110000000000003e-06 2024-09-15T03:47:54.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:54.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156723+0000", 2024-09-15T03:47:54.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6209000000000003e-05 2024-09-15T03:47:54.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:54.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156734+0000", 2024-09-15T03:47:54.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0475999999999999e-05 2024-09-15T03:47:54.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:54.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156868+0000", 2024-09-15T03:47:54.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013402999999999999 2024-09-15T03:47:54.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:54.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156889+0000", 2024-09-15T03:47:54.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1146e-05 2024-09-15T03:47:54.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:54.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156937+0000", 2024-09-15T03:47:54.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8031999999999997e-05 2024-09-15T03:47:54.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:54.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156970+0000", 2024-09-15T03:47:54.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2462999999999997e-05 2024-09-15T03:47:54.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:54.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:54.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:54.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142540+0000", 2024-09-15T03:47:54.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98557050700000004 2024-09-15T03:47:54.749 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.749 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.749 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.749 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:54.749 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:54.750 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:54.751 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:47:54.752 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:54.750+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:54.840 INFO:tasks.rados.rados.0.smithi001.stdout:355: done (5 left) 2024-09-15T03:47:54.840 INFO:tasks.rados.rados.0.smithi001.stdout:357: done (4 left) 2024-09-15T03:47:54.840 INFO:tasks.rados.rados.0.smithi001.stdout:358: done (3 left) 2024-09-15T03:47:54.840 INFO:tasks.rados.rados.0.smithi001.stdout:361: done (2 left) 2024-09-15T03:47:54.840 INFO:tasks.rados.rados.0.smithi001.stdout:362: done (1 left) 2024-09-15T03:47:54.840 INFO:tasks.rados.rados.0.smithi001.stdout:363: done (0 left) 2024-09-15T03:47:54.840 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 40 current snap is 48 2024-09-15T03:47:54.840 INFO:tasks.rados.rados.0.smithi001.stdout:364: seq_num 137 ranges {0=753664,753664=786432,1540096=655360,2195456=114688} 2024-09-15T03:47:54.850 INFO:tasks.rados.rados.0.smithi001.stdout:364: writing smithi00132172-40 from 0 to 753664 tid 1 2024-09-15T03:47:54.851 INFO:tasks.rados.rados.0.smithi001.stdout:364: writing smithi00132172-40 from 753664 to 1540096 tid 2 2024-09-15T03:47:54.852 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:54.850+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:54.855 INFO:tasks.rados.rados.0.smithi001.stdout:364: writing smithi00132172-40 from 1540096 to 2195456 tid 3 2024-09-15T03:47:54.856 INFO:tasks.rados.rados.0.smithi001.stdout:364: writing smithi00132172-40 from 2195456 to 2310144 tid 4 2024-09-15T03:47:54.856 INFO:tasks.rados.rados.0.smithi001.stdout:365: read oid 1 snap 34 2024-09-15T03:47:54.856 INFO:tasks.rados.rados.0.smithi001.stdout:365: expect deleted 2024-09-15T03:47:54.856 INFO:tasks.rados.rados.0.smithi001.stdout:366: delete oid 26 current snap is 48 2024-09-15T03:47:54.861 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:54.861 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:54.861 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:54.862 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:54.870 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:47:54.952 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:54.952+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:54.970 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:54.971 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:54.971 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:54.971 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:54.971 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:54.979 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:47:55.054 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:55.053+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:55.097 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:47:55.121 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:55.121 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:55.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=21, reqid=client.4521.0:184, at_version=33'6, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 75.012728136999996, 2024-09-15T03:47:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010901256999999999, 2024-09-15T03:47:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:55.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:55.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106509+0000", 2024-09-15T03:47:55.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0200000000000001e-07 2024-09-15T03:47:55.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106529+0000", 2024-09-15T03:47:55.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9539e-05 2024-09-15T03:47:55.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106542+0000", 2024-09-15T03:47:55.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2731999999999999e-05 2024-09-15T03:47:55.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106549+0000", 2024-09-15T03:47:55.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3339999999999996e-06 2024-09-15T03:47:55.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106613+0000", 2024-09-15T03:47:55.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4527000000000001e-05 2024-09-15T03:47:55.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106622+0000", 2024-09-15T03:47:55.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.755e-06 2024-09-15T03:47:55.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106841+0000", 2024-09-15T03:47:55.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021906200000000001 2024-09-15T03:47:55.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106869+0000", 2024-09-15T03:47:55.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.739e-05 2024-09-15T03:47:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106948+0000", 2024-09-15T03:47:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9422e-05 2024-09-15T03:47:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.106965+0000", 2024-09-15T03:47:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7010000000000001e-05 2024-09-15T03:47:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.117410+0000", 2024-09-15T03:47:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010445284000000001 2024-09-15T03:47:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=23, reqid=client.4521.0:188, at_version=33'9, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 75.01081465, 2024-09-15T03:47:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015296944, 2024-09-15T03:47:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.129 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:55.130 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:55.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108423+0000", 2024-09-15T03:47:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5499999999999998e-07 2024-09-15T03:47:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108447+0000", 2024-09-15T03:47:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3314000000000001e-05 2024-09-15T03:47:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108460+0000", 2024-09-15T03:47:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3444e-05 2024-09-15T03:47:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108464+0000", 2024-09-15T03:47:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3610000000000003e-06 2024-09-15T03:47:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108494+0000", 2024-09-15T03:47:55.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9688000000000001e-05 2024-09-15T03:47:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108502+0000", 2024-09-15T03:47:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.678e-06 2024-09-15T03:47:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108601+0000", 2024-09-15T03:47:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9748999999999997e-05 2024-09-15T03:47:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108610+0000", 2024-09-15T03:47:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2490000000000001e-06 2024-09-15T03:47:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109977+0000", 2024-09-15T03:47:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013669450000000001 2024-09-15T03:47:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109993+0000", 2024-09-15T03:47:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6688e-05 2024-09-15T03:47:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123720+0000", 2024-09-15T03:47:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013726172999999999 2024-09-15T03:47:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=24, reqid=client.4521.0:185, at_version=33'7, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 75.010764988999995, 2024-09-15T03:47:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015105210000000001, 2024-09-15T03:47:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.135 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108472+0000", 2024-09-15T03:47:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108473+0000", 2024-09-15T03:47:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.84e-07 2024-09-15T03:47:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108486+0000", 2024-09-15T03:47:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3845999999999999e-05 2024-09-15T03:47:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108491+0000", 2024-09-15T03:47:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4719999999999997e-06 2024-09-15T03:47:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108495+0000", 2024-09-15T03:47:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.822e-06 2024-09-15T03:47:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108546+0000", 2024-09-15T03:47:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0853000000000003e-05 2024-09-15T03:47:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108555+0000", 2024-09-15T03:47:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3319999999999998e-06 2024-09-15T03:47:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108638+0000", 2024-09-15T03:47:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3580999999999998e-05 2024-09-15T03:47:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109983+0000", 2024-09-15T03:47:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001344392 2024-09-15T03:47:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110023+0000", 2024-09-15T03:47:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9895999999999999e-05 2024-09-15T03:47:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110040+0000", 2024-09-15T03:47:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.747e-05 2024-09-15T03:47:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123577+0000", 2024-09-15T03:47:55.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013537262 2024-09-15T03:47:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.fs7 33/31 ECSubWrite(tid=25, reqid=client.4521.0:189, at_version=33'10, trim_to=0'0, roll_forward_to=32'6))", 2024-09-15T03:47:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 75.010731109999995, 2024-09-15T03:47:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015311313999999999, 2024-09-15T03:47:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:55.141 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108506+0000", 2024-09-15T03:47:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8500000000000002e-07 2024-09-15T03:47:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108517+0000", 2024-09-15T03:47:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0543000000000001e-05 2024-09-15T03:47:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108524+0000", 2024-09-15T03:47:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7170000000000002e-06 2024-09-15T03:47:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108528+0000", 2024-09-15T03:47:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3970000000000004e-06 2024-09-15T03:47:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108632+0000", 2024-09-15T03:47:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103687 2024-09-15T03:47:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108637+0000", 2024-09-15T03:47:55.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4389999999999999e-06 2024-09-15T03:47:55.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.108713+0000", 2024-09-15T03:47:55.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6223000000000005e-05 2024-09-15T03:47:55.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109984+0000", 2024-09-15T03:47:55.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012700719999999999 2024-09-15T03:47:55.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110023+0000", 2024-09-15T03:47:55.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9864999999999999e-05 2024-09-15T03:47:55.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110041+0000", 2024-09-15T03:47:55.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8040999999999999e-05 2024-09-15T03:47:55.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123817+0000", 2024-09-15T03:47:55.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013776045000000001 2024-09-15T03:47:55.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.147 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.147 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.147 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 33/28 ECSubWrite(tid=26, reqid=client.4521.0:186, at_version=33'8, trim_to=0'0, roll_forward_to=32'4))", 2024-09-15T03:47:55.147 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:55.147 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 75.010237501999995, 2024-09-15T03:47:55.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014864502, 2024-09-15T03:47:55.147 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.147 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:55.147 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:55.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:55.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109000+0000", 2024-09-15T03:47:55.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:47:55.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109016+0000", 2024-09-15T03:47:55.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5889999999999999e-05 2024-09-15T03:47:55.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109029+0000", 2024-09-15T03:47:55.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2812e-05 2024-09-15T03:47:55.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109035+0000", 2024-09-15T03:47:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0109999999999997e-06 2024-09-15T03:47:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109073+0000", 2024-09-15T03:47:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8260000000000003e-05 2024-09-15T03:47:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109080+0000", 2024-09-15T03:47:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0190000000000001e-06 2024-09-15T03:47:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109165+0000", 2024-09-15T03:47:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4957000000000007e-05 2024-09-15T03:47:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.109984+0000", 2024-09-15T03:47:55.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081866399999999998 2024-09-15T03:47:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110024+0000", 2024-09-15T03:47:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9644999999999997e-05 2024-09-15T03:47:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.110072+0000", 2024-09-15T03:47:55.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8418999999999998e-05 2024-09-15T03:47:55.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:40.123864+0000", 2024-09-15T03:47:55.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013792234 2024-09-15T03:47:55.153 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.153 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.153 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.153 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=34, reqid=client.4521.0:197, at_version=34'21, trim_to=0'0, roll_forward_to=34'18))", 2024-09-15T03:47:55.153 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.995225622000007, 2024-09-15T03:47:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012992399, 2024-09-15T03:47:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124012+0000", 2024-09-15T03:47:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124013+0000", 2024-09-15T03:47:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6299999999999993e-07 2024-09-15T03:47:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124044+0000", 2024-09-15T03:47:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0995999999999999e-05 2024-09-15T03:47:55.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124062+0000", 2024-09-15T03:47:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8413000000000001e-05 2024-09-15T03:47:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124069+0000", 2024-09-15T03:47:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3930000000000002e-06 2024-09-15T03:47:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124150+0000", 2024-09-15T03:47:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1097e-05 2024-09-15T03:47:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124161+0000", 2024-09-15T03:47:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0720000000000001e-05 2024-09-15T03:47:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124402+0000", 2024-09-15T03:47:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000240442 2024-09-15T03:47:55.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125141+0000", 2024-09-15T03:47:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00073915500000000002 2024-09-15T03:47:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125191+0000", 2024-09-15T03:47:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9951999999999999e-05 2024-09-15T03:47:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125251+0000", 2024-09-15T03:47:55.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9938000000000001e-05 2024-09-15T03:47:55.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.137004+0000", 2024-09-15T03:47:55.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011753329999999999 2024-09-15T03:47:55.159 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.159 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.160 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.160 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/31 ECSubWrite(tid=35, reqid=client.4521.0:197, at_version=0'0, trim_to=0'0, roll_forward_to=34'21))", 2024-09-15T03:47:55.161 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:55.161 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.993285577999998, 2024-09-15T03:47:55.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011202456, 2024-09-15T03:47:55.161 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.161 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:55.161 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125952+0000", 2024-09-15T03:47:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125953+0000", 2024-09-15T03:47:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9100000000000002e-07 2024-09-15T03:47:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125964+0000", 2024-09-15T03:47:55.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1399999999999999e-05 2024-09-15T03:47:55.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125979+0000", 2024-09-15T03:47:55.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4810000000000001e-05 2024-09-15T03:47:55.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126004+0000", 2024-09-15T03:47:55.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4913e-05 2024-09-15T03:47:55.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126059+0000", 2024-09-15T03:47:55.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5668999999999998e-05 2024-09-15T03:47:55.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126071+0000", 2024-09-15T03:47:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1219999999999999e-05 2024-09-15T03:47:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126151+0000", 2024-09-15T03:47:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0572e-05 2024-09-15T03:47:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.126163+0000", 2024-09-15T03:47:55.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2101e-05 2024-09-15T03:47:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.127945+0000", 2024-09-15T03:47:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001781803 2024-09-15T03:47:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.127982+0000", 2024-09-15T03:47:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7129999999999999e-05 2024-09-15T03:47:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.137154+0000", 2024-09-15T03:47:55.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0091719469999999997 2024-09-15T03:47:55.167 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.167 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.167 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.167 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:243 3.3s0 3:c7c3ef16:::smithi00132172-25:head [create,setxattr iZnWfebab`ptjURR\\f`psW]tpS`qpUaeWkY[Oqep (1330) in=40b,setxattr f[mZ_qQmcZejQ`fbdeic_tkPUaTYcnm[WprjVmc] (12488) in=40b,setxattr tkZgQ]S_pdlhZnqrVaZRa`oQQr_USmUfhado^dXT (1121) in=40b,setxattr srWjTqXhcth`hfZmQY_fg]UqNjWofa^dfgOV`sYP (455) in=40b,setxattr ^_mdiRroj_TtQ^gcNj`XtkW[ooqY`PWXqYmcOk_n (138) in=40b,setxattr jQlajdtcalhkcfYaakZfNb[^_TjkfcS`Vrk]q_d[ (806) in=40b,setxattr ZfYgkVcrsXcnpchgfcbldNrUTasnQNmfZW]Nbh\\o (561) in=40b,setxattr _hkYOqjPNhmhUlSNktfeQpsqSWjO\\fbf[fpTlZiO (1227) in=40b,setxattr ]mdV^Xdc\\sViXdkXNU[js_Qhh[b[sbfnZVXkdids (11788) in=40b,setxattr tb^mbOrNhfrhtce^g (455) in=17b,setxattr koTkVgrfU\\m`StnZ`Rirjhbhg`PhVqm^a^Xgaaqn (35) in=40b,setxattr aUtZPUPXPnQmT]\\\\n\\lZpnXakdNkTldVQoQhPUf] (559) in=40b,setxattr mrhR]^U`Tteo[Topmql`SNma\\pqfk`if]OppnYt` (770) in=40b,setxattr Oetk]RrnWeXbVctVtVrhXf_RfTdelqX^lWp[sTkp (1226) in=40b,setxattr WtedXoOd`VqnbksOqQkatk`gdTnW^lXbUtrlYgkl (3576) in=40b,setxattr dddOWN\\j\\XXg^fr[rZcskeVX]jOQ^ZmoWepk[]\\a (17824) in=40b,setxattr TeRO^djPgNZiemQ_aRXgP[]jgUZa\\^\\iUgesaqaN (208) in=40b,setxattr teqm]mUobZbWjcdQ_jTa[f[k[Sd\\lh]n\\osigqkN (7436) in=40b,setxattr dphaol[XkeitbkomWYaQctWfataUhbSiNntkWo`P (1893) in=40b,setxattr oQsnhO]ZklSVYQ[WUpaelY[p[Qie^UepOt`Z[q (455) in=38b] snapc 7=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-15T03:47:55.167 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:55.167 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.950321689999996, 2024-09-15T03:47:55.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012365956000000001, 2024-09-15T03:47:55.167 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.167 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:55.168 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.168 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.168 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.168 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 243 2024-09-15T03:47:55.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.168 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:55.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168916+0000", 2024-09-15T03:47:55.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.168917+0000", 2024-09-15T03:47:55.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.725e-06 2024-09-15T03:47:55.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169030+0000", 2024-09-15T03:47:55.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113035 2024-09-15T03:47:55.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169047+0000", 2024-09-15T03:47:55.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6701000000000002e-05 2024-09-15T03:47:55.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169054+0000", 2024-09-15T03:47:55.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1369999999999996e-06 2024-09-15T03:47:55.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169140+0000", 2024-09-15T03:47:55.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6069999999999994e-05 2024-09-15T03:47:55.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.169270+0000", 2024-09-15T03:47:55.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130071 2024-09-15T03:47:55.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.170661+0000", 2024-09-15T03:47:55.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001390666 2024-09-15T03:47:55.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171751+0000", 2024-09-15T03:47:55.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010899079999999999 2024-09-15T03:47:55.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.171812+0000", 2024-09-15T03:47:55.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0797000000000001e-05 2024-09-15T03:47:55.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172279+0000", 2024-09-15T03:47:55.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046704500000000003 2024-09-15T03:47:55.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.172299+0000", 2024-09-15T03:47:55.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0387999999999999e-05 2024-09-15T03:47:55.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:55.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.176110+0000", 2024-09-15T03:47:55.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038112340000000001 2024-09-15T03:47:55.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.181282+0000", 2024-09-15T03:47:55.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051711789999999997 2024-09-15T03:47:55.174 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.174 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.174 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.174 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/28 ECSubWrite(tid=15, reqid=client.4521.0:337, at_version=49'10, trim_to=0'0, roll_forward_to=43'8))", 2024-09-15T03:47:55.175 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:55.175 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.261331781000003, 2024-09-15T03:47:55.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010500893000000001, 2024-09-15T03:47:55.175 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.175 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:55.175 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:55.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857905+0000", 2024-09-15T03:47:55.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857906+0000", 2024-09-15T03:47:55.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.63e-07 2024-09-15T03:47:55.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857928+0000", 2024-09-15T03:47:55.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2155999999999999e-05 2024-09-15T03:47:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857937+0000", 2024-09-15T03:47:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2359999999999993e-06 2024-09-15T03:47:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857941+0000", 2024-09-15T03:47:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0960000000000003e-06 2024-09-15T03:47:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857999+0000", 2024-09-15T03:47:55.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7407000000000001e-05 2024-09-15T03:47:55.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858011+0000", 2024-09-15T03:47:55.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1759999999999999e-05 2024-09-15T03:47:55.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858240+0000", 2024-09-15T03:47:55.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022949299999999999 2024-09-15T03:47:55.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858251+0000", 2024-09-15T03:47:55.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1168999999999999e-05 2024-09-15T03:47:55.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858332+0000", 2024-09-15T03:47:55.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1204000000000004e-05 2024-09-15T03:47:55.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858360+0000", 2024-09-15T03:47:55.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7767999999999999e-05 2024-09-15T03:47:55.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.868406+0000", 2024-09-15T03:47:55.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010046141 2024-09-15T03:47:55.180 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.180 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.180 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.180 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:349 3.3s0 3:c7c3ef16:::smithi00132172-25:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:55.180 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:55.180 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.280755755000001, 2024-09-15T03:47:55.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005145406, 2024-09-15T03:47:55.181 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.181 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:55.181 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.181 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.181 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.181 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 349 2024-09-15T03:47:55.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.181 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:55.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838481+0000", 2024-09-15T03:47:55.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838485+0000", 2024-09-15T03:47:55.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0620000000000001e-06 2024-09-15T03:47:55.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838494+0000", 2024-09-15T03:47:55.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3179999999999996e-06 2024-09-15T03:47:55.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:55.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3689999999999998e-05 2024-09-15T03:47:55.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838812+0000", 2024-09-15T03:47:55.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029437300000000001 2024-09-15T03:47:55.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838905+0000", 2024-09-15T03:47:55.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3196999999999996e-05 2024-09-15T03:47:55.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:55.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838914+0000", 2024-09-15T03:47:55.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0890000000000006e-06 2024-09-15T03:47:55.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840773+0000", 2024-09-15T03:47:55.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018584820000000001 2024-09-15T03:47:55.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840889+0000", 2024-09-15T03:47:55.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115986 2024-09-15T03:47:55.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842016+0000", 2024-09-15T03:47:55.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001126878 2024-09-15T03:47:55.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842320+0000", 2024-09-15T03:47:55.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030409799999999999 2024-09-15T03:47:55.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842360+0000", 2024-09-15T03:47:55.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0750000000000001e-05 2024-09-15T03:47:55.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842448+0000", 2024-09-15T03:47:55.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7842000000000004e-05 2024-09-15T03:47:55.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844580+0000", 2024-09-15T03:47:55.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021313719999999999 2024-09-15T03:47:55.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:55.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843574+0000", 2024-09-15T03:47:55.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99899468199999997 2024-09-15T03:47:55.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843627+0000", 2024-09-15T03:47:55.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2587000000000003e-05 2024-09-15T03:47:55.188 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.188 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.188 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.189 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:350 3.3s0 3:c7c3ef16:::smithi00132172-25:head [rollback 14] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:55.189 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:55.189 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.280404093999998, 2024-09-15T03:47:55.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0050383839999999, 2024-09-15T03:47:55.189 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.189 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:55.189 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.189 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.189 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.189 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 350 2024-09-15T03:47:55.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.190 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:55.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838833+0000", 2024-09-15T03:47:55.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838834+0000", 2024-09-15T03:47:55.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0200000000000001e-07 2024-09-15T03:47:55.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838839+0000", 2024-09-15T03:47:55.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9790000000000001e-06 2024-09-15T03:47:55.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838852+0000", 2024-09-15T03:47:55.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3138e-05 2024-09-15T03:47:55.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838858+0000", 2024-09-15T03:47:55.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8320000000000002e-06 2024-09-15T03:47:55.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839003+0000", 2024-09-15T03:47:55.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014513500000000001 2024-09-15T03:47:55.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:55.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839011+0000", 2024-09-15T03:47:55.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7719999999999994e-06 2024-09-15T03:47:55.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842395+0000", 2024-09-15T03:47:55.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033840110000000001 2024-09-15T03:47:55.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842580+0000", 2024-09-15T03:47:55.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000185067 2024-09-15T03:47:55.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843394+0000", 2024-09-15T03:47:55.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081377299999999995 2024-09-15T03:47:55.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843510+0000", 2024-09-15T03:47:55.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0001165 2024-09-15T03:47:55.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843524+0000", 2024-09-15T03:47:55.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4022999999999999e-05 2024-09-15T03:47:55.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843579+0000", 2024-09-15T03:47:55.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4654999999999998e-05 2024-09-15T03:47:55.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844583+0000", 2024-09-15T03:47:55.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010042429999999999 2024-09-15T03:47:55.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:55.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843772+0000", 2024-09-15T03:47:55.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99918911200000005 2024-09-15T03:47:55.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843872+0000", 2024-09-15T03:47:55.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9341999999999999e-05 2024-09-15T03:47:55.196 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.196 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.196 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.197 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:351 3.3s0 3:c7c3ef16:::smithi00132172-25:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:55.197 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:55.197 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.280346461000001, 2024-09-15T03:47:55.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0051167679999999, 2024-09-15T03:47:55.197 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.197 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:55.197 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.197 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.197 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.197 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 351 2024-09-15T03:47:55.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.197 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:55.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:55.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838891+0000", 2024-09-15T03:47:55.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0800000000000004e-07 2024-09-15T03:47:55.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838895+0000", 2024-09-15T03:47:55.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.241e-06 2024-09-15T03:47:55.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838904+0000", 2024-09-15T03:47:55.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9549999999999998e-06 2024-09-15T03:47:55.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838909+0000", 2024-09-15T03:47:55.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7869999999999998e-06 2024-09-15T03:47:55.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839045+0000", 2024-09-15T03:47:55.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013556399999999999 2024-09-15T03:47:55.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:55.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.839052+0000", 2024-09-15T03:47:55.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8199999999999999e-06 2024-09-15T03:47:55.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843587+0000", 2024-09-15T03:47:55.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045349739999999998 2024-09-15T03:47:55.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843708+0000", 2024-09-15T03:47:55.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120776 2024-09-15T03:47:55.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844246+0000", 2024-09-15T03:47:55.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053838200000000001 2024-09-15T03:47:55.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844335+0000", 2024-09-15T03:47:55.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9240999999999995e-05 2024-09-15T03:47:55.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844347+0000", 2024-09-15T03:47:55.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2187e-05 2024-09-15T03:47:55.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844460+0000", 2024-09-15T03:47:55.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112571 2024-09-15T03:47:55.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844584+0000", 2024-09-15T03:47:55.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012403800000000001 2024-09-15T03:47:55.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:55.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843972+0000", 2024-09-15T03:47:55.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99938787299999998 2024-09-15T03:47:55.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844008+0000", 2024-09-15T03:47:55.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5750999999999997e-05 2024-09-15T03:47:55.204 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.204 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.204 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.204 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/31 ECSubWrite(tid=91, reqid=client.4521.0:347, at_version=51'13, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:55.204 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:55.205 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.27321251, 2024-09-15T03:47:55.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013010173, 2024-09-15T03:47:55.205 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.205 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:55.205 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:55.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846025+0000", 2024-09-15T03:47:55.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846026+0000", 2024-09-15T03:47:55.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1230000000000001e-06 2024-09-15T03:47:55.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846047+0000", 2024-09-15T03:47:55.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0774000000000001e-05 2024-09-15T03:47:55.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846064+0000", 2024-09-15T03:47:55.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7560000000000001e-05 2024-09-15T03:47:55.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846072+0000", 2024-09-15T03:47:55.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5020000000000003e-06 2024-09-15T03:47:55.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846187+0000", 2024-09-15T03:47:55.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115717 2024-09-15T03:47:55.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846201+0000", 2024-09-15T03:47:55.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3329999999999999e-05 2024-09-15T03:47:55.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846303+0000", 2024-09-15T03:47:55.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101807 2024-09-15T03:47:55.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846312+0000", 2024-09-15T03:47:55.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5529999999999998e-06 2024-09-15T03:47:55.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846382+0000", 2024-09-15T03:47:55.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0146000000000002e-05 2024-09-15T03:47:55.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846520+0000", 2024-09-15T03:47:55.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013814099999999999 2024-09-15T03:47:55.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.859035+0000", 2024-09-15T03:47:55.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012514519999999999 2024-09-15T03:47:55.210 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.210 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.210 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.210 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/31 ECSubWrite(tid=92, reqid=client.4521.0:348, at_version=51'14, trim_to=0'0, roll_forward_to=34'9))", 2024-09-15T03:47:55.210 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:55.210 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.273115763, 2024-09-15T03:47:55.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013044694000000001, 2024-09-15T03:47:55.211 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.211 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:55.211 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:55.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846121+0000", 2024-09-15T03:47:55.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846122+0000", 2024-09-15T03:47:55.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0399999999999996e-07 2024-09-15T03:47:55.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846174+0000", 2024-09-15T03:47:55.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1672999999999998e-05 2024-09-15T03:47:55.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846184+0000", 2024-09-15T03:47:55.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0514e-05 2024-09-15T03:47:55.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846190+0000", 2024-09-15T03:47:55.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2820000000000002e-06 2024-09-15T03:47:55.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846341+0000", 2024-09-15T03:47:55.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150803 2024-09-15T03:47:55.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846351+0000", 2024-09-15T03:47:55.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3899999999999999e-06 2024-09-15T03:47:55.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846450+0000", 2024-09-15T03:47:55.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9839000000000004e-05 2024-09-15T03:47:55.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846482+0000", 2024-09-15T03:47:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1089e-05 2024-09-15T03:47:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846615+0000", 2024-09-15T03:47:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013305899999999999 2024-09-15T03:47:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.846634+0000", 2024-09-15T03:47:55.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8899999999999999e-05 2024-09-15T03:47:55.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.859166+0000", 2024-09-15T03:47:55.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012532641000000001 2024-09-15T03:47:55.218 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.218 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.218 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.218 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as6 60/28 ECSubWrite(tid=73, reqid=client.4521.0:424, at_version=60'41, trim_to=0'0, roll_forward_to=47'38))", 2024-09-15T03:47:55.218 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:55.219 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 48.218806915999998, 2024-09-15T03:47:55.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011124821, 2024-09-15T03:47:55.219 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.219 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:55.219 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:55.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900430+0000", 2024-09-15T03:47:55.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900431+0000", 2024-09-15T03:47:55.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2900000000000003e-07 2024-09-15T03:47:55.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900451+0000", 2024-09-15T03:47:55.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0066999999999999e-05 2024-09-15T03:47:55.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900466+0000", 2024-09-15T03:47:55.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5062000000000001e-05 2024-09-15T03:47:55.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900474+0000", 2024-09-15T03:47:55.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6310000000000003e-06 2024-09-15T03:47:55.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900524+0000", 2024-09-15T03:47:55.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0037000000000002e-05 2024-09-15T03:47:55.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900535+0000", 2024-09-15T03:47:55.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1209000000000001e-05 2024-09-15T03:47:55.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900659+0000", 2024-09-15T03:47:55.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012441000000000001 2024-09-15T03:47:55.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.900971+0000", 2024-09-15T03:47:55.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031183799999999999 2024-09-15T03:47:55.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901144+0000", 2024-09-15T03:47:55.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172261 2024-09-15T03:47:55.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.901734+0000", 2024-09-15T03:47:55.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059069700000000005 2024-09-15T03:47:55.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:06.911555+0000", 2024-09-15T03:47:55.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0098208800000000006 2024-09-15T03:47:55.224 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.224 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.224 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.224 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPush(2.2 88/87 [PushOp(2:5cea7035:::rbd_info:head, version: 24'2, data_included: [0~19], data_size: 19, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:5cea7035:::rbd_info:head@24'2, size: 19, copy_subset: [0~19], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:19, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-15T03:47:55.224 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.846903+0000", 2024-09-15T03:47:55.224 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.272333790000001, 2024-09-15T03:47:55.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013354107, 2024-09-15T03:47:55.225 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.225 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:55.225 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846903+0000", 2024-09-15T03:47:55.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846903+0000", 2024-09-15T03:47:55.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846904+0000", 2024-09-15T03:47:55.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4099999999999998e-07 2024-09-15T03:47:55.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846920+0000", 2024-09-15T03:47:55.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6061000000000001e-05 2024-09-15T03:47:55.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846945+0000", 2024-09-15T03:47:55.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4519e-05 2024-09-15T03:47:55.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.846958+0000", 2024-09-15T03:47:55.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3546999999999999e-05 2024-09-15T03:47:55.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.847077+0000", 2024-09-15T03:47:55.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011864 2024-09-15T03:47:55.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for flush", 2024-09-15T03:47:55.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.847094+0000", 2024-09-15T03:47:55.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7017e-05 2024-09-15T03:47:55.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.859991+0000", 2024-09-15T03:47:55.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012897136 2024-09-15T03:47:55.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.860001+0000", 2024-09-15T03:47:55.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5089999999999999e-06 2024-09-15T03:47:55.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.860258+0000", 2024-09-15T03:47:55.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000256937 2024-09-15T03:47:55.229 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.229 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.229 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.229 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.bs1 88/87 ECSubWrite(tid=210, reqid=client.4521.0:740, at_version=88'113, trim_to=0'0, roll_forward_to=88'111))", 2024-09-15T03:47:55.230 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.915188+0000", 2024-09-15T03:47:55.230 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.204049664999999, 2024-09-15T03:47:55.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012691618999999999, 2024-09-15T03:47:55.230 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.230 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:55.230 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915188+0000", 2024-09-15T03:47:55.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915188+0000", 2024-09-15T03:47:55.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915188+0000", 2024-09-15T03:47:55.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3399999999999999e-07 2024-09-15T03:47:55.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915216+0000", 2024-09-15T03:47:55.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8067000000000001e-05 2024-09-15T03:47:55.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915233+0000", 2024-09-15T03:47:55.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.683e-05 2024-09-15T03:47:55.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915239+0000", 2024-09-15T03:47:55.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5720000000000002e-06 2024-09-15T03:47:55.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915313+0000", 2024-09-15T03:47:55.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4066000000000005e-05 2024-09-15T03:47:55.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915325+0000", 2024-09-15T03:47:55.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.252e-05 2024-09-15T03:47:55.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915603+0000", 2024-09-15T03:47:55.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027800699999999998 2024-09-15T03:47:55.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915645+0000", 2024-09-15T03:47:55.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1777000000000003e-05 2024-09-15T03:47:55.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.915687+0000", 2024-09-15T03:47:55.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2181000000000002e-05 2024-09-15T03:47:55.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.917623+0000", 2024-09-15T03:47:55.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019357210000000001 2024-09-15T03:47:55.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.927879+0000", 2024-09-15T03:47:55.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010256243999999999 2024-09-15T03:47:55.235 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.235 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.235 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.235 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:866 3.5s1 3:af1dd10e:::smithi00132172-27:head [append 0~0] snapc 3e=[3e,3d,3b,3a,38,37,35,34,2d,2b,25,24,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-15T03:47:55.235 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:45.860999+0000", 2024-09-15T03:47:55.235 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.2582378629999997, 2024-09-15T03:47:55.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016187865999999999, 2024-09-15T03:47:55.235 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.235 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:55.236 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.236 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.236 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.236 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 866 2024-09-15T03:47:55.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.236 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.860999+0000", 2024-09-15T03:47:55.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.860999+0000", 2024-09-15T03:47:55.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861001+0000", 2024-09-15T03:47:55.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5519999999999999e-06 2024-09-15T03:47:55.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861008+0000", 2024-09-15T03:47:55.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2710000000000003e-06 2024-09-15T03:47:55.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861027+0000", 2024-09-15T03:47:55.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8416000000000001e-05 2024-09-15T03:47:55.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861036+0000", 2024-09-15T03:47:55.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8899999999999996e-06 2024-09-15T03:47:55.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861092+0000", 2024-09-15T03:47:55.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6746000000000002e-05 2024-09-15T03:47:55.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:55.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861102+0000", 2024-09-15T03:47:55.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8130000000000007e-06 2024-09-15T03:47:55.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861811+0000", 2024-09-15T03:47:55.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070912899999999997 2024-09-15T03:47:55.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861936+0000", 2024-09-15T03:47:55.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012470999999999999 2024-09-15T03:47:55.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.862842+0000", 2024-09-15T03:47:55.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00090626499999999996 2024-09-15T03:47:55.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.863195+0000", 2024-09-15T03:47:55.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035236600000000001 2024-09-15T03:47:55.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.863208+0000", 2024-09-15T03:47:55.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3848e-05 2024-09-15T03:47:55.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.863243+0000", 2024-09-15T03:47:55.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4307000000000002e-05 2024-09-15T03:47:55.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.866214+0000", 2024-09-15T03:47:55.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002970838 2024-09-15T03:47:55.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:55.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.866515+0000", 2024-09-15T03:47:55.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000301682 2024-09-15T03:47:55.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.877187+0000", 2024-09-15T03:47:55.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010672032999999999 2024-09-15T03:47:55.242 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.243 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.243 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.243 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:867 3.5s1 3:af1dd10e:::smithi00132172-27:head [rollback 2d] snapc 3e=[3e,3d,3b,3a,38,37,35,34,2d,2b,25,24,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-15T03:47:55.243 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:45.861150+0000", 2024-09-15T03:47:55.243 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.2580870849999997, 2024-09-15T03:47:55.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018882498000000001, 2024-09-15T03:47:55.243 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.243 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:55.243 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.243 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.243 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.244 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 867 2024-09-15T03:47:55.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.244 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861150+0000", 2024-09-15T03:47:55.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861150+0000", 2024-09-15T03:47:55.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861151+0000", 2024-09-15T03:47:55.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.92e-07 2024-09-15T03:47:55.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861155+0000", 2024-09-15T03:47:55.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7189999999999999e-06 2024-09-15T03:47:55.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861163+0000", 2024-09-15T03:47:55.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8769999999999999e-06 2024-09-15T03:47:55.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861169+0000", 2024-09-15T03:47:55.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0579999999999997e-06 2024-09-15T03:47:55.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861213+0000", 2024-09-15T03:47:55.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4249000000000003e-05 2024-09-15T03:47:55.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:55.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861221+0000", 2024-09-15T03:47:55.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4950000000000008e-06 2024-09-15T03:47:55.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.863319+0000", 2024-09-15T03:47:55.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020973709999999998 2024-09-15T03:47:55.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.863432+0000", 2024-09-15T03:47:55.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113465 2024-09-15T03:47:55.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865365+0000", 2024-09-15T03:47:55.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001932546 2024-09-15T03:47:55.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865488+0000", 2024-09-15T03:47:55.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122929 2024-09-15T03:47:55.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865517+0000", 2024-09-15T03:47:55.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9091999999999999e-05 2024-09-15T03:47:55.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.870930+0000", 2024-09-15T03:47:55.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054136760000000001 2024-09-15T03:47:55.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.870964+0000", 2024-09-15T03:47:55.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3377999999999998e-05 2024-09-15T03:47:55.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:55.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.871243+0000", 2024-09-15T03:47:55.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027943199999999998 2024-09-15T03:47:55.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.880033+0000", 2024-09-15T03:47:55.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087895190000000008 2024-09-15T03:47:55.250 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.250 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.250 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.250 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:868 3.5s1 3:af1dd10e:::smithi00132172-27:head [append 0~0] snapc 3e=[3e,3d,3b,3a,38,37,35,34,2d,2b,25,24,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-15T03:47:55.250 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:45.861231+0000", 2024-09-15T03:47:55.251 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.2580060960000008, 2024-09-15T03:47:55.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018901188999999999, 2024-09-15T03:47:55.251 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.251 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:55.251 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.251 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.251 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.251 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 868 2024-09-15T03:47:55.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.251 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861231+0000", 2024-09-15T03:47:55.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861231+0000", 2024-09-15T03:47:55.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861232+0000", 2024-09-15T03:47:55.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3799999999999997e-07 2024-09-15T03:47:55.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861234+0000", 2024-09-15T03:47:55.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7640000000000001e-06 2024-09-15T03:47:55.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861242+0000", 2024-09-15T03:47:55.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6830000000000008e-06 2024-09-15T03:47:55.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.861247+0000", 2024-09-15T03:47:55.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3719999999999998e-06 2024-09-15T03:47:55.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865555+0000", 2024-09-15T03:47:55.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004308976 2024-09-15T03:47:55.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865641+0000", 2024-09-15T03:47:55.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5801000000000001e-05 2024-09-15T03:47:55.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.866070+0000", 2024-09-15T03:47:55.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042870300000000001 2024-09-15T03:47:55.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.866174+0000", 2024-09-15T03:47:55.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104422 2024-09-15T03:47:55.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.870933+0000", 2024-09-15T03:47:55.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047587000000000003 2024-09-15T03:47:55.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.871013+0000", 2024-09-15T03:47:55.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0059e-05 2024-09-15T03:47:55.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.871046+0000", 2024-09-15T03:47:55.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3281999999999997e-05 2024-09-15T03:47:55.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:55.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.871558+0000", 2024-09-15T03:47:55.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051141000000000001 2024-09-15T03:47:55.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.880132+0000", 2024-09-15T03:47:55.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085744789999999994 2024-09-15T03:47:55.259 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.259 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.259 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.259 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.259 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.259 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:55.261 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-15T03:47:55.261 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-09-15T03:47:55.261 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-09-15T03:47:55.261 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-15T03:47:55.262 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:47:55.262 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-15T03:47:55.263 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:55.260+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:55.362 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:55.361+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:55.364 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:55.364 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:55.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.364 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:931 3.bs0 3.aaf5f4ab (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e108)", 2024-09-15T03:47:55.364 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:54.852841+0000", 2024-09-15T03:47:55.364 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.51120285700000001, 2024-09-15T03:47:55.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.51122314000000002, 2024-09-15T03:47:55.365 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.365 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "waiting for peered", 2024-09-15T03:47:55.365 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.365 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.365 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.365 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 931 2024-09-15T03:47:55.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.365 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.852841+0000", 2024-09-15T03:47:55.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.852841+0000", 2024-09-15T03:47:55.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.852842+0000", 2024-09-15T03:47:55.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0399999999999995e-07 2024-09-15T03:47:55.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.853757+0000", 2024-09-15T03:47:55.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00091496700000000004 2024-09-15T03:47:55.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.853767+0000", 2024-09-15T03:47:55.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0550999999999999e-05 2024-09-15T03:47:55.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.853772+0000", 2024-09-15T03:47:55.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.853e-06 2024-09-15T03:47:55.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.853825+0000", 2024-09-15T03:47:55.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2995999999999999e-05 2024-09-15T03:47:55.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:55.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.853828+0000", 2024-09-15T03:47:55.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.943e-06 2024-09-15T03:47:55.368 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.368 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.368 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.369 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:932 3.bs0 3.aaf5f4ab (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e108)", 2024-09-15T03:47:55.369 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:54.853778+0000", 2024-09-15T03:47:55.369 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.51026624600000003, 2024-09-15T03:47:55.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.51032845199999999, 2024-09-15T03:47:55.369 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.369 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "waiting for peered", 2024-09-15T03:47:55.369 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.369 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.369 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.369 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 932 2024-09-15T03:47:55.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.370 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.853778+0000", 2024-09-15T03:47:55.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.853778+0000", 2024-09-15T03:47:55.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.853778+0000", 2024-09-15T03:47:55.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8000000000000001e-07 2024-09-15T03:47:55.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.854415+0000", 2024-09-15T03:47:55.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00063681000000000002 2024-09-15T03:47:55.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.854423+0000", 2024-09-15T03:47:55.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9969999999999998e-06 2024-09-15T03:47:55.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.854426+0000", 2024-09-15T03:47:55.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4630000000000001e-06 2024-09-15T03:47:55.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.854464+0000", 2024-09-15T03:47:55.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7561000000000002e-05 2024-09-15T03:47:55.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:55.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.854466+0000", 2024-09-15T03:47:55.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4810000000000001e-06 2024-09-15T03:47:55.373 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.373 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.373 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.373 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:933 3.bs0 3.aaf5f4ab (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-15T03:47:55.373 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:54.855803+0000", 2024-09-15T03:47:55.373 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.50824114200000003, 2024-09-15T03:47:55.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.508338242, 2024-09-15T03:47:55.373 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.373 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "waiting for peered", 2024-09-15T03:47:55.374 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.374 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.374 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.374 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 933 2024-09-15T03:47:55.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.374 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.855803+0000", 2024-09-15T03:47:55.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.855803+0000", 2024-09-15T03:47:55.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.855804+0000", 2024-09-15T03:47:55.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3799999999999996e-07 2024-09-15T03:47:55.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856391+0000", 2024-09-15T03:47:55.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00058760400000000003 2024-09-15T03:47:55.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856402+0000", 2024-09-15T03:47:55.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0125999999999999e-05 2024-09-15T03:47:55.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856405+0000", 2024-09-15T03:47:55.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7969999999999998e-06 2024-09-15T03:47:55.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856441+0000", 2024-09-15T03:47:55.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5828000000000003e-05 2024-09-15T03:47:55.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:55.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856443+0000", 2024-09-15T03:47:55.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5030000000000001e-06 2024-09-15T03:47:55.377 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.377 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.377 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.377 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:934 3.bs0 3.aaf5f4ab (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-15T03:47:55.377 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:54.856450+0000", 2024-09-15T03:47:55.377 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.50759397500000003, 2024-09-15T03:47:55.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.50772468900000001, 2024-09-15T03:47:55.378 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.378 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "waiting for peered", 2024-09-15T03:47:55.378 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.378 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.378 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.378 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 934 2024-09-15T03:47:55.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.378 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856450+0000", 2024-09-15T03:47:55.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856450+0000", 2024-09-15T03:47:55.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856451+0000", 2024-09-15T03:47:55.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7800000000000002e-07 2024-09-15T03:47:55.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856578+0000", 2024-09-15T03:47:55.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127019 2024-09-15T03:47:55.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856584+0000", 2024-09-15T03:47:55.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3759999999999996e-06 2024-09-15T03:47:55.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856587+0000", 2024-09-15T03:47:55.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8329999999999998e-06 2024-09-15T03:47:55.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856624+0000", 2024-09-15T03:47:55.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6774999999999999e-05 2024-09-15T03:47:55.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:55.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856625+0000", 2024-09-15T03:47:55.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2839999999999999e-06 2024-09-15T03:47:55.381 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.381 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.381 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.382 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:935 3.bs0 3.aaf5f4ab (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-15T03:47:55.382 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:54.856591+0000", 2024-09-15T03:47:55.382 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.50745294200000002, 2024-09-15T03:47:55.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.50761749, 2024-09-15T03:47:55.382 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.382 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "waiting for peered", 2024-09-15T03:47:55.382 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.382 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.382 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.382 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 935 2024-09-15T03:47:55.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.382 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856591+0000", 2024-09-15T03:47:55.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856591+0000", 2024-09-15T03:47:55.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856592+0000", 2024-09-15T03:47:55.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0900000000000002e-07 2024-09-15T03:47:55.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856594+0000", 2024-09-15T03:47:55.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5490000000000001e-06 2024-09-15T03:47:55.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856605+0000", 2024-09-15T03:47:55.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0552e-05 2024-09-15T03:47:55.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856609+0000", 2024-09-15T03:47:55.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3000000000000003e-06 2024-09-15T03:47:55.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856662+0000", 2024-09-15T03:47:55.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3331000000000002e-05 2024-09-15T03:47:55.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:55.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856664+0000", 2024-09-15T03:47:55.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9489999999999999e-06 2024-09-15T03:47:55.385 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.385 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.386 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.386 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:936 3.bs0 3.aaf5f4ab (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e108)", 2024-09-15T03:47:55.386 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:54.856624+0000", 2024-09-15T03:47:55.386 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.50741971200000002, 2024-09-15T03:47:55.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.50761918900000003, 2024-09-15T03:47:55.386 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.386 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "waiting for peered", 2024-09-15T03:47:55.386 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.386 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.386 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.386 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 936 2024-09-15T03:47:55.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.387 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856624+0000", 2024-09-15T03:47:55.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856624+0000", 2024-09-15T03:47:55.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856625+0000", 2024-09-15T03:47:55.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7800000000000002e-07 2024-09-15T03:47:55.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856626+0000", 2024-09-15T03:47:55.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.733e-06 2024-09-15T03:47:55.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856634+0000", 2024-09-15T03:47:55.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5610000000000001e-06 2024-09-15T03:47:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856638+0000", 2024-09-15T03:47:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1899999999999997e-06 2024-09-15T03:47:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856687+0000", 2024-09-15T03:47:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8687999999999999e-05 2024-09-15T03:47:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:55.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.856689+0000", 2024-09-15T03:47:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.733e-06 2024-09-15T03:47:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:931 3.bs0 3.aaf5f4ab (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-15T03:47:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:54.847138+0000", 2024-09-15T03:47:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.51690639599999999, 2024-09-15T03:47:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.51714011199999999, 2024-09-15T03:47:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "waiting for peered", 2024-09-15T03:47:55.390 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 931 2024-09-15T03:47:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.847138+0000", 2024-09-15T03:47:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.847138+0000", 2024-09-15T03:47:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.847140+0000", 2024-09-15T03:47:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.902e-06 2024-09-15T03:47:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.849702+0000", 2024-09-15T03:47:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025625890000000001 2024-09-15T03:47:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.849716+0000", 2024-09-15T03:47:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3358e-05 2024-09-15T03:47:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.849724+0000", 2024-09-15T03:47:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1459999999999996e-06 2024-09-15T03:47:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.849946+0000", 2024-09-15T03:47:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000222625 2024-09-15T03:47:55.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.849950+0000", 2024-09-15T03:47:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6629999999999999e-06 2024-09-15T03:47:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-15T03:47:55.394 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 7 2024-09-15T03:47:55.394 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:55.396 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:47:55.480 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:55.479+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:55.536 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:55.536 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:55.536 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:55.537 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:55.537 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:55.544 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:47:55.565 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:55.564+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:55.659 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:47:55.665 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:55.665+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:55.688 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:55.688 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:55.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:55.688 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:55.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.689 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:197 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [delete] snapc 3=[3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e34)", 2024-09-15T03:47:55.689 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:55.689 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.572699168, 2024-09-15T03:47:55.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021934704999999999, 2024-09-15T03:47:55.689 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.689 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:55.690 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.690 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.690 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.690 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197 2024-09-15T03:47:55.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.690 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:55.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112265+0000", 2024-09-15T03:47:55.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112266+0000", 2024-09-15T03:47:55.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5000000000000002e-07 2024-09-15T03:47:55.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112269+0000", 2024-09-15T03:47:55.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.066e-06 2024-09-15T03:47:55.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112276+0000", 2024-09-15T03:47:55.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-09-15T03:47:55.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.112281+0000", 2024-09-15T03:47:55.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3970000000000004e-06 2024-09-15T03:47:55.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123281+0000", 2024-09-15T03:47:55.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011000575 2024-09-15T03:47:55.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123395+0000", 2024-09-15T03:47:55.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113398 2024-09-15T03:47:55.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.123963+0000", 2024-09-15T03:47:55.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000568101 2024-09-15T03:47:55.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124220+0000", 2024-09-15T03:47:55.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025704900000000003 2024-09-15T03:47:55.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124229+0000", 2024-09-15T03:47:55.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4860000000000003e-06 2024-09-15T03:47:55.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124277+0000", 2024-09-15T03:47:55.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7574000000000002e-05 2024-09-15T03:47:55.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.124312+0000", 2024-09-15T03:47:55.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5256000000000003e-05 2024-09-15T03:47:55.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:55.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.125716+0000", 2024-09-15T03:47:55.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014043720000000001 2024-09-15T03:47:55.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:41.134200+0000", 2024-09-15T03:47:55.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084835120000000003 2024-09-15T03:47:55.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.702 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.703 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:220 3.bs0 3:d99ddedc:::smithi00132172-44:head [copy-from ver 22] snapc 7=[7,6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:55.703 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:55.703 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.557319497999998, 2024-09-15T03:47:55.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024273429999999999, 2024-09-15T03:47:55.704 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.704 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:55.704 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.704 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.704 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.705 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 220 2024-09-15T03:47:55.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.705 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:55.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127645+0000", 2024-09-15T03:47:55.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127647+0000", 2024-09-15T03:47:55.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1799999999999999e-06 2024-09-15T03:47:55.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127656+0000", 2024-09-15T03:47:55.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3759999999999997e-06 2024-09-15T03:47:55.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127674+0000", 2024-09-15T03:47:55.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7374e-05 2024-09-15T03:47:55.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127680+0000", 2024-09-15T03:47:55.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6949999999999997e-06 2024-09-15T03:47:55.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127736+0000", 2024-09-15T03:47:55.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6011999999999998e-05 2024-09-15T03:47:55.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:55.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127744+0000", 2024-09-15T03:47:55.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2870000000000004e-06 2024-09-15T03:47:55.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130338+0000", 2024-09-15T03:47:55.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025940249999999998 2024-09-15T03:47:55.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130423+0000", 2024-09-15T03:47:55.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5303000000000006e-05 2024-09-15T03:47:55.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.147840+0000", 2024-09-15T03:47:55.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017416580000000001 2024-09-15T03:47:55.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148318+0000", 2024-09-15T03:47:55.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047822899999999999 2024-09-15T03:47:55.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148329+0000", 2024-09-15T03:47:55.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0611000000000001e-05 2024-09-15T03:47:55.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148488+0000", 2024-09-15T03:47:55.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015938399999999999 2024-09-15T03:47:55.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148870+0000", 2024-09-15T03:47:55.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000382405 2024-09-15T03:47:55.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:55.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.151092+0000", 2024-09-15T03:47:55.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022212529999999999 2024-09-15T03:47:55.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.151918+0000", 2024-09-15T03:47:55.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082671600000000002 2024-09-15T03:47:55.722 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.722 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.722 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.723 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:221 3.bs0 3:d99ddedc:::smithi00132172-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-15T03:47:55.723 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:55.723 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.557275343000001, 2024-09-15T03:47:55.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021194474000000001, 2024-09-15T03:47:55.724 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.724 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:55.724 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.724 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.724 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.724 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 221 2024-09-15T03:47:55.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.725 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:55.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127689+0000", 2024-09-15T03:47:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127690+0000", 2024-09-15T03:47:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:47:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127692+0000", 2024-09-15T03:47:55.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6589999999999999e-06 2024-09-15T03:47:55.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127700+0000", 2024-09-15T03:47:55.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4769999999999997e-06 2024-09-15T03:47:55.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127704+0000", 2024-09-15T03:47:55.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3819999999999997e-06 2024-09-15T03:47:55.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127805+0000", 2024-09-15T03:47:55.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100639 2024-09-15T03:47:55.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:55.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.127809+0000", 2024-09-15T03:47:55.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8870000000000002e-06 2024-09-15T03:47:55.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130527+0000", 2024-09-15T03:47:55.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002718579 2024-09-15T03:47:55.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:55.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130558+0000", 2024-09-15T03:47:55.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0827999999999997e-05 2024-09-15T03:47:55.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148546+0000", 2024-09-15T03:47:55.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017988356 2024-09-15T03:47:55.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148683+0000", 2024-09-15T03:47:55.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013650899999999999 2024-09-15T03:47:55.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.148884+0000", 2024-09-15T03:47:55.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020061499999999999 2024-09-15T03:47:55.730 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.730 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.730 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.730 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:357 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:55.730 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:55.730 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.835923809000001, 2024-09-15T03:47:55.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.02017917, 2024-09-15T03:47:55.731 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.731 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:55.731 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.731 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.731 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.731 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 357 2024-09-15T03:47:55.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.731 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:55.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849041+0000", 2024-09-15T03:47:55.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849043+0000", 2024-09-15T03:47:55.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.295e-06 2024-09-15T03:47:55.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849051+0000", 2024-09-15T03:47:55.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4079999999999998e-06 2024-09-15T03:47:55.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849069+0000", 2024-09-15T03:47:55.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7462999999999999e-05 2024-09-15T03:47:55.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849077+0000", 2024-09-15T03:47:55.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8869999999999992e-06 2024-09-15T03:47:55.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849194+0000", 2024-09-15T03:47:55.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011723899999999999 2024-09-15T03:47:55.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:55.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849296+0000", 2024-09-15T03:47:55.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102011 2024-09-15T03:47:55.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861106+0000", 2024-09-15T03:47:55.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0118097610000001 2024-09-15T03:47:55.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861222+0000", 2024-09-15T03:47:55.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116561 2024-09-15T03:47:55.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.869220+0000", 2024-09-15T03:47:55.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007997545 2024-09-15T03:47:55.735 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.736 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.736 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.736 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:358 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:55.736 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:55.736 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.835868863000002, 2024-09-15T03:47:55.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0228638050000001, 2024-09-15T03:47:55.736 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.736 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:55.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.737 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 358 2024-09-15T03:47:55.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.737 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:55.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:55.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849096+0000", 2024-09-15T03:47:55.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4099999999999997e-07 2024-09-15T03:47:55.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849101+0000", 2024-09-15T03:47:55.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8180000000000003e-06 2024-09-15T03:47:55.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849110+0000", 2024-09-15T03:47:55.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1409999999999994e-06 2024-09-15T03:47:55.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849116+0000", 2024-09-15T03:47:55.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2390000000000002e-06 2024-09-15T03:47:55.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849326+0000", 2024-09-15T03:47:55.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000210029 2024-09-15T03:47:55.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:55.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849421+0000", 2024-09-15T03:47:55.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5775999999999999e-05 2024-09-15T03:47:55.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861647+0000", 2024-09-15T03:47:55.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0122257139999999 2024-09-15T03:47:55.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861777+0000", 2024-09-15T03:47:55.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130087 2024-09-15T03:47:55.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.871959+0000", 2024-09-15T03:47:55.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010182160000000001 2024-09-15T03:47:55.741 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.741 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.741 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.741 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:359 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~2441216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:55.741 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:55.741 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.835843711000003, 2024-09-15T03:47:55.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0240468149999999, 2024-09-15T03:47:55.741 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.741 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:55.742 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.742 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.742 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.742 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 359 2024-09-15T03:47:55.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.742 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:55.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:55.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849121+0000", 2024-09-15T03:47:55.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:47:55.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849124+0000", 2024-09-15T03:47:55.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2679999999999999e-06 2024-09-15T03:47:55.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849132+0000", 2024-09-15T03:47:55.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4030000000000001e-06 2024-09-15T03:47:55.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849136+0000", 2024-09-15T03:47:55.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2289999999999997e-06 2024-09-15T03:47:55.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849446+0000", 2024-09-15T03:47:55.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031043299999999999 2024-09-15T03:47:55.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:55.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849553+0000", 2024-09-15T03:47:55.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106415 2024-09-15T03:47:55.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.863404+0000", 2024-09-15T03:47:55.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0138511029999999 2024-09-15T03:47:55.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.863475+0000", 2024-09-15T03:47:55.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0814000000000003e-05 2024-09-15T03:47:55.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.873168+0000", 2024-09-15T03:47:55.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00969274 2024-09-15T03:47:55.746 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.746 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.746 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.747 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:474 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~786432] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:55.747 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:55.747 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.809919035999997, 2024-09-15T03:47:55.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98604943700000003, 2024-09-15T03:47:55.747 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.747 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:55.747 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.747 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.747 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.747 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 474 2024-09-15T03:47:55.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.747 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:55.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875045+0000", 2024-09-15T03:47:55.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875048+0000", 2024-09-15T03:47:55.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1600000000000001e-06 2024-09-15T03:47:55.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875844+0000", 2024-09-15T03:47:55.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079686299999999998 2024-09-15T03:47:55.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875857+0000", 2024-09-15T03:47:55.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2889e-05 2024-09-15T03:47:55.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875864+0000", 2024-09-15T03:47:55.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6139999999999997e-06 2024-09-15T03:47:55.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875916+0000", 2024-09-15T03:47:55.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1992e-05 2024-09-15T03:47:55.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:55.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.875918+0000", 2024-09-15T03:47:55.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3300000000000001e-06 2024-09-15T03:47:55.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.845918+0000", 2024-09-15T03:47:55.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96999959199999997 2024-09-15T03:47:55.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.846229+0000", 2024-09-15T03:47:55.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000311601 2024-09-15T03:47:55.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848445+0000", 2024-09-15T03:47:55.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022160499999999998 2024-09-15T03:47:55.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848880+0000", 2024-09-15T03:47:55.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043432600000000002 2024-09-15T03:47:55.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.848952+0000", 2024-09-15T03:47:55.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2509000000000007e-05 2024-09-15T03:47:55.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.849311+0000", 2024-09-15T03:47:55.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000358611 2024-09-15T03:47:55.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851131+0000", 2024-09-15T03:47:55.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001820115 2024-09-15T03:47:55.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:55.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861048+0000", 2024-09-15T03:47:55.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099164830000000002 2024-09-15T03:47:55.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861095+0000", 2024-09-15T03:47:55.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7302000000000002e-05 2024-09-15T03:47:55.754 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.754 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.754 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.754 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:475 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~491520] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:55.754 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:55.754 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.807152565000003, 2024-09-15T03:47:55.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98407319000000004, 2024-09-15T03:47:55.754 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.754 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:55.754 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.754 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.755 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.755 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 475 2024-09-15T03:47:55.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.755 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:55.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877812+0000", 2024-09-15T03:47:55.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.877813+0000", 2024-09-15T03:47:55.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8800000000000002e-07 2024-09-15T03:47:55.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878260+0000", 2024-09-15T03:47:55.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044732099999999998 2024-09-15T03:47:55.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878270+0000", 2024-09-15T03:47:55.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9799999999999993e-06 2024-09-15T03:47:55.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878276+0000", 2024-09-15T03:47:55.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2269999999999998e-06 2024-09-15T03:47:55.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878313+0000", 2024-09-15T03:47:55.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6570999999999999e-05 2024-09-15T03:47:55.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:55.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.878315+0000", 2024-09-15T03:47:55.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8729999999999999e-06 2024-09-15T03:47:55.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851316+0000", 2024-09-15T03:47:55.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97300154800000005 2024-09-15T03:47:55.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.851440+0000", 2024-09-15T03:47:55.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124287 2024-09-15T03:47:55.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852732+0000", 2024-09-15T03:47:55.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001291174 2024-09-15T03:47:55.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.852991+0000", 2024-09-15T03:47:55.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025905299999999997 2024-09-15T03:47:55.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853003+0000", 2024-09-15T03:47:55.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2908000000000001e-05 2024-09-15T03:47:55.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853104+0000", 2024-09-15T03:47:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010011800000000001 2024-09-15T03:47:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854342+0000", 2024-09-15T03:47:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012380189999999999 2024-09-15T03:47:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861206+0000", 2024-09-15T03:47:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068648499999999996 2024-09-15T03:47:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861885+0000", 2024-09-15T03:47:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00067857300000000002 2024-09-15T03:47:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:476 3.bs0 3:d52faf55:::smithi00132172-40:head [append 0~278528] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.805525551999999, 2024-09-15T03:47:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98268067299999995, 2024-09-15T03:47:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 476 2024-09-15T03:47:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879439+0000", 2024-09-15T03:47:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.13e-07 2024-09-15T03:47:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879706+0000", 2024-09-15T03:47:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026656100000000001 2024-09-15T03:47:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879715+0000", 2024-09-15T03:47:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9330000000000007e-06 2024-09-15T03:47:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879720+0000", 2024-09-15T03:47:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0880000000000002e-06 2024-09-15T03:47:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879743+0000", 2024-09-15T03:47:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2654e-05 2024-09-15T03:47:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879743+0000", 2024-09-15T03:47:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.99e-07 2024-09-15T03:47:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853170+0000", 2024-09-15T03:47:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97342687900000002 2024-09-15T03:47:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.853297+0000", 2024-09-15T03:47:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127157 2024-09-15T03:47:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854150+0000", 2024-09-15T03:47:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085288900000000001 2024-09-15T03:47:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854292+0000", 2024-09-15T03:47:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014199300000000001 2024-09-15T03:47:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854559+0000", 2024-09-15T03:47:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026668999999999998 2024-09-15T03:47:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855998+0000", 2024-09-15T03:47:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001438714 2024-09-15T03:47:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856547+0000", 2024-09-15T03:47:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054880399999999996 2024-09-15T03:47:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.861858+0000", 2024-09-15T03:47:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053110320000000003 2024-09-15T03:47:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.862120+0000", 2024-09-15T03:47:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026196699999999999 2024-09-15T03:47:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:477 3.bs0 3:d52faf55:::smithi00132172-40:head [setxattr _header (57) in=7b] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.805237615000003, 2024-09-15T03:47:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98562198000000001, 2024-09-15T03:47:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 477 2024-09-15T03:47:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879727+0000", 2024-09-15T03:47:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2800000000000002e-07 2024-09-15T03:47:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879730+0000", 2024-09-15T03:47:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0589999999999998e-06 2024-09-15T03:47:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879737+0000", 2024-09-15T03:47:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0419999999999997e-06 2024-09-15T03:47:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879741+0000", 2024-09-15T03:47:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9960000000000004e-06 2024-09-15T03:47:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879756+0000", 2024-09-15T03:47:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4681999999999999e-05 2024-09-15T03:47:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879756+0000", 2024-09-15T03:47:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.22e-07 2024-09-15T03:47:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854590+0000", 2024-09-15T03:47:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97483319000000002 2024-09-15T03:47:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.854718+0000", 2024-09-15T03:47:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012842100000000001 2024-09-15T03:47:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855416+0000", 2024-09-15T03:47:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069759200000000003 2024-09-15T03:47:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855551+0000", 2024-09-15T03:47:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013517600000000001 2024-09-15T03:47:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856002+0000", 2024-09-15T03:47:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045169399999999999 2024-09-15T03:47:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856065+0000", 2024-09-15T03:47:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3028999999999996e-05 2024-09-15T03:47:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856554+0000", 2024-09-15T03:47:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000488486 2024-09-15T03:47:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865278+0000", 2024-09-15T03:47:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087241929999999999 2024-09-15T03:47:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865349+0000", 2024-09-15T03:47:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0669999999999999e-05 2024-09-15T03:47:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:478 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.80521882, 2024-09-15T03:47:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99471090500000003, 2024-09-15T03:47:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 478 2024-09-15T03:47:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879746+0000", 2024-09-15T03:47:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2599999999999998e-07 2024-09-15T03:47:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879748+0000", 2024-09-15T03:47:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9609999999999999e-06 2024-09-15T03:47:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879755+0000", 2024-09-15T03:47:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1879999999999996e-06 2024-09-15T03:47:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879760+0000", 2024-09-15T03:47:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2560000000000002e-06 2024-09-15T03:47:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879797+0000", 2024-09-15T03:47:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6452999999999998e-05 2024-09-15T03:47:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.879799+0000", 2024-09-15T03:47:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6619999999999999e-06 2024-09-15T03:47:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855637+0000", 2024-09-15T03:47:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97583850999999999 2024-09-15T03:47:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.855763+0000", 2024-09-15T03:47:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012612699999999999 2024-09-15T03:47:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865707+0000", 2024-09-15T03:47:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099439660000000003 2024-09-15T03:47:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.865803+0000", 2024-09-15T03:47:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5946999999999995e-05 2024-09-15T03:47:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.874456+0000", 2024-09-15T03:47:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0086534090000000008 2024-09-15T03:47:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:486 3.bs0 3:df8b18a3:::smithi00132172-17:head [delete] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e64)", 2024-09-15T03:47:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.802406341000001, 2024-09-15T03:47:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98362617100000005, 2024-09-15T03:47:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 486 2024-09-15T03:47:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882558+0000", 2024-09-15T03:47:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882559+0000", 2024-09-15T03:47:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0359999999999999e-06 2024-09-15T03:47:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882564+0000", 2024-09-15T03:47:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6530000000000002e-06 2024-09-15T03:47:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882574+0000", 2024-09-15T03:47:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0464e-05 2024-09-15T03:47:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882578+0000", 2024-09-15T03:47:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2459999999999997e-06 2024-09-15T03:47:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882624+0000", 2024-09-15T03:47:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5698000000000002e-05 2024-09-15T03:47:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:10.882627+0000", 2024-09-15T03:47:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3350000000000001e-06 2024-09-15T03:47:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856574+0000", 2024-09-15T03:47:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97394739399999997 2024-09-15T03:47:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.856853+0000", 2024-09-15T03:47:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027918399999999998 2024-09-15T03:47:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.866184+0000", 2024-09-15T03:47:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093311609999999993 2024-09-15T03:47:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:495 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [copy-from ver 51] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-15T03:47:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.796505291999999, 2024-09-15T03:47:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023555178, 2024-09-15T03:47:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 495 2024-09-15T03:47:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888459+0000", 2024-09-15T03:47:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888461+0000", 2024-09-15T03:47:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.719e-06 2024-09-15T03:47:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888471+0000", 2024-09-15T03:47:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0161e-05 2024-09-15T03:47:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888500+0000", 2024-09-15T03:47:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8989e-05 2024-09-15T03:47:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888508+0000", 2024-09-15T03:47:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5299999999999999e-06 2024-09-15T03:47:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888554+0000", 2024-09-15T03:47:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6489000000000002e-05 2024-09-15T03:47:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.888809+0000", 2024-09-15T03:47:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000255467 2024-09-15T03:47:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.905724+0000", 2024-09-15T03:47:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016914348999999999 2024-09-15T03:47:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906244+0000", 2024-09-15T03:47:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052054900000000003 2024-09-15T03:47:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.906262+0000", 2024-09-15T03:47:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7499999999999998e-05 2024-09-15T03:47:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.907367+0000", 2024-09-15T03:47:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011049460000000001 2024-09-15T03:47:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.907384+0000", 2024-09-15T03:47:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7000999999999999e-05 2024-09-15T03:47:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.911965+0000", 2024-09-15T03:47:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045809470000000001 2024-09-15T03:47:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:11.912014+0000", 2024-09-15T03:47:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9530999999999997e-05 2024-09-15T03:47:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:689 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~5144576,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.088846+0000", 2024-09-15T03:47:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.596118872000002, 2024-09-15T03:47:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020311780000000002, 2024-09-15T03:47:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 689 2024-09-15T03:47:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088846+0000", 2024-09-15T03:47:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088846+0000", 2024-09-15T03:47:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088847+0000", 2024-09-15T03:47:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9300000000000006e-07 2024-09-15T03:47:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088852+0000", 2024-09-15T03:47:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1850000000000003e-06 2024-09-15T03:47:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088863+0000", 2024-09-15T03:47:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1418e-05 2024-09-15T03:47:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088873+0000", 2024-09-15T03:47:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4020000000000008e-06 2024-09-15T03:47:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.089560+0000", 2024-09-15T03:47:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068774899999999998 2024-09-15T03:47:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090073+0000", 2024-09-15T03:47:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051222799999999999 2024-09-15T03:47:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.109157+0000", 2024-09-15T03:47:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019084805 2024-09-15T03:47:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:690 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~5144576,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.596085733999999, 2024-09-15T03:47:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025340499999999998, 2024-09-15T03:47:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 690 2024-09-15T03:47:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088879+0000", 2024-09-15T03:47:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7999999999999996e-07 2024-09-15T03:47:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088882+0000", 2024-09-15T03:47:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1839999999999999e-06 2024-09-15T03:47:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088892+0000", 2024-09-15T03:47:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3540000000000006e-06 2024-09-15T03:47:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088896+0000", 2024-09-15T03:47:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5210000000000004e-06 2024-09-15T03:47:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.093968+0000", 2024-09-15T03:47:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050715170000000002 2024-09-15T03:47:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.094098+0000", 2024-09-15T03:47:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013070000000000001 2024-09-15T03:47:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.114219+0000", 2024-09-15T03:47:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020120743999999999 2024-09-15T03:47:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:691 3.bs0 3:d52faf55:::smithi00132172-40:head [read 0~5144576,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.088900+0000", 2024-09-15T03:47:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.596063991000001, 2024-09-15T03:47:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031407274999999998, 2024-09-15T03:47:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 691 2024-09-15T03:47:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088900+0000", 2024-09-15T03:47:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088900+0000", 2024-09-15T03:47:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088901+0000", 2024-09-15T03:47:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9099999999999999e-07 2024-09-15T03:47:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088904+0000", 2024-09-15T03:47:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0299999999999998e-06 2024-09-15T03:47:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088911+0000", 2024-09-15T03:47:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.708e-06 2024-09-15T03:47:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.088915+0000", 2024-09-15T03:47:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0579999999999999e-06 2024-09-15T03:47:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.096143+0000", 2024-09-15T03:47:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007228099 2024-09-15T03:47:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.096202+0000", 2024-09-15T03:47:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9667999999999999e-05 2024-09-15T03:47:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.120308+0000", 2024-09-15T03:47:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024105320999999999 2024-09-15T03:47:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(osd.0.17:2 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [assert-version v51,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:30.090507+0000", 2024-09-15T03:47:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.594457523999999, 2024-09-15T03:47:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025010186, 2024-09-15T03:47:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090507+0000", 2024-09-15T03:47:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090507+0000", 2024-09-15T03:47:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090508+0000", 2024-09-15T03:47:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.176e-06 2024-09-15T03:47:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090512+0000", 2024-09-15T03:47:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2989999999999998e-06 2024-09-15T03:47:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090522+0000", 2024-09-15T03:47:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6549999999999999e-06 2024-09-15T03:47:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.090526+0000", 2024-09-15T03:47:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8500000000000004e-06 2024-09-15T03:47:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.094385+0000", 2024-09-15T03:47:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038591670000000002 2024-09-15T03:47:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.094541+0000", 2024-09-15T03:47:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015590999999999999 2024-09-15T03:47:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:30.115517+0000", 2024-09-15T03:47:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020976129 2024-09-15T03:47:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:749 3.bs0 3:d99ddedc:::smithi00132172-44:head [read 0~737280,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-15T03:47:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.935286+0000", 2024-09-15T03:47:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.749677984000002, 2024-09-15T03:47:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021703044000000001, 2024-09-15T03:47:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 749 2024-09-15T03:47:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935286+0000", 2024-09-15T03:47:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935286+0000", 2024-09-15T03:47:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935287+0000", 2024-09-15T03:47:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.51e-07 2024-09-15T03:47:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935290+0000", 2024-09-15T03:47:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8710000000000001e-06 2024-09-15T03:47:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935297+0000", 2024-09-15T03:47:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3390000000000004e-06 2024-09-15T03:47:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935302+0000", 2024-09-15T03:47:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5560000000000001e-06 2024-09-15T03:47:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936361+0000", 2024-09-15T03:47:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010593200000000001 2024-09-15T03:47:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:47:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936486+0000", 2024-09-15T03:47:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124657 2024-09-15T03:47:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.952979+0000", 2024-09-15T03:47:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016493154999999999 2024-09-15T03:47:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.953132+0000", 2024-09-15T03:47:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015346500000000001 2024-09-15T03:47:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.956989+0000", 2024-09-15T03:47:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038571299999999998 2024-09-15T03:47:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:750 3.bs0 3:d1d6a7bc:::smithi00132172-3:head [create,setxattr \\^tkqj^qnPPfYPq\\WZoebWOkaebalTko_OhaYsrZ (1787) in=40b,setxattr cg]RoaNaVsOQTrQfp[VeRj_tmeegdoc`PtRYpskN (1998) in=40b,setxattr \\_N_s\\f^o_Ojh]gqWPkfsZ\\WTSbY]SOlX`mlV[^q (6944) in=40b,setxattr nsdiO]arkaqtYt (33) in=14b,setxattr ti`p[nirg_rWRRSe[oZhhn]b]ZsTimedg]]p[sQY (1049) in=40b,setxattr i[YNharOrm`bbP]UtYhjlqVWP\\XnccYlfhV_deTf (383) in=40b,setxattr ka_q[S]a]naRePWgcgUWmZSaW^RVV\\_rNXdZefal (875) in=40b,setxattr gaoaanUtdfqUrimS^_NfUtarRjNSVRf^_PbUpdtf (1645) in=40b,setxattr qNb^mqtq^Vsg]m\\aWdRTbbgOlZYptsW]O[_R`Vaa (1155) in=40b,setxattr gUPP_ttp`PdrngYsWrVdfZZVgff^Q`YRSjgOXTao (417) in=40b,setxattr mi`p\\f[PUQf^[[skf^\\Za_iXO_oTscOWNnQqcR[[ (4560) in=40b,setxattr N^_ka_abmi]seatXYWSQtibSdeOsfV\\OXsmmXlYm (1893) in=40b,setxattr aoqZmYaUpNW (33) in=11b,setxattr Z[bbsdRZTfkrP`]akaXifm`oV[YTOPrPaeeXV[fl (1611) in=40b,setxattr oZ`SalVqYt^T_[b_a]r[rlaaWOa_WZQ^^opf\\aN_ (243) in=40b,setxattr k\\V^enpkstj^VpoopX\\PcoTY]eogbs`pQQW]ofrm (347) in=40b,setxattr cfXfpnr`SogqjNa^cTVYtpgdciPp_eTUg]Ujn\\at (1926) in=40b,setxattr t`ojSZgRjbb[aaUXSepgQf_nc`kaOetle^n^`Vfd (1087) in=40b,setxattr ONYdltaQhe\\a\\da\\ehQRpsS^Ugdgmf\\WWigfUarV (631) in=40b,setxattr [Tj`Rbkt]qjatpSf_fPsanNr^ipYnOQlPnUZlmRU (4068) in=40b] snapc 32=[32,2d,2b,26,25,24,23,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T03:47:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:34.935912+0000", 2024-09-15T03:47:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.74905266, 2024-09-15T03:47:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03915623, 2024-09-15T03:47:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 750 2024-09-15T03:47:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935912+0000", 2024-09-15T03:47:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935912+0000", 2024-09-15T03:47:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935913+0000", 2024-09-15T03:47:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7599999999999996e-07 2024-09-15T03:47:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935984+0000", 2024-09-15T03:47:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1136000000000005e-05 2024-09-15T03:47:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.935997+0000", 2024-09-15T03:47:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3482000000000001e-05 2024-09-15T03:47:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936004+0000", 2024-09-15T03:47:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3260000000000001e-06 2024-09-15T03:47:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.936878+0000", 2024-09-15T03:47:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087483999999999999 2024-09-15T03:47:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:47:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.937895+0000", 2024-09-15T03:47:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010163209999999999 2024-09-15T03:47:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.965388+0000", 2024-09-15T03:47:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027492986000000001 2024-09-15T03:47:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.965630+0000", 2024-09-15T03:47:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000242012 2024-09-15T03:47:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.966983+0000", 2024-09-15T03:47:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013535349999999999 2024-09-15T03:47:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.967360+0000", 2024-09-15T03:47:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037704399999999999 2024-09-15T03:47:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.967371+0000", 2024-09-15T03:47:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0988999999999999e-05 2024-09-15T03:47:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.967436+0000", 2024-09-15T03:47:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4679999999999997e-05 2024-09-15T03:47:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.969080+0000", 2024-09-15T03:47:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001644017 2024-09-15T03:47:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.969532+0000", 2024-09-15T03:47:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045157699999999999 2024-09-15T03:47:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:34.975068+0000", 2024-09-15T03:47:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055364089999999999 2024-09-15T03:47:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.5s5 99/89 ECSubWrite(tid=134, reqid=client.4521.0:867, at_version=99'50, trim_to=0'0, roll_forward_to=96'47))", 2024-09-15T03:47:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:45.865389+0000", 2024-09-15T03:47:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.81957536, 2024-09-15T03:47:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019027228, 2024-09-15T03:47:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:47:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:55.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865389+0000", 2024-09-15T03:47:55.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:55.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865389+0000", 2024-09-15T03:47:55.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:55.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865390+0000", 2024-09-15T03:47:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:47:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865411+0000", 2024-09-15T03:47:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1005e-05 2024-09-15T03:47:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865424+0000", 2024-09-15T03:47:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3671e-05 2024-09-15T03:47:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865431+0000", 2024-09-15T03:47:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5989999999999998e-06 2024-09-15T03:47:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865487+0000", 2024-09-15T03:47:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5933999999999997e-05 2024-09-15T03:47:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865497+0000", 2024-09-15T03:47:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0079e-05 2024-09-15T03:47:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865620+0000", 2024-09-15T03:47:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012282499999999999 2024-09-15T03:47:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.865632+0000", 2024-09-15T03:47:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1841999999999999e-05 2024-09-15T03:47:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.870932+0000", 2024-09-15T03:47:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005300478 2024-09-15T03:47:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.870953+0000", 2024-09-15T03:47:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0797000000000001e-05 2024-09-15T03:47:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:45.884416+0000", 2024-09-15T03:47:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013463454999999999 2024-09-15T03:47:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:55.833 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:55.833 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:55.834 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:55.834 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:55.835 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:55.835 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:55.835 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:47:55.836 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:55.833+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:55.836 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:55.836 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:55.837 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:55.837 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:55.837 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:55.837 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.0 to 1 (10000) 2024-09-15T03:47:55.866 INFO:tasks.rados.rados.0.smithi001.stdout:365: done (2 left) 2024-09-15T03:47:55.866 INFO:tasks.rados.rados.0.smithi001.stdout:366: done (1 left) 2024-09-15T03:47:55.866 INFO:tasks.rados.rados.0.smithi001.stdout:367: delete oid 6 current snap is 48 2024-09-15T03:47:55.867 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-15T03:47:55.867 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:55.868 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:55.875 INFO:tasks.rados.rados.0.smithi001.stdout:364: finishing write tid 1 to smithi00132172-40 2024-09-15T03:47:55.875 INFO:tasks.rados.rados.0.smithi001.stdout:364: finishing write tid 2 to smithi00132172-40 2024-09-15T03:47:55.875 INFO:tasks.rados.rados.0.smithi001.stdout:364: finishing write tid 3 to smithi00132172-40 2024-09-15T03:47:55.875 INFO:tasks.rados.rados.0.smithi001.stdout:364: finishing write tid 4 to smithi00132172-40 2024-09-15T03:47:55.876 INFO:tasks.rados.rados.0.smithi001.stdout:364: finishing write tid 5 to smithi00132172-40 2024-09-15T03:47:55.881 INFO:tasks.rados.rados.0.smithi001.stdout:367: done (1 left) 2024-09-15T03:47:55.881 INFO:tasks.rados.rados.0.smithi001.stdout:368: copy_from oid 8 from oid 16 current snap is 48 2024-09-15T03:47:55.881 INFO:tasks.rados.rados.0.smithi001.stdout:369: read oid 47 snap 33 2024-09-15T03:47:55.881 INFO:tasks.rados.rados.0.smithi001.stdout:369: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-15T03:47:55.881 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 3 current snap is 48 2024-09-15T03:47:55.881 INFO:tasks.rados.rados.0.smithi001.stdout:370: seq_num 140 ranges {2621440=802816,3424256=425984,3850240=720896,4571136=409600} 2024-09-15T03:47:55.903 INFO:tasks.rados.rados.0.smithi001.stdout:370: writing smithi00132172-3 from 2621440 to 3424256 tid 1 2024-09-15T03:47:55.904 INFO:tasks.rados.rados.0.smithi001.stdout:370: writing smithi00132172-3 from 3424256 to 3850240 tid 2 2024-09-15T03:47:55.910 INFO:tasks.rados.rados.0.smithi001.stdout:370: writing smithi00132172-3 from 3850240 to 4571136 tid 3 2024-09-15T03:47:55.913 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:55.913 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:55.914 INFO:tasks.rados.rados.0.smithi001.stdout:370: writing smithi00132172-3 from 4571136 to 4980736 tid 4 2024-09-15T03:47:55.914 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 27 current snap is 48 2024-09-15T03:47:55.914 INFO:tasks.rados.rados.0.smithi001.stdout:364: finishing write tid 6 to smithi00132172-40 2024-09-15T03:47:55.914 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 40 v 129 (ObjNum 137 snap 48 seq_num 137) dirty exists 2024-09-15T03:47:55.914 INFO:tasks.rados.rados.0.smithi001.stdout:364: left oid 40 (ObjNum 137 snap 48 seq_num 137) 2024-09-15T03:47:55.914 INFO:tasks.rados.rados.0.smithi001.stdout:369: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-15T03:47:55.934 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T03:47:55.936 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:55.934+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:55.945 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:55.945 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:55.962 INFO:tasks.rados.rados.0.smithi001.stdout:368: finishing copy_from racing read to smithi00132172-8 2024-09-15T03:47:55.963 INFO:tasks.rados.rados.0.smithi001.stdout:368: finishing copy_from to smithi00132172-8 2024-09-15T03:47:55.963 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 8 v 46 (ObjNum 119 snap 33 seq_num 119) dirty exists 2024-09-15T03:47:55.963 INFO:tasks.rados.rados.0.smithi001.stdout:370: finishing write tid 1 to smithi00132172-3 2024-09-15T03:47:55.963 INFO:tasks.rados.rados.0.smithi001.stdout:370: finishing write tid 2 to smithi00132172-3 2024-09-15T03:47:55.963 INFO:tasks.rados.rados.0.smithi001.stdout:370: finishing write tid 3 to smithi00132172-3 2024-09-15T03:47:55.963 INFO:tasks.rados.rados.0.smithi001.stdout:370: finishing write tid 4 to smithi00132172-3 2024-09-15T03:47:55.963 INFO:tasks.rados.rados.0.smithi001.stdout:370: finishing write tid 5 to smithi00132172-3 2024-09-15T03:47:55.963 INFO:tasks.rados.rados.0.smithi001.stdout:370: finishing write tid 6 to smithi00132172-3 2024-09-15T03:47:55.963 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 3 v 145 (ObjNum 140 snap 48 seq_num 140) dirty exists 2024-09-15T03:47:55.964 INFO:tasks.rados.rados.0.smithi001.stdout:370: left oid 3 (ObjNum 140 snap 48 seq_num 140) 2024-09-15T03:47:55.964 INFO:tasks.rados.rados.0.smithi001.stdout:371: seq_num 141 ranges {5210112=425984,5636096=573440,6209536=425984,6635520=524288,7159808=425984} 2024-09-15T03:47:55.977 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:55.977 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:55.993 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:55.993 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:55.994 INFO:tasks.rados.rados.0.smithi001.stdout:371: writing smithi00132172-27 from 5210112 to 5636096 tid 1 2024-09-15T03:47:55.997 INFO:tasks.rados.rados.0.smithi001.stdout:371: writing smithi00132172-27 from 5636096 to 6209536 tid 2 2024-09-15T03:47:56.000 INFO:tasks.rados.rados.0.smithi001.stdout:371: writing smithi00132172-27 from 6209536 to 6635520 tid 3 2024-09-15T03:47:56.003 INFO:tasks.rados.rados.0.smithi001.stdout:371: writing smithi00132172-27 from 6635520 to 7159808 tid 4 2024-09-15T03:47:56.005 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:47:56.006 INFO:tasks.rados.rados.0.smithi001.stdout:371: writing smithi00132172-27 from 7159808 to 7585792 tid 5 2024-09-15T03:47:56.006 INFO:tasks.rados.rados.0.smithi001.stdout:364: done (4 left) 2024-09-15T03:47:56.007 INFO:tasks.rados.rados.0.smithi001.stdout:368: done (3 left) 2024-09-15T03:47:56.007 INFO:tasks.rados.rados.0.smithi001.stdout:369: done (2 left) 2024-09-15T03:47:56.007 INFO:tasks.rados.rados.0.smithi001.stdout:370: done (1 left) 2024-09-15T03:47:56.007 INFO:tasks.rados.rados.0.smithi001.stdout:372: setattr oid 11 current snap is 48 2024-09-15T03:47:56.007 INFO:tasks.rados.rados.0.smithi001.stdout:371: finishing write tid 1 to smithi00132172-27 2024-09-15T03:47:56.007 INFO:tasks.rados.rados.0.smithi001.stdout:371: finishing write tid 2 to smithi00132172-27 2024-09-15T03:47:56.007 INFO:tasks.rados.rados.0.smithi001.stdout:371: finishing write tid 3 to smithi00132172-27 2024-09-15T03:47:56.007 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:56.008 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:56.008 INFO:tasks.rados.rados.0.smithi001.stdout:371: finishing write tid 4 to smithi00132172-27 2024-09-15T03:47:56.008 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 5 current snap is 48 2024-09-15T03:47:56.008 INFO:tasks.rados.rados.0.smithi001.stdout:373: seq_num 142 ranges {4063232=442368,4505600=720896,5226496=540672,5767168=688128} 2024-09-15T03:47:56.033 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:47:56.034 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:47:56.034 INFO:tasks.rados.rados.0.smithi001.stdout:373: writing smithi00132172-5 from 4063232 to 4505600 tid 1 2024-09-15T03:47:56.035 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:56.034+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:56.040 INFO:tasks.rados.rados.0.smithi001.stdout:373: writing smithi00132172-5 from 4505600 to 5226496 tid 2 2024-09-15T03:47:56.044 INFO:tasks.rados.rados.0.smithi001.stdout:373: writing smithi00132172-5 from 5226496 to 5767168 tid 3 2024-09-15T03:47:56.051 INFO:tasks.rados.rados.0.smithi001.stdout:373: writing smithi00132172-5 from 5767168 to 6455296 tid 4 2024-09-15T03:47:56.051 INFO:tasks.rados.rados.0.smithi001.stdout:374: rollback oid 25 current snap is 48 2024-09-15T03:47:56.051 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 25 to 34 2024-09-15T03:47:56.051 INFO:tasks.rados.rados.0.smithi001.stdout:371: finishing write tid 5 to smithi00132172-27 2024-09-15T03:47:56.051 INFO:tasks.rados.rados.0.smithi001.stdout:371: finishing write tid 6 to smithi00132172-27 2024-09-15T03:47:56.051 INFO:tasks.rados.rados.0.smithi001.stdout:371: finishing write tid 7 to smithi00132172-27 2024-09-15T03:47:56.051 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 27 v 68 (ObjNum 141 snap 48 seq_num 141) dirty exists 2024-09-15T03:47:56.051 INFO:tasks.rados.rados.0.smithi001.stdout:371: left oid 27 (ObjNum 141 snap 48 seq_num 141) 2024-09-15T03:47:56.051 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 11 v 113 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-15T03:47:56.052 INFO:tasks.rados.rados.0.smithi001.stdout:373: finishing write tid 1 to smithi00132172-5 2024-09-15T03:47:56.052 INFO:tasks.rados.rados.0.smithi001.stdout:373: finishing write tid 2 to smithi00132172-5 2024-09-15T03:47:56.052 INFO:tasks.rados.rados.0.smithi001.stdout:373: finishing write tid 3 to smithi00132172-5 2024-09-15T03:47:56.052 INFO:tasks.rados.rados.0.smithi001.stdout:371: done (3 left) 2024-09-15T03:47:56.052 INFO:tasks.rados.rados.0.smithi001.stdout:372: done (2 left) 2024-09-15T03:47:56.052 INFO:tasks.rados.rados.0.smithi001.stdout:375: copy_from oid 16 from oid 28 current snap is 48 2024-09-15T03:47:56.052 INFO:tasks.rados.rados.0.smithi001.stdout:376: copy_from oid 32 from oid 44 current snap is 48 2024-09-15T03:47:56.052 INFO:tasks.rados.rados.0.smithi001.stdout:377: rollback oid 27 current snap is 48 2024-09-15T03:47:56.052 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 27 to 24 2024-09-15T03:47:56.052 INFO:tasks.rados.rados.0.smithi001.stdout:378: snap_remove snap 11 2024-09-15T03:47:56.057 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:56.057 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:56.135 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:56.135+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:56.147 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:56.147 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:56.155 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:47:56.236 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:56.235+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:56.253 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:56.253 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:56.262 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:47:56.337 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:56.337+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:56.358 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:56.358 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:56.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:47:56.438 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:56.437+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:56.463 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:56.463 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:56.471 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:47:56.540 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:56.539+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:56.608 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:56.609 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:56.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:47:56.641 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:56.641+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:56.713 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:56.713 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:56.725 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:47:56.768 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:56.767+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:56.821 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:56.821 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:56.829 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:47:56.843 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:56.842+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:56.931 INFO:tasks.rados.rados.0.smithi001.stdout:378: done (5 left) 2024-09-15T03:47:56.931 INFO:tasks.rados.rados.0.smithi001.stdout:379: copy_from oid 49 from oid 10 current snap is 48 2024-09-15T03:47:56.931 INFO:tasks.rados.rados.0.smithi001.stdout:377: finishing rollback tid 0 to smithi00132172-27 2024-09-15T03:47:56.931 INFO:tasks.rados.rados.0.smithi001.stdout:374: finishing rollback tid 0 to smithi00132172-25 2024-09-15T03:47:56.931 INFO:tasks.rados.rados.0.smithi001.stdout:377: finishing rollback tid 1 to smithi00132172-27 2024-09-15T03:47:56.931 INFO:tasks.rados.rados.0.smithi001.stdout:374: finishing rollback tid 1 to smithi00132172-25 2024-09-15T03:47:56.931 INFO:tasks.rados.rados.0.smithi001.stdout:376: finishing copy_from racing read to smithi00132172-32 2024-09-15T03:47:56.931 INFO:tasks.rados.rados.0.smithi001.stdout:373: finishing write tid 4 to smithi00132172-5 2024-09-15T03:47:56.931 INFO:tasks.rados.rados.0.smithi001.stdout:373: finishing write tid 5 to smithi00132172-5 2024-09-15T03:47:56.931 INFO:tasks.rados.rados.0.smithi001.stdout:377: finishing rollback tid 2 to smithi00132172-27 2024-09-15T03:47:56.931 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 27 v 71 (ObjNum 85 snap 18 seq_num 85) dirty exists 2024-09-15T03:47:56.931 INFO:tasks.rados.rados.0.smithi001.stdout:374: finishing rollback tid 2 to smithi00132172-25 2024-09-15T03:47:56.932 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 25 v 156 (ObjNum 66 snap 6 seq_num 66) dirty exists 2024-09-15T03:47:56.932 INFO:tasks.rados.rados.0.smithi001.stdout:376: finishing copy_from to smithi00132172-32 2024-09-15T03:47:56.932 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 32 v 80 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-15T03:47:56.932 INFO:tasks.rados.rados.0.smithi001.stdout:373: finishing write tid 6 to smithi00132172-5 2024-09-15T03:47:56.932 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 5 v 68 (ObjNum 142 snap 48 seq_num 142) dirty exists 2024-09-15T03:47:56.932 INFO:tasks.rados.rados.0.smithi001.stdout:373: left oid 5 (ObjNum 142 snap 48 seq_num 142) 2024-09-15T03:47:56.932 INFO:tasks.rados.rados.0.smithi001.stdout:375: finishing copy_from racing read to smithi00132172-16 2024-09-15T03:47:56.932 INFO:tasks.rados.rados.0.smithi001.stdout:373: done (5 left) 2024-09-15T03:47:56.932 INFO:tasks.rados.rados.0.smithi001.stdout:374: done (4 left) 2024-09-15T03:47:56.932 INFO:tasks.rados.rados.0.smithi001.stdout:376: done (3 left) 2024-09-15T03:47:56.932 INFO:tasks.rados.rados.0.smithi001.stdout:377: done (2 left) 2024-09-15T03:47:56.932 INFO:tasks.rados.rados.0.smithi001.stdout:380: snap_remove snap 19 2024-09-15T03:47:56.932 INFO:tasks.rados.rados.0.smithi001.stdout:375: finishing copy_from to smithi00132172-16 2024-09-15T03:47:56.933 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 16 v 107 (ObjNum 112 snap 32 seq_num 112) dirty exists 2024-09-15T03:47:56.933 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-15T03:47:56.933 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.1 to 1 (10000) 2024-09-15T03:47:56.944 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:56.943+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:56.970 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-15T03:47:56.988 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:56.988 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:56.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:47:57.045 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:47:57.044+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:47:57.093 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:57.094 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:57.102 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:47:57.147 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:57.145+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:57.241 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:57.241 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:57.246 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:57.245+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:57.249 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:47:57.347 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:57.347+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:57.386 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:57.386 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:57.395 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:47:57.449 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:57.448+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:57.496 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:57.496 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:57.504 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:47:57.549 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:57.548+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:57.605 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:57.605 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:57.614 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:47:57.650 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:57.649+0000 7fd62abf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33736) UID: 0 2024-09-15T03:47:57.715 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:57.715 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:57.723 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:47:57.751 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:57.750+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:57.824 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:57.825 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:57.833 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:47:57.852 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:57.851+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:57.931 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.2 to 1 (10000) 2024-09-15T03:47:57.932 INFO:tasks.rados.rados.0.smithi001.stdout:375: done (2 left) 2024-09-15T03:47:57.932 INFO:tasks.rados.rados.0.smithi001.stdout:380: done (1 left) 2024-09-15T03:47:57.932 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 43 current snap is 48 2024-09-15T03:47:57.932 INFO:tasks.rados.rados.0.smithi001.stdout:381: seq_num 143 ranges {4816896=458752,5275648=475136,5750784=655360,6406144=671744,7077888=147456} 2024-09-15T03:47:57.936 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:57.936 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:57.953 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:57.952+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:57.954 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:47:57.962 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-15T03:47:57.965 INFO:tasks.rados.rados.0.smithi001.stdout:381: writing smithi00132172-43 from 4816896 to 5275648 tid 1 2024-09-15T03:47:57.969 INFO:tasks.rados.rados.0.smithi001.stdout:381: writing smithi00132172-43 from 5275648 to 5750784 tid 2 2024-09-15T03:47:57.973 INFO:tasks.rados.rados.0.smithi001.stdout:381: writing smithi00132172-43 from 5750784 to 6406144 tid 3 2024-09-15T03:47:57.977 INFO:tasks.rados.rados.0.smithi001.stdout:381: writing smithi00132172-43 from 6406144 to 7077888 tid 4 2024-09-15T03:47:57.979 INFO:tasks.rados.rados.0.smithi001.stdout:381: writing smithi00132172-43 from 7077888 to 7225344 tid 5 2024-09-15T03:47:57.979 INFO:tasks.rados.rados.0.smithi001.stdout:382: copy_from oid 8 from oid 20 current snap is 48 2024-09-15T03:47:57.979 INFO:tasks.rados.rados.0.smithi001.stdout:379: finishing copy_from racing read to smithi00132172-49 2024-09-15T03:47:57.979 INFO:tasks.rados.rados.0.smithi001.stdout:379: finishing copy_from to smithi00132172-49 2024-09-15T03:47:57.979 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 49 v 31 (ObjNum 102 snap 28 seq_num 102) dirty exists 2024-09-15T03:47:57.979 INFO:tasks.rados.rados.0.smithi001.stdout:381: finishing write tid 1 to smithi00132172-43 2024-09-15T03:47:57.979 INFO:tasks.rados.rados.0.smithi001.stdout:381: finishing write tid 2 to smithi00132172-43 2024-09-15T03:47:57.979 INFO:tasks.rados.rados.0.smithi001.stdout:381: finishing write tid 3 to smithi00132172-43 2024-09-15T03:47:57.980 INFO:tasks.rados.rados.0.smithi001.stdout:379: done (2 left) 2024-09-15T03:47:57.980 INFO:tasks.rados.rados.0.smithi001.stdout:383: read oid 5 snap 29 2024-09-15T03:47:57.980 INFO:tasks.rados.rados.0.smithi001.stdout:383: expect (ObjNum 105 snap 29 seq_num 105) 2024-09-15T03:47:57.980 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 4 current snap is 48 2024-09-15T03:47:57.980 INFO:tasks.rados.rados.0.smithi001.stdout:384: seq_num 144 ranges {1179648=16384} 2024-09-15T03:47:57.987 INFO:tasks.rados.rados.0.smithi001.stdout:384: writing smithi00132172-4 from 1179648 to 1196032 tid 1 2024-09-15T03:47:57.987 INFO:tasks.rados.rados.0.smithi001.stdout:385: rollback oid 29 current snap is 48 2024-09-15T03:47:57.987 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 29 to 20 2024-09-15T03:47:57.987 INFO:tasks.rados.rados.0.smithi001.stdout:381: finishing write tid 4 to smithi00132172-43 2024-09-15T03:47:57.987 INFO:tasks.rados.rados.0.smithi001.stdout:381: finishing write tid 5 to smithi00132172-43 2024-09-15T03:47:57.988 INFO:tasks.rados.rados.0.smithi001.stdout:386: rollback oid 9 current snap is 48 2024-09-15T03:47:57.988 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 9 to 41 2024-09-15T03:47:57.988 INFO:tasks.rados.rados.0.smithi001.stdout:387: read oid 34 snap 43 2024-09-15T03:47:57.988 INFO:tasks.rados.rados.0.smithi001.stdout:387: expect (ObjNum 70 snap 8 seq_num 70) 2024-09-15T03:47:57.988 INFO:tasks.rados.rados.0.smithi001.stdout:388: snap_create 2024-09-15T03:47:57.989 INFO:tasks.rados.rados.0.smithi001.stdout:383: expect (ObjNum 105 snap 29 seq_num 105) 2024-09-15T03:47:58.044 INFO:tasks.rados.rados.0.smithi001.stdout:381: finishing write tid 6 to smithi00132172-43 2024-09-15T03:47:58.045 INFO:tasks.rados.rados.0.smithi001.stdout:381: finishing write tid 7 to smithi00132172-43 2024-09-15T03:47:58.045 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 43 v 136 (ObjNum 143 snap 48 seq_num 143) dirty exists 2024-09-15T03:47:58.045 INFO:tasks.rados.rados.0.smithi001.stdout:381: left oid 43 (ObjNum 143 snap 48 seq_num 143) 2024-09-15T03:47:58.045 INFO:tasks.rados.rados.0.smithi001.stdout:385: finishing rollback tid 0 to smithi00132172-29 2024-09-15T03:47:58.045 INFO:tasks.rados.rados.0.smithi001.stdout:386: finishing rollback tid 0 to smithi00132172-9 2024-09-15T03:47:58.045 INFO:tasks.rados.rados.0.smithi001.stdout:385: finishing rollback tid 1 to smithi00132172-29 2024-09-15T03:47:58.045 INFO:tasks.rados.rados.0.smithi001.stdout:386: finishing rollback tid 1 to smithi00132172-9 2024-09-15T03:47:58.045 INFO:tasks.rados.rados.0.smithi001.stdout:385: finishing rollback tid 2 to smithi00132172-29 2024-09-15T03:47:58.045 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 29 v 201 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-15T03:47:58.045 INFO:tasks.rados.rados.0.smithi001.stdout:386: finishing rollback tid 2 to smithi00132172-9 2024-09-15T03:47:58.045 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 9 v 180 (ObjNum 103 snap 28 seq_num 103) dirty exists 2024-09-15T03:47:58.046 INFO:tasks.rados.rados.0.smithi001.stdout:384: finishing write tid 1 to smithi00132172-4 2024-09-15T03:47:58.046 INFO:tasks.rados.rados.0.smithi001.stdout:384: finishing write tid 2 to smithi00132172-4 2024-09-15T03:47:58.046 INFO:tasks.rados.rados.0.smithi001.stdout:387: expect (ObjNum 70 snap 8 seq_num 70) 2024-09-15T03:47:58.053 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:58.052+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:58.075 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:47:58.076 INFO:tasks.rados.rados.0.smithi001.stdout:382: finishing copy_from racing read to smithi00132172-8 2024-09-15T03:47:58.076 INFO:tasks.rados.rados.0.smithi001.stdout:382: finishing copy_from to smithi00132172-8 2024-09-15T03:47:58.076 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 8 v 28 (ObjNum 99 snap 24 seq_num 99) dirty exists 2024-09-15T03:47:58.076 INFO:tasks.rados.rados.0.smithi001.stdout:384: finishing write tid 3 to smithi00132172-4 2024-09-15T03:47:58.076 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 4 v 200 (ObjNum 144 snap 48 seq_num 144) dirty exists 2024-09-15T03:47:58.076 INFO:tasks.rados.rados.0.smithi001.stdout:384: left oid 4 (ObjNum 144 snap 48 seq_num 144) 2024-09-15T03:47:58.077 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:47:58.077 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:47:58.085 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:47:58.154 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:47:58.154+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:47:58.198 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:47:58.199 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:47:58.215 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:47:58.215 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:47:58.231 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:47:58.231 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:47:58.232 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:58.232 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:58.232 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-15T03:47:58.232 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:58.242 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:47:58.246 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:47:58.246 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:47:58.255 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:47:58.255+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:47:58.262 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:47:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:47:58.279 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:47:58.280 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:47:58.294 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:47:58.294 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:47:58.309 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:47:58.309 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:47:58.324 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:47:58.324 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:47:58.339 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:47:58.339 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:47:58.345 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:58.345 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:58.345 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-15T03:47:58.345 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-15T03:47:58.345 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:58.354 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:47:58.354 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:47:58.355 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:47:58.355 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:58.355+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:58.369 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:47:58.369 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:47:58.457 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:47:58.456+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:47:58.471 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:58.471 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-15T03:47:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-15T03:47:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-15T03:47:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:195 3.es0 3:7f0816e1:::smithi00132172-31:head [copy-from ver 4] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:47:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:41.112078+0000", 2024-09-15T03:47:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 77.357340653999998, 2024-09-15T03:47:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012535605, 2024-09-15T03:47:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:58.472 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 195 2024-09-15T03:47:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112078+0000", 2024-09-15T03:47:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112078+0000", 2024-09-15T03:47:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112080+0000", 2024-09-15T03:47:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4399999999999999e-06 2024-09-15T03:47:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112090+0000", 2024-09-15T03:47:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7310000000000005e-06 2024-09-15T03:47:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:58.474 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112107+0000", 2024-09-15T03:47:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7557000000000001e-05 2024-09-15T03:47:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112113+0000", 2024-09-15T03:47:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6910000000000002e-06 2024-09-15T03:47:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112176+0000", 2024-09-15T03:47:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3014000000000003e-05 2024-09-15T03:47:58.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112308+0000", 2024-09-15T03:47:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000131402 2024-09-15T03:47:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118088+0000", 2024-09-15T03:47:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0057801570000000002 2024-09-15T03:47:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.476 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118560+0000", 2024-09-15T03:47:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00047206199999999998 2024-09-15T03:47:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118573+0000", 2024-09-15T03:47:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3521e-05 2024-09-15T03:47:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118667+0000", 2024-09-15T03:47:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.323e-05 2024-09-15T03:47:58.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.119046+0000", 2024-09-15T03:47:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00037924699999999998 2024-09-15T03:47:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.124533+0000", 2024-09-15T03:47:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0054869849999999998 2024-09-15T03:47:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:58.478 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.124613+0000", 2024-09-15T03:47:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0568000000000005e-05 2024-09-15T03:47:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:384 3.es0 3:77a7da34:::smithi00132172-4:head [read 0~1146880,checksum,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e56)", 2024-09-15T03:47:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:02.864202+0000", 2024-09-15T03:47:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.605216874999996, 2024-09-15T03:47:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012715746999999999, 2024-09-15T03:47:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:58.479 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 384 2024-09-15T03:47:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864202+0000", 2024-09-15T03:47:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864202+0000", 2024-09-15T03:47:58.480 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864202+0000", 2024-09-15T03:47:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9400000000000004e-07 2024-09-15T03:47:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864206+0000", 2024-09-15T03:47:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9770000000000002e-06 2024-09-15T03:47:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.481 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864215+0000", 2024-09-15T03:47:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.337e-06 2024-09-15T03:47:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864221+0000", 2024-09-15T03:47:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4570000000000002e-06 2024-09-15T03:47:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864255+0000", 2024-09-15T03:47:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3725e-05 2024-09-15T03:47:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864260+0000", 2024-09-15T03:47:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.395e-06 2024-09-15T03:47:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.871305+0000", 2024-09-15T03:47:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0070446800000000002 2024-09-15T03:47:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.871413+0000", 2024-09-15T03:47:58.483 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000107731 2024-09-15T03:47:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.876917+0000", 2024-09-15T03:47:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0055046510000000002 2024-09-15T03:47:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:385 3.es0 3:77a7da34:::smithi00132172-4:head [read 0~1146880,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e56)", 2024-09-15T03:47:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:02.864230+0000", 2024-09-15T03:47:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.605187944000001, 2024-09-15T03:47:58.484 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.014027420000000001, 2024-09-15T03:47:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 385 2024-09-15T03:47:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864230+0000", 2024-09-15T03:47:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864230+0000", 2024-09-15T03:47:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864231+0000", 2024-09-15T03:47:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7700000000000005e-07 2024-09-15T03:47:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:58.486 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864234+0000", 2024-09-15T03:47:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0369999999999999e-06 2024-09-15T03:47:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864242+0000", 2024-09-15T03:47:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4200000000000007e-06 2024-09-15T03:47:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864247+0000", 2024-09-15T03:47:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4789999999999998e-06 2024-09-15T03:47:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.487 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864290+0000", 2024-09-15T03:47:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3594000000000003e-05 2024-09-15T03:47:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864297+0000", 2024-09-15T03:47:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3579999999999999e-06 2024-09-15T03:47:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.872828+0000", 2024-09-15T03:47:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0085315449999999998 2024-09-15T03:47:58.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:58.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.872956+0000", 2024-09-15T03:47:58.489 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000127818 2024-09-15T03:47:58.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.489 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:58.489 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.878258+0000", 2024-09-15T03:47:58.493 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0053016920000000002 2024-09-15T03:47:58.493 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.494 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:58.494 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.494 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:386 3.es0 3:77a7da34:::smithi00132172-4:head [read 0~1146880,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e56)", 2024-09-15T03:47:58.494 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:02.864259+0000", 2024-09-15T03:47:58.494 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.605159149999999, 2024-09-15T03:47:58.494 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.014933481, 2024-09-15T03:47:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 386 2024-09-15T03:47:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864259+0000", 2024-09-15T03:47:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864259+0000", 2024-09-15T03:47:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864260+0000", 2024-09-15T03:47:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3799999999999998e-07 2024-09-15T03:47:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864262+0000", 2024-09-15T03:47:58.496 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7209999999999999e-06 2024-09-15T03:47:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864270+0000", 2024-09-15T03:47:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6240000000000002e-06 2024-09-15T03:47:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864274+0000", 2024-09-15T03:47:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1699999999999999e-06 2024-09-15T03:47:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.497 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864316+0000", 2024-09-15T03:47:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1489999999999997e-05 2024-09-15T03:47:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864321+0000", 2024-09-15T03:47:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1100000000000002e-06 2024-09-15T03:47:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.873888+0000", 2024-09-15T03:47:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0095676790000000008 2024-09-15T03:47:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.874019+0000", 2024-09-15T03:47:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013085199999999999 2024-09-15T03:47:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.879193+0000", 2024-09-15T03:47:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0051733969999999997 2024-09-15T03:47:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.499 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 60/31 ECSubWrite(tid=70, reqid=client.4521.0:421, at_version=60'22, trim_to=0'0, roll_forward_to=47'20))", 2024-09-15T03:47:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.837585+0000", 2024-09-15T03:47:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 51.631833858999997, 2024-09-15T03:47:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011527037, 2024-09-15T03:47:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837585+0000", 2024-09-15T03:47:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:58.500 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837585+0000", 2024-09-15T03:47:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837585+0000", 2024-09-15T03:47:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1199999999999999e-07 2024-09-15T03:47:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837609+0000", 2024-09-15T03:47:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4647999999999998e-05 2024-09-15T03:47:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.501 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837631+0000", 2024-09-15T03:47:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1840000000000001e-05 2024-09-15T03:47:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837641+0000", 2024-09-15T03:47:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0263e-05 2024-09-15T03:47:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.838036+0000", 2024-09-15T03:47:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00039492500000000002 2024-09-15T03:47:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.502 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.838048+0000", 2024-09-15T03:47:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1599e-05 2024-09-15T03:47:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.838445+0000", 2024-09-15T03:47:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000397313 2024-09-15T03:47:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.838479+0000", 2024-09-15T03:47:58.503 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3368000000000003e-05 2024-09-15T03:47:58.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.504 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.838525+0000", 2024-09-15T03:47:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6100999999999998e-05 2024-09-15T03:47:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.841589+0000", 2024-09-15T03:47:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003063957 2024-09-15T03:47:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.849112+0000", 2024-09-15T03:47:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0075228109999999999 2024-09-15T03:47:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.505 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:446 3.es0 3:75485f06:::smithi00132172-19:head [copy-from ver 38] snapc 1a=[1a,18,17,16,15,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:47:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.951492+0000", 2024-09-15T03:47:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 51.517926764000002, 2024-09-15T03:47:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013140193, 2024-09-15T03:47:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 446 2024-09-15T03:47:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951492+0000", 2024-09-15T03:47:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:58.507 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951492+0000", 2024-09-15T03:47:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951493+0000", 2024-09-15T03:47:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.525e-06 2024-09-15T03:47:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951500+0000", 2024-09-15T03:47:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6649999999999997e-06 2024-09-15T03:47:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.508 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951514+0000", 2024-09-15T03:47:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3713999999999999e-05 2024-09-15T03:47:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951518+0000", 2024-09-15T03:47:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.921e-06 2024-09-15T03:47:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951569+0000", 2024-09-15T03:47:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0281999999999997e-05 2024-09-15T03:47:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.509 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951676+0000", 2024-09-15T03:47:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000107527 2024-09-15T03:47:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.961625+0000", 2024-09-15T03:47:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0099492209999999994 2024-09-15T03:47:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.510 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.962100+0000", 2024-09-15T03:47:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00047458100000000002 2024-09-15T03:47:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.962112+0000", 2024-09-15T03:47:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2069000000000001e-05 2024-09-15T03:47:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.962343+0000", 2024-09-15T03:47:58.511 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000230962 2024-09-15T03:47:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.962561+0000", 2024-09-15T03:47:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000217899 2024-09-15T03:47:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.964176+0000", 2024-09-15T03:47:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001615109 2024-09-15T03:47:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:58.512 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.964632+0000", 2024-09-15T03:47:58.513 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00045571799999999998 2024-09-15T03:47:58.513 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.513 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:58.513 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.514 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.514 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:447 3.es0 3:75485f06:::smithi00132172-19:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:47:58.514 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.951527+0000", 2024-09-15T03:47:58.514 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 51.517891298000002, 2024-09-15T03:47:58.514 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011030607, 2024-09-15T03:47:58.514 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:58.514 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 447 2024-09-15T03:47:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951527+0000", 2024-09-15T03:47:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951527+0000", 2024-09-15T03:47:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951528+0000", 2024-09-15T03:47:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.63e-07 2024-09-15T03:47:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951530+0000", 2024-09-15T03:47:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4499999999999998e-06 2024-09-15T03:47:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.516 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951539+0000", 2024-09-15T03:47:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2760000000000006e-06 2024-09-15T03:47:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951543+0000", 2024-09-15T03:47:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0149999999999997e-06 2024-09-15T03:47:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951845+0000", 2024-09-15T03:47:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00030196500000000001 2024-09-15T03:47:58.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951915+0000", 2024-09-15T03:47:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9814000000000006e-05 2024-09-15T03:47:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.962316+0000", 2024-09-15T03:47:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010401405000000001 2024-09-15T03:47:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.962449+0000", 2024-09-15T03:47:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013275900000000001 2024-09-15T03:47:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.962558+0000", 2024-09-15T03:47:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010846 2024-09-15T03:47:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 65/64 ECSubWrite(tid=103, reqid=client.4521.0:463, at_version=65'24, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.847624+0000", 2024-09-15T03:47:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 46.621794199, 2024-09-15T03:47:58.519 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010501285000000001, 2024-09-15T03:47:58.520 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:58.520 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:58.520 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:58.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:58.521 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.847624+0000", 2024-09-15T03:47:58.521 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.521 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:58.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.847624+0000", 2024-09-15T03:47:58.522 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.522 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:58.522 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.847625+0000", 2024-09-15T03:47:58.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7599999999999998e-07 2024-09-15T03:47:58.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.523 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:58.523 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.847661+0000", 2024-09-15T03:47:58.523 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5936000000000003e-05 2024-09-15T03:47:58.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.524 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:58.524 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.847671+0000", 2024-09-15T03:47:58.524 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0835e-05 2024-09-15T03:47:58.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.525 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:58.525 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.847681+0000", 2024-09-15T03:47:58.525 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8190000000000003e-06 2024-09-15T03:47:58.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.526 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.847733+0000", 2024-09-15T03:47:58.526 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2256999999999998e-05 2024-09-15T03:47:58.526 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.526 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.526 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:58.527 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.847745+0000", 2024-09-15T03:47:58.527 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1851999999999999e-05 2024-09-15T03:47:58.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.528 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:58.528 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848063+0000", 2024-09-15T03:47:58.528 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031761300000000001 2024-09-15T03:47:58.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.529 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:58.529 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848218+0000", 2024-09-15T03:47:58.529 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000155341 2024-09-15T03:47:58.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:58.530 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848261+0000", 2024-09-15T03:47:58.530 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3176999999999997e-05 2024-09-15T03:47:58.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.530 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.849619+0000", 2024-09-15T03:47:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001358064 2024-09-15T03:47:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858125+0000", 2024-09-15T03:47:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0085060150000000005 2024-09-15T03:47:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 65/64 ECSubWrite(tid=106, reqid=client.4521.0:464, at_version=65'25, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.852924+0000", 2024-09-15T03:47:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 46.616494236999998, 2024-09-15T03:47:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013666606, 2024-09-15T03:47:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852924+0000", 2024-09-15T03:47:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852924+0000", 2024-09-15T03:47:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852925+0000", 2024-09-15T03:47:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9999999999999997e-07 2024-09-15T03:47:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852945+0000", 2024-09-15T03:47:58.533 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9952000000000001e-05 2024-09-15T03:47:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852957+0000", 2024-09-15T03:47:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2532e-05 2024-09-15T03:47:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852965+0000", 2024-09-15T03:47:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6359999999999994e-06 2024-09-15T03:47:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.534 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.853100+0000", 2024-09-15T03:47:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013520900000000001 2024-09-15T03:47:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.853111+0000", 2024-09-15T03:47:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0508e-05 2024-09-15T03:47:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.853220+0000", 2024-09-15T03:47:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000109053 2024-09-15T03:47:58.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.853247+0000", 2024-09-15T03:47:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7025999999999998e-05 2024-09-15T03:47:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854114+0000", 2024-09-15T03:47:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00086709299999999999 2024-09-15T03:47:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854129+0000", 2024-09-15T03:47:58.536 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5602000000000001e-05 2024-09-15T03:47:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866591+0000", 2024-09-15T03:47:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012461395 2024-09-15T03:47:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 65/64 ECSubWrite(tid=107, reqid=client.4521.0:465, at_version=65'26, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.854248+0000", 2024-09-15T03:47:58.537 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 46.615169971, 2024-09-15T03:47:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012733421999999999, 2024-09-15T03:47:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854248+0000", 2024-09-15T03:47:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854248+0000", 2024-09-15T03:47:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854249+0000", 2024-09-15T03:47:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8999999999999997e-07 2024-09-15T03:47:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854274+0000", 2024-09-15T03:47:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4678e-05 2024-09-15T03:47:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854289+0000", 2024-09-15T03:47:58.539 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5622000000000001e-05 2024-09-15T03:47:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854296+0000", 2024-09-15T03:47:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2869999999999999e-06 2024-09-15T03:47:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859614+0000", 2024-09-15T03:47:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0053176869999999998 2024-09-15T03:47:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.540 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859625+0000", 2024-09-15T03:47:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0475000000000001e-05 2024-09-15T03:47:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859728+0000", 2024-09-15T03:47:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010383 2024-09-15T03:47:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859769+0000", 2024-09-15T03:47:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0132999999999997e-05 2024-09-15T03:47:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859855+0000", 2024-09-15T03:47:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6188999999999998e-05 2024-09-15T03:47:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860468+0000", 2024-09-15T03:47:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00061328800000000003 2024-09-15T03:47:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866982+0000", 2024-09-15T03:47:58.542 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0065137429999999998 2024-09-15T03:47:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 65/64 ECSubWrite(tid=108, reqid=client.4521.0:466, at_version=65'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.858803+0000", 2024-09-15T03:47:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 46.610615549999999, 2024-09-15T03:47:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012489828999999999, 2024-09-15T03:47:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:58.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858803+0000", 2024-09-15T03:47:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858803+0000", 2024-09-15T03:47:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858803+0000", 2024-09-15T03:47:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1699999999999999e-07 2024-09-15T03:47:58.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858970+0000", 2024-09-15T03:47:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000166844 2024-09-15T03:47:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858982+0000", 2024-09-15T03:47:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2092999999999999e-05 2024-09-15T03:47:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:58.545 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858988+0000", 2024-09-15T03:47:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1999999999999999e-06 2024-09-15T03:47:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859762+0000", 2024-09-15T03:47:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00077351199999999996 2024-09-15T03:47:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859770+0000", 2024-09-15T03:47:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0360000000000006e-06 2024-09-15T03:47:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.546 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860086+0000", 2024-09-15T03:47:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031557999999999998 2024-09-15T03:47:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860099+0000", 2024-09-15T03:47:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3329999999999999e-05 2024-09-15T03:47:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860225+0000", 2024-09-15T03:47:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012638300000000001 2024-09-15T03:47:58.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861591+0000", 2024-09-15T03:47:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0013660059999999999 2024-09-15T03:47:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871293+0000", 2024-09-15T03:47:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0097014279999999998 2024-09-15T03:47:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 65/64 ECSubWrite(tid=111, reqid=client.4521.0:467, at_version=65'29, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.858996+0000", 2024-09-15T03:47:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 46.610422819999997, 2024-09-15T03:47:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012595857, 2024-09-15T03:47:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858996+0000", 2024-09-15T03:47:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858996+0000", 2024-09-15T03:47:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858996+0000", 2024-09-15T03:47:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9900000000000001e-07 2024-09-15T03:47:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859088+0000", 2024-09-15T03:47:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2473999999999996e-05 2024-09-15T03:47:58.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859102+0000", 2024-09-15T03:47:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3933e-05 2024-09-15T03:47:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859114+0000", 2024-09-15T03:47:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1262e-05 2024-09-15T03:47:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.551 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860136+0000", 2024-09-15T03:47:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010221169999999999 2024-09-15T03:47:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860144+0000", 2024-09-15T03:47:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6270000000000006e-06 2024-09-15T03:47:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860264+0000", 2024-09-15T03:47:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000119954 2024-09-15T03:47:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.552 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860274+0000", 2024-09-15T03:47:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8959999999999998e-06 2024-09-15T03:47:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860365+0000", 2024-09-15T03:47:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1195999999999999e-05 2024-09-15T03:47:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861615+0000", 2024-09-15T03:47:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012495690000000001 2024-09-15T03:47:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871591+0000", 2024-09-15T03:47:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0099764299999999997 2024-09-15T03:47:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 65/64 ECSubWrite(tid=112, reqid=client.4521.0:468, at_version=65'30, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.859121+0000", 2024-09-15T03:47:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 46.610297557999999, 2024-09-15T03:47:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012988807, 2024-09-15T03:47:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:58.554 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859121+0000", 2024-09-15T03:47:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859121+0000", 2024-09-15T03:47:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.555 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859121+0000", 2024-09-15T03:47:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4499999999999998e-07 2024-09-15T03:47:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859170+0000", 2024-09-15T03:47:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9008999999999998e-05 2024-09-15T03:47:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859179+0000", 2024-09-15T03:47:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5809999999999997e-06 2024-09-15T03:47:58.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859185+0000", 2024-09-15T03:47:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7350000000000001e-06 2024-09-15T03:47:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860314+0000", 2024-09-15T03:47:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001129403 2024-09-15T03:47:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860324+0000", 2024-09-15T03:47:58.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0274e-05 2024-09-15T03:47:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860440+0000", 2024-09-15T03:47:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011603 2024-09-15T03:47:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860450+0000", 2024-09-15T03:47:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0243e-05 2024-09-15T03:47:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860518+0000", 2024-09-15T03:47:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7780000000000005e-05 2024-09-15T03:47:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861625+0000", 2024-09-15T03:47:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0011072460000000001 2024-09-15T03:47:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.872110+0000", 2024-09-15T03:47:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010484161000000001 2024-09-15T03:47:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 65/64 ECSubWrite(tid=113, reqid=client.4521.0:469, at_version=65'31, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:47:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.860777+0000", 2024-09-15T03:47:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 46.608640975999997, 2024-09-15T03:47:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012054540000000001, 2024-09-15T03:47:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:58.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860777+0000", 2024-09-15T03:47:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860777+0000", 2024-09-15T03:47:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860778+0000", 2024-09-15T03:47:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3399999999999999e-07 2024-09-15T03:47:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860795+0000", 2024-09-15T03:47:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6691e-05 2024-09-15T03:47:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860807+0000", 2024-09-15T03:47:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2149e-05 2024-09-15T03:47:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860814+0000", 2024-09-15T03:47:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.255e-06 2024-09-15T03:47:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861681+0000", 2024-09-15T03:47:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00086670300000000003 2024-09-15T03:47:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861690+0000", 2024-09-15T03:47:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3039999999999994e-06 2024-09-15T03:47:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861789+0000", 2024-09-15T03:47:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8851999999999995e-05 2024-09-15T03:47:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861819+0000", 2024-09-15T03:47:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9646e-05 2024-09-15T03:47:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861852+0000", 2024-09-15T03:47:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3436000000000003e-05 2024-09-15T03:47:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861871+0000", 2024-09-15T03:47:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9202e-05 2024-09-15T03:47:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.872832+0000", 2024-09-15T03:47:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010960668 2024-09-15T03:47:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.4s1 65/64 ECSubWrite(tid=70, reqid=osd.3.0:68, at_version=0'0, trim_to=0'0, roll_forward_to=65'26))", 2024-09-15T03:47:58.565 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.861273+0000", 2024-09-15T03:47:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 46.608145559, 2024-09-15T03:47:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010446211, 2024-09-15T03:47:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861273+0000", 2024-09-15T03:47:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861273+0000", 2024-09-15T03:47:58.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861274+0000", 2024-09-15T03:47:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0240000000000001e-06 2024-09-15T03:47:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861288+0000", 2024-09-15T03:47:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4036000000000001e-05 2024-09-15T03:47:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:58.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861318+0000", 2024-09-15T03:47:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0590999999999999e-05 2024-09-15T03:47:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861335+0000", 2024-09-15T03:47:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6679000000000001e-05 2024-09-15T03:47:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861415+0000", 2024-09-15T03:47:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9375000000000005e-05 2024-09-15T03:47:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861425+0000", 2024-09-15T03:47:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0101999999999999e-05 2024-09-15T03:47:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861533+0000", 2024-09-15T03:47:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010840699999999999 2024-09-15T03:47:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861623+0000", 2024-09-15T03:47:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9856000000000002e-05 2024-09-15T03:47:58.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861671+0000", 2024-09-15T03:47:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8346000000000001e-05 2024-09-15T03:47:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861686+0000", 2024-09-15T03:47:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4792e-05 2024-09-15T03:47:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:58.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871719+0000", 2024-09-15T03:47:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010033003 2024-09-15T03:47:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 65/64 ECSubWrite(tid=116, reqid=client.4521.0:469, at_version=0'0, trim_to=0'0, roll_forward_to=65'31))", 2024-09-15T03:47:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.866498+0000", 2024-09-15T03:47:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 46.602920267000002, 2024-09-15T03:47:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013459561, 2024-09-15T03:47:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:47:58.571 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866498+0000", 2024-09-15T03:47:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866498+0000", 2024-09-15T03:47:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:58.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866499+0000", 2024-09-15T03:47:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9800000000000003e-07 2024-09-15T03:47:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866509+0000", 2024-09-15T03:47:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8980000000000008e-06 2024-09-15T03:47:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866520+0000", 2024-09-15T03:47:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0927000000000001e-05 2024-09-15T03:47:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866528+0000", 2024-09-15T03:47:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.388e-06 2024-09-15T03:47:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870788+0000", 2024-09-15T03:47:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.004259657 2024-09-15T03:47:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870798+0000", 2024-09-15T03:47:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0156999999999999e-05 2024-09-15T03:47:58.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871002+0000", 2024-09-15T03:47:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000204174 2024-09-15T03:47:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871015+0000", 2024-09-15T03:47:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3422e-05 2024-09-15T03:47:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871057+0000", 2024-09-15T03:47:58.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1270999999999998e-05 2024-09-15T03:47:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.872441+0000", 2024-09-15T03:47:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0013840980000000001 2024-09-15T03:47:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.879958+0000", 2024-09-15T03:47:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0075169709999999999 2024-09-15T03:47:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:58.576 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:488 3.es0 3:75485f06:::smithi00132172-19:head [rollback 12] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-15T03:47:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.867857+0000", 2024-09-15T03:47:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 46.601561021999999, 2024-09-15T03:47:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011714672000000001, 2024-09-15T03:47:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 488 2024-09-15T03:47:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.577 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867857+0000", 2024-09-15T03:47:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867857+0000", 2024-09-15T03:47:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867859+0000", 2024-09-15T03:47:58.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.562e-06 2024-09-15T03:47:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867874+0000", 2024-09-15T03:47:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5170999999999999e-05 2024-09-15T03:47:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867886+0000", 2024-09-15T03:47:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1819e-05 2024-09-15T03:47:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867892+0000", 2024-09-15T03:47:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0449999999999997e-06 2024-09-15T03:47:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868553+0000", 2024-09-15T03:47:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00066058700000000002 2024-09-15T03:47:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868660+0000", 2024-09-15T03:47:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000107878 2024-09-15T03:47:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869303+0000", 2024-09-15T03:47:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00064280399999999997 2024-09-15T03:47:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869419+0000", 2024-09-15T03:47:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011619400000000001 2024-09-15T03:47:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869431+0000", 2024-09-15T03:47:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1941e-05 2024-09-15T03:47:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869461+0000", 2024-09-15T03:47:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9478000000000001e-05 2024-09-15T03:47:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870370+0000", 2024-09-15T03:47:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00090957899999999999 2024-09-15T03:47:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.873494+0000", 2024-09-15T03:47:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0031240679999999998 2024-09-15T03:47:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.879572+0000", 2024-09-15T03:47:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0060775459999999996 2024-09-15T03:47:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:489 3.es0 3:75485f06:::smithi00132172-19:head [append 0~0] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-15T03:47:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.867898+0000", 2024-09-15T03:47:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 46.601520413000003, 2024-09-15T03:47:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012220945, 2024-09-15T03:47:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 489 2024-09-15T03:47:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867898+0000", 2024-09-15T03:47:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867898+0000", 2024-09-15T03:47:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867899+0000", 2024-09-15T03:47:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0900000000000001e-07 2024-09-15T03:47:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867901+0000", 2024-09-15T03:47:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.65e-06 2024-09-15T03:47:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867908+0000", 2024-09-15T03:47:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2790000000000004e-06 2024-09-15T03:47:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867911+0000", 2024-09-15T03:47:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9700000000000001e-06 2024-09-15T03:47:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869480+0000", 2024-09-15T03:47:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001568051 2024-09-15T03:47:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869565+0000", 2024-09-15T03:47:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5500999999999993e-05 2024-09-15T03:47:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870027+0000", 2024-09-15T03:47:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00046226900000000002 2024-09-15T03:47:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870140+0000", 2024-09-15T03:47:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112223 2024-09-15T03:47:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870168+0000", 2024-09-15T03:47:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8031e-05 2024-09-15T03:47:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870202+0000", 2024-09-15T03:47:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4598999999999999e-05 2024-09-15T03:47:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870372+0000", 2024-09-15T03:47:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016990199999999999 2024-09-15T03:47:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.879822+0000", 2024-09-15T03:47:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0094500739999999993 2024-09-15T03:47:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.880119+0000", 2024-09-15T03:47:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029678700000000002 2024-09-15T03:47:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:830 3.es0 3:79526987:::smithi00132172-28:head [copy-from ver 107] snapc 3d=[3d,3b,3a,38,37,35,34,32,2d,2b,25,24,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-15T03:47:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:43.912967+0000", 2024-09-15T03:47:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 14.556451110999999, 2024-09-15T03:47:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.026454678999999998, 2024-09-15T03:47:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 830 2024-09-15T03:47:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.912967+0000", 2024-09-15T03:47:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.912967+0000", 2024-09-15T03:47:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.912969+0000", 2024-09-15T03:47:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.939e-06 2024-09-15T03:47:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.912980+0000", 2024-09-15T03:47:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.04e-05 2024-09-15T03:47:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.912999+0000", 2024-09-15T03:47:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9391000000000001e-05 2024-09-15T03:47:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913015+0000", 2024-09-15T03:47:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5568000000000001e-05 2024-09-15T03:47:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913068+0000", 2024-09-15T03:47:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3270999999999999e-05 2024-09-15T03:47:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913152+0000", 2024-09-15T03:47:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4155999999999999e-05 2024-09-15T03:47:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.934430+0000", 2024-09-15T03:47:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.021278372 2024-09-15T03:47:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.934773+0000", 2024-09-15T03:47:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00034307700000000001 2024-09-15T03:47:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.934792+0000", 2024-09-15T03:47:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8640000000000001e-05 2024-09-15T03:47:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.935203+0000", 2024-09-15T03:47:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00041061100000000002 2024-09-15T03:47:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.935232+0000", 2024-09-15T03:47:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9093000000000001e-05 2024-09-15T03:47:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.939341+0000", 2024-09-15T03:47:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041089849999999999 2024-09-15T03:47:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.939422+0000", 2024-09-15T03:47:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1175999999999997e-05 2024-09-15T03:47:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:831 3.es0 3:79526987:::smithi00132172-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2024-09-15T03:47:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:43.913039+0000", 2024-09-15T03:47:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 14.556379143999999, 2024-09-15T03:47:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.023987538999999999, 2024-09-15T03:47:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 831 2024-09-15T03:47:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913039+0000", 2024-09-15T03:47:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913039+0000", 2024-09-15T03:47:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913040+0000", 2024-09-15T03:47:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4900000000000005e-07 2024-09-15T03:47:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913044+0000", 2024-09-15T03:47:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0550000000000001e-06 2024-09-15T03:47:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913053+0000", 2024-09-15T03:47:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4150000000000005e-06 2024-09-15T03:47:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913058+0000", 2024-09-15T03:47:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8609999999999997e-06 2024-09-15T03:47:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913284+0000", 2024-09-15T03:47:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022525400000000001 2024-09-15T03:47:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913347+0000", 2024-09-15T03:47:58.600 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3400999999999998e-05 2024-09-15T03:47:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.936662+0000", 2024-09-15T03:47:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.023315361 2024-09-15T03:47:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.936840+0000", 2024-09-15T03:47:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017759699999999999 2024-09-15T03:47:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:58.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:58.602 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.937027+0000", 2024-09-15T03:47:58.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018684600000000001 2024-09-15T03:47:58.602 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.602 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:58.602 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.602 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:58.602 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:58.602 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:58.604 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:47:58.604 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:47:58.603+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:47:58.704 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:58.704+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:58.746 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:58.746 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:58.746 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-15T03:47:58.746 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:58.754 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:47:58.839 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:58.838+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:58.896 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:58.896 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:47:58.896 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-15T03:47:58.896 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-15T03:47:58.897 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:58.908 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:47:58.908 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:58.907+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:58.936 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.3 to 1 (10000) 2024-09-15T03:47:58.937 INFO:tasks.rados.rados.0.smithi001.stdout:381: done (7 left) 2024-09-15T03:47:58.937 INFO:tasks.rados.rados.0.smithi001.stdout:382: done (6 left) 2024-09-15T03:47:58.938 INFO:tasks.rados.rados.0.smithi001.stdout:383: done (5 left) 2024-09-15T03:47:58.938 INFO:tasks.rados.rados.0.smithi001.stdout:384: done (4 left) 2024-09-15T03:47:58.938 INFO:tasks.rados.rados.0.smithi001.stdout:385: done (3 left) 2024-09-15T03:47:58.938 INFO:tasks.rados.rados.0.smithi001.stdout:386: done (2 left) 2024-09-15T03:47:58.938 INFO:tasks.rados.rados.0.smithi001.stdout:387: done (1 left) 2024-09-15T03:47:58.938 INFO:tasks.rados.rados.0.smithi001.stdout:388: done (0 left) 2024-09-15T03:47:58.938 INFO:tasks.rados.rados.0.smithi001.stdout:389: delete oid 50 current snap is 49 2024-09-15T03:47:58.942 INFO:tasks.rados.rados.0.smithi001.stdout:389: done (0 left) 2024-09-15T03:47:58.942 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 41 current snap is 49 2024-09-15T03:47:58.942 INFO:tasks.rados.rados.0.smithi001.stdout:390: seq_num 146 ranges {2621440=49152} 2024-09-15T03:47:58.951 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-15T03:47:58.957 INFO:tasks.rados.rados.0.smithi001.stdout:390: writing smithi00132172-41 from 2621440 to 2670592 tid 1 2024-09-15T03:47:58.957 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 10 current snap is 49 2024-09-15T03:47:58.957 INFO:tasks.rados.rados.0.smithi001.stdout:391: seq_num 147 ranges {1720320=65536} 2024-09-15T03:47:58.967 INFO:tasks.rados.rados.0.smithi001.stdout:391: writing smithi00132172-10 from 1720320 to 1785856 tid 1 2024-09-15T03:47:58.967 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 2 current snap is 49 2024-09-15T03:47:58.967 INFO:tasks.rados.rados.0.smithi001.stdout:390: finishing write tid 1 to smithi00132172-41 2024-09-15T03:47:58.967 INFO:tasks.rados.rados.0.smithi001.stdout:392: seq_num 148 ranges {2195456=81920} 2024-09-15T03:47:58.968 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T03:47:58.980 INFO:tasks.rados.rados.0.smithi001.stdout:392: writing smithi00132172-2 from 2195456 to 2277376 tid 1 2024-09-15T03:47:58.980 INFO:tasks.rados.rados.0.smithi001.stdout:393: read oid 14 snap -1 2024-09-15T03:47:58.980 INFO:tasks.rados.rados.0.smithi001.stdout:393: expect (ObjNum 116 snap 33 seq_num 116) 2024-09-15T03:47:58.980 INFO:tasks.rados.rados.0.smithi001.stdout:394: snap_remove snap 23 2024-09-15T03:47:59.009 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:47:59.008+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:47:59.055 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:47:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-15T03:47:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-15T03:47:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-15T03:47:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.055 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:333 3.9s0 3:94e9e019:::smithi00132172-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.856848+0000", 2024-09-15T03:47:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 63.196133234999998, 2024-09-15T03:47:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012151039000000001, 2024-09-15T03:47:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 333 2024-09-15T03:47:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:59.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.057 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:59.057 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856848+0000", 2024-09-15T03:47:59.057 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.057 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.057 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:59.057 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856848+0000", 2024-09-15T03:47:59.057 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.057 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.057 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:59.057 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856849+0000", 2024-09-15T03:47:59.057 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5499999999999999e-07 2024-09-15T03:47:59.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.058 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:59.058 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:59.058 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.378e-06 2024-09-15T03:47:59.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.058 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:59.058 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856854+0000", 2024-09-15T03:47:59.058 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.653e-06 2024-09-15T03:47:59.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.058 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.059 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856856+0000", 2024-09-15T03:47:59.059 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4059999999999999e-06 2024-09-15T03:47:59.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.059 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.059 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.059 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.862404+0000", 2024-09-15T03:47:59.059 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0055481310000000004 2024-09-15T03:47:59.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.059 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.059 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:59.059 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.862519+0000", 2024-09-15T03:47:59.059 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011449400000000001 2024-09-15T03:47:59.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.060 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.060 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.060 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864592+0000", 2024-09-15T03:47:59.060 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020730200000000001 2024-09-15T03:47:59.060 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.060 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.060 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:59.060 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864687+0000", 2024-09-15T03:47:59.060 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5531000000000004e-05 2024-09-15T03:47:59.060 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.060 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.060 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:59.060 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.868999+0000", 2024-09-15T03:47:59.061 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0043121710000000001 2024-09-15T03:47:59.061 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.061 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:59.061 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.061 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.061 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.061 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:458 3.9s0 3:97177042:::smithi00132172-18:head [append 0~753664] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:59.061 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.841113+0000", 2024-09-15T03:47:59.061 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 48.21186883, 2024-09-15T03:47:59.061 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0334256719999999, 2024-09-15T03:47:59.061 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:59.061 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.061 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:59.061 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:59.062 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.062 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 458 2024-09-15T03:47:59.062 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.062 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:59.062 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.062 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:59.062 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841113+0000", 2024-09-15T03:47:59.062 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.062 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.062 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.062 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:59.062 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841113+0000", 2024-09-15T03:47:59.062 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.063 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.063 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.063 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:59.063 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841114+0000", 2024-09-15T03:47:59.063 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.491e-06 2024-09-15T03:47:59.063 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.063 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.063 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:59.063 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841577+0000", 2024-09-15T03:47:59.063 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000462571 2024-09-15T03:47:59.063 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.063 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.063 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:59.064 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841589+0000", 2024-09-15T03:47:59.064 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2119999999999999e-05 2024-09-15T03:47:59.064 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.064 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.064 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.064 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841596+0000", 2024-09-15T03:47:59.064 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0740000000000004e-06 2024-09-15T03:47:59.064 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.064 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.064 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.064 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841708+0000", 2024-09-15T03:47:59.064 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112126 2024-09-15T03:47:59.064 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.065 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.065 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:47:59.065 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841712+0000", 2024-09-15T03:47:59.065 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3639999999999999e-06 2024-09-15T03:47:59.065 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.065 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.065 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.065 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.856485+0000", 2024-09-15T03:47:59.065 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0147736949999999 2024-09-15T03:47:59.065 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.065 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.065 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:59.065 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.856699+0000", 2024-09-15T03:47:59.065 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000213372 2024-09-15T03:47:59.066 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.066 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.066 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.066 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.862261+0000", 2024-09-15T03:47:59.066 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0055625450000000003 2024-09-15T03:47:59.066 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.066 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.066 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.066 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.862781+0000", 2024-09-15T03:47:59.066 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00052013500000000002 2024-09-15T03:47:59.066 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.066 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.066 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.067 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.862885+0000", 2024-09-15T03:47:59.067 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103455 2024-09-15T03:47:59.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.067 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.067 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.067 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.863028+0000", 2024-09-15T03:47:59.067 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014313999999999999 2024-09-15T03:47:59.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.067 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.067 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.067 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.864079+0000", 2024-09-15T03:47:59.067 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010511229999999999 2024-09-15T03:47:59.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.067 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.068 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:59.068 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874458+0000", 2024-09-15T03:47:59.068 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010379019999999999 2024-09-15T03:47:59.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.068 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.068 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:59.068 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874538+0000", 2024-09-15T03:47:59.068 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0440999999999998e-05 2024-09-15T03:47:59.068 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.068 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:59.068 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.068 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.069 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:459 3.9s0 3:97177042:::smithi00132172-18:head [append 0~720896] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:59.069 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.846766+0000", 2024-09-15T03:47:59.069 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 48.206215207, 2024-09-15T03:47:59.069 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0281560110000001, 2024-09-15T03:47:59.069 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:59.069 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.069 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:59.069 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:59.069 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.069 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 459 2024-09-15T03:47:59.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.069 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:59.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.069 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:59.070 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.846766+0000", 2024-09-15T03:47:59.070 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.070 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:59.070 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.846766+0000", 2024-09-15T03:47:59.070 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.070 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:59.070 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.846769+0000", 2024-09-15T03:47:59.070 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.323e-06 2024-09-15T03:47:59.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.071 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:59.071 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847323+0000", 2024-09-15T03:47:59.071 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000554267 2024-09-15T03:47:59.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.071 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:59.071 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847338+0000", 2024-09-15T03:47:59.071 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.465e-05 2024-09-15T03:47:59.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.071 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.071 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847345+0000", 2024-09-15T03:47:59.071 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6029999999999999e-06 2024-09-15T03:47:59.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.072 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.072 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847494+0000", 2024-09-15T03:47:59.072 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000148541 2024-09-15T03:47:59.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.072 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:47:59.072 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847498+0000", 2024-09-15T03:47:59.072 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7629999999999998e-06 2024-09-15T03:47:59.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.072 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.073 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.864944+0000", 2024-09-15T03:47:59.073 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0174461050000001 2024-09-15T03:47:59.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.073 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:59.073 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.865122+0000", 2024-09-15T03:47:59.073 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000178123 2024-09-15T03:47:59.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.073 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.073 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867632+0000", 2024-09-15T03:47:59.073 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002510249 2024-09-15T03:47:59.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.074 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.074 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867775+0000", 2024-09-15T03:47:59.074 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014326000000000001 2024-09-15T03:47:59.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.074 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.074 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868663+0000", 2024-09-15T03:47:59.074 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00088783799999999995 2024-09-15T03:47:59.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.074 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.074 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868802+0000", 2024-09-15T03:47:59.074 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000139173 2024-09-15T03:47:59.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.075 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.075 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874167+0000", 2024-09-15T03:47:59.075 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.005365117 2024-09-15T03:47:59.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.075 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:59.075 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874842+0000", 2024-09-15T03:47:59.075 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00067414199999999997 2024-09-15T03:47:59.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.075 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:59.076 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874922+0000", 2024-09-15T03:47:59.076 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0857000000000002e-05 2024-09-15T03:47:59.076 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.076 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:59.076 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.076 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:460 3.9s0 3:97177042:::smithi00132172-18:head [append 0~49152] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:59.076 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.847359+0000", 2024-09-15T03:47:59.076 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 48.205622833, 2024-09-15T03:47:59.076 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0278644889999999, 2024-09-15T03:47:59.076 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:59.076 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.077 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:59.077 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:59.077 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.077 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 460 2024-09-15T03:47:59.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.077 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:59.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.077 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:59.077 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847359+0000", 2024-09-15T03:47:59.077 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.077 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:59.077 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847359+0000", 2024-09-15T03:47:59.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.078 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:59.078 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847360+0000", 2024-09-15T03:47:59.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.26e-06 2024-09-15T03:47:59.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.078 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:59.078 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847390+0000", 2024-09-15T03:47:59.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9533999999999999e-05 2024-09-15T03:47:59.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.079 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:59.079 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847406+0000", 2024-09-15T03:47:59.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6725999999999999e-05 2024-09-15T03:47:59.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.079 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.079 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847412+0000", 2024-09-15T03:47:59.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4149999999999998e-06 2024-09-15T03:47:59.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.079 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.079 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847522+0000", 2024-09-15T03:47:59.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011041500000000001 2024-09-15T03:47:59.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:47:59.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847523+0000", 2024-09-15T03:47:59.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.09e-07 2024-09-15T03:47:59.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868928+0000", 2024-09-15T03:47:59.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0214053089999999 2024-09-15T03:47:59.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:59.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869043+0000", 2024-09-15T03:47:59.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000115021 2024-09-15T03:47:59.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869635+0000", 2024-09-15T03:47:59.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00059164199999999997 2024-09-15T03:47:59.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869755+0000", 2024-09-15T03:47:59.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0001205 2024-09-15T03:47:59.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870274+0000", 2024-09-15T03:47:59.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00051842900000000005 2024-09-15T03:47:59.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870322+0000", 2024-09-15T03:47:59.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8164000000000002e-05 2024-09-15T03:47:59.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874941+0000", 2024-09-15T03:47:59.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0046186140000000001 2024-09-15T03:47:59.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.083 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:59.083 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875140+0000", 2024-09-15T03:47:59.083 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019890700000000001 2024-09-15T03:47:59.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.083 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:59.083 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875223+0000", 2024-09-15T03:47:59.083 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3644000000000001e-05 2024-09-15T03:47:59.083 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.083 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:59.083 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.084 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:461 3.9s0 3:97177042:::smithi00132172-18:head [setxattr _header (57) in=7b] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:59.084 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.847417+0000", 2024-09-15T03:47:59.084 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 48.205564729000002, 2024-09-15T03:47:59.084 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0332682499999999, 2024-09-15T03:47:59.084 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:59.084 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.084 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:59.084 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:59.084 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.084 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 461 2024-09-15T03:47:59.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.084 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:59.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:59.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847417+0000", 2024-09-15T03:47:59.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:59.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847417+0000", 2024-09-15T03:47:59.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:59.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847418+0000", 2024-09-15T03:47:59.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8999999999999995e-07 2024-09-15T03:47:59.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.086 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:59.086 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847421+0000", 2024-09-15T03:47:59.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1769999999999998e-06 2024-09-15T03:47:59.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.086 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:59.086 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847428+0000", 2024-09-15T03:47:59.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6969999999999996e-06 2024-09-15T03:47:59.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.086 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.087 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847432+0000", 2024-09-15T03:47:59.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3209999999999998e-06 2024-09-15T03:47:59.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.087 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847542+0000", 2024-09-15T03:47:59.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011035999999999999 2024-09-15T03:47:59.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:47:59.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847543+0000", 2024-09-15T03:47:59.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1200000000000002e-07 2024-09-15T03:47:59.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875612+0000", 2024-09-15T03:47:59.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0280692330000001 2024-09-15T03:47:59.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:59.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875735+0000", 2024-09-15T03:47:59.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012276600000000001 2024-09-15T03:47:59.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876307+0000", 2024-09-15T03:47:59.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00057198600000000002 2024-09-15T03:47:59.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876408+0000", 2024-09-15T03:47:59.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010148400000000001 2024-09-15T03:47:59.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.091 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.091 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876508+0000", 2024-09-15T03:47:59.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9328999999999997e-05 2024-09-15T03:47:59.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.091 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.091 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876546+0000", 2024-09-15T03:47:59.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8361000000000001e-05 2024-09-15T03:47:59.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.091 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.092 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876602+0000", 2024-09-15T03:47:59.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6412999999999997e-05 2024-09-15T03:47:59.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.092 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:59.092 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.880589+0000", 2024-09-15T03:47:59.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0039866190000000003 2024-09-15T03:47:59.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.092 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:59.092 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.880685+0000", 2024-09-15T03:47:59.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6001999999999993e-05 2024-09-15T03:47:59.092 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.093 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:59.093 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.093 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.093 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:462 3.9s0 3:97177042:::smithi00132172-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:47:59.093 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.847436+0000", 2024-09-15T03:47:59.093 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 48.205545770999997, 2024-09-15T03:47:59.093 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.03734576, 2024-09-15T03:47:59.093 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:59.093 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:59.093 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:59.093 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:59.093 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.094 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 462 2024-09-15T03:47:59.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.094 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:59.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:59.094 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847436+0000", 2024-09-15T03:47:59.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:59.094 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847436+0000", 2024-09-15T03:47:59.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.095 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:59.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847436+0000", 2024-09-15T03:47:59.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3099999999999998e-07 2024-09-15T03:47:59.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.095 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:59.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847438+0000", 2024-09-15T03:47:59.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9570000000000001e-06 2024-09-15T03:47:59.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.095 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:59.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847445+0000", 2024-09-15T03:47:59.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.238e-06 2024-09-15T03:47:59.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847454+0000", 2024-09-15T03:47:59.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5930000000000006e-06 2024-09-15T03:47:59.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847562+0000", 2024-09-15T03:47:59.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108448 2024-09-15T03:47:59.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:47:59.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847563+0000", 2024-09-15T03:47:59.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0200000000000002e-07 2024-09-15T03:47:59.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876482+0000", 2024-09-15T03:47:59.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.028918658 2024-09-15T03:47:59.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:59.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876593+0000", 2024-09-15T03:47:59.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011145300000000001 2024-09-15T03:47:59.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.098 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.098 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.881189+0000", 2024-09-15T03:47:59.098 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0045956139999999996 2024-09-15T03:47:59.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.098 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:59.098 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.881293+0000", 2024-09-15T03:47:59.098 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103875 2024-09-15T03:47:59.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.098 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:59.098 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.884782+0000", 2024-09-15T03:47:59.098 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003488891 2024-09-15T03:47:59.099 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.099 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:59.099 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.099 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.099 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:561 3.9s0 3:9cd57211:::smithi00132172-16:head [append 0~557056] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:59.099 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:20.935324+0000", 2024-09-15T03:47:59.099 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.117658069999997, 2024-09-15T03:47:59.099 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012459909999999999, 2024-09-15T03:47:59.099 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:59.099 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.099 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:59.099 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:59.099 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.100 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 561 2024-09-15T03:47:59.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.100 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:59.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.100 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:59.100 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935324+0000", 2024-09-15T03:47:59.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.100 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:59.100 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935324+0000", 2024-09-15T03:47:59.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.101 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:59.101 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935326+0000", 2024-09-15T03:47:59.101 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8909999999999999e-06 2024-09-15T03:47:59.101 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.101 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:59.101 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935587+0000", 2024-09-15T03:47:59.101 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026019699999999999 2024-09-15T03:47:59.101 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.101 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:59.101 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935610+0000", 2024-09-15T03:47:59.102 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2949e-05 2024-09-15T03:47:59.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.102 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935618+0000", 2024-09-15T03:47:59.102 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6289999999999999e-06 2024-09-15T03:47:59.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.102 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935688+0000", 2024-09-15T03:47:59.102 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9801999999999993e-05 2024-09-15T03:47:59.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-15T03:47:59.103 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935698+0000", 2024-09-15T03:47:59.103 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0103e-05 2024-09-15T03:47:59.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.103 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.103 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.103 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939626+0000", 2024-09-15T03:47:59.103 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0039276759999999997 2024-09-15T03:47:59.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.103 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.103 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:59.103 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939844+0000", 2024-09-15T03:47:59.103 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000217741 2024-09-15T03:47:59.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.104 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.104 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.104 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941127+0000", 2024-09-15T03:47:59.104 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001283719 2024-09-15T03:47:59.104 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.104 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.104 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.104 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941395+0000", 2024-09-15T03:47:59.104 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026823800000000002 2024-09-15T03:47:59.104 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.104 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.104 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.104 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941448+0000", 2024-09-15T03:47:59.105 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2751999999999999e-05 2024-09-15T03:47:59.105 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.105 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.109 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.109 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941646+0000", 2024-09-15T03:47:59.109 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019804100000000001 2024-09-15T03:47:59.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.109 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.109 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943664+0000", 2024-09-15T03:47:59.109 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020181090000000001 2024-09-15T03:47:59.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.110 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:59.110 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.947709+0000", 2024-09-15T03:47:59.110 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0040445059999999998 2024-09-15T03:47:59.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.110 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:59.110 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.947783+0000", 2024-09-15T03:47:59.110 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4556999999999998e-05 2024-09-15T03:47:59.110 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.110 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:59.110 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.111 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:562 3.9s0 3:9cd57211:::smithi00132172-16:head [append 0~638976] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:59.111 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:20.938590+0000", 2024-09-15T03:47:59.111 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.114391286, 2024-09-15T03:47:59.111 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011368382, 2024-09-15T03:47:59.111 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:59.111 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.111 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:59.111 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:59.111 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.111 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 562 2024-09-15T03:47:59.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.112 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:59.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.112 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:59.112 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.938590+0000", 2024-09-15T03:47:59.112 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.112 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:59.112 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.938590+0000", 2024-09-15T03:47:59.112 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.113 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:59.113 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.938592+0000", 2024-09-15T03:47:59.113 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.623e-06 2024-09-15T03:47:59.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.113 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.113 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:59.113 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939253+0000", 2024-09-15T03:47:59.113 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00066056899999999996 2024-09-15T03:47:59.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.113 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.113 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:59.113 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939271+0000", 2024-09-15T03:47:59.113 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8482e-05 2024-09-15T03:47:59.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.114 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.114 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939280+0000", 2024-09-15T03:47:59.114 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5569999999999996e-06 2024-09-15T03:47:59.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.114 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.114 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941736+0000", 2024-09-15T03:47:59.115 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024565640000000001 2024-09-15T03:47:59.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.115 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:59.115 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941861+0000", 2024-09-15T03:47:59.115 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012484 2024-09-15T03:47:59.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.116 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.116 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943223+0000", 2024-09-15T03:47:59.116 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0013619260000000001 2024-09-15T03:47:59.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.116 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.116 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943354+0000", 2024-09-15T03:47:59.116 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013100399999999999 2024-09-15T03:47:59.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.116 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.117 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943376+0000", 2024-09-15T03:47:59.117 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1829999999999999e-05 2024-09-15T03:47:59.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.117 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.117 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.117 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943579+0000", 2024-09-15T03:47:59.117 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020374400000000001 2024-09-15T03:47:59.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.117 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.117 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.117 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943904+0000", 2024-09-15T03:47:59.117 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000324846 2024-09-15T03:47:59.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.118 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:59.118 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949897+0000", 2024-09-15T03:47:59.118 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0059927879999999998 2024-09-15T03:47:59.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.118 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:59.118 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949959+0000", 2024-09-15T03:47:59.118 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1608999999999999e-05 2024-09-15T03:47:59.118 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.118 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:59.119 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.119 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.119 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:565 3.9s0 3:9cd57211:::smithi00132172-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:59.119 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:20.943038+0000", 2024-09-15T03:47:59.119 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.109943942000001, 2024-09-15T03:47:59.119 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011190904, 2024-09-15T03:47:59.119 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:59.119 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:59.119 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:59.119 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:59.120 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.120 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 565 2024-09-15T03:47:59.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.120 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:59.120 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.120 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:59.120 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943038+0000", 2024-09-15T03:47:59.120 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.120 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.120 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:59.120 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943038+0000", 2024-09-15T03:47:59.121 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.121 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.121 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.121 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:59.121 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943038+0000", 2024-09-15T03:47:59.121 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6800000000000001e-07 2024-09-15T03:47:59.121 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.121 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.121 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:59.121 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943040+0000", 2024-09-15T03:47:59.121 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3700000000000002e-06 2024-09-15T03:47:59.121 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.122 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:59.122 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943047+0000", 2024-09-15T03:47:59.122 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1299999999999998e-06 2024-09-15T03:47:59.122 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.122 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.122 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943050+0000", 2024-09-15T03:47:59.122 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6229999999999999e-06 2024-09-15T03:47:59.122 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.123 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.123 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.947990+0000", 2024-09-15T03:47:59.123 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0049398690000000004 2024-09-15T03:47:59.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.123 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.123 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:59.123 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.948080+0000", 2024-09-15T03:47:59.123 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9721000000000004e-05 2024-09-15T03:47:59.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.123 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.123 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.123 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.952263+0000", 2024-09-15T03:47:59.123 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.004183417 2024-09-15T03:47:59.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.124 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.124 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:59.124 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.952374+0000", 2024-09-15T03:47:59.124 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110435 2024-09-15T03:47:59.124 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.124 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.124 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:59.124 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.954229+0000", 2024-09-15T03:47:59.124 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018548709999999999 2024-09-15T03:47:59.124 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.124 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:59.125 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.125 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.125 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.125 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:569 3.9s0 3:97177042:::smithi00132172-18:head [rmxattr Z in=1b,rmxattr ZVpala^a[hcaro^RXmp_SYlliaqih_SpraepZfQX in=40b,rmxattr aNhZ\\ePhXcfSNf\\hQ^orWZst^jcpVp[skbdkms]j in=40b,rmxattr ehO_catUjgbjTPQOspNgZXioh\\UidVPlqcarXe]r in=40b,rmxattr hTUN^iUTTOa\\lp\\napoiZNkQnN_jfWcgjVeQRahm in=40b] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:47:59.125 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:20.943054+0000", 2024-09-15T03:47:59.125 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.109927755999998, 2024-09-15T03:47:59.125 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010701584, 2024-09-15T03:47:59.125 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:59.125 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.125 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:59.125 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:59.125 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.125 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 569 2024-09-15T03:47:59.125 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.126 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:59.126 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.126 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:59.126 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943054+0000", 2024-09-15T03:47:59.126 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.126 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.126 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:59.126 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943054+0000", 2024-09-15T03:47:59.126 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.126 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.126 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:59.126 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943054+0000", 2024-09-15T03:47:59.127 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:47:59.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.127 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:59.127 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943058+0000", 2024-09-15T03:47:59.127 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.631e-06 2024-09-15T03:47:59.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.127 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:59.127 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943063+0000", 2024-09-15T03:47:59.127 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3800000000000002e-06 2024-09-15T03:47:59.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.127 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.127 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943067+0000", 2024-09-15T03:47:59.127 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4829999999999999e-06 2024-09-15T03:47:59.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.128 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.128 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.948103+0000", 2024-09-15T03:47:59.128 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0050365490000000004 2024-09-15T03:47:59.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.128 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:59.128 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.948602+0000", 2024-09-15T03:47:59.128 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00049903600000000001 2024-09-15T03:47:59.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.128 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.128 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949373+0000", 2024-09-15T03:47:59.129 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00077054200000000004 2024-09-15T03:47:59.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.129 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.129 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949630+0000", 2024-09-15T03:47:59.129 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00025681499999999997 2024-09-15T03:47:59.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.129 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.129 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949645+0000", 2024-09-15T03:47:59.129 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5704000000000001e-05 2024-09-15T03:47:59.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.129 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.129 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949690+0000", 2024-09-15T03:47:59.130 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4413999999999999e-05 2024-09-15T03:47:59.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.130 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.130 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949979+0000", 2024-09-15T03:47:59.130 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00028949100000000001 2024-09-15T03:47:59.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.130 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:59.130 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.951859+0000", 2024-09-15T03:47:59.130 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018798179999999999 2024-09-15T03:47:59.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.131 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:59.131 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.953755+0000", 2024-09-15T03:47:59.131 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018963109999999999 2024-09-15T03:47:59.131 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.131 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:59.131 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.131 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:667 3.4s0 3:2d1c787c:::smithi00132172-45:head [read 0~1703936,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:59.131 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:29.978831+0000", 2024-09-15T03:47:59.131 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 29.074150340999999, 2024-09-15T03:47:59.131 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0098251169999999995, 2024-09-15T03:47:59.131 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:59.132 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:59.132 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:59.132 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:59.132 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.133 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 667 2024-09-15T03:47:59.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.133 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:59.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.133 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:59.133 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978831+0000", 2024-09-15T03:47:59.133 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.133 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:59.133 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978831+0000", 2024-09-15T03:47:59.133 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.134 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:59.134 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978832+0000", 2024-09-15T03:47:59.134 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9299999999999999e-07 2024-09-15T03:47:59.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.134 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:59.134 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978834+0000", 2024-09-15T03:47:59.134 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3429999999999998e-06 2024-09-15T03:47:59.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.134 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:59.134 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978838+0000", 2024-09-15T03:47:59.134 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8809999999999998e-06 2024-09-15T03:47:59.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.135 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.135 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978840+0000", 2024-09-15T03:47:59.135 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.216e-06 2024-09-15T03:47:59.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.135 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.135 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978980+0000", 2024-09-15T03:47:59.135 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000140214 2024-09-15T03:47:59.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.135 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:59.135 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978989+0000", 2024-09-15T03:47:59.136 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4079999999999998e-06 2024-09-15T03:47:59.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.136 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.136 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.981199+0000", 2024-09-15T03:47:59.136 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022103539999999999 2024-09-15T03:47:59.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.136 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:59.136 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.981288+0000", 2024-09-15T03:47:59.136 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8819000000000005e-05 2024-09-15T03:47:59.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.137 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:59.137 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.988656+0000", 2024-09-15T03:47:59.137 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0073684889999999998 2024-09-15T03:47:59.137 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.137 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:59.137 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.137 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:668 3.4s0 3:2d1c787c:::smithi00132172-45:head [read 0~1703936,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:59.137 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:29.978875+0000", 2024-09-15T03:47:59.137 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 29.074106225000001, 2024-09-15T03:47:59.137 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011314753, 2024-09-15T03:47:59.137 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:59.137 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:59.138 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:59.138 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:59.138 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.138 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 668 2024-09-15T03:47:59.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.138 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:59.138 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.138 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:59.138 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978875+0000", 2024-09-15T03:47:59.138 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.138 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.138 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:59.138 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978875+0000", 2024-09-15T03:47:59.139 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.139 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:59.139 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978876+0000", 2024-09-15T03:47:59.139 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.46e-07 2024-09-15T03:47:59.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.139 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:59.139 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978878+0000", 2024-09-15T03:47:59.139 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4399999999999999e-06 2024-09-15T03:47:59.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.140 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:59.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978884+0000", 2024-09-15T03:47:59.140 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8930000000000002e-06 2024-09-15T03:47:59.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.140 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978887+0000", 2024-09-15T03:47:59.140 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8040000000000002e-06 2024-09-15T03:47:59.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.140 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.979013+0000", 2024-09-15T03:47:59.140 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012650899999999999 2024-09-15T03:47:59.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.141 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:59.141 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.979020+0000", 2024-09-15T03:47:59.141 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8159999999999996e-06 2024-09-15T03:47:59.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.141 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.141 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.981562+0000", 2024-09-15T03:47:59.141 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002542267 2024-09-15T03:47:59.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.141 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:59.141 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.981653+0000", 2024-09-15T03:47:59.141 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0509999999999999e-05 2024-09-15T03:47:59.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.142 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:59.142 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.990190+0000", 2024-09-15T03:47:59.142 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0085371679999999995 2024-09-15T03:47:59.142 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.142 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:59.142 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.142 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:692 3.9s0 3:97177042:::smithi00132172-18:head [copy-from ver 4] snapc 2f=[2d,2b,29,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:59.142 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.088835+0000", 2024-09-15T03:47:59.142 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 28.964146919000001, 2024-09-15T03:47:59.143 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.016647229999999999, 2024-09-15T03:47:59.143 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:59.143 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.143 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:59.143 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:59.143 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.143 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 692 2024-09-15T03:47:59.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.143 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:59.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.143 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:59.143 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088835+0000", 2024-09-15T03:47:59.143 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.144 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:59.144 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088835+0000", 2024-09-15T03:47:59.144 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.144 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:59.144 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088837+0000", 2024-09-15T03:47:59.144 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8670000000000001e-06 2024-09-15T03:47:59.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.144 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:59.144 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088843+0000", 2024-09-15T03:47:59.144 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8179999999999998e-06 2024-09-15T03:47:59.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:59.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088860+0000", 2024-09-15T03:47:59.145 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6566999999999999e-05 2024-09-15T03:47:59.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088881+0000", 2024-09-15T03:47:59.145 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0619999999999999e-05 2024-09-15T03:47:59.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088963+0000", 2024-09-15T03:47:59.146 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2650000000000003e-05 2024-09-15T03:47:59.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.146 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:59.146 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089325+0000", 2024-09-15T03:47:59.146 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00036168700000000002 2024-09-15T03:47:59.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.146 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.146 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102185+0000", 2024-09-15T03:47:59.146 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012859786 2024-09-15T03:47:59.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.146 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.147 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102633+0000", 2024-09-15T03:47:59.147 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00044844699999999998 2024-09-15T03:47:59.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.147 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.147 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102713+0000", 2024-09-15T03:47:59.147 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9616999999999993e-05 2024-09-15T03:47:59.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.147 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.147 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102771+0000", 2024-09-15T03:47:59.147 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7797999999999998e-05 2024-09-15T03:47:59.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.148 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.148 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102793+0000", 2024-09-15T03:47:59.148 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2827999999999999e-05 2024-09-15T03:47:59.148 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.148 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.148 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:59.148 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.104072+0000", 2024-09-15T03:47:59.148 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012787759999999999 2024-09-15T03:47:59.148 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.148 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.148 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:59.148 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.105482+0000", 2024-09-15T03:47:59.148 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001409769 2024-09-15T03:47:59.149 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.149 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:59.149 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.149 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:693 3.9s0 3:97177042:::smithi00132172-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:47:59.149 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.089498+0000", 2024-09-15T03:47:59.149 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 28.963483984, 2024-09-15T03:47:59.149 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013753056, 2024-09-15T03:47:59.149 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:59.149 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:59.149 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:59.149 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:59.149 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.150 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 693 2024-09-15T03:47:59.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.150 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:59.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.150 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:59.150 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089498+0000", 2024-09-15T03:47:59.150 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.150 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:59.150 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089498+0000", 2024-09-15T03:47:59.150 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.151 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:59.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089499+0000", 2024-09-15T03:47:59.151 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.074e-06 2024-09-15T03:47:59.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.151 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:59.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089503+0000", 2024-09-15T03:47:59.151 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1470000000000003e-06 2024-09-15T03:47:59.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.151 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:59.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089513+0000", 2024-09-15T03:47:59.151 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0248000000000001e-05 2024-09-15T03:47:59.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.152 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.152 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089519+0000", 2024-09-15T03:47:59.152 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2149999999999998e-06 2024-09-15T03:47:59.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.152 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.152 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089719+0000", 2024-09-15T03:47:59.152 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019960400000000001 2024-09-15T03:47:59.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.152 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:59.152 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089776+0000", 2024-09-15T03:47:59.153 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7123000000000002e-05 2024-09-15T03:47:59.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.153 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.153 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102827+0000", 2024-09-15T03:47:59.153 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013050858 2024-09-15T03:47:59.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.153 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:59.153 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102976+0000", 2024-09-15T03:47:59.153 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014900800000000001 2024-09-15T03:47:59.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.153 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:59.154 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.103251+0000", 2024-09-15T03:47:59.154 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00027477900000000001 2024-09-15T03:47:59.154 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.154 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:59.154 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.154 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.154 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:704 3.8s0 3:1af4eaf3:::smithi00132172-20:head [rollback 2d] snapc 2f=[2d,2b,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-15T03:47:59.154 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.982834+0000", 2024-09-15T03:47:59.154 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 28.070147878, 2024-09-15T03:47:59.154 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011049209000000001, 2024-09-15T03:47:59.154 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:59.154 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.155 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:59.155 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:59.155 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.155 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 704 2024-09-15T03:47:59.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.155 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:59.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.155 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:59.155 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982834+0000", 2024-09-15T03:47:59.155 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.155 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:59.155 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982834+0000", 2024-09-15T03:47:59.156 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.156 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:59.156 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982834+0000", 2024-09-15T03:47:59.156 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3000000000000001e-07 2024-09-15T03:47:59.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.156 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:59.156 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982836+0000", 2024-09-15T03:47:59.156 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0870000000000001e-06 2024-09-15T03:47:59.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.156 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:59.157 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982842+0000", 2024-09-15T03:47:59.157 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.879e-06 2024-09-15T03:47:59.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.157 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.157 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982846+0000", 2024-09-15T03:47:59.157 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8500000000000004e-06 2024-09-15T03:47:59.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.157 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.157 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982967+0000", 2024-09-15T03:47:59.157 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012074999999999999 2024-09-15T03:47:59.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.158 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:59.158 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982977+0000", 2024-09-15T03:47:59.158 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0397999999999999e-05 2024-09-15T03:47:59.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.158 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.158 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.985406+0000", 2024-09-15T03:47:59.158 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024286619999999998 2024-09-15T03:47:59.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.158 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:59.158 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.985533+0000", 2024-09-15T03:47:59.158 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012677399999999999 2024-09-15T03:47:59.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.159 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.159 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986275+0000", 2024-09-15T03:47:59.159 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0007427 2024-09-15T03:47:59.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.159 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.159 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986403+0000", 2024-09-15T03:47:59.159 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012725799999999999 2024-09-15T03:47:59.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.159 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.159 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986644+0000", 2024-09-15T03:47:59.159 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000241876 2024-09-15T03:47:59.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.160 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.160 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986705+0000", 2024-09-15T03:47:59.160 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0350999999999999e-05 2024-09-15T03:47:59.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.160 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.160 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987544+0000", 2024-09-15T03:47:59.160 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00083908500000000005 2024-09-15T03:47:59.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.160 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:59.160 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.990656+0000", 2024-09-15T03:47:59.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0031123750000000001 2024-09-15T03:47:59.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:59.161 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.993883+0000", 2024-09-15T03:47:59.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032266339999999999 2024-09-15T03:47:59.161 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.161 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:59.161 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.161 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:705 3.8s0 3:1af4eaf3:::smithi00132172-20:head [append 0~0] snapc 2f=[2d,2b,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-15T03:47:59.161 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.982870+0000", 2024-09-15T03:47:59.161 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 28.070111572999998, 2024-09-15T03:47:59.162 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.01140421, 2024-09-15T03:47:59.162 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:59.162 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.162 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:59.162 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:59.162 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.162 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 705 2024-09-15T03:47:59.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.162 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:59.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.162 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:59.162 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982870+0000", 2024-09-15T03:47:59.162 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.163 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:59.163 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982870+0000", 2024-09-15T03:47:59.163 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.163 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:59.163 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982871+0000", 2024-09-15T03:47:59.163 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2900000000000004e-07 2024-09-15T03:47:59.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.163 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:59.163 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982874+0000", 2024-09-15T03:47:59.164 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9610000000000001e-06 2024-09-15T03:47:59.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.164 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:59.164 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982878+0000", 2024-09-15T03:47:59.164 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7620000000000003e-06 2024-09-15T03:47:59.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.164 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.164 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982881+0000", 2024-09-15T03:47:59.164 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.112e-06 2024-09-15T03:47:59.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.164 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.165 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982995+0000", 2024-09-15T03:47:59.165 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000113544 2024-09-15T03:47:59.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.165 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:59.165 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982999+0000", 2024-09-15T03:47:59.165 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9339999999999999e-06 2024-09-15T03:47:59.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.165 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.165 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986479+0000", 2024-09-15T03:47:59.165 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0034799399999999999 2024-09-15T03:47:59.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.166 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:59.166 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986599+0000", 2024-09-15T03:47:59.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000120364 2024-09-15T03:47:59.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.166 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.166 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987178+0000", 2024-09-15T03:47:59.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00057913799999999998 2024-09-15T03:47:59.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.166 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.166 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987321+0000", 2024-09-15T03:47:59.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000142208 2024-09-15T03:47:59.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.167 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.167 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987335+0000", 2024-09-15T03:47:59.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4578e-05 2024-09-15T03:47:59.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.167 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.167 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987690+0000", 2024-09-15T03:47:59.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00035494600000000003 2024-09-15T03:47:59.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.167 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.167 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987799+0000", 2024-09-15T03:47:59.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000109379 2024-09-15T03:47:59.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.168 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:59.168 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.991063+0000", 2024-09-15T03:47:59.168 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032640249999999998 2024-09-15T03:47:59.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.168 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:59.168 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.994274+0000", 2024-09-15T03:47:59.168 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032107899999999998 2024-09-15T03:47:59.168 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.168 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:59.168 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.169 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:708 3.4s0 3:2d1c787c:::smithi00132172-45:head [copy-from ver 28] snapc 32=[32,2d,2b,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-15T03:47:59.169 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:31.983317+0000", 2024-09-15T03:47:59.169 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.06966504, 2024-09-15T03:47:59.169 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.021360130000000001, 2024-09-15T03:47:59.169 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:59.169 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.169 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:59.169 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:59.169 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.169 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 708 2024-09-15T03:47:59.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.169 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:59.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.170 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:59.170 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983317+0000", 2024-09-15T03:47:59.170 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.170 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:59.170 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983317+0000", 2024-09-15T03:47:59.170 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.170 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:59.170 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983318+0000", 2024-09-15T03:47:59.170 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2780000000000001e-06 2024-09-15T03:47:59.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.171 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:59.171 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983331+0000", 2024-09-15T03:47:59.171 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2821e-05 2024-09-15T03:47:59.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.171 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:59.171 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983349+0000", 2024-09-15T03:47:59.171 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8544999999999999e-05 2024-09-15T03:47:59.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.171 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.171 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983359+0000", 2024-09-15T03:47:59.171 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8570000000000007e-06 2024-09-15T03:47:59.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.172 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.172 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983434+0000", 2024-09-15T03:47:59.172 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4556999999999998e-05 2024-09-15T03:47:59.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.172 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-15T03:47:59.172 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983452+0000", 2024-09-15T03:47:59.172 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8439000000000001e-05 2024-09-15T03:47:59.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.172 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.172 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.985636+0000", 2024-09-15T03:47:59.173 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021841320000000001 2024-09-15T03:47:59.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.173 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:59.173 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.985755+0000", 2024-09-15T03:47:59.173 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118363 2024-09-15T03:47:59.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.173 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.173 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.998959+0000", 2024-09-15T03:47:59.173 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013204053 2024-09-15T03:47:59.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.173 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.174 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.999917+0000", 2024-09-15T03:47:59.174 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095851300000000003 2024-09-15T03:47:59.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.174 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.174 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.999931+0000", 2024-09-15T03:47:59.174 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3631999999999999e-05 2024-09-15T03:47:59.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.174 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.174 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000553+0000", 2024-09-15T03:47:59.174 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00062246600000000001 2024-09-15T03:47:59.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.175 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.175 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000695+0000", 2024-09-15T03:47:59.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014162499999999999 2024-09-15T03:47:59.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.175 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:59.175 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.004291+0000", 2024-09-15T03:47:59.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0035959289999999999 2024-09-15T03:47:59.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.175 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:59.175 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.004677+0000", 2024-09-15T03:47:59.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00038591999999999999 2024-09-15T03:47:59.176 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.176 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:59.176 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.176 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:709 3.4s0 3:2d1c787c:::smithi00132172-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-15T03:47:59.176 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:31.983373+0000", 2024-09-15T03:47:59.176 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 27.069608199000001, 2024-09-15T03:47:59.176 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.017317583000000001, 2024-09-15T03:47:59.176 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:59.176 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:59.176 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:59.176 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:59.177 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.177 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 709 2024-09-15T03:47:59.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.177 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:59.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.177 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:59.177 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983373+0000", 2024-09-15T03:47:59.177 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.177 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:59.177 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983373+0000", 2024-09-15T03:47:59.177 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.178 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:59.178 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983374+0000", 2024-09-15T03:47:59.178 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8599999999999998e-07 2024-09-15T03:47:59.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.178 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:59.178 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983377+0000", 2024-09-15T03:47:59.178 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0139999999999999e-06 2024-09-15T03:47:59.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.178 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:59.178 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983388+0000", 2024-09-15T03:47:59.179 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0988999999999999e-05 2024-09-15T03:47:59.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.179 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.179 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983392+0000", 2024-09-15T03:47:59.179 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5059999999999997e-06 2024-09-15T03:47:59.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.179 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.179 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983542+0000", 2024-09-15T03:47:59.179 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000149887 2024-09-15T03:47:59.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.179 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:59.180 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983550+0000", 2024-09-15T03:47:59.180 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9589999999999995e-06 2024-09-15T03:47:59.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.180 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.180 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.985938+0000", 2024-09-15T03:47:59.180 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023879930000000001 2024-09-15T03:47:59.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.180 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:59.180 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.986005+0000", 2024-09-15T03:47:59.180 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6926000000000003e-05 2024-09-15T03:47:59.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.181 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000370+0000", 2024-09-15T03:47:59.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.014364656999999999 2024-09-15T03:47:59.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:59.181 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000485+0000", 2024-09-15T03:47:59.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011524899999999999 2024-09-15T03:47:59.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:59.181 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000691+0000", 2024-09-15T03:47:59.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020581700000000001 2024-09-15T03:47:59.181 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.182 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:59.182 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.182 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:761 3.9s0 3:9cd57211:::smithi00132172-16:head [append 0~507904] snapc 32=[32,2d,2b,26,25,24,23,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T03:47:59.182 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:34.955885+0000", 2024-09-15T03:47:59.182 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 24.097096256, 2024-09-15T03:47:59.182 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011160715, 2024-09-15T03:47:59.182 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:59.182 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.182 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:47:59.182 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:47:59.182 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.183 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 761 2024-09-15T03:47:59.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.183 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:59.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.183 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:59.183 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.955885+0000", 2024-09-15T03:47:59.183 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.183 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:59.183 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.955885+0000", 2024-09-15T03:47:59.183 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.183 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:59.184 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.955887+0000", 2024-09-15T03:47:59.184 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8169999999999999e-06 2024-09-15T03:47:59.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.184 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:59.184 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956332+0000", 2024-09-15T03:47:59.184 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00044459600000000001 2024-09-15T03:47:59.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.184 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:59.184 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956347+0000", 2024-09-15T03:47:59.184 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5685999999999999e-05 2024-09-15T03:47:59.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956354+0000", 2024-09-15T03:47:59.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6810000000000001e-06 2024-09-15T03:47:59.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956805+0000", 2024-09-15T03:47:59.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00045055300000000002 2024-09-15T03:47:59.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:59.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956981+0000", 2024-09-15T03:47:59.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017590100000000001 2024-09-15T03:47:59.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.958362+0000", 2024-09-15T03:47:59.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001381415 2024-09-15T03:47:59.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.958524+0000", 2024-09-15T03:47:59.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016167800000000001 2024-09-15T03:47:59.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.958797+0000", 2024-09-15T03:47:59.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000273274 2024-09-15T03:47:59.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.959021+0000", 2024-09-15T03:47:59.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022373099999999999 2024-09-15T03:47:59.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.961002+0000", 2024-09-15T03:47:59.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019813069999999999 2024-09-15T03:47:59.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:47:59.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.966647+0000", 2024-09-15T03:47:59.188 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00564541 2024-09-15T03:47:59.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.188 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:59.188 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.967046+0000", 2024-09-15T03:47:59.188 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00039866599999999999 2024-09-15T03:47:59.188 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.188 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:59.188 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.189 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(osd.9.17:1 3.9s0 3:94e9e019:::smithi00132172-30:head [assert-version v16,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e107)", 2024-09-15T03:47:59.189 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:53.919800+0000", 2024-09-15T03:47:59.189 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 5.133181532, 2024-09-15T03:47:59.189 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27396664300000001, 2024-09-15T03:47:59.189 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:47:59.189 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:47:59.189 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:47:59.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.189 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:47:59.189 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.919800+0000", 2024-09-15T03:47:59.189 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.190 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:47:59.190 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.919800+0000", 2024-09-15T03:47:59.190 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:47:59.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.190 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:47:59.190 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.919802+0000", 2024-09-15T03:47:59.190 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.787e-06 2024-09-15T03:47:59.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.190 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:47:59.190 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.919807+0000", 2024-09-15T03:47:59.190 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4519999999999996e-06 2024-09-15T03:47:59.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.191 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:47:59.191 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.919823+0000", 2024-09-15T03:47:59.191 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5349000000000002e-05 2024-09-15T03:47:59.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.191 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.191 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.919829+0000", 2024-09-15T03:47:59.191 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.562e-06 2024-09-15T03:47:59.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.192 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.192 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.920022+0000", 2024-09-15T03:47:59.192 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000192504 2024-09-15T03:47:59.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.192 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:47:59.192 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.920025+0000", 2024-09-15T03:47:59.192 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5609999999999999e-06 2024-09-15T03:47:59.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.192 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:47:59.192 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:54.183165+0000", 2024-09-15T03:47:59.193 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26313930800000002 2024-09-15T03:47:59.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.193 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:47:59.193 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:54.183525+0000", 2024-09-15T03:47:59.193 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00036087700000000001 2024-09-15T03:47:59.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:47:59.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:47:59.193 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:47:59.193 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:54.193767+0000", 2024-09-15T03:47:59.194 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010241243000000001 2024-09-15T03:47:59.194 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.194 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:59.194 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.194 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:47:59.194 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:47:59.194 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:47:59.195 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:47:59.196 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:59.195+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:59.296 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:47:59.296+0000 7fd62abf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33736) UID: 0 2024-09-15T03:47:59.313 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:59.314 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:59.314 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:59.314 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:59.319 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:47:59.398 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:47:59.397+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:47:59.422 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:59.422 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:59.422 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:47:59.422 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:47:59.422 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:59.430 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:47:59.499 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:47:59.498+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:47:59.534 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:59.534 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:47:59.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:47:59.534 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:47:59.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.534 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:324 3.7s0 3:e40d1694:::smithi00132172-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:59.535 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:59.535 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.687061821999997, 2024-09-15T03:47:59.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0031668960000002, 2024-09-15T03:47:59.535 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:59.535 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:59.535 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:59.535 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:59.535 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.535 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 324 2024-09-15T03:47:59.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.536 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:59.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:59.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:59.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:59.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:59.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:59.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:47:59.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:47:59.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:59.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844125+0000", 2024-09-15T03:47:59.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.142e-06 2024-09-15T03:47:59.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:59.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844132+0000", 2024-09-15T03:47:59.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0439999999999998e-06 2024-09-15T03:47:59.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844136+0000", 2024-09-15T03:47:59.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.126e-06 2024-09-15T03:47:59.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.847618+0000", 2024-09-15T03:47:59.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0034813790000000002 2024-09-15T03:47:59.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:59.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.847709+0000", 2024-09-15T03:47:59.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0835999999999996e-05 2024-09-15T03:47:59.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.849483+0000", 2024-09-15T03:47:59.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001774929 2024-09-15T03:47:59.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:59.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.849574+0000", 2024-09-15T03:47:59.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0847999999999995e-05 2024-09-15T03:47:59.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:59.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.847290+0000", 2024-09-15T03:47:59.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9977151819999999 2024-09-15T03:47:59.544 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.544 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:59.544 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.545 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:334 3.7s0 3:ece1472c:::smithi00132172-43:head [append 0~0] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:59.545 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:59.545 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.674334733999999, 2024-09-15T03:47:59.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.997592021, 2024-09-15T03:47:59.546 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:59.546 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.546 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:59.546 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:59.546 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.547 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 334 2024-09-15T03:47:59.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.547 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:59.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:59.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:59.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:59.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:47:59.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:59.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856851+0000", 2024-09-15T03:47:59.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.238e-06 2024-09-15T03:47:59.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:59.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856855+0000", 2024-09-15T03:47:59.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3640000000000001e-06 2024-09-15T03:47:59.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:59.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856868+0000", 2024-09-15T03:47:59.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2588999999999999e-05 2024-09-15T03:47:59.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856873+0000", 2024-09-15T03:47:59.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5350000000000004e-06 2024-09-15T03:47:59.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856916+0000", 2024-09-15T03:47:59.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2889000000000002e-05 2024-09-15T03:47:59.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:59.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856983+0000", 2024-09-15T03:47:59.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6746000000000001e-05 2024-09-15T03:47:59.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857448+0000", 2024-09-15T03:47:59.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046498500000000001 2024-09-15T03:47:59.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857720+0000", 2024-09-15T03:47:59.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027227600000000001 2024-09-15T03:47:59.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857741+0000", 2024-09-15T03:47:59.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1149999999999999e-05 2024-09-15T03:47:59.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857795+0000", 2024-09-15T03:47:59.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3919999999999999e-05 2024-09-15T03:47:59.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859582+0000", 2024-09-15T03:47:59.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017862640000000001 2024-09-15T03:47:59.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:59.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854347+0000", 2024-09-15T03:47:59.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9947654780000001 2024-09-15T03:47:59.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:59.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854442+0000", 2024-09-15T03:47:59.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4586999999999994e-05 2024-09-15T03:47:59.559 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.559 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:59.559 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.560 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:335 3.7s0 3:ece1472c:::smithi00132172-43:head [rollback 12] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:59.560 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:59.560 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.674304716999998, 2024-09-15T03:47:59.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9977859200000001, 2024-09-15T03:47:59.560 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:59.560 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.561 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:59.561 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:59.561 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.561 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 335 2024-09-15T03:47:59.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.561 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:59.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:59.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:59.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:59.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:59.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:59.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:47:59.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8099999999999999e-07 2024-09-15T03:47:59.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:59.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856882+0000", 2024-09-15T03:47:59.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.649e-06 2024-09-15T03:47:59.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:59.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856885+0000", 2024-09-15T03:47:59.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.664e-06 2024-09-15T03:47:59.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856888+0000", 2024-09-15T03:47:59.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3910000000000001e-06 2024-09-15T03:47:59.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857826+0000", 2024-09-15T03:47:59.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00093817900000000003 2024-09-15T03:47:59.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:59.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857940+0000", 2024-09-15T03:47:59.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114424 2024-09-15T03:47:59.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858504+0000", 2024-09-15T03:47:59.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056400899999999997 2024-09-15T03:47:59.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858593+0000", 2024-09-15T03:47:59.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8463999999999999e-05 2024-09-15T03:47:59.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858604+0000", 2024-09-15T03:47:59.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1567e-05 2024-09-15T03:47:59.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858642+0000", 2024-09-15T03:47:59.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7369000000000001e-05 2024-09-15T03:47:59.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859588+0000", 2024-09-15T03:47:59.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00094661799999999998 2024-09-15T03:47:59.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:59.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854594+0000", 2024-09-15T03:47:59.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9950055550000001 2024-09-15T03:47:59.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:59.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854666+0000", 2024-09-15T03:47:59.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1749999999999996e-05 2024-09-15T03:47:59.569 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.569 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:59.569 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.569 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:336 3.7s0 3:ece1472c:::smithi00132172-43:head [append 0~0] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:47:59.569 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:59.569 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.674273667000001, 2024-09-15T03:47:59.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9980017820000002, 2024-09-15T03:47:59.570 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:59.570 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.570 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:59.570 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:59.570 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.570 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 336 2024-09-15T03:47:59.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.570 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:59.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:59.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:59.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:59.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:59.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:59.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:47:59.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8000000000000002e-07 2024-09-15T03:47:59.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:59.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856913+0000", 2024-09-15T03:47:59.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5400000000000001e-06 2024-09-15T03:47:59.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:59.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856918+0000", 2024-09-15T03:47:59.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1540000000000001e-06 2024-09-15T03:47:59.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856921+0000", 2024-09-15T03:47:59.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2550000000000002e-06 2024-09-15T03:47:59.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858655+0000", 2024-09-15T03:47:59.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017336070000000001 2024-09-15T03:47:59.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:59.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858741+0000", 2024-09-15T03:47:59.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6305999999999997e-05 2024-09-15T03:47:59.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859193+0000", 2024-09-15T03:47:59.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045221200000000001 2024-09-15T03:47:59.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859319+0000", 2024-09-15T03:47:59.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012542900000000001 2024-09-15T03:47:59.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859331+0000", 2024-09-15T03:47:59.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2028000000000001e-05 2024-09-15T03:47:59.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859369+0000", 2024-09-15T03:47:59.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8522999999999997e-05 2024-09-15T03:47:59.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859649+0000", 2024-09-15T03:47:59.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028008200000000002 2024-09-15T03:47:59.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:59.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854800+0000", 2024-09-15T03:47:59.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9951506540000001 2024-09-15T03:47:59.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:59.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854913+0000", 2024-09-15T03:47:59.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112712 2024-09-15T03:47:59.576 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.576 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:59.576 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.576 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:346 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:59.576 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:47:59.576 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.692777296999999, 2024-09-15T03:47:59.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005631948, 2024-09-15T03:47:59.576 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:59.577 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.577 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:59.577 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:59.577 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.577 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 346 2024-09-15T03:47:59.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.577 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:59.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:59.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:47:59.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:59.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:47:59.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:59.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838410+0000", 2024-09-15T03:47:59.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4430000000000002e-06 2024-09-15T03:47:59.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:59.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838419+0000", 2024-09-15T03:47:59.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1730000000000001e-06 2024-09-15T03:47:59.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:59.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838444+0000", 2024-09-15T03:47:59.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5111000000000001e-05 2024-09-15T03:47:59.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838455+0000", 2024-09-15T03:47:59.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0839000000000001e-05 2024-09-15T03:47:59.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:59.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3415000000000002e-05 2024-09-15T03:47:59.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:47:59.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838527+0000", 2024-09-15T03:47:59.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0550000000000005e-06 2024-09-15T03:47:59.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840638+0000", 2024-09-15T03:47:59.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021105379999999999 2024-09-15T03:47:59.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:59.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840729+0000", 2024-09-15T03:47:59.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0977999999999995e-05 2024-09-15T03:47:59.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.841834+0000", 2024-09-15T03:47:59.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011055640000000001 2024-09-15T03:47:59.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842223+0000", 2024-09-15T03:47:59.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038899900000000001 2024-09-15T03:47:59.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842241+0000", 2024-09-15T03:47:59.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7317e-05 2024-09-15T03:47:59.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842298+0000", 2024-09-15T03:47:59.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7825000000000003e-05 2024-09-15T03:47:59.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844866+0000", 2024-09-15T03:47:59.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025675289999999998 2024-09-15T03:47:59.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:59.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843973+0000", 2024-09-15T03:47:59.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99910710599999997 2024-09-15T03:47:59.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:59.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844039+0000", 2024-09-15T03:47:59.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6056000000000006e-05 2024-09-15T03:47:59.583 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.584 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:59.584 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.584 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:347 3.2s0 3:48ceb02f:::smithi00132172-5:head [rollback c] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:59.584 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:47:59.584 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.692692501000003, 2024-09-15T03:47:59.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005723841, 2024-09-15T03:47:59.584 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:59.584 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.584 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:59.585 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:59.585 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.585 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 347 2024-09-15T03:47:59.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.585 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:59.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:59.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:47:59.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:59.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:47:59.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:59.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838493+0000", 2024-09-15T03:47:59.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7300000000000005e-07 2024-09-15T03:47:59.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:59.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838498+0000", 2024-09-15T03:47:59.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9300000000000002e-06 2024-09-15T03:47:59.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:59.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838508+0000", 2024-09-15T03:47:59.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9790000000000005e-06 2024-09-15T03:47:59.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838513+0000", 2024-09-15T03:47:59.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8890000000000001e-06 2024-09-15T03:47:59.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838583+0000", 2024-09-15T03:47:59.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0696000000000002e-05 2024-09-15T03:47:59.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:59.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838590+0000", 2024-09-15T03:47:59.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2990000000000002e-06 2024-09-15T03:47:59.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842313+0000", 2024-09-15T03:47:59.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037238810000000001 2024-09-15T03:47:59.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:59.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842432+0000", 2024-09-15T03:47:59.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118474 2024-09-15T03:47:59.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843108+0000", 2024-09-15T03:47:59.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00067639299999999996 2024-09-15T03:47:59.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843227+0000", 2024-09-15T03:47:59.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119092 2024-09-15T03:47:59.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843242+0000", 2024-09-15T03:47:59.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4216000000000001e-05 2024-09-15T03:47:59.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843292+0000", 2024-09-15T03:47:59.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0800999999999997e-05 2024-09-15T03:47:59.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844878+0000", 2024-09-15T03:47:59.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015852749999999999 2024-09-15T03:47:59.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:59.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844147+0000", 2024-09-15T03:47:59.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99926949200000004 2024-09-15T03:47:59.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:59.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844216+0000", 2024-09-15T03:47:59.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8551000000000001e-05 2024-09-15T03:47:59.592 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.592 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:59.592 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.592 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:348 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:59.592 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:59.592 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.692666557999999, 2024-09-15T03:47:59.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.006019768, 2024-09-15T03:47:59.592 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:59.592 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.593 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:59.593 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:59.593 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.593 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 348 2024-09-15T03:47:59.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.593 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:59.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:59.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:59.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:59.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:59.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:59.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:47:59.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8700000000000001e-07 2024-09-15T03:47:59.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:59.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838521+0000", 2024-09-15T03:47:59.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3740000000000001e-06 2024-09-15T03:47:59.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:59.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838527+0000", 2024-09-15T03:47:59.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1999999999999999e-06 2024-09-15T03:47:59.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838532+0000", 2024-09-15T03:47:59.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2059999999999998e-06 2024-09-15T03:47:59.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838609+0000", 2024-09-15T03:47:59.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7193000000000004e-05 2024-09-15T03:47:59.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:47:59.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838615+0000", 2024-09-15T03:47:59.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7980000000000002e-06 2024-09-15T03:47:59.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843986+0000", 2024-09-15T03:47:59.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053706450000000003 2024-09-15T03:47:59.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:59.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844106+0000", 2024-09-15T03:47:59.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120079 2024-09-15T03:47:59.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844669+0000", 2024-09-15T03:47:59.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056290300000000001 2024-09-15T03:47:59.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844804+0000", 2024-09-15T03:47:59.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013575600000000001 2024-09-15T03:47:59.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844820+0000", 2024-09-15T03:47:59.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5145e-05 2024-09-15T03:47:59.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844893+0000", 2024-09-15T03:47:59.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3270999999999997e-05 2024-09-15T03:47:59.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845043+0000", 2024-09-15T03:47:59.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150144 2024-09-15T03:47:59.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:59.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844473+0000", 2024-09-15T03:47:59.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99943048700000003 2024-09-15T03:47:59.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:59.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844538+0000", 2024-09-15T03:47:59.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4179999999999999e-05 2024-09-15T03:47:59.600 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.600 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:59.600 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.600 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:352 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~557056] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:59.600 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:47:59.600 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.686468914000002, 2024-09-15T03:47:59.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.016265065, 2024-09-15T03:47:59.600 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:59.600 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.601 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:59.601 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:59.601 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.601 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 352 2024-09-15T03:47:59.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.601 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:59.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:59.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:47:59.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:59.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:47:59.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:59.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844718+0000", 2024-09-15T03:47:59.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2340000000000001e-06 2024-09-15T03:47:59.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:59.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845237+0000", 2024-09-15T03:47:59.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051953399999999999 2024-09-15T03:47:59.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:59.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845273+0000", 2024-09-15T03:47:59.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5683999999999999e-05 2024-09-15T03:47:59.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845281+0000", 2024-09-15T03:47:59.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8439999999999998e-06 2024-09-15T03:47:59.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845365+0000", 2024-09-15T03:47:59.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4651e-05 2024-09-15T03:47:59.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:59.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845486+0000", 2024-09-15T03:47:59.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012009499999999999 2024-09-15T03:47:59.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847101+0000", 2024-09-15T03:47:59.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016157350000000001 2024-09-15T03:47:59.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847602+0000", 2024-09-15T03:47:59.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050066200000000003 2024-09-15T03:47:59.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847622+0000", 2024-09-15T03:47:59.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9749999999999999e-05 2024-09-15T03:47:59.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848616+0000", 2024-09-15T03:47:59.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099390799999999994 2024-09-15T03:47:59.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848709+0000", 2024-09-15T03:47:59.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2874000000000006e-05 2024-09-15T03:47:59.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:59.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.860888+0000", 2024-09-15T03:47:59.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.012179707 2024-09-15T03:47:59.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:59.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.860981+0000", 2024-09-15T03:47:59.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2386999999999995e-05 2024-09-15T03:47:59.607 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.607 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:59.607 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.607 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:353 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~671744] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:59.607 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:47:59.607 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.682906721000002, 2024-09-15T03:47:59.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.012912206, 2024-09-15T03:47:59.607 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:59.607 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.607 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:59.607 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:59.607 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.608 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 353 2024-09-15T03:47:59.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.608 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:59.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:59.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:47:59.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:59.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:47:59.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:59.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848279+0000", 2024-09-15T03:47:59.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.362e-06 2024-09-15T03:47:59.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:59.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849024+0000", 2024-09-15T03:47:59.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00074495500000000005 2024-09-15T03:47:59.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:59.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849039+0000", 2024-09-15T03:47:59.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5119e-05 2024-09-15T03:47:59.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849046+0000", 2024-09-15T03:47:59.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6359999999999997e-06 2024-09-15T03:47:59.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849115+0000", 2024-09-15T03:47:59.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8998999999999994e-05 2024-09-15T03:47:59.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:59.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849251+0000", 2024-09-15T03:47:59.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013589799999999999 2024-09-15T03:47:59.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850605+0000", 2024-09-15T03:47:59.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013544659999999999 2024-09-15T03:47:59.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850703+0000", 2024-09-15T03:47:59.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7576000000000002e-05 2024-09-15T03:47:59.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850719+0000", 2024-09-15T03:47:59.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5767e-05 2024-09-15T03:47:59.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851562+0000", 2024-09-15T03:47:59.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084361899999999999 2024-09-15T03:47:59.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851705+0000", 2024-09-15T03:47:59.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000142893 2024-09-15T03:47:59.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:59.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861112+0000", 2024-09-15T03:47:59.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.009407349 2024-09-15T03:47:59.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:59.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861190+0000", 2024-09-15T03:47:59.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7566999999999998e-05 2024-09-15T03:47:59.613 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.613 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:59.614 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.614 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:354 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~98304] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:59.614 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:47:59.614 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.682104285000001, 2024-09-15T03:47:59.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0177595669999999, 2024-09-15T03:47:59.614 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:59.614 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.614 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:59.614 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:59.614 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.615 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 354 2024-09-15T03:47:59.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.615 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:59.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:59.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:47:59.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:59.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:47:59.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:59.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849081+0000", 2024-09-15T03:47:59.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0610000000000001e-06 2024-09-15T03:47:59.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:59.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849157+0000", 2024-09-15T03:47:59.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5804999999999997e-05 2024-09-15T03:47:59.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:59.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849168+0000", 2024-09-15T03:47:59.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0730000000000001e-05 2024-09-15T03:47:59.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849174+0000", 2024-09-15T03:47:59.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3609999999999997e-06 2024-09-15T03:47:59.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850814+0000", 2024-09-15T03:47:59.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001640124 2024-09-15T03:47:59.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:59.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850894+0000", 2024-09-15T03:47:59.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0167999999999995e-05 2024-09-15T03:47:59.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851406+0000", 2024-09-15T03:47:59.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051203200000000005 2024-09-15T03:47:59.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851506+0000", 2024-09-15T03:47:59.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9457999999999995e-05 2024-09-15T03:47:59.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851574+0000", 2024-09-15T03:47:59.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8145000000000006e-05 2024-09-15T03:47:59.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851632+0000", 2024-09-15T03:47:59.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8108e-05 2024-09-15T03:47:59.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851712+0000", 2024-09-15T03:47:59.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9380000000000002e-05 2024-09-15T03:47:59.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:59.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.866753+0000", 2024-09-15T03:47:59.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0150417979999999 2024-09-15T03:47:59.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:59.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.866840+0000", 2024-09-15T03:47:59.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6397000000000006e-05 2024-09-15T03:47:59.620 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.620 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:59.620 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.621 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:355 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [setxattr _header (57) in=7b] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:59.621 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:47:59.621 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.681974971999999, 2024-09-15T03:47:59.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.018024966, 2024-09-15T03:47:59.621 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:59.621 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.621 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:59.621 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:59.621 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.621 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 355 2024-09-15T03:47:59.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.621 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:59.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:59.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:47:59.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:59.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:47:59.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:59.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849210+0000", 2024-09-15T03:47:59.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.33e-07 2024-09-15T03:47:59.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:59.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849215+0000", 2024-09-15T03:47:59.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4379999999999997e-06 2024-09-15T03:47:59.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:59.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849234+0000", 2024-09-15T03:47:59.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9409999999999999e-05 2024-09-15T03:47:59.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849241+0000", 2024-09-15T03:47:59.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6370000000000002e-06 2024-09-15T03:47:59.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851823+0000", 2024-09-15T03:47:59.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025817689999999998 2024-09-15T03:47:59.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:59.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851934+0000", 2024-09-15T03:47:59.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111357 2024-09-15T03:47:59.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852466+0000", 2024-09-15T03:47:59.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053248100000000001 2024-09-15T03:47:59.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852580+0000", 2024-09-15T03:47:59.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011379699999999999 2024-09-15T03:47:59.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852607+0000", 2024-09-15T03:47:59.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279e-05 2024-09-15T03:47:59.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852693+0000", 2024-09-15T03:47:59.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6792999999999994e-05 2024-09-15T03:47:59.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852869+0000", 2024-09-15T03:47:59.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017537499999999999 2024-09-15T03:47:59.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:59.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867006+0000", 2024-09-15T03:47:59.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.014136996 2024-09-15T03:47:59.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:59.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867234+0000", 2024-09-15T03:47:59.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022870099999999999 2024-09-15T03:47:59.627 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.627 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:59.627 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:356 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:47:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:47:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.681938209000002, 2024-09-15T03:47:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.020403521, 2024-09-15T03:47:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 356 2024-09-15T03:47:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:59.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:47:59.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:59.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:47:59.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:59.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849247+0000", 2024-09-15T03:47:59.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2699999999999999e-07 2024-09-15T03:47:59.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:59.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849250+0000", 2024-09-15T03:47:59.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2610000000000002e-06 2024-09-15T03:47:59.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:59.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849257+0000", 2024-09-15T03:47:59.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.703e-06 2024-09-15T03:47:59.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849261+0000", 2024-09-15T03:47:59.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7910000000000002e-06 2024-09-15T03:47:59.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852672+0000", 2024-09-15T03:47:59.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003411627 2024-09-15T03:47:59.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:59.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852801+0000", 2024-09-15T03:47:59.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000128918 2024-09-15T03:47:59.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867347+0000", 2024-09-15T03:47:59.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0145458380000001 2024-09-15T03:47:59.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:59.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867476+0000", 2024-09-15T03:47:59.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012915100000000001 2024-09-15T03:47:59.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:59.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.869650+0000", 2024-09-15T03:47:59.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021737050000000002 2024-09-15T03:47:59.633 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.633 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:59.633 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.633 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:527 3.7s0 3:e4b8e19d:::smithi00132172-41:head [copy-from ver 51] snapc 23=[23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:59.633 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:47:59.633 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.582369247000003, 2024-09-15T03:47:59.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030362388000000001, 2024-09-15T03:47:59.633 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:59.633 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.633 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:59.634 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:59.634 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.634 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 527 2024-09-15T03:47:59.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.634 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:59.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:59.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:47:59.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:59.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:47:59.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:59.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948817+0000", 2024-09-15T03:47:59.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3230000000000001e-06 2024-09-15T03:47:59.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:59.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948825+0000", 2024-09-15T03:47:59.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5900000000000008e-06 2024-09-15T03:47:59.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:59.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948838+0000", 2024-09-15T03:47:59.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2712e-05 2024-09-15T03:47:59.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948842+0000", 2024-09-15T03:47:59.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4170000000000002e-06 2024-09-15T03:47:59.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948923+0000", 2024-09-15T03:47:59.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0686000000000006e-05 2024-09-15T03:47:59.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:47:59.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949020+0000", 2024-09-15T03:47:59.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6984999999999994e-05 2024-09-15T03:47:59.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954390+0000", 2024-09-15T03:47:59.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053704790000000001 2024-09-15T03:47:59.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:59.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954558+0000", 2024-09-15T03:47:59.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000167202 2024-09-15T03:47:59.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.973998+0000", 2024-09-15T03:47:59.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019440316999999999 2024-09-15T03:47:59.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974527+0000", 2024-09-15T03:47:59.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052943800000000002 2024-09-15T03:47:59.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974541+0000", 2024-09-15T03:47:59.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3774999999999999e-05 2024-09-15T03:47:59.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974841+0000", 2024-09-15T03:47:59.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029942899999999998 2024-09-15T03:47:59.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.976067+0000", 2024-09-15T03:47:59.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012265399999999999 2024-09-15T03:47:59.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:59.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.978952+0000", 2024-09-15T03:47:59.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028848839999999999 2024-09-15T03:47:59.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:59.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.979178+0000", 2024-09-15T03:47:59.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022561099999999999 2024-09-15T03:47:59.640 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.640 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:59.640 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.641 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:528 3.7s0 3:e4b8e19d:::smithi00132172-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:47:59.641 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:59.641 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.582334838000001, 2024-09-15T03:47:59.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026927036000000001, 2024-09-15T03:47:59.641 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:59.641 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:59.641 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:59.641 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:59.641 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.641 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 528 2024-09-15T03:47:59.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.642 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:59.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:59.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:59.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:59.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:59.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:59.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:47:59.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9700000000000003e-07 2024-09-15T03:47:59.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:59.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948852+0000", 2024-09-15T03:47:59.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.604e-06 2024-09-15T03:47:59.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:59.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948855+0000", 2024-09-15T03:47:59.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9369999999999997e-06 2024-09-15T03:47:59.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948858+0000", 2024-09-15T03:47:59.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2709999999999999e-06 2024-09-15T03:47:59.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949080+0000", 2024-09-15T03:47:59.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000222148 2024-09-15T03:47:59.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:47:59.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949156+0000", 2024-09-15T03:47:59.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6215000000000001e-05 2024-09-15T03:47:59.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954754+0000", 2024-09-15T03:47:59.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055978060000000003 2024-09-15T03:47:59.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:47:59.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954858+0000", 2024-09-15T03:47:59.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010419199999999999 2024-09-15T03:47:59.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974861+0000", 2024-09-15T03:47:59.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020002671999999999 2024-09-15T03:47:59.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:59.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974987+0000", 2024-09-15T03:47:59.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012638699999999999 2024-09-15T03:47:59.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:59.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.975777+0000", 2024-09-15T03:47:59.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00078950699999999995 2024-09-15T03:47:59.647 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.647 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:59.647 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.647 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:924 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~737280] snapc 45=[45,44,43,42,41,40,3e,3d,3b,3a,38,37,35,34,2d,2b,25,24,22,21,20,16,12,11,3] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-15T03:47:59.647 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:53.925245+0000", 2024-09-15T03:47:59.648 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.6059391859999996, 2024-09-15T03:47:59.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.29827357799999998, 2024-09-15T03:47:59.648 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:59.648 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.648 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:59.648 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:59.648 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.648 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 924 2024-09-15T03:47:59.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.648 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:59.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:59.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.925245+0000", 2024-09-15T03:47:59.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:59.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.925245+0000", 2024-09-15T03:47:59.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:59.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.925247+0000", 2024-09-15T03:47:59.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9999999999999999e-06 2024-09-15T03:47:59.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:59.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.926127+0000", 2024-09-15T03:47:59.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000879957 2024-09-15T03:47:59.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:59.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.926146+0000", 2024-09-15T03:47:59.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8349999999999999e-05 2024-09-15T03:47:59.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.926153+0000", 2024-09-15T03:47:59.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3889999999999999e-06 2024-09-15T03:47:59.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.926220+0000", 2024-09-15T03:47:59.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7490000000000006e-05 2024-09-15T03:47:59.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:59.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.926224+0000", 2024-09-15T03:47:59.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5590000000000001e-06 2024-09-15T03:47:59.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.209778+0000", 2024-09-15T03:47:59.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.28355351899999998 2024-09-15T03:47:59.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:59.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.209967+0000", 2024-09-15T03:47:59.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018960400000000001 2024-09-15T03:47:59.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.211917+0000", 2024-09-15T03:47:59.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019494320000000001 2024-09-15T03:47:59.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.212779+0000", 2024-09-15T03:47:59.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086249099999999995 2024-09-15T03:47:59.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.212798+0000", 2024-09-15T03:47:59.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9015e-05 2024-09-15T03:47:59.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.212925+0000", 2024-09-15T03:47:59.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012736099999999999 2024-09-15T03:47:59.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.212946+0000", 2024-09-15T03:47:59.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0919000000000001e-05 2024-09-15T03:47:59.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:59.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.215488+0000", 2024-09-15T03:47:59.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025418350000000001 2024-09-15T03:47:59.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:59.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.223519+0000", 2024-09-15T03:47:59.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0080306570000000001 2024-09-15T03:47:59.655 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.655 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:59.655 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.655 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:925 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~770048] snapc 45=[45,44,43,42,41,40,3e,3d,3b,3a,38,37,35,34,2d,2b,25,24,22,21,20,16,12,11,3] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-15T03:47:59.655 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:53.930327+0000", 2024-09-15T03:47:59.656 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.6008574719999995, 2024-09-15T03:47:59.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.29345302099999998, 2024-09-15T03:47:59.656 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:59.656 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.656 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:59.656 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:59.656 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.656 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 925 2024-09-15T03:47:59.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.656 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:59.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:59.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.930327+0000", 2024-09-15T03:47:59.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:59.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.930327+0000", 2024-09-15T03:47:59.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:59.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.930328+0000", 2024-09-15T03:47:59.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.15e-06 2024-09-15T03:47:59.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:59.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.930988+0000", 2024-09-15T03:47:59.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000659594 2024-09-15T03:47:59.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:59.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.931004+0000", 2024-09-15T03:47:59.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6101000000000001e-05 2024-09-15T03:47:59.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.931011+0000", 2024-09-15T03:47:59.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8940000000000004e-06 2024-09-15T03:47:59.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.931050+0000", 2024-09-15T03:47:59.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9085000000000003e-05 2024-09-15T03:47:59.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:59.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.931052+0000", 2024-09-15T03:47:59.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1789999999999998e-06 2024-09-15T03:47:59.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.213039+0000", 2024-09-15T03:47:59.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.281986827 2024-09-15T03:47:59.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:59.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.213228+0000", 2024-09-15T03:47:59.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000189159 2024-09-15T03:47:59.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.214530+0000", 2024-09-15T03:47:59.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001302147 2024-09-15T03:47:59.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.214723+0000", 2024-09-15T03:47:59.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000192998 2024-09-15T03:47:59.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.214741+0000", 2024-09-15T03:47:59.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7762e-05 2024-09-15T03:47:59.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.215136+0000", 2024-09-15T03:47:59.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039510400000000002 2024-09-15T03:47:59.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.215269+0000", 2024-09-15T03:47:59.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013283300000000001 2024-09-15T03:47:59.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:59.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.216320+0000", 2024-09-15T03:47:59.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001050888 2024-09-15T03:47:59.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:59.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.223780+0000", 2024-09-15T03:47:59.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074602999999999996 2024-09-15T03:47:59.663 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.663 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:59.663 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.663 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:926 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~540672] snapc 45=[45,44,43,42,41,40,3e,3d,3b,3a,38,37,35,34,2d,2b,25,24,22,21,20,16,12,11,3] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-15T03:47:59.663 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:53.933251+0000", 2024-09-15T03:47:59.664 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.5979338109999999, 2024-09-15T03:47:59.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.29574198400000001, 2024-09-15T03:47:59.664 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:59.664 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.664 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:59.664 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:59.664 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.664 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 926 2024-09-15T03:47:59.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.664 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:59.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:59.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933251+0000", 2024-09-15T03:47:59.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:59.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933251+0000", 2024-09-15T03:47:59.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:59.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933252+0000", 2024-09-15T03:47:59.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.569999999999999e-07 2024-09-15T03:47:59.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:59.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933849+0000", 2024-09-15T03:47:59.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059778100000000003 2024-09-15T03:47:59.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:59.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933865+0000", 2024-09-15T03:47:59.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5170999999999999e-05 2024-09-15T03:47:59.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933871+0000", 2024-09-15T03:47:59.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2899999999999999e-06 2024-09-15T03:47:59.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933904+0000", 2024-09-15T03:47:59.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3519000000000002e-05 2024-09-15T03:47:59.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:59.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933905+0000", 2024-09-15T03:47:59.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.096e-06 2024-09-15T03:47:59.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.217183+0000", 2024-09-15T03:47:59.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.28327724799999998 2024-09-15T03:47:59.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:59.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.217399+0000", 2024-09-15T03:47:59.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000216429 2024-09-15T03:47:59.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.224559+0000", 2024-09-15T03:47:59.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071598429999999999 2024-09-15T03:47:59.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.224765+0000", 2024-09-15T03:47:59.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020648500000000001 2024-09-15T03:47:59.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.224794+0000", 2024-09-15T03:47:59.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8645000000000001e-05 2024-09-15T03:47:59.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.224949+0000", 2024-09-15T03:47:59.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015473200000000001 2024-09-15T03:47:59.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.227877+0000", 2024-09-15T03:47:59.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029284110000000001 2024-09-15T03:47:59.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:59.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.228692+0000", 2024-09-15T03:47:59.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081477700000000004 2024-09-15T03:47:59.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:59.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.228993+0000", 2024-09-15T03:47:59.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030059999999999999 2024-09-15T03:47:59.671 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.671 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:59.671 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.671 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:927 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~245760] snapc 45=[45,44,43,42,41,40,3e,3d,3b,3a,38,37,35,34,2d,2b,25,24,22,21,20,16,12,11,3] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-15T03:47:59.671 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:53.934611+0000", 2024-09-15T03:47:59.671 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.5965729609999997, 2024-09-15T03:47:59.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.29515997300000002, 2024-09-15T03:47:59.671 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:59.671 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.671 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:59.671 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:59.671 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.672 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 927 2024-09-15T03:47:59.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.672 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:59.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:59.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934611+0000", 2024-09-15T03:47:59.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:59.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934611+0000", 2024-09-15T03:47:59.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:59.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934612+0000", 2024-09-15T03:47:59.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3099999999999996e-07 2024-09-15T03:47:59.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:59.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934841+0000", 2024-09-15T03:47:59.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022889699999999999 2024-09-15T03:47:59.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:59.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934855+0000", 2024-09-15T03:47:59.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3998999999999999e-05 2024-09-15T03:47:59.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934862+0000", 2024-09-15T03:47:59.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3300000000000004e-06 2024-09-15T03:47:59.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934894+0000", 2024-09-15T03:47:59.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1995000000000001e-05 2024-09-15T03:47:59.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:59.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934894+0000", 2024-09-15T03:47:59.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6199999999999998e-07 2024-09-15T03:47:59.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.224939+0000", 2024-09-15T03:47:59.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.29004481300000001 2024-09-15T03:47:59.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:59.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.225469+0000", 2024-09-15T03:47:59.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052985499999999997 2024-09-15T03:47:59.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.226886+0000", 2024-09-15T03:47:59.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001416785 2024-09-15T03:47:59.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.227699+0000", 2024-09-15T03:47:59.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000813363 2024-09-15T03:47:59.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.227716+0000", 2024-09-15T03:47:59.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7017e-05 2024-09-15T03:47:59.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.227783+0000", 2024-09-15T03:47:59.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7057999999999994e-05 2024-09-15T03:47:59.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.227879+0000", 2024-09-15T03:47:59.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6417999999999997e-05 2024-09-15T03:47:59.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:59.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.229372+0000", 2024-09-15T03:47:59.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014929019999999999 2024-09-15T03:47:59.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:59.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.229771+0000", 2024-09-15T03:47:59.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039914799999999999 2024-09-15T03:47:59.678 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.678 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:59.678 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.679 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:928 3.2s0 3:48ceb02f:::smithi00132172-5:head [setxattr _header (56) in=7b] snapc 45=[45,44,43,42,41,40,3e,3d,3b,3a,38,37,35,34,2d,2b,25,24,22,21,20,16,12,11,3] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-15T03:47:59.679 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:53.934871+0000", 2024-09-15T03:47:59.679 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.5963130469999998, 2024-09-15T03:47:59.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.298571425, 2024-09-15T03:47:59.679 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:59.679 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:47:59.679 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:59.679 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:59.680 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.680 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 928 2024-09-15T03:47:59.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.680 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:59.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:59.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934871+0000", 2024-09-15T03:47:59.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:59.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934871+0000", 2024-09-15T03:47:59.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:59.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934872+0000", 2024-09-15T03:47:59.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9100000000000003e-07 2024-09-15T03:47:59.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:59.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934876+0000", 2024-09-15T03:47:59.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.687e-06 2024-09-15T03:47:59.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:59.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934884+0000", 2024-09-15T03:47:59.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7049999999999999e-06 2024-09-15T03:47:59.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934888+0000", 2024-09-15T03:47:59.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.639e-06 2024-09-15T03:47:59.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934911+0000", 2024-09-15T03:47:59.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3224999999999999e-05 2024-09-15T03:47:59.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:59.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934912+0000", 2024-09-15T03:47:59.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9999999999999999e-07 2024-09-15T03:47:59.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.230182+0000", 2024-09-15T03:47:59.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.29527004699999998 2024-09-15T03:47:59.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:59.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.230320+0000", 2024-09-15T03:47:59.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013794499999999999 2024-09-15T03:47:59.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:47:59.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.231057+0000", 2024-09-15T03:47:59.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00073703400000000002 2024-09-15T03:47:59.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:47:59.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.231199+0000", 2024-09-15T03:47:59.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014196099999999999 2024-09-15T03:47:59.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:47:59.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.231309+0000", 2024-09-15T03:47:59.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011001399999999999 2024-09-15T03:47:59.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:47:59.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.231349+0000", 2024-09-15T03:47:59.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0068999999999999e-05 2024-09-15T03:47:59.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:47:59.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.231658+0000", 2024-09-15T03:47:59.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030921799999999999 2024-09-15T03:47:59.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:47:59.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.232928+0000", 2024-09-15T03:47:59.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012703409999999999 2024-09-15T03:47:59.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:59.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.233443+0000", 2024-09-15T03:47:59.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051444900000000005 2024-09-15T03:47:59.687 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.687 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:59.687 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.687 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:929 3.2s0 3:48ceb02f:::smithi00132172-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2024-09-15T03:47:59.687 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:53.934895+0000", 2024-09-15T03:47:59.687 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.5962896049999999, 2024-09-15T03:47:59.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.30125257599999999, 2024-09-15T03:47:59.687 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:47:59.687 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:47:59.687 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:47:59.687 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:47:59.688 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:47:59.688 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 929 2024-09-15T03:47:59.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.688 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:47:59.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:47:59.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934895+0000", 2024-09-15T03:47:59.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:47:59.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934895+0000", 2024-09-15T03:47:59.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:47:59.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:47:59.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934895+0000", 2024-09-15T03:47:59.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2699999999999999e-07 2024-09-15T03:47:59.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:47:59.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934898+0000", 2024-09-15T03:47:59.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3549999999999999e-06 2024-09-15T03:47:59.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:47:59.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934905+0000", 2024-09-15T03:47:59.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0060000000000003e-06 2024-09-15T03:47:59.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:47:59.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934910+0000", 2024-09-15T03:47:59.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1499999999999998e-06 2024-09-15T03:47:59.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934935+0000", 2024-09-15T03:47:59.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4641e-05 2024-09-15T03:47:59.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:47:59.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934935+0000", 2024-09-15T03:47:59.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7099999999999997e-07 2024-09-15T03:47:59.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.231303+0000", 2024-09-15T03:47:59.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.29636852000000002 2024-09-15T03:47:59.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:47:59.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.231460+0000", 2024-09-15T03:47:59.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015613700000000001 2024-09-15T03:47:59.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:47:59.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.233845+0000", 2024-09-15T03:47:59.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023851660000000002 2024-09-15T03:47:59.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:47:59.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.233998+0000", 2024-09-15T03:47:59.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015345699999999999 2024-09-15T03:47:59.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:47:59.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:47:59.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:47:59.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.236147+0000", 2024-09-15T03:47:59.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021492460000000001 2024-09-15T03:47:59.693 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.693 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:59.693 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.693 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:47:59.693 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:47:59.693 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:59.695 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:47:59.696 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:47:59.695+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:47:59.796 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:47:59.795+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:47:59.838 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:47:59.838 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:47:59.838 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:47:59.839 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:47:59.850 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:47:59.896 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:47:59.896+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:47:59.941 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.4 to 1 (10000) 2024-09-15T03:47:59.942 INFO:tasks.rados.rados.0.smithi001.stdout:394: done (4 left) 2024-09-15T03:47:59.942 INFO:tasks.rados.rados.0.smithi001.stdout:append oid 38 current snap is 49 2024-09-15T03:47:59.942 INFO:tasks.rados.rados.0.smithi001.stdout:390: finishing write tid 2 to smithi00132172-41 2024-09-15T03:47:59.942 INFO:tasks.rados.rados.0.smithi001.stdout:395: seq_num 149 ranges {5537792=98304} 2024-09-15T03:47:59.942 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-15T03:47:59.975 INFO:tasks.rados.rados.0.smithi001.stdout:395: writing smithi00132172-38 from 5537792 to 5636096 tid 1 2024-09-15T03:47:59.975 INFO:tasks.rados.rados.0.smithi001.stdout:390: finishing write tid 3 to smithi00132172-41 2024-09-15T03:47:59.976 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 41 v 141 (ObjNum 146 snap 49 seq_num 146) dirty exists 2024-09-15T03:47:59.976 INFO:tasks.rados.rados.0.smithi001.stdout:390: left oid 41 (ObjNum 146 snap 49 seq_num 146) 2024-09-15T03:47:59.976 INFO:tasks.rados.rados.0.smithi001.stdout:391: finishing write tid 1 to smithi00132172-10 2024-09-15T03:47:59.976 INFO:tasks.rados.rados.0.smithi001.stdout:391: finishing write tid 2 to smithi00132172-10 2024-09-15T03:47:59.976 INFO:tasks.rados.rados.0.smithi001.stdout:391: finishing write tid 3 to smithi00132172-10 2024-09-15T03:47:59.976 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 10 v 183 (ObjNum 147 snap 49 seq_num 147) dirty exists 2024-09-15T03:47:59.976 INFO:tasks.rados.rados.0.smithi001.stdout:391: left oid 10 (ObjNum 147 snap 49 seq_num 147) 2024-09-15T03:47:59.976 INFO:tasks.rados.rados.0.smithi001.stdout:392: finishing write tid 1 to smithi00132172-2 2024-09-15T03:47:59.976 INFO:tasks.rados.rados.0.smithi001.stdout:392: finishing write tid 2 to smithi00132172-2 2024-09-15T03:47:59.976 INFO:tasks.rados.rados.0.smithi001.stdout:392: finishing write tid 3 to smithi00132172-2 2024-09-15T03:47:59.976 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 2 v 186 (ObjNum 148 snap 49 seq_num 148) dirty exists 2024-09-15T03:47:59.976 INFO:tasks.rados.rados.0.smithi001.stdout:392: left oid 2 (ObjNum 148 snap 49 seq_num 148) 2024-09-15T03:47:59.976 INFO:tasks.rados.rados.0.smithi001.stdout:393: expect (ObjNum 116 snap 33 seq_num 116) 2024-09-15T03:47:59.984 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-15T03:48:00.008 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:48:00.008 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:48:00.008 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:48:00.008 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:48:00.009 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:48:00.017 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:48:00.028 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:48:00.027+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:48:00.098 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:48:00.097+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:48:00.119 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:48:00.119 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:48:00.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:48:00.119 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:48:00.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.119 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 38/28 ECSubWrite(tid=33, reqid=client.4521.0:222, at_version=38'14, trim_to=0'0, roll_forward_to=35'12))", 2024-09-15T03:48:00.119 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:48:00.119 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.986163657999995, 2024-09-15T03:48:00.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9949187450000001, 2024-09-15T03:48:00.119 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:00.119 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:00.120 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:00.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:00.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:48:00.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:00.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:48:00.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:00.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130996+0000", 2024-09-15T03:48:00.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8809999999999998e-06 2024-09-15T03:48:00.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:00.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131020+0000", 2024-09-15T03:48:00.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4190999999999999e-05 2024-09-15T03:48:00.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:00.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131039+0000", 2024-09-15T03:48:00.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9228999999999998e-05 2024-09-15T03:48:00.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:00.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131048+0000", 2024-09-15T03:48:00.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2380000000000003e-06 2024-09-15T03:48:00.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:00.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131122+0000", 2024-09-15T03:48:00.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3146999999999996e-05 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131132+0000", 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.08e-05 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131381+0000", 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000248936 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:00.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131392+0000", 2024-09-15T03:48:00.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0987e-05 2024-09-15T03:48:00.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131444+0000", 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1585000000000002e-05 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131465+0000", 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1365999999999998e-05 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:00.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.125911+0000", 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9944453850000001 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 38/28 ECSubWrite(tid=34, reqid=client.4521.0:222, at_version=0'0, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.984764558999998, 2024-09-15T03:48:00.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.002752697, 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:48:00.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132392+0000", 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5899999999999995e-07 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132415+0000", 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3561e-05 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132436+0000", 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0879000000000001e-05 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132444+0000", 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9200000000000004e-06 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132497+0000", 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2299000000000002e-05 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132508+0000", 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1100999999999999e-05 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132615+0000", 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107546 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132635+0000", 2024-09-15T03:48:00.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9374000000000001e-05 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132677+0000", 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2456999999999997e-05 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132711+0000", 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4118000000000001e-05 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135144+0000", 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.002432583 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=45, reqid=client.4521.0:227, at_version=38'24, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.961838017000005, 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.974765339, 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:00.133 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:00.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.43e-07 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156110+0000", 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000792409 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156139+0000", 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9207e-05 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156148+0000", 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1269999999999992e-06 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156248+0000", 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9421999999999998e-05 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:00.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156258+0000", 2024-09-15T03:48:00.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6989999999999998e-06 2024-09-15T03:48:00.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:00.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156674+0000", 2024-09-15T03:48:00.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041682600000000002 2024-09-15T03:48:00.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:00.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156705+0000", 2024-09-15T03:48:00.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0525000000000003e-05 2024-09-15T03:48:00.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:00.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157742+0000", 2024-09-15T03:48:00.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010374 2024-09-15T03:48:00.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:00.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157777+0000", 2024-09-15T03:48:00.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4878e-05 2024-09-15T03:48:00.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:00.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.130083+0000", 2024-09-15T03:48:00.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972305703 2024-09-15T03:48:00.138 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.139 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:00.139 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.139 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=37, reqid=client.4521.0:236, at_version=38'5, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:48:00.139 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:48:00.139 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.960224561000004, 2024-09-15T03:48:00.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9740836640000001, 2024-09-15T03:48:00.139 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:00.139 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:00.139 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:00.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1300000000000001e-07 2024-09-15T03:48:00.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156953+0000", 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1585000000000001e-05 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156974+0000", 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0764e-05 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:00.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156981+0000", 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2760000000000003e-06 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157061+0000", 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9583e-05 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157075+0000", 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4019000000000001e-05 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157351+0000", 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027641100000000001 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157746+0000", 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039553199999999998 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157838+0000", 2024-09-15T03:48:00.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1632000000000006e-05 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157856+0000", 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7873999999999999e-05 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.131015+0000", 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9731586750000001 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:00.144 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=46, reqid=client.4521.0:228, at_version=38'25, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.960165778999993, 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9732838510000001, 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:48:00.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0999999999999998e-07 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:00.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157149+0000", 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000158763 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157165+0000", 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5883999999999999e-05 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157175+0000", 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4949999999999997e-06 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:00.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157219+0000", 2024-09-15T03:48:00.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4069000000000001e-05 2024-09-15T03:48:00.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:00.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157228+0000", 2024-09-15T03:48:00.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7829999999999993e-06 2024-09-15T03:48:00.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:00.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157345+0000", 2024-09-15T03:48:00.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116655 2024-09-15T03:48:00.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:00.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157746+0000", 2024-09-15T03:48:00.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040045100000000002 2024-09-15T03:48:00.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:00.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157838+0000", 2024-09-15T03:48:00.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1955999999999998e-05 2024-09-15T03:48:00.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:00.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157888+0000", 2024-09-15T03:48:00.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0943000000000003e-05 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.130274+0000", 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972385142 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=38, reqid=client.4521.0:237, at_version=38'6, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:48:00.150 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.958823209000002, 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9764011529999999, 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158333+0000", 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8599999999999997e-07 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158353+0000", 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9504000000000002e-05 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:00.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158368+0000", 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4966999999999999e-05 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158374+0000", 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.798e-06 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158430+0000", 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5744e-05 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158440+0000", 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4560000000000006e-06 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158529+0000", 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9172999999999995e-05 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158543+0000", 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.395e-05 2024-09-15T03:48:00.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158654+0000", 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011110700000000001 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158671+0000", 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7068999999999999e-05 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:00.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134733+0000", 2024-09-15T03:48:00.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.976062599 2024-09-15T03:48:00.156 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.156 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:00.156 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.156 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=47, reqid=client.4521.0:229, at_version=38'26, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:48:00.156 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:48:00.156 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.958819246999994, 2024-09-15T03:48:00.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9727810969999999, 2024-09-15T03:48:00.156 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:00.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158337+0000", 2024-09-15T03:48:00.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.15e-07 2024-09-15T03:48:00.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:00.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158351+0000", 2024-09-15T03:48:00.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3871e-05 2024-09-15T03:48:00.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:00.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158372+0000", 2024-09-15T03:48:00.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1058999999999999e-05 2024-09-15T03:48:00.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:00.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158376+0000", 2024-09-15T03:48:00.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5249999999999999e-06 2024-09-15T03:48:00.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:00.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158420+0000", 2024-09-15T03:48:00.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3918000000000002e-05 2024-09-15T03:48:00.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:00.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158429+0000", 2024-09-15T03:48:00.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5150000000000006e-06 2024-09-15T03:48:00.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:00.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158514+0000", 2024-09-15T03:48:00.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5252000000000003e-05 2024-09-15T03:48:00.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:00.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158542+0000", 2024-09-15T03:48:00.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.798e-05 2024-09-15T03:48:00.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:00.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158653+0000", 2024-09-15T03:48:00.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111315 2024-09-15T03:48:00.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:00.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158701+0000", 2024-09-15T03:48:00.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8241000000000001e-05 2024-09-15T03:48:00.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:00.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.131117+0000", 2024-09-15T03:48:00.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972416006 2024-09-15T03:48:00.161 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.162 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:00.162 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.162 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=39, reqid=client.4521.0:238, at_version=38'7, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:48:00.162 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:48:00.162 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.958132090999996, 2024-09-15T03:48:00.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9758214509999998, 2024-09-15T03:48:00.162 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:00.162 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:00.162 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:00.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:00.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:48:00.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:00.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:48:00.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:00.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159024+0000", 2024-09-15T03:48:00.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6700000000000003e-07 2024-09-15T03:48:00.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:00.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159096+0000", 2024-09-15T03:48:00.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1386999999999999e-05 2024-09-15T03:48:00.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:00.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159113+0000", 2024-09-15T03:48:00.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7020999999999998e-05 2024-09-15T03:48:00.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:00.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159122+0000", 2024-09-15T03:48:00.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4720000000000001e-06 2024-09-15T03:48:00.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:00.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159168+0000", 2024-09-15T03:48:00.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5766000000000002e-05 2024-09-15T03:48:00.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:00.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159176+0000", 2024-09-15T03:48:00.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3070000000000003e-06 2024-09-15T03:48:00.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:00.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159258+0000", 2024-09-15T03:48:00.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1398999999999998e-05 2024-09-15T03:48:00.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:00.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159292+0000", 2024-09-15T03:48:00.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4480000000000002e-05 2024-09-15T03:48:00.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:00.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159342+0000", 2024-09-15T03:48:00.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9450999999999998e-05 2024-09-15T03:48:00.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:00.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159362+0000", 2024-09-15T03:48:00.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0633e-05 2024-09-15T03:48:00.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:00.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134845+0000", 2024-09-15T03:48:00.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.975482768 2024-09-15T03:48:00.167 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.167 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:00.167 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.168 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=48, reqid=client.4521.0:229, at_version=0'0, trim_to=0'0, roll_forward_to=38'26))", 2024-09-15T03:48:00.168 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:48:00.168 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.956765294999997, 2024-09-15T03:48:00.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.974485472, 2024-09-15T03:48:00.168 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:00.168 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:00.168 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:00.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:00.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:48:00.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:00.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:48:00.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:00.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160391+0000", 2024-09-15T03:48:00.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.136e-06 2024-09-15T03:48:00.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:00.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160405+0000", 2024-09-15T03:48:00.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3613e-05 2024-09-15T03:48:00.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:00.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160426+0000", 2024-09-15T03:48:00.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1234999999999999e-05 2024-09-15T03:48:00.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:00.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160434+0000", 2024-09-15T03:48:00.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.61e-06 2024-09-15T03:48:00.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:00.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160496+0000", 2024-09-15T03:48:00.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2384000000000004e-05 2024-09-15T03:48:00.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:00.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160507+0000", 2024-09-15T03:48:00.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1005000000000001e-05 2024-09-15T03:48:00.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:00.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160632+0000", 2024-09-15T03:48:00.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124566 2024-09-15T03:48:00.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:00.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160653+0000", 2024-09-15T03:48:00.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0707999999999999e-05 2024-09-15T03:48:00.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:00.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160701+0000", 2024-09-15T03:48:00.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8751999999999997e-05 2024-09-15T03:48:00.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:00.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160739+0000", 2024-09-15T03:48:00.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7486e-05 2024-09-15T03:48:00.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:00.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134876+0000", 2024-09-15T03:48:00.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9741369770000001 2024-09-15T03:48:00.173 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.173 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:00.173 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.173 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=40, reqid=client.4521.0:238, at_version=0'0, trim_to=0'0, roll_forward_to=38'7))", 2024-09-15T03:48:00.173 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:48:00.173 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.954918874000001, 2024-09-15T03:48:00.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9727862730000001, 2024-09-15T03:48:00.174 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:00.174 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:00.174 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:00.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:00.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:48:00.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:00.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:48:00.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:00.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162238+0000", 2024-09-15T03:48:00.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.672e-06 2024-09-15T03:48:00.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:00.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162253+0000", 2024-09-15T03:48:00.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4932e-05 2024-09-15T03:48:00.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:00.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162284+0000", 2024-09-15T03:48:00.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1211000000000003e-05 2024-09-15T03:48:00.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:00.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162296+0000", 2024-09-15T03:48:00.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1229999999999999e-05 2024-09-15T03:48:00.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:00.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162373+0000", 2024-09-15T03:48:00.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7146999999999998e-05 2024-09-15T03:48:00.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:00.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162387+0000", 2024-09-15T03:48:00.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4290999999999999e-05 2024-09-15T03:48:00.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:00.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162502+0000", 2024-09-15T03:48:00.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115054 2024-09-15T03:48:00.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:00.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162516+0000", 2024-09-15T03:48:00.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3597999999999999e-05 2024-09-15T03:48:00.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:00.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162558+0000", 2024-09-15T03:48:00.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2562999999999999e-05 2024-09-15T03:48:00.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:00.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162584+0000", 2024-09-15T03:48:00.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5738e-05 2024-09-15T03:48:00.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:00.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135023+0000", 2024-09-15T03:48:00.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9724388369999999 2024-09-15T03:48:00.179 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.179 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:00.179 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.179 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 39/28 ECSubWrite(tid=38, reqid=osd.11.0:38, at_version=39'16, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T03:48:00.179 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:48:00.179 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.973536573999993, 2024-09-15T03:48:00.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99210653800000004, 2024-09-15T03:48:00.179 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:00.179 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:00.180 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:00.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:00.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:48:00.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:00.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:48:00.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:00.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:48:00.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1800000000000002e-07 2024-09-15T03:48:00.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:00.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143648+0000", 2024-09-15T03:48:00.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.849e-05 2024-09-15T03:48:00.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:00.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143670+0000", 2024-09-15T03:48:00.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2036e-05 2024-09-15T03:48:00.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:00.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143680+0000", 2024-09-15T03:48:00.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8320000000000001e-06 2024-09-15T03:48:00.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:00.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143783+0000", 2024-09-15T03:48:00.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103569 2024-09-15T03:48:00.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:00.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143801+0000", 2024-09-15T03:48:00.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8077999999999999e-05 2024-09-15T03:48:00.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:00.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144077+0000", 2024-09-15T03:48:00.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027569500000000001 2024-09-15T03:48:00.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:00.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144459+0000", 2024-09-15T03:48:00.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038227299999999998 2024-09-15T03:48:00.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:00.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144519+0000", 2024-09-15T03:48:00.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9907000000000001e-05 2024-09-15T03:48:00.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:00.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144628+0000", 2024-09-15T03:48:00.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108958 2024-09-15T03:48:00.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:00.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135726+0000", 2024-09-15T03:48:00.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99109738199999997 2024-09-15T03:48:00.185 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.185 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:00.185 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.185 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 39/31 ECSubWrite(tid=41, reqid=osd.6.0:41, at_version=39'9, trim_to=0'0, roll_forward_to=38'7))", 2024-09-15T03:48:00.185 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:48:00.185 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.972886318999997, 2024-09-15T03:48:00.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99106653099999997, 2024-09-15T03:48:00.185 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:00.185 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:00.185 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:00.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:00.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:48:00.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:00.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:48:00.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:00.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:48:00.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.97e-07 2024-09-15T03:48:00.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:00.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144297+0000", 2024-09-15T03:48:00.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7815000000000001e-05 2024-09-15T03:48:00.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:00.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144314+0000", 2024-09-15T03:48:00.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6269e-05 2024-09-15T03:48:00.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:00.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144322+0000", 2024-09-15T03:48:00.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1880000000000002e-06 2024-09-15T03:48:00.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:00.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144767+0000", 2024-09-15T03:48:00.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044555300000000001 2024-09-15T03:48:00.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:00.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144777+0000", 2024-09-15T03:48:00.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5950000000000005e-06 2024-09-15T03:48:00.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:00.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144992+0000", 2024-09-15T03:48:00.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000215479 2024-09-15T03:48:00.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:00.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145142+0000", 2024-09-15T03:48:00.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149302 2024-09-15T03:48:00.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:00.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145249+0000", 2024-09-15T03:48:00.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106971 2024-09-15T03:48:00.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:00.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145299+0000", 2024-09-15T03:48:00.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0413e-05 2024-09-15T03:48:00.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:00.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135336+0000", 2024-09-15T03:48:00.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99003674900000005 2024-09-15T03:48:00.190 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.190 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:00.190 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.191 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 39/28 ECSubWrite(tid=50, reqid=osd.2.0:50, at_version=39'28, trim_to=0'0, roll_forward_to=38'26))", 2024-09-15T03:48:00.191 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:48:00.191 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.972588837000004, 2024-09-15T03:48:00.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99095576299999999, 2024-09-15T03:48:00.191 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:00.191 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:00.191 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:00.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:00.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:48:00.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:00.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:48:00.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:00.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:48:00.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.03e-07 2024-09-15T03:48:00.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:00.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144593+0000", 2024-09-15T03:48:00.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6520999999999999e-05 2024-09-15T03:48:00.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:00.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144611+0000", 2024-09-15T03:48:00.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7963000000000001e-05 2024-09-15T03:48:00.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:00.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144619+0000", 2024-09-15T03:48:00.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7409999999999992e-06 2024-09-15T03:48:00.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:00.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144821+0000", 2024-09-15T03:48:00.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000201652 2024-09-15T03:48:00.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:00.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144832+0000", 2024-09-15T03:48:00.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0827e-05 2024-09-15T03:48:00.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:00.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145099+0000", 2024-09-15T03:48:00.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026688299999999999 2024-09-15T03:48:00.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:00.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145143+0000", 2024-09-15T03:48:00.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.481e-05 2024-09-15T03:48:00.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:00.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145249+0000", 2024-09-15T03:48:00.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010618799999999999 2024-09-15T03:48:00.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:00.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145340+0000", 2024-09-15T03:48:00.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0340000000000006e-05 2024-09-15T03:48:00.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:00.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135522+0000", 2024-09-15T03:48:00.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99018263500000003 2024-09-15T03:48:00.196 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.196 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:00.196 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.196 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=43, reqid=client.4521.0:240, at_version=39'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:48:00.196 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:48:00.197 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.969248117000006, 2024-09-15T03:48:00.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99104975500000003, 2024-09-15T03:48:00.197 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:00.197 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:00.197 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:00.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:00.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:48:00.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:00.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:48:00.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:00.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147908+0000", 2024-09-15T03:48:00.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.05e-07 2024-09-15T03:48:00.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:00.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147931+0000", 2024-09-15T03:48:00.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3621e-05 2024-09-15T03:48:00.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:00.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147945+0000", 2024-09-15T03:48:00.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3648999999999999e-05 2024-09-15T03:48:00.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:00.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147951+0000", 2024-09-15T03:48:00.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0510000000000002e-06 2024-09-15T03:48:00.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:00.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148055+0000", 2024-09-15T03:48:00.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104421 2024-09-15T03:48:00.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:00.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148068+0000", 2024-09-15T03:48:00.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3019e-05 2024-09-15T03:48:00.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:00.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148375+0000", 2024-09-15T03:48:00.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030683400000000002 2024-09-15T03:48:00.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:00.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148399+0000", 2024-09-15T03:48:00.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3760999999999998e-05 2024-09-15T03:48:00.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:00.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148453+0000", 2024-09-15T03:48:00.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3851999999999999e-05 2024-09-15T03:48:00.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:00.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148481+0000", 2024-09-15T03:48:00.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8637999999999999e-05 2024-09-15T03:48:00.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:00.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.138957+0000", 2024-09-15T03:48:00.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99047580400000002 2024-09-15T03:48:00.202 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.202 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:00.202 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.202 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 39/28 ECSubWrite(tid=39, reqid=osd.11.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:48:00.202 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:48:00.202 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.969115627999997, 2024-09-15T03:48:00.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99452509, 2024-09-15T03:48:00.203 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:00.203 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:00.203 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:00.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:00.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:48:00.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:00.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:48:00.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:00.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148041+0000", 2024-09-15T03:48:00.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0999999999999998e-07 2024-09-15T03:48:00.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:00.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148052+0000", 2024-09-15T03:48:00.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.129e-05 2024-09-15T03:48:00.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:00.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148064+0000", 2024-09-15T03:48:00.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173e-05 2024-09-15T03:48:00.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:00.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148068+0000", 2024-09-15T03:48:00.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5680000000000001e-06 2024-09-15T03:48:00.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:00.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148136+0000", 2024-09-15T03:48:00.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7508999999999993e-05 2024-09-15T03:48:00.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:00.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148148+0000", 2024-09-15T03:48:00.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2002e-05 2024-09-15T03:48:00.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:00.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148265+0000", 2024-09-15T03:48:00.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117669 2024-09-15T03:48:00.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:00.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148289+0000", 2024-09-15T03:48:00.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3672e-05 2024-09-15T03:48:00.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:00.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148348+0000", 2024-09-15T03:48:00.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8838000000000001e-05 2024-09-15T03:48:00.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:00.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148368+0000", 2024-09-15T03:48:00.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9942999999999999e-05 2024-09-15T03:48:00.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:00.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142565+0000", 2024-09-15T03:48:00.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99419715900000005 2024-09-15T03:48:00.208 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.208 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:00.208 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.208 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=45, reqid=client.4521.0:241, at_version=39'17, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:48:00.208 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:48:00.208 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.966112605000006, 2024-09-15T03:48:00.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99118202200000005, 2024-09-15T03:48:00.208 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:00.208 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:00.209 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:00.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:00.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:48:00.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:00.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:48:00.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:00.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151044+0000", 2024-09-15T03:48:00.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1340000000000001e-06 2024-09-15T03:48:00.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:00.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151073+0000", 2024-09-15T03:48:00.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8894000000000002e-05 2024-09-15T03:48:00.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:00.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151095+0000", 2024-09-15T03:48:00.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1889e-05 2024-09-15T03:48:00.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:00.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151107+0000", 2024-09-15T03:48:00.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2136999999999999e-05 2024-09-15T03:48:00.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:00.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151189+0000", 2024-09-15T03:48:00.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1883000000000002e-05 2024-09-15T03:48:00.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:00.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151203+0000", 2024-09-15T03:48:00.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3825999999999999e-05 2024-09-15T03:48:00.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:00.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151305+0000", 2024-09-15T03:48:00.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101816 2024-09-15T03:48:00.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:00.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151337+0000", 2024-09-15T03:48:00.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2703000000000002e-05 2024-09-15T03:48:00.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:00.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151387+0000", 2024-09-15T03:48:00.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9366000000000001e-05 2024-09-15T03:48:00.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:00.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151405+0000", 2024-09-15T03:48:00.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8748000000000001e-05 2024-09-15T03:48:00.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:00.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142225+0000", 2024-09-15T03:48:00.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99081962599999995 2024-09-15T03:48:00.214 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.214 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:00.214 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.214 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 39/31 ECSubWrite(tid=42, reqid=osd.6.0:41, at_version=0'0, trim_to=0'0, roll_forward_to=39'9))", 2024-09-15T03:48:00.214 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:48:00.214 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.965816563999994, 2024-09-15T03:48:00.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99092639999999999, 2024-09-15T03:48:00.214 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:00.214 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:00.214 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:00.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:00.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:48:00.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:00.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:48:00.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:00.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151340+0000", 2024-09-15T03:48:00.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8299999999999996e-07 2024-09-15T03:48:00.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:00.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151350+0000", 2024-09-15T03:48:00.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0675e-05 2024-09-15T03:48:00.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:00.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151362+0000", 2024-09-15T03:48:00.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2048999999999999e-05 2024-09-15T03:48:00.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:00.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151369+0000", 2024-09-15T03:48:00.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3230000000000003e-06 2024-09-15T03:48:00.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:00.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151425+0000", 2024-09-15T03:48:00.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5964000000000002e-05 2024-09-15T03:48:00.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:00.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151435+0000", 2024-09-15T03:48:00.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9240000000000002e-06 2024-09-15T03:48:00.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:00.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151530+0000", 2024-09-15T03:48:00.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5649000000000005e-05 2024-09-15T03:48:00.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:00.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151558+0000", 2024-09-15T03:48:00.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7976000000000001e-05 2024-09-15T03:48:00.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:00.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151599+0000", 2024-09-15T03:48:00.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1152000000000002e-05 2024-09-15T03:48:00.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:00.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151617+0000", 2024-09-15T03:48:00.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7676000000000002e-05 2024-09-15T03:48:00.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:00.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142265+0000", 2024-09-15T03:48:00.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99064822900000005 2024-09-15T03:48:00.219 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.219 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:00.220 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.220 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=46, reqid=client.4521.0:242, at_version=39'18, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:48:00.220 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:48:00.220 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.965790928000004, 2024-09-15T03:48:00.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.991000204, 2024-09-15T03:48:00.220 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:00.220 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:00.220 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:00.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:00.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:48:00.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:00.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:48:00.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:00.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:48:00.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3399999999999999e-07 2024-09-15T03:48:00.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:00.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151393+0000", 2024-09-15T03:48:00.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8149000000000001e-05 2024-09-15T03:48:00.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:00.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151405+0000", 2024-09-15T03:48:00.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2063e-05 2024-09-15T03:48:00.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:00.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151415+0000", 2024-09-15T03:48:00.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4579999999999999e-06 2024-09-15T03:48:00.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:00.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151472+0000", 2024-09-15T03:48:00.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7018999999999997e-05 2024-09-15T03:48:00.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:00.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151480+0000", 2024-09-15T03:48:00.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1410000000000005e-06 2024-09-15T03:48:00.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:00.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151571+0000", 2024-09-15T03:48:00.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1243999999999996e-05 2024-09-15T03:48:00.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:00.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151602+0000", 2024-09-15T03:48:00.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0834000000000003e-05 2024-09-15T03:48:00.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:00.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151632+0000", 2024-09-15T03:48:00.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9527e-05 2024-09-15T03:48:00.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:00.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151685+0000", 2024-09-15T03:48:00.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3599e-05 2024-09-15T03:48:00.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:00.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142365+0000", 2024-09-15T03:48:00.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99067953600000003 2024-09-15T03:48:00.226 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.226 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:00.226 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.226 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 39/28 ECSubWrite(tid=51, reqid=osd.2.0:50, at_version=0'0, trim_to=0'0, roll_forward_to=39'28))", 2024-09-15T03:48:00.226 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:48:00.226 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.962427489999996, 2024-09-15T03:48:00.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98766307399999997, 2024-09-15T03:48:00.226 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:00.226 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:00.226 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:00.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:00.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:48:00.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:00.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:48:00.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:00.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154729+0000", 2024-09-15T03:48:00.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3799999999999996e-07 2024-09-15T03:48:00.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:00.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154742+0000", 2024-09-15T03:48:00.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2909e-05 2024-09-15T03:48:00.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:00.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154755+0000", 2024-09-15T03:48:00.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3251999999999999e-05 2024-09-15T03:48:00.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:00.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154761+0000", 2024-09-15T03:48:00.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6559999999999997e-06 2024-09-15T03:48:00.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:00.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154829+0000", 2024-09-15T03:48:00.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8549999999999999e-05 2024-09-15T03:48:00.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:00.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154840+0000", 2024-09-15T03:48:00.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0159999999999999e-05 2024-09-15T03:48:00.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:00.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154943+0000", 2024-09-15T03:48:00.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010347400000000001 2024-09-15T03:48:00.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:00.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154955+0000", 2024-09-15T03:48:00.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1935e-05 2024-09-15T03:48:00.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:00.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156442+0000", 2024-09-15T03:48:00.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014867859999999999 2024-09-15T03:48:00.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:00.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156466+0000", 2024-09-15T03:48:00.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4077999999999999e-05 2024-09-15T03:48:00.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:00.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142391+0000", 2024-09-15T03:48:00.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98592533599999999 2024-09-15T03:48:00.232 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.232 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:00.232 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.232 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=48, reqid=client.4521.0:242, at_version=0'0, trim_to=0'0, roll_forward_to=39'18))", 2024-09-15T03:48:00.232 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:48:00.232 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.960540471000002, 2024-09-15T03:48:00.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98592495099999999, 2024-09-15T03:48:00.232 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:00.232 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:00.233 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:00.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:00.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:48:00.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:00.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:48:00.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:00.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:00.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156616+0000", 2024-09-15T03:48:00.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1999999999999998e-07 2024-09-15T03:48:00.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:00.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156629+0000", 2024-09-15T03:48:00.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2870999999999999e-05 2024-09-15T03:48:00.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:00.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156649+0000", 2024-09-15T03:48:00.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9786e-05 2024-09-15T03:48:00.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:00.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156657+0000", 2024-09-15T03:48:00.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5110000000000003e-06 2024-09-15T03:48:00.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:00.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156723+0000", 2024-09-15T03:48:00.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6209000000000003e-05 2024-09-15T03:48:00.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:00.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156734+0000", 2024-09-15T03:48:00.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0475999999999999e-05 2024-09-15T03:48:00.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:00.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156868+0000", 2024-09-15T03:48:00.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013402999999999999 2024-09-15T03:48:00.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:00.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156889+0000", 2024-09-15T03:48:00.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1146e-05 2024-09-15T03:48:00.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:00.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156937+0000", 2024-09-15T03:48:00.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8031999999999997e-05 2024-09-15T03:48:00.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:00.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156970+0000", 2024-09-15T03:48:00.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2462999999999997e-05 2024-09-15T03:48:00.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:00.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:00.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:00.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142540+0000", 2024-09-15T03:48:00.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98557050700000004 2024-09-15T03:48:00.238 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.238 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:00.238 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.238 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:00.238 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:00.238 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:48:00.241 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:48:00.241 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:48:00.239+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:48:00.242 INFO:tasks.rados.rados.0.smithi001.stdout:395: finishing write tid 1 to smithi00132172-38 2024-09-15T03:48:00.242 INFO:tasks.rados.rados.0.smithi001.stdout:395: finishing write tid 2 to smithi00132172-38 2024-09-15T03:48:00.242 INFO:tasks.rados.rados.0.smithi001.stdout:395: finishing write tid 3 to smithi00132172-38 2024-09-15T03:48:00.242 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 38 v 84 (ObjNum 149 snap 49 seq_num 149) dirty exists 2024-09-15T03:48:00.242 INFO:tasks.rados.rados.0.smithi001.stdout:395: left oid 38 (ObjNum 149 snap 49 seq_num 149) 2024-09-15T03:48:00.243 INFO:tasks.rados.rados.0.smithi001.stdout:390: done (4 left) 2024-09-15T03:48:00.243 INFO:tasks.rados.rados.0.smithi001.stdout:391: done (3 left) 2024-09-15T03:48:00.243 INFO:tasks.rados.rados.0.smithi001.stdout:392: done (2 left) 2024-09-15T03:48:00.243 INFO:tasks.rados.rados.0.smithi001.stdout:393: done (1 left) 2024-09-15T03:48:00.243 INFO:tasks.rados.rados.0.smithi001.stdout:395: done (0 left) 2024-09-15T03:48:00.243 INFO:tasks.rados.rados.0.smithi001.stdout:396: rollback oid 22 current snap is 49 2024-09-15T03:48:00.243 INFO:tasks.rados.rados.0.smithi001.stdout:rollback oid 22 to 44 2024-09-15T03:48:00.243 INFO:tasks.rados.rados.0.smithi001.stdout:397: read oid 26 snap -1 2024-09-15T03:48:00.243 INFO:tasks.rados.rados.0.smithi001.stdout:397: expect deleted 2024-09-15T03:48:00.243 INFO:tasks.rados.rados.0.smithi001.stdout:398: copy_from oid 43 from oid 39 current snap is 49 2024-09-15T03:48:00.243 INFO:tasks.rados.rados.0.smithi001.stdout:399: copy_from oid 27 from oid 25 current snap is 49 2024-09-15T03:48:00.243 INFO:tasks.rados.rados.0.smithi001.stdout: waiting on 4 2024-09-15T03:48:00.243 INFO:tasks.rados.rados.0.smithi001.stdout:397: done (3 left) 2024-09-15T03:48:00.244 INFO:tasks.rados.rados.0.smithi001.stdout: waiting on 3 2024-09-15T03:48:00.244 INFO:tasks.rados.rados.0.smithi001.stdout:396: finishing rollback tid 1 to smithi00132172-22 2024-09-15T03:48:00.244 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 22 v 117 (ObjNum 33 snap 0 seq_num 3470555216) dirty dne 2024-09-15T03:48:00.244 INFO:tasks.rados.rados.0.smithi001.stdout:396: done (2 left) 2024-09-15T03:48:00.244 INFO:tasks.rados.rados.0.smithi001.stdout: waiting on 2 2024-09-15T03:48:00.244 INFO:tasks.rados.rados.0.smithi001.stdout:398: finishing copy_from racing read to smithi00132172-43 2024-09-15T03:48:00.244 INFO:tasks.rados.rados.0.smithi001.stdout:398: finishing copy_from to smithi00132172-43 2024-09-15T03:48:00.244 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 43 v 116 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-15T03:48:00.244 INFO:tasks.rados.rados.0.smithi001.stdout:398: done (1 left) 2024-09-15T03:48:00.244 INFO:tasks.rados.rados.0.smithi001.stdout: waiting on 1 2024-09-15T03:48:00.244 INFO:tasks.rados.rados.0.smithi001.stdout:399: finishing copy_from racing read to smithi00132172-27 2024-09-15T03:48:00.244 INFO:tasks.rados.rados.0.smithi001.stdout:399: finishing copy_from to smithi00132172-27 2024-09-15T03:48:00.244 INFO:tasks.rados.rados.0.smithi001.stdout:update_object_version oid 27 v 156 (ObjNum 66 snap 6 seq_num 66) dirty exists 2024-09-15T03:48:00.244 INFO:tasks.rados.rados.0.smithi001.stdout:399: done (0 left) 2024-09-15T03:48:00.245 INFO:tasks.rados.rados.0.smithi001.stderr:0 errors. 2024-09-15T03:48:00.245 INFO:tasks.rados.rados.0.smithi001.stderr: 2024-09-15T03:48:00.246 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:48:00.340 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:48:00.340+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:48:00.381 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:48:00.382 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:48:00.382 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:48:00.382 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:48:00.382 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:48:00.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:48:00.441 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:48:00.440+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:48:00.490 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:48:00.491 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:48:00.491 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:48:00.491 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:48:00.491 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:48:00.499 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:48:00.499 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:48:00.500 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:48:00.515 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:48:00.515 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:48:00.530 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:48:00.531 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:48:00.542 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:48:00.542+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:48:00.545 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:48:00.545 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:48:00.559 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:48:00.559 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:48:00.567 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:48:00.567 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":113,"fsid":"f27b540e-e5e3-4e95-afe8-c95ee66b0a62","created":"2024-09-15T03:46:09.843094+0000","modified":"2024-09-15T03:47:59.935524+0000","last_up_change":"2024-09-15T03:47:51.883425+0000","last_in_change":"2024-09-15T03:47:32.982084+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:46:16.185530+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:46:22.284406+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":24,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-09-15T03:46:34.105566+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":8,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":76,"snap_epoch":113,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lrcprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"cd433a6a-c875-4e6e-b89c-938caecbae3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":28908},{"type":"v1","addr":"172.21.15.1:6801","nonce":28908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":28908},{"type":"v1","addr":"172.21.15.1:6803","nonce":28908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":28908},{"type":"v1","addr":"172.21.15.1:6807","nonce":28908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":28908},{"type":"v1","addr":"172.21.15.1:6805","nonce":28908}]},"public_addr":"172.21.15.1:6801/28908","cluster_addr":"172.21.15.1:6803/28908","heartbeat_back_addr":"172.21.15.1:6807/28908","heartbeat_front_addr":"172.21.15.1:6805/28908","state":["exists","up"]},{"osd":1,"uuid":"789930f9-42bf-476b-9f09-a21bffe33a79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":28913},{"type":"v1","addr":"172.21.15.1:6809","nonce":28913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":28913},{"type":"v1","addr":"172.21.15.1:6811","nonce":28913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":28913},{"type":"v1","addr":"172.21.15.1:6815","nonce":28913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":28913},{"type":"v1","addr":"172.21.15.1:6813","nonce":28913}]},"public_addr":"172.21.15.1:6809/28913","cluster_addr":"172.21.15.1:6811/28913","heartbeat_back_addr":"172.21.15.1:6815/28913","heartbeat_front_addr":"172.21.15.1:6813/28913","state":["exists","up"]},{"osd":2,"uuid":"9a83fec5-0d63-461c-b516-289443b3882b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":28911},{"type":"v1","addr":"172.21.15.1:6823","nonce":28911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":28911},{"type":"v1","addr":"172.21.15.1:6827","nonce":28911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":28911},{"type":"v1","addr":"172.21.15.1:6831","nonce":28911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":28911},{"type":"v1","addr":"172.21.15.1:6829","nonce":28911}]},"public_addr":"172.21.15.1:6823/28911","cluster_addr":"172.21.15.1:6827/28911","heartbeat_back_addr":"172.21.15.1:6831/28911","heartbeat_front_addr":"172.21.15.1:6829/28911","state":["exists","up"]},{"osd":3,"uuid":"d944bf89-e183-496a-9ca3-7949741b4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":28914},{"type":"v1","addr":"172.21.15.1:6817","nonce":28914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":28914},{"type":"v1","addr":"172.21.15.1:6819","nonce":28914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":28914},{"type":"v1","addr":"172.21.15.1:6826","nonce":28914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":28914},{"type":"v1","addr":"172.21.15.1:6822","nonce":28914}]},"public_addr":"172.21.15.1:6817/28914","cluster_addr":"172.21.15.1:6819/28914","heartbeat_back_addr":"172.21.15.1:6826/28914","heartbeat_front_addr":"172.21.15.1:6822/28914","state":["exists","up"]},{"osd":4,"uuid":"15ec2dbd-d099-4aff-94b2-1bf9dbb00b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":28468},{"type":"v1","addr":"172.21.15.67:6805","nonce":28468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6807","nonce":28468},{"type":"v1","addr":"172.21.15.67:6809","nonce":28468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":28468},{"type":"v1","addr":"172.21.15.67:6815","nonce":28468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":28468},{"type":"v1","addr":"172.21.15.67:6813","nonce":28468}]},"public_addr":"172.21.15.67:6805/28468","cluster_addr":"172.21.15.67:6809/28468","heartbeat_back_addr":"172.21.15.67:6815/28468","heartbeat_front_addr":"172.21.15.67:6813/28468","state":["exists","up"]},{"osd":5,"uuid":"1e45de07-746b-4612-9a52-d05509cae9a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":28495},{"type":"v1","addr":"172.21.15.67:6825","nonce":28495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":28495},{"type":"v1","addr":"172.21.15.67:6827","nonce":28495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":28495},{"type":"v1","addr":"172.21.15.67:6831","nonce":28495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":28495},{"type":"v1","addr":"172.21.15.67:6829","nonce":28495}]},"public_addr":"172.21.15.67:6825/28495","cluster_addr":"172.21.15.67:6827/28495","heartbeat_back_addr":"172.21.15.67:6831/28495","heartbeat_front_addr":"172.21.15.67:6829/28495","state":["exists","up"]},{"osd":6,"uuid":"ccf785f8-751d-473e-b17c-3df9346be9f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":28469},{"type":"v1","addr":"172.21.15.67:6801","nonce":28469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":28469},{"type":"v1","addr":"172.21.15.67:6804","nonce":28469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":28469},{"type":"v1","addr":"172.21.15.67:6812","nonce":28469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":28469},{"type":"v1","addr":"172.21.15.67:6808","nonce":28469}]},"public_addr":"172.21.15.67:6801/28469","cluster_addr":"172.21.15.67:6804/28469","heartbeat_back_addr":"172.21.15.67:6812/28469","heartbeat_front_addr":"172.21.15.67:6808/28469","state":["exists","up"]},{"osd":7,"uuid":"f9f59987-869d-42eb-9eb6-08c39e3afe6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":28470},{"type":"v1","addr":"172.21.15.67:6817","nonce":28470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":28470},{"type":"v1","addr":"172.21.15.67:6819","nonce":28470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":28470},{"type":"v1","addr":"172.21.15.67:6823","nonce":28470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":28470},{"type":"v1","addr":"172.21.15.67:6821","nonce":28470}]},"public_addr":"172.21.15.67:6817/28470","cluster_addr":"172.21.15.67:6819/28470","heartbeat_back_addr":"172.21.15.67:6823/28470","heartbeat_front_addr":"172.21.15.67:6821/28470","state":["exists","up"]},{"osd":8,"uuid":"69ddfe52-11df-4608-9a10-e8907c4cd87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":29472},{"type":"v1","addr":"172.21.15.190:6809","nonce":29472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":29472},{"type":"v1","addr":"172.21.15.190:6811","nonce":29472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":29472},{"type":"v1","addr":"172.21.15.190:6815","nonce":29472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":29472},{"type":"v1","addr":"172.21.15.190:6813","nonce":29472}]},"public_addr":"172.21.15.190:6809/29472","cluster_addr":"172.21.15.190:6811/29472","heartbeat_back_addr":"172.21.15.190:6815/29472","heartbeat_front_addr":"172.21.15.190:6813/29472","state":["exists","up"]},{"osd":9,"uuid":"f92e1609-2a1a-4814-9ac0-44a73373d229","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":29473},{"type":"v1","addr":"172.21.15.190:6801","nonce":29473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":29473},{"type":"v1","addr":"172.21.15.190:6803","nonce":29473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":29473},{"type":"v1","addr":"172.21.15.190:6807","nonce":29473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":29473},{"type":"v1","addr":"172.21.15.190:6805","nonce":29473}]},"public_addr":"172.21.15.190:6801/29473","cluster_addr":"172.21.15.190:6803/29473","heartbeat_back_addr":"172.21.15.190:6807/29473","heartbeat_front_addr":"172.21.15.190:6805/29473","state":["exists","up"]},{"osd":10,"uuid":"5f2e8ca3-8c65-4595-8e82-27dc3768ab99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":78,"up_from":105,"up_thru":108,"down_at":80,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":33741},{"type":"v1","addr":"172.21.15.190:6817","nonce":33741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":33741},{"type":"v1","addr":"172.21.15.190:6819","nonce":33741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":33741},{"type":"v1","addr":"172.21.15.190:6823","nonce":33741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":33741},{"type":"v1","addr":"172.21.15.190:6821","nonce":33741}]},"public_addr":"172.21.15.190:6817/33741","cluster_addr":"172.21.15.190:6819/33741","heartbeat_back_addr":"172.21.15.190:6823/33741","heartbeat_front_addr":"172.21.15.190:6821/33741","state":["exists","up"]},{"osd":11,"uuid":"cedf7df4-f875-42f5-a8cf-13cb82f05cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":29475},{"type":"v1","addr":"172.21.15.190:6825","nonce":29475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":29475},{"type":"v1","addr":"172.21.15.190:6827","nonce":29475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":29475},{"type":"v1","addr":"172.21.15.190:6831","nonce":29475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":29475},{"type":"v1","addr":"172.21.15.190:6829","nonce":29475}]},"public_addr":"172.21.15.190:6825/29475","cluster_addr":"172.21.15.190:6827/29475","heartbeat_back_addr":"172.21.15.190:6831/29475","heartbeat_front_addr":"172.21.15.190:6829/29475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:37.565633+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:18.975241+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:25.983954+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:26.751379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:25.908552+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:34.158155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:37.031163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:18.853134+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:29.597174+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:37.409413+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:51.125326+0000","dead_epoch":80},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:27.962448+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.a","osds":[4,10,8,9,5,6,2,0]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lrcprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","l":"3","m":"2","name":"lrcprofile","plugin":"lrc"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":1},{"begin":12,"length":1},{"begin":18,"length":1},{"begin":23,"length":2},{"begin":32,"length":1},{"begin":35,"length":2},{"begin":38,"length":1},{"begin":41,"length":1},{"begin":47,"length":1},{"begin":50,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":36,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:48:00.573 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:48:00.574 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:48:00.580 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:48:00.588 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:48:00.589 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:48:00.601 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:00.601 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:00.603 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:48:00.603 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:48:00.609 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:48:00.618 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:48:00.619 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:48:00.633 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:48:00.634 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:48:00.643 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:48:00.642+0000 7fd62abf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33736) UID: 0 2024-09-15T03:48:00.648 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:48:00.649 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:48:00.664 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:48:00.664 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:48:00.744 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:48:00.744+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:48:00.751 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:00.751 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:00.760 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:48:00.845 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:48:00.844+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:48:00.866 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:48:00.867 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-15T03:48:00.877 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-09-15T03:48:00.162054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":174866995,"num_objects":183,"num_object_clones":129,"num_object_copies":1440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":9,"num_read":509,"num_read_kb":217205,"num_write":994,"num_write_kb":161880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":267,"num_bytes_recovered":285786150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1284,"ondisk_log_size":1284,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":12,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":1124155392,"kb_used":10198188,"kb_used_data":1113989424,"kb_used_omap":45956,"kb_used_meta":0,"kb_avail":1113957204,"statfs":{"total":1151135121408,"available":1140692176896,"internally_reserved":32993280,"allocated":1140725170176,"data_stored":1140725170176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47059824,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,7],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":20,"commit_latency_ns":0,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2375680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000839"},"pg_stats":[{"pgid":"3.f","version":"109'59","reported_seq":419,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-15T03:47:55.916775+0000","last_change":"2024-09-15T03:47:55.917413+0000","last_active":"2024-09-15T03:47:55.916775+0000","last_peered":"2024-09-15T03:47:55.916775+0000","last_clean":"2024-09-15T03:47:55.916775+0000","last_became_active":"2024-09-15T03:47:55.850051+0000","last_became_peered":"2024-09-15T03:47:55.850051+0000","last_unstale":"2024-09-15T03:47:55.916775+0000","last_undegraded":"2024-09-15T03:47:55.916775+0000","last_fullsized":"2024-09-15T03:47:55.916775+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.064057156000000004,"stat_sum":{"num_bytes":8028160,"num_objects":8,"num_object_clones":5,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":32,"num_read_kb":42506,"num_write":26,"num_write_kb":7840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0,7,2,8,4,10],"acting":[11,1,0,7,2,8,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"c","length":"1"},{"start":"17","length":"2"},{"start":"23","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"},{"start":"2f","length":"1"},{"start":"32","length":"1"}]},{"pgid":"3.e","version":"107'173","reported_seq":386,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-15T03:47:55.844164+0000","last_change":"2024-09-15T03:47:54.108846+0000","last_active":"2024-09-15T03:47:55.844164+0000","last_peered":"2024-09-15T03:47:55.844164+0000","last_clean":"2024-09-15T03:47:55.844164+0000","last_became_active":"2024-09-15T03:46:36.170536+0000","last_became_peered":"2024-09-15T03:46:36.170536+0000","last_unstale":"2024-09-15T03:47:55.844164+0000","last_undegraded":"2024-09-15T03:47:55.844164+0000","last_fullsized":"2024-09-15T03:47:55.844164+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'173","last_scrub_stamp":"2024-09-15T03:47:54.108780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:47:54.108780+0000","objects_scrubbed":21,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:11.038305+0000","scrub_duration":0.0088561600000000001,"objects_trimmed":0,"snaptrim_duration":0.0152789,"stat_sum":{"num_bytes":17596416,"num_objects":21,"num_object_clones":14,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":63,"num_read_kb":20614,"num_write":49,"num_write_kb":15504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,11,7,8,5,9],"acting":[2,0,1,11,7,8,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"},{"start":"c","length":"1"},{"start":"17","length":"2"},{"start":"23","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"},{"start":"2f","length":"1"},{"start":"32","length":"1"}]},{"pgid":"3.d","version":"109'62","reported_seq":356,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-15T03:47:55.918059+0000","last_change":"2024-09-15T03:47:55.918959+0000","last_active":"2024-09-15T03:47:55.918059+0000","last_peered":"2024-09-15T03:47:55.918059+0000","last_clean":"2024-09-15T03:47:55.918059+0000","last_became_active":"2024-09-15T03:47:55.850104+0000","last_became_peered":"2024-09-15T03:47:55.850104+0000","last_unstale":"2024-09-15T03:47:55.918059+0000","last_undegraded":"2024-09-15T03:47:55.918059+0000","last_fullsized":"2024-09-15T03:47:55.918059+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.068063701000000004,"stat_sum":{"num_bytes":5931008,"num_objects":10,"num_object_clones":8,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":19,"num_read_kb":8406,"num_write":42,"num_write_kb":5792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":10780672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,1,3,10,7,9,4],"acting":[8,11,1,3,10,7,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"c","length":"1"},{"start":"17","length":"2"},{"start":"23","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"},{"start":"2f","length":"1"},{"start":"32","length":"1"}]},{"pgid":"3.c","version":"107'20","reported_seq":351,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-15T03:47:54.847412+0000","last_change":"2024-09-15T03:47:53.917525+0000","last_active":"2024-09-15T03:47:54.847412+0000","last_peered":"2024-09-15T03:47:54.847412+0000","last_clean":"2024-09-15T03:47:54.847412+0000","last_became_active":"2024-09-15T03:47:36.848722+0000","last_became_peered":"2024-09-15T03:47:36.848722+0000","last_unstale":"2024-09-15T03:47:54.847412+0000","last_undegraded":"2024-09-15T03:47:54.847412+0000","last_fullsized":"2024-09-15T03:47:54.847412+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'16","last_scrub_stamp":"2024-09-15T03:47:41.493521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:47:41.493521+0000","objects_scrubbed":3,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:09.644586+0000","scrub_duration":0.0039888800000000002,"objects_trimmed":0,"snaptrim_duration":0.0016633049999999999,"stat_sum":{"num_bytes":2965504,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":14,"num_read_kb":5045,"num_write":10,"num_write_kb":3440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,11,1,0,6,5,2],"acting":[9,7,11,1,0,6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"},{"start":"c","length":"1"},{"start":"17","length":"2"},{"start":"23","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"},{"start":"2f","length":"1"},{"start":"32","length":"1"}]},{"pgid":"3.b","version":"109'175","reported_seq":703,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-15T03:47:55.979678+0000","last_change":"2024-09-15T03:47:55.979678+0000","last_active":"2024-09-15T03:47:55.979678+0000","last_peered":"2024-09-15T03:47:55.979678+0000","last_clean":"2024-09-15T03:47:55.979678+0000","last_became_active":"2024-09-15T03:47:55.850099+0000","last_became_peered":"2024-09-15T03:47:55.850099+0000","last_unstale":"2024-09-15T03:47:55.979678+0000","last_undegraded":"2024-09-15T03:47:55.979678+0000","last_fullsized":"2024-09-15T03:47:55.979678+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.12836329799999999,"stat_sum":{"num_bytes":24117248,"num_objects":26,"num_object_clones":19,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":2,"num_read":65,"num_read_kb":36889,"num_write":168,"num_write_kb":22832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":74006528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,8,4,1,10,11,3],"acting":[7,6,8,4,1,10,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"c","length":"1"},{"start":"17","length":"2"},{"start":"23","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"},{"start":"2f","length":"1"},{"start":"32","length":"1"}]},{"pgid":"3.a","version":"109'73","reported_seq":198,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-15T03:47:55.922453+0000","last_change":"2024-09-15T03:47:55.923564+0000","last_active":"2024-09-15T03:47:55.922453+0000","last_peered":"2024-09-15T03:47:55.922453+0000","last_clean":"2024-09-15T03:47:55.922453+0000","last_became_active":"2024-09-15T03:47:55.854826+0000","last_became_peered":"2024-09-15T03:47:55.854826+0000","last_unstale":"2024-09-15T03:47:55.922453+0000","last_undegraded":"2024-09-15T03:47:55.922453+0000","last_fullsized":"2024-09-15T03:47:55.922453+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.067400532999999999,"stat_sum":{"num_bytes":6897664,"num_objects":7,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":15,"num_read_kb":6727,"num_write":28,"num_write_kb":6736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":30703616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8,9,5,6,2,0],"acting":[4,10,8,9,5,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"c","length":"1"},{"start":"17","length":"2"},{"start":"23","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"},{"start":"2f","length":"1"},{"start":"32","length":"1"}]},{"pgid":"3.9","version":"109'72","reported_seq":463,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-15T03:47:56.070155+0000","last_change":"2024-09-15T03:47:55.958401+0000","last_active":"2024-09-15T03:47:56.070155+0000","last_peered":"2024-09-15T03:47:56.070155+0000","last_clean":"2024-09-15T03:47:56.070155+0000","last_became_active":"2024-09-15T03:47:55.851394+0000","last_became_peered":"2024-09-15T03:47:55.851394+0000","last_unstale":"2024-09-15T03:47:56.070155+0000","last_undegraded":"2024-09-15T03:47:56.070155+0000","last_fullsized":"2024-09-15T03:47:56.070155+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:02.003280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105531164,"stat_sum":{"num_bytes":12025856,"num_objects":13,"num_object_clones":10,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":15,"num_read_kb":8,"num_write":59,"num_write_kb":11744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":27590656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,10,1,9,6,8,11],"acting":[3,2,10,1,9,6,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"c","length":"1"},{"start":"17","length":"2"},{"start":"23","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"},{"start":"2f","length":"1"},{"start":"32","length":"1"}]},{"pgid":"3.8","version":"109'40","reported_seq":345,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-15T03:47:55.938749+0000","last_change":"2024-09-15T03:47:55.939172+0000","last_active":"2024-09-15T03:47:55.938749+0000","last_peered":"2024-09-15T03:47:55.938749+0000","last_clean":"2024-09-15T03:47:55.938749+0000","last_became_active":"2024-09-15T03:47:55.848776+0000","last_became_peered":"2024-09-15T03:47:55.848776+0000","last_unstale":"2024-09-15T03:47:55.938749+0000","last_undegraded":"2024-09-15T03:47:55.938749+0000","last_fullsized":"2024-09-15T03:47:55.938749+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.088992801999999996,"stat_sum":{"num_bytes":4079616,"num_objects":6,"num_object_clones":4,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":23,"num_read_kb":4616,"num_write":54,"num_write_kb":3984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":11223040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,11,4,1,0,10],"acting":[3,7,2,11,4,1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"c","length":"1"},{"start":"17","length":"2"},{"start":"23","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"},{"start":"2f","length":"1"},{"start":"32","length":"1"}]},{"pgid":"1.0","version":"20'92","reported_seq":217,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-15T03:47:54.841001+0000","last_change":"2024-09-15T03:47:39.774326+0000","last_active":"2024-09-15T03:47:54.841001+0000","last_peered":"2024-09-15T03:47:54.841001+0000","last_clean":"2024-09-15T03:47:54.841001+0000","last_became_active":"2024-09-15T03:46:18.011130+0000","last_became_peered":"2024-09-15T03:46:18.011130+0000","last_unstale":"2024-09-15T03:47:54.841001+0000","last_undegraded":"2024-09-15T03:47:54.841001+0000","last_fullsized":"2024-09-15T03:47:54.841001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-15T03:47:39.774262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_clean_scrub_stamp":"2024-09-15T03:47:39.774262+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:43.691013+0000","scrub_duration":0.0027413759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"22'1","reported_seq":109,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-15T03:47:54.841072+0000","last_change":"2024-09-15T03:47:52.292975+0000","last_active":"2024-09-15T03:47:54.841072+0000","last_peered":"2024-09-15T03:47:54.841072+0000","last_clean":"2024-09-15T03:47:54.841072+0000","last_became_active":"2024-09-15T03:46:24.038633+0000","last_became_peered":"2024-09-15T03:46:24.038633+0000","last_unstale":"2024-09-15T03:47:54.841072+0000","last_undegraded":"2024-09-15T03:47:54.841072+0000","last_fullsized":"2024-09-15T03:47:54.841072+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'1","last_scrub_stamp":"2024-09-15T03:47:52.292926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:52.292926+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:56.362597+0000","scrub_duration":0.001409429,"objects_trimmed":0,"snaptrim_duration":0.00030178200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"109'68","reported_seq":479,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-15T03:47:56.059792+0000","last_change":"2024-09-15T03:47:55.951927+0000","last_active":"2024-09-15T03:47:56.059792+0000","last_peered":"2024-09-15T03:47:56.059792+0000","last_clean":"2024-09-15T03:47:56.059792+0000","last_became_active":"2024-09-15T03:47:55.854075+0000","last_became_peered":"2024-09-15T03:47:55.854075+0000","last_unstale":"2024-09-15T03:47:56.059792+0000","last_undegraded":"2024-09-15T03:47:56.059792+0000","last_fullsized":"2024-09-15T03:47:56.059792+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:57.786019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.096480027999999995,"stat_sum":{"num_bytes":13320192,"num_objects":10,"num_object_clones":7,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":33,"num_read_kb":11534,"num_write":39,"num_write_kb":13088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":18628608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,7,8,6,11,1],"acting":[4,2,10,7,8,6,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"c","length":"1"},{"start":"17","length":"2"},{"start":"23","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"},{"start":"2f","length":"1"},{"start":"32","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":106,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-15T03:47:54.840918+0000","last_change":"2024-09-15T03:47:27.779257+0000","last_active":"2024-09-15T03:47:54.840918+0000","last_peered":"2024-09-15T03:47:54.840918+0000","last_clean":"2024-09-15T03:47:54.840918+0000","last_became_active":"2024-09-15T03:46:24.063106+0000","last_became_peered":"2024-09-15T03:46:24.063106+0000","last_unstale":"2024-09-15T03:47:54.840918+0000","last_undegraded":"2024-09-15T03:47:54.840918+0000","last_fullsized":"2024-09-15T03:47:54.840918+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.779210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.779210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:42.717969+0000","scrub_duration":0.0021030710000000002,"objects_trimmed":0,"snaptrim_duration":0.000393075,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"109'23","reported_seq":136,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-15T03:47:56.058237+0000","last_change":"2024-09-15T03:47:55.877263+0000","last_active":"2024-09-15T03:47:56.058237+0000","last_peered":"2024-09-15T03:47:56.058237+0000","last_clean":"2024-09-15T03:47:56.058237+0000","last_became_active":"2024-09-15T03:47:55.848033+0000","last_became_peered":"2024-09-15T03:47:55.848033+0000","last_unstale":"2024-09-15T03:47:56.058237+0000","last_undegraded":"2024-09-15T03:47:56.058237+0000","last_fullsized":"2024-09-15T03:47:56.058237+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:00.864034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.028277915000000001,"stat_sum":{"num_bytes":4653056,"num_objects":5,"num_object_clones":4,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":3075,"num_write":12,"num_write_kb":4032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":6045696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3,10,5,2,9,7],"acting":[6,11,3,10,5,2,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"},{"start":"c","length":"1"},{"start":"17","length":"2"},{"start":"23","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"},{"start":"2f","length":"1"},{"start":"32","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":110,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-15T03:47:54.838596+0000","last_change":"2024-09-15T03:47:42.997012+0000","last_active":"2024-09-15T03:47:54.838596+0000","last_peered":"2024-09-15T03:47:54.838596+0000","last_clean":"2024-09-15T03:47:54.838596+0000","last_became_active":"2024-09-15T03:46:24.037471+0000","last_became_peered":"2024-09-15T03:46:24.037471+0000","last_unstale":"2024-09-15T03:47:54.838596+0000","last_undegraded":"2024-09-15T03:47:54.838596+0000","last_fullsized":"2024-09-15T03:47:54.838596+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:42.996957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:42.996957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:53.443125+0000","scrub_duration":0.0014164309999999999,"objects_trimmed":0,"snaptrim_duration":0.00026490299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"111'54","reported_seq":435,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-15T03:47:57.943707+0000","last_change":"2024-09-15T03:47:57.944789+0000","last_active":"2024-09-15T03:47:57.943707+0000","last_peered":"2024-09-15T03:47:57.943707+0000","last_clean":"2024-09-15T03:47:57.943707+0000","last_became_active":"2024-09-15T03:47:55.848512+0000","last_became_peered":"2024-09-15T03:47:55.848512+0000","last_unstale":"2024-09-15T03:47:57.943707+0000","last_undegraded":"2024-09-15T03:47:57.943707+0000","last_fullsized":"2024-09-15T03:47:57.943707+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:02.552307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012879249000000001,"stat_sum":{"num_bytes":4997120,"num_objects":8,"num_object_clones":5,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":20,"num_read_kb":7496,"num_write":43,"num_write_kb":4880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":16449536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,9,4,3,6,0,10],"acting":[1,2,9,4,3,6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"c","length":"1"},{"start":"12","length":"1"},{"start":"17","length":"2"},{"start":"20","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"},{"start":"2f","length":"1"},{"start":"32","length":"1"}]},{"pgid":"2.2","version":"24'2","reported_seq":142,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-15T03:47:54.843021+0000","last_change":"2024-09-15T03:47:36.848438+0000","last_active":"2024-09-15T03:47:54.843021+0000","last_peered":"2024-09-15T03:47:54.843021+0000","last_clean":"2024-09-15T03:47:54.843021+0000","last_became_active":"2024-09-15T03:47:36.848076+0000","last_became_peered":"2024-09-15T03:47:36.848076+0000","last_unstale":"2024-09-15T03:47:54.843021+0000","last_undegraded":"2024-09-15T03:47:54.843021+0000","last_fullsized":"2024-09-15T03:47:54.843021+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:46:23.024039+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000944677,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"109'132","reported_seq":340,"reported_epoch":109,"state":"active+clean+snaptrim","last_fresh":"2024-09-15T03:47:55.897006+0000","last_change":"2024-09-15T03:47:55.856095+0000","last_active":"2024-09-15T03:47:55.897006+0000","last_peered":"2024-09-15T03:47:55.897006+0000","last_clean":"2024-09-15T03:47:55.897006+0000","last_became_active":"2024-09-15T03:47:55.855061+0000","last_became_peered":"2024-09-15T03:47:55.855061+0000","last_unstale":"2024-09-15T03:47:55.897006+0000","last_undegraded":"2024-09-15T03:47:55.897006+0000","last_fullsized":"2024-09-15T03:47:55.897006+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047245120000000002,"stat_sum":{"num_bytes":20398080,"num_objects":23,"num_object_clones":17,"num_object_copies":184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":25,"num_read_kb":350,"num_write":165,"num_write_kb":21296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,4,7,1,2,9,0],"acting":[10,8,4,7,1,2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.5","version":"109'55","reported_seq":303,"reported_epoch":109,"state":"active+clean+snaptrim","last_fresh":"2024-09-15T03:47:55.888378+0000","last_change":"2024-09-15T03:47:55.888415+0000","last_active":"2024-09-15T03:47:55.888378+0000","last_peered":"2024-09-15T03:47:55.888378+0000","last_clean":"2024-09-15T03:47:55.888378+0000","last_became_active":"2024-09-15T03:47:55.860659+0000","last_became_peered":"2024-09-15T03:47:55.860659+0000","last_unstale":"2024-09-15T03:47:55.888378+0000","last_undegraded":"2024-09-15T03:47:55.888378+0000","last_fullsized":"2024-09-15T03:47:55.888378+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:59.584012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025439059999999999,"stat_sum":{"num_bytes":20332544,"num_objects":8,"num_object_clones":7,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":12,"num_read_kb":11047,"num_write":25,"num_write_kb":9680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":23789568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4,9,8,7,3,0],"acting":[10,6,4,9,8,7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"c","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":128,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-15T03:47:54.838928+0000","last_change":"2024-09-15T03:47:39.930675+0000","last_active":"2024-09-15T03:47:54.838928+0000","last_peered":"2024-09-15T03:47:54.838928+0000","last_clean":"2024-09-15T03:47:54.838928+0000","last_became_active":"2024-09-15T03:47:33.998802+0000","last_became_peered":"2024-09-15T03:47:33.998802+0000","last_unstale":"2024-09-15T03:47:54.838928+0000","last_undegraded":"2024-09-15T03:47:54.838928+0000","last_fullsized":"2024-09-15T03:47:54.838928+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:39.930628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:39.930628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:03.535396+0000","scrub_duration":0.001146333,"objects_trimmed":0,"snaptrim_duration":0.00034123999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"109'36","reported_seq":361,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-15T03:47:55.889836+0000","last_change":"2024-09-15T03:47:55.890132+0000","last_active":"2024-09-15T03:47:55.889836+0000","last_peered":"2024-09-15T03:47:55.889836+0000","last_clean":"2024-09-15T03:47:55.889836+0000","last_became_active":"2024-09-15T03:47:55.846248+0000","last_became_peered":"2024-09-15T03:47:55.846248+0000","last_unstale":"2024-09-15T03:47:55.889836+0000","last_undegraded":"2024-09-15T03:47:55.889836+0000","last_fullsized":"2024-09-15T03:47:55.889836+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.043040152999999998,"stat_sum":{"num_bytes":7389184,"num_objects":5,"num_object_clones":4,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":6052,"num_write":11,"num_write_kb":7952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":9814016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5,8,0,4,7,10],"acting":[6,2,5,8,0,4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"},{"start":"c","length":"1"},{"start":"17","length":"2"},{"start":"23","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"},{"start":"2f","length":"1"},{"start":"32","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":108,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-15T03:47:54.839996+0000","last_change":"2024-09-15T03:47:30.025153+0000","last_active":"2024-09-15T03:47:54.839996+0000","last_peered":"2024-09-15T03:47:54.839996+0000","last_clean":"2024-09-15T03:47:54.839996+0000","last_became_active":"2024-09-15T03:46:24.038945+0000","last_became_peered":"2024-09-15T03:46:24.038945+0000","last_unstale":"2024-09-15T03:47:54.839996+0000","last_undegraded":"2024-09-15T03:47:54.839996+0000","last_fullsized":"2024-09-15T03:47:54.839996+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:30.025107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:30.025107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:36.231260+0000","scrub_duration":0.0013039429999999999,"objects_trimmed":0,"snaptrim_duration":0.00032787499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"109'113","reported_seq":455,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-15T03:47:55.964316+0000","last_change":"2024-09-15T03:47:55.964316+0000","last_active":"2024-09-15T03:47:55.964316+0000","last_peered":"2024-09-15T03:47:55.964316+0000","last_clean":"2024-09-15T03:47:55.964316+0000","last_became_active":"2024-09-15T03:47:55.848466+0000","last_became_peered":"2024-09-15T03:47:55.848466+0000","last_unstale":"2024-09-15T03:47:55.964316+0000","last_undegraded":"2024-09-15T03:47:55.964316+0000","last_fullsized":"2024-09-15T03:47:55.964316+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.111384919,"stat_sum":{"num_bytes":16449536,"num_objects":18,"num_object_clones":13,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":60,"num_read_kb":52261,"num_write":108,"num_write_kb":15952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":27639808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,7,2,1,11,10],"acting":[4,3,9,7,2,1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"c","length":"1"},{"start":"17","length":"2"},{"start":"23","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"},{"start":"2f","length":"1"},{"start":"32","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":109,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-15T03:47:54.847630+0000","last_change":"2024-09-15T03:47:27.297139+0000","last_active":"2024-09-15T03:47:54.847630+0000","last_peered":"2024-09-15T03:47:54.847630+0000","last_clean":"2024-09-15T03:47:54.847630+0000","last_became_active":"2024-09-15T03:46:24.063380+0000","last_became_peered":"2024-09-15T03:46:24.063380+0000","last_unstale":"2024-09-15T03:47:54.847630+0000","last_undegraded":"2024-09-15T03:47:54.847630+0000","last_fullsized":"2024-09-15T03:47:54.847630+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.297083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.297083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:48.875998+0000","scrub_duration":0.0018328789999999999,"objects_trimmed":0,"snaptrim_duration":0.00034490600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"109'34","reported_seq":454,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-15T03:47:55.951898+0000","last_change":"2024-09-15T03:47:55.952365+0000","last_active":"2024-09-15T03:47:55.951898+0000","last_peered":"2024-09-15T03:47:55.951898+0000","last_clean":"2024-09-15T03:47:55.951898+0000","last_became_active":"2024-09-15T03:47:55.847868+0000","last_became_peered":"2024-09-15T03:47:55.847868+0000","last_unstale":"2024-09-15T03:47:55.951898+0000","last_undegraded":"2024-09-15T03:47:55.951898+0000","last_fullsized":"2024-09-15T03:47:55.951898+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:02.071140+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.10330456,"stat_sum":{"num_bytes":5095424,"num_objects":7,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":12,"num_read_kb":515,"num_write":30,"num_write_kb":4976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":7176192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,9,8,3,11,1,10],"acting":[4,7,9,8,3,11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"c","length":"1"},{"start":"17","length":"2"},{"start":"23","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"},{"start":"2f","length":"1"},{"start":"32","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":106,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-15T03:47:54.843094+0000","last_change":"2024-09-15T03:47:42.675933+0000","last_active":"2024-09-15T03:47:54.843094+0000","last_peered":"2024-09-15T03:47:54.843094+0000","last_clean":"2024-09-15T03:47:54.843094+0000","last_became_active":"2024-09-15T03:46:24.063335+0000","last_became_peered":"2024-09-15T03:46:24.063335+0000","last_unstale":"2024-09-15T03:47:54.843094+0000","last_undegraded":"2024-09-15T03:47:54.843094+0000","last_fullsized":"2024-09-15T03:47:54.843094+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:42.675882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:42.675882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:02.697036+0000","scrub_duration":0.001445478,"objects_trimmed":0,"snaptrim_duration":0.00065928000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":174276608,"num_objects":179,"num_object_clones":129,"num_object_copies":1432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":9,"num_read":433,"num_read_kb":217141,"num_write":869,"num_write_kb":159728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":265,"num_bytes_recovered":285786112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1189,"ondisk_log_size":1189,"up":128,"acting":128,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":11,"up_from":17,"seq":73014444054,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":842472,"kb_used_data":92838188,"kb_used_omap":3786,"kb_used_meta":0,"kb_avail":92837144,"statfs":{"total":95927926784,"available":95065235456,"internally_reserved":1069056,"allocated":95066304512,"data_stored":95066304512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3877481,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.41699999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":121.575,"5min":121.575,"15min":121.575},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2060.797,"5min":2060.797,"15min":2060.797},"last":0.36299999999999999},{"interface":"front","average":{"1min":121.56,"5min":121.56,"15min":121.56},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2060.7159999999999,"5min":2060.7159999999999,"15min":2060.7159999999999},"last":0.50700000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.433},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.45300000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.379},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.47499999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.754},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.72599999999999998}]},{"osd":5,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.84599999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.69199999999999995}]},{"osd":7,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.67600000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.46800000000000003}]},{"osd":9,"last update":"Sun Sep 15 03:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.57099999999999995}]},{"osd":10,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.496},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.61099999999999999}]}]},{"osd":10,"up_from":105,"seq":450971566082,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":855452,"kb_used_data":92824164,"kb_used_omap":2243,"kb_used_meta":0,"kb_avail":92824164,"statfs":{"total":95927926784,"available":95051943936,"internally_reserved":0,"allocated":95051943936,"data_stored":95051943936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2297134,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]}]},{"osd":9,"up_from":17,"seq":73014444054,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":860264,"kb_used_data":92820520,"kb_used_omap":4103,"kb_used_meta":0,"kb_avail":92819352,"statfs":{"total":95927926784,"available":95047016448,"internally_reserved":1196032,"allocated":95048212480,"data_stored":95048212480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4202475,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.67200000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.71199999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.42299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.752},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.73799999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.65800000000000003}]},{"osd":6,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.628}]},{"osd":7,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.61799999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.55700000000000005}]},{"osd":10,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.54200000000000004}]},{"osd":11,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.57499999999999996}]}]},{"osd":8,"up_from":17,"seq":73014444055,"num_pgs":17,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":859932,"kb_used_data":92821696,"kb_used_omap":4132,"kb_used_meta":0,"kb_avail":92819684,"statfs":{"total":95927926784,"available":95047356416,"internally_reserved":2060288,"allocated":95049416704,"data_stored":95049416704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4231298,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.432},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.53900000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":42.381,"5min":42.381,"15min":42.381},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":963.98400000000004,"5min":963.98400000000004,"15min":963.98400000000004},"last":0.497},{"interface":"front","average":{"1min":42.447000000000003,"5min":42.447000000000003,"15min":42.447000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":964.79600000000005,"5min":964.79600000000005,"15min":964.79600000000005},"last":0.41399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.59299999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.69399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:47:25 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.71899999999999997}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.60499999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.65800000000000003}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.72799999999999998}]},{"osd":11,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.52000000000000002}]}]},{"osd":7,"up_from":17,"seq":73014444054,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":861576,"kb_used_data":92822272,"kb_used_omap":4702,"kb_used_meta":0,"kb_avail":92818040,"statfs":{"total":95927926784,"available":95045672960,"internally_reserved":4333568,"allocated":95050006528,"data_stored":95050006528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4815839,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,7],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.56899999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.63400000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.65400000000000003}]},{"osd":4,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.32700000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:47:27 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.38100000000000001}]},{"osd":6,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.39300000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.621}]},{"osd":9,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.61299999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.46999999999999997}]},{"osd":11,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.58999999999999997}]}]},{"osd":6,"up_from":17,"seq":73014444054,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":836692,"kb_used_data":92845160,"kb_used_omap":3056,"kb_used_meta":0,"kb_avail":92842924,"statfs":{"total":95927926784,"available":95071154176,"internally_reserved":2289664,"allocated":95073443840,"data_stored":95073443840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3129739,"internal_metadata":0},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.436},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.57499999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:47:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.56299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.53200000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.38300000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.40799999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.502},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.61399999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.54900000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.63200000000000001}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.78400000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":11,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.73399999999999999}]}]},{"osd":5,"up_from":17,"seq":73014444054,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":814244,"kb_used_data":92867232,"kb_used_omap":1654,"kb_used_meta":0,"kb_avail":92865372,"statfs":{"total":95927926784,"available":95094140928,"internally_reserved":1904640,"allocated":95096045568,"data_stored":95096045568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1694267,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.54800000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":113.717,"5min":113.717,"15min":113.717},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2037.425,"5min":2037.425,"15min":2037.425},"last":0.53200000000000003},{"interface":"front","average":{"1min":113.752,"5min":113.752,"15min":113.752},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2037.3230000000001,"5min":2037.3230000000001,"15min":2037.3230000000001},"last":0.56799999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.68799999999999994}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.32100000000000001}]},{"osd":6,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.38800000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.34699999999999998}]},{"osd":8,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.625},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.70299999999999996}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.64300000000000002}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.83399999999999996}]},{"osd":11,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.55900000000000005}]}]},{"osd":4,"up_from":17,"seq":73014444054,"num_pgs":15,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":861572,"kb_used_data":92820020,"kb_used_omap":4313,"kb_used_meta":0,"kb_avail":92818044,"statfs":{"total":95927926784,"available":95045677056,"internally_reserved":2023424,"allocated":95047700480,"data_stored":95047700480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416750,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.747}]},{"osd":1,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":84.314999999999998,"5min":84.314999999999998,"15min":84.314999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1758.9780000000001,"5min":1758.9780000000001,"15min":1758.9780000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":84.364000000000004,"5min":84.364000000000004,"15min":84.364000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1758.671,"5min":1758.671,"15min":1758.671},"last":0.84799999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.872}]},{"osd":3,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.75800000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.501}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":7,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.60999999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.79100000000000004}]},{"osd":9,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":0.69299999999999995}]},{"osd":10,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.59599999999999997}]},{"osd":11,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.67500000000000004}]}]},{"osd":3,"up_from":17,"seq":73014444054,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":846780,"kb_used_data":92837052,"kb_used_omap":4162,"kb_used_meta":0,"kb_avail":92832836,"statfs":{"total":95927926784,"available":95060824064,"internally_reserved":4317184,"allocated":95065141248,"data_stored":95065141248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4262010,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":5,"commit_latency_ns":0,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":1.0529999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":1.736}]},{"osd":1,"last update":"Sun Sep 15 03:47:38 2024","interfaces":[{"interface":"back","average":{"1min":35.600000000000001,"5min":35.600000000000001,"15min":35.600000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":564.05100000000004,"5min":564.05100000000004,"15min":564.05100000000004},"last":7.0350000000000001},{"interface":"front","average":{"1min":35.677,"5min":35.677,"15min":35.677},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":564.57000000000005,"5min":564.57000000000005,"15min":564.57000000000005},"last":2.448}]},{"osd":2,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":5.1799999999999997},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":2.5179999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":5.1399999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":2.3639999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":2.3940000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":5.2830000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":1.7729999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":5.2489999999999997}]},{"osd":7,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":1.827},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":5.3090000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":2.4900000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":1.8}]},{"osd":9,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":5.3399999999999999},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":2.4239999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.113}]},{"osd":11,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":1.859},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":2.5430000000000001}]}]},{"osd":2,"up_from":17,"seq":73014444054,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":851376,"kb_used_data":92830556,"kb_used_omap":4128,"kb_used_meta":0,"kb_avail":92828240,"statfs":{"total":95927926784,"available":95056117760,"internally_reserved":2371584,"allocated":95058489344,"data_stored":95058489344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4227620,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:37 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.57899999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":34.457999999999998,"5min":34.457999999999998,"15min":34.457999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":647.92399999999998,"5min":647.92399999999998,"15min":647.92399999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":34.475000000000001,"5min":34.475000000000001,"15min":34.475000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":648.02700000000004,"5min":648.02700000000004,"15min":648.02700000000004},"last":0.56599999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.504}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.68300000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.746},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.79200000000000004}]},{"osd":7,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.71399999999999997}]},{"osd":8,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.343,"15min":1.343},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":14.721,"5min":14.721,"15min":14.721},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.371,"5min":1.371,"15min":1.371},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":14.970000000000001,"5min":14.970000000000001,"15min":14.970000000000001},"last":0.84599999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.377,"15min":1.377},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.875},{"interface":"front","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":14.888,"5min":14.888,"15min":14.888},"last":0.85999999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":14.654999999999999,"5min":14.654999999999999,"15min":14.654999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.409,"5min":1.409,"15min":1.409},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":15.096,"5min":15.096,"15min":15.096},"last":0.89400000000000002}]}]},{"osd":1,"up_from":17,"seq":73014444055,"num_pgs":13,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":868812,"kb_used_data":92819516,"kb_used_omap":5763,"kb_used_meta":0,"kb_avail":92810804,"statfs":{"total":95927926784,"available":95038263296,"internally_reserved":8921088,"allocated":95047184384,"data_stored":95047184384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5901589,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.372},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.30099999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.34499999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.60799999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":2.6920000000000002,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":0.48699999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.7440000000000002,"5min":2.7440000000000002,"15min":2.7440000000000002},"last":0.62}]},{"osd":6,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":3.278,"5min":3.278,"15min":3.278},"last":0.67400000000000004}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.81399999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":119.93899999999999,"5min":119.93899999999999,"15min":119.93899999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2861.7190000000001,"5min":2861.7190000000001,"15min":2861.7190000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.946,"5min":1.946,"15min":1.946},"last":0.40799999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":119.937,"5min":119.937,"15min":119.937},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":2861.6010000000001,"5min":2861.6010000000001,"15min":2861.6010000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.44900000000000001}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":119.96599999999999,"5min":119.96599999999999,"15min":119.96599999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2861.5639999999999,"5min":2861.5639999999999,"15min":2861.5639999999999},"last":0.58699999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]}]},{"osd":0,"up_from":17,"seq":73014444054,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":839016,"kb_used_data":92843048,"kb_used_omap":3909,"kb_used_meta":0,"kb_avail":92840600,"statfs":{"total":95927926784,"available":95068774400,"internally_reserved":2506752,"allocated":95071281152,"data_stored":95071281152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4003622,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.51700000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.804,"5min":1.804,"15min":1.804},"last":0.379}]},{"osd":3,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.52,"5min":1.52,"15min":1.52},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.65000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.62},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.58499999999999996}]},{"osd":6,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.502}]},{"osd":7,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.56899999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.71599999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":10,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.76100000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.81799999999999995}]}]}],"pool_statfs":[]}} 2024-09-15T03:48:00.881 INFO:tasks.ceph.ceph_manager.ceph:pg 3.3 in trimming, state: active+clean+snaptrim 2024-09-15T03:48:00.881 INFO:tasks.ceph.ceph_manager.ceph:pg 3.5 in trimming, state: active+clean+snaptrim 2024-09-15T03:48:00.881 INFO:tasks.ceph.ceph_manager.ceph:unique_pool_0 still trimming, waiting 2024-09-15T03:48:00.901 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:00.901 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:00.910 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:48:00.942 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.5 to 1 (10000) 2024-09-15T03:48:00.947 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:48:00.946+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:48:00.955 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-15T03:48:01.011 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:01.011 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:01.020 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:48:01.048 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:48:01.047+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:48:01.127 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:01.127 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:01.136 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:48:01.181 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:48:01.180+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:48:01.235 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:01.236 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:01.244 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:48:01.250 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:48:01.249+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:48:01.351 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:48:01.351+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:48:01.382 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:01.382 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:01.391 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:48:01.452 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:48:01.452+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:48:01.487 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:01.487 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:01.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:48:01.553 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:48:01.552+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:48:01.591 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:01.592 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:01.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:48:01.654 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:48:01.653+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:48:01.695 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:01.695 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:01.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:48:01.755 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:48:01.754+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:48:01.799 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:01.799 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:01.807 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:48:01.856 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:48:01.856+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:48:01.943 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:01.944 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:01.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:48:01.952 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.7 to 1 (10000) 2024-09-15T03:48:01.957 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:48:01.956+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:48:01.968 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-15T03:48:01.987 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T03:48:02.048 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:02.048 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:02.057 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:48:02.059 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:48:02.058+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:48:02.155 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:02.155 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:02.159 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:48:02.159+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:48:02.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:48:02.260 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:02.260 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:02.261 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:48:02.260+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:48:02.269 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:48:02.362 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:48:02.361+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:48:02.365 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:02.366 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:02.374 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:48:02.463 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:48:02.462+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:48:02.477 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:02.477 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:02.485 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:48:02.564 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:48:02.564+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:48:02.585 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:02.585 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:02.594 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:48:02.665 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:48:02.664+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:48:02.675 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:48:02.696 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:02.696 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:02.705 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:48:02.766 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:48:02.765+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:48:02.847 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:02.847 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:02.855 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:48:02.867 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:48:02.867+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:48:02.956 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:02.957 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:02.963 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.8 to 1 (10000) 2024-09-15T03:48:02.963 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-15T03:48:02.966 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:48:02.968 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:48:02.967+0000 7fd62abf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33736) UID: 0 2024-09-15T03:48:02.976 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-15T03:48:02.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:48:02.976+0000 7fa6d2d9d640 0 --2- 172.21.15.1:0/3246391333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa6d4067a70 0x7fa6d406d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-15T03:48:02.997 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:48:02.997 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:48:03.012 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:48:03.013 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:48:03.028 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:48:03.028 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:48:03.042 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:48:03.043 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:48:03.058 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:48:03.058 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:48:03.069 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:48:03.068+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:48:03.073 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:48:03.073 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:48:03.077 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:03.077 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-15T03:48:03.077 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-15T03:48:03.077 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-15T03:48:03.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.077 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=80, reqid=client.4521.0:320, at_version=49'15, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:48:03.077 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.840885+0000", 2024-09-15T03:48:03.077 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.233828363000001, 2024-09-15T03:48:03.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0033862459999998, 2024-09-15T03:48:03.078 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.078 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.078 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.078 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.078 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.840885+0000", 2024-09-15T03:48:03.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.078 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.078 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.840885+0000", 2024-09-15T03:48:03.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.079 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.079 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.840885+0000", 2024-09-15T03:48:03.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1100000000000002e-07 2024-09-15T03:48:03.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.079 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.079 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841181+0000", 2024-09-15T03:48:03.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029618199999999998 2024-09-15T03:48:03.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.079 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.079 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841195+0000", 2024-09-15T03:48:03.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3424000000000001e-05 2024-09-15T03:48:03.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841204+0000", 2024-09-15T03:48:03.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0170000000000002e-06 2024-09-15T03:48:03.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841274+0000", 2024-09-15T03:48:03.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0561000000000004e-05 2024-09-15T03:48:03.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841289+0000", 2024-09-15T03:48:03.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4127e-05 2024-09-15T03:48:03.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841717+0000", 2024-09-15T03:48:03.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00042811800000000001 2024-09-15T03:48:03.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841745+0000", 2024-09-15T03:48:03.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8297e-05 2024-09-15T03:48:03.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841852+0000", 2024-09-15T03:48:03.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010660199999999999 2024-09-15T03:48:03.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.842419+0000", 2024-09-15T03:48:03.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00056784499999999998 2024-09-15T03:48:03.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.083 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.844271+0000", 2024-09-15T03:48:03.083 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0018517619999998 2024-09-15T03:48:03.083 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.083 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.083 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.083 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=30, reqid=client.4521.0:325, at_version=49'12, trim_to=0'0, roll_forward_to=40'10))", 2024-09-15T03:48:03.083 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.844912+0000", 2024-09-15T03:48:03.083 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.229801781000006, 2024-09-15T03:48:03.083 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0007286940000002, 2024-09-15T03:48:03.083 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.083 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.083 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.084 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.084 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844912+0000", 2024-09-15T03:48:03.084 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.084 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.084 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844912+0000", 2024-09-15T03:48:03.084 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.084 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.084 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844912+0000", 2024-09-15T03:48:03.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.84e-07 2024-09-15T03:48:03.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844937+0000", 2024-09-15T03:48:03.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4729e-05 2024-09-15T03:48:03.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844957+0000", 2024-09-15T03:48:03.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0285999999999999e-05 2024-09-15T03:48:03.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.086 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844965+0000", 2024-09-15T03:48:03.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5709999999999998e-06 2024-09-15T03:48:03.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.086 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.086 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845086+0000", 2024-09-15T03:48:03.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000120856 2024-09-15T03:48:03.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.086 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.086 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845121+0000", 2024-09-15T03:48:03.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4292999999999998e-05 2024-09-15T03:48:03.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.087 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845492+0000", 2024-09-15T03:48:03.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00037167400000000001 2024-09-15T03:48:03.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.087 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845529+0000", 2024-09-15T03:48:03.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7092000000000003e-05 2024-09-15T03:48:03.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.087 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845591+0000", 2024-09-15T03:48:03.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1824999999999998e-05 2024-09-15T03:48:03.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845612+0000", 2024-09-15T03:48:03.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.084e-05 2024-09-15T03:48:03.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845640+0000", 2024-09-15T03:48:03.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0000281439999998 2024-09-15T03:48:03.088 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.088 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.088 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.089 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=81, reqid=client.4521.0:321, at_version=49'16, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:48:03.089 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.845113+0000", 2024-09-15T03:48:03.089 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.229600374, 2024-09-15T03:48:03.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.999459995, 2024-09-15T03:48:03.089 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.089 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.089 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845113+0000", 2024-09-15T03:48:03.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845113+0000", 2024-09-15T03:48:03.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.091 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845114+0000", 2024-09-15T03:48:03.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0210000000000001e-06 2024-09-15T03:48:03.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.091 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.091 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845422+0000", 2024-09-15T03:48:03.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00030762999999999998 2024-09-15T03:48:03.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.091 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.091 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845436+0000", 2024-09-15T03:48:03.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3998999999999999e-05 2024-09-15T03:48:03.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.092 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.092 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845445+0000", 2024-09-15T03:48:03.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5799999999999998e-06 2024-09-15T03:48:03.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.092 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.092 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845538+0000", 2024-09-15T03:48:03.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2739999999999996e-05 2024-09-15T03:48:03.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.092 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.092 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845548+0000", 2024-09-15T03:48:03.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0548e-05 2024-09-15T03:48:03.093 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.093 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.093 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845803+0000", 2024-09-15T03:48:03.093 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00025495599999999998 2024-09-15T03:48:03.093 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.093 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.093 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845998+0000", 2024-09-15T03:48:03.093 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019432799999999999 2024-09-15T03:48:03.093 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.093 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.093 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846122+0000", 2024-09-15T03:48:03.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012434600000000001 2024-09-15T03:48:03.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.094 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846178+0000", 2024-09-15T03:48:03.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5943999999999998e-05 2024-09-15T03:48:03.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.094 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.844573+0000", 2024-09-15T03:48:03.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9983949029999999 2024-09-15T03:48:03.094 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.094 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.095 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.095 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=82, reqid=client.4521.0:322, at_version=49'17, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:48:03.095 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.846151+0000", 2024-09-15T03:48:03.095 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.228562351999997, 2024-09-15T03:48:03.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9989014119999999, 2024-09-15T03:48:03.095 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.095 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.095 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.095 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846151+0000", 2024-09-15T03:48:03.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846151+0000", 2024-09-15T03:48:03.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846152+0000", 2024-09-15T03:48:03.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0800000000000004e-07 2024-09-15T03:48:03.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846238+0000", 2024-09-15T03:48:03.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5866999999999996e-05 2024-09-15T03:48:03.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846255+0000", 2024-09-15T03:48:03.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6954999999999999e-05 2024-09-15T03:48:03.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846262+0000", 2024-09-15T03:48:03.098 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7680000000000008e-06 2024-09-15T03:48:03.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.098 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.098 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846325+0000", 2024-09-15T03:48:03.098 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2191000000000001e-05 2024-09-15T03:48:03.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.098 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.098 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846335+0000", 2024-09-15T03:48:03.098 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0767e-05 2024-09-15T03:48:03.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.099 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.099 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846446+0000", 2024-09-15T03:48:03.099 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110522 2024-09-15T03:48:03.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.099 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.099 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.099 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846458+0000", 2024-09-15T03:48:03.099 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2023e-05 2024-09-15T03:48:03.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.099 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.099 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.099 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846508+0000", 2024-09-15T03:48:03.099 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0479000000000002e-05 2024-09-15T03:48:03.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.100 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.100 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846526+0000", 2024-09-15T03:48:03.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7836e-05 2024-09-15T03:48:03.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.100 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.100 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845052+0000", 2024-09-15T03:48:03.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9985261960000003 2024-09-15T03:48:03.100 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.100 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.100 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.101 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=31, reqid=client.4521.0:325, at_version=0'0, trim_to=0'0, roll_forward_to=49'12))", 2024-09-15T03:48:03.101 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.847407+0000", 2024-09-15T03:48:03.101 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.227306149, 2024-09-15T03:48:03.101 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9983494909999999, 2024-09-15T03:48:03.101 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.101 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.101 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.101 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.101 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847407+0000", 2024-09-15T03:48:03.101 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.101 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.102 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847407+0000", 2024-09-15T03:48:03.102 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.102 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847408+0000", 2024-09-15T03:48:03.102 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0070000000000001e-06 2024-09-15T03:48:03.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.102 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847421+0000", 2024-09-15T03:48:03.102 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3203999999999999e-05 2024-09-15T03:48:03.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.103 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.103 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.103 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847443+0000", 2024-09-15T03:48:03.103 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1539000000000002e-05 2024-09-15T03:48:03.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.103 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.103 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.103 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847452+0000", 2024-09-15T03:48:03.103 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6570000000000003e-06 2024-09-15T03:48:03.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.103 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.103 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.103 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847534+0000", 2024-09-15T03:48:03.103 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2309000000000001e-05 2024-09-15T03:48:03.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.104 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.104 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.104 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847544+0000", 2024-09-15T03:48:03.104 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0482000000000001e-05 2024-09-15T03:48:03.104 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.104 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.104 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.104 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847645+0000", 2024-09-15T03:48:03.104 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010021700000000001 2024-09-15T03:48:03.104 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.104 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.104 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.104 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847665+0000", 2024-09-15T03:48:03.105 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0271999999999998e-05 2024-09-15T03:48:03.105 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.105 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.105 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.105 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847708+0000", 2024-09-15T03:48:03.105 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2828999999999999e-05 2024-09-15T03:48:03.105 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.105 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.105 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.105 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847724+0000", 2024-09-15T03:48:03.105 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6334e-05 2024-09-15T03:48:03.105 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.105 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.105 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.106 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845757+0000", 2024-09-15T03:48:03.106 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.998032641 2024-09-15T03:48:03.106 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.106 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.106 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.106 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.106 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.106 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=83, reqid=client.4521.0:323, at_version=49'18, trim_to=0'0, roll_forward_to=49'15))", 2024-09-15T03:48:03.106 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.847691+0000", 2024-09-15T03:48:03.106 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.227022301999995, 2024-09-15T03:48:03.106 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9981305969999998, 2024-09-15T03:48:03.106 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.106 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.106 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.107 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.107 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847691+0000", 2024-09-15T03:48:03.107 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.107 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.107 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847691+0000", 2024-09-15T03:48:03.107 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.107 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.107 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847692+0000", 2024-09-15T03:48:03.107 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9899999999999999e-07 2024-09-15T03:48:03.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.108 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.108 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847715+0000", 2024-09-15T03:48:03.108 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2818000000000001e-05 2024-09-15T03:48:03.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.108 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.108 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847731+0000", 2024-09-15T03:48:03.108 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6401000000000001e-05 2024-09-15T03:48:03.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.108 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.109 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847738+0000", 2024-09-15T03:48:03.109 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3699999999999997e-06 2024-09-15T03:48:03.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.109 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.109 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847791+0000", 2024-09-15T03:48:03.109 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2584999999999999e-05 2024-09-15T03:48:03.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.109 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.109 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847801+0000", 2024-09-15T03:48:03.109 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0295999999999999e-05 2024-09-15T03:48:03.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.110 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.110 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847931+0000", 2024-09-15T03:48:03.110 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0001295 2024-09-15T03:48:03.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.110 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.110 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847946+0000", 2024-09-15T03:48:03.110 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4732000000000001e-05 2024-09-15T03:48:03.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.110 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.110 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847989+0000", 2024-09-15T03:48:03.110 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3625000000000003e-05 2024-09-15T03:48:03.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.111 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.111 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.848004+0000", 2024-09-15T03:48:03.111 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4385e-05 2024-09-15T03:48:03.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.111 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.111 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845822+0000", 2024-09-15T03:48:03.111 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9978179860000003 2024-09-15T03:48:03.111 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.111 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.111 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.112 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=84, reqid=client.4521.0:323, at_version=0'0, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:48:03.112 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.849295+0000", 2024-09-15T03:48:03.112 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.225418222000002, 2024-09-15T03:48:03.112 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9965517450000001, 2024-09-15T03:48:03.112 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.112 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.112 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.112 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.112 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849295+0000", 2024-09-15T03:48:03.112 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.113 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.113 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849295+0000", 2024-09-15T03:48:03.113 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.113 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.113 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.113 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849296+0000", 2024-09-15T03:48:03.113 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6300000000000004e-07 2024-09-15T03:48:03.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.113 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.113 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.113 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849308+0000", 2024-09-15T03:48:03.113 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2261e-05 2024-09-15T03:48:03.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.114 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.114 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849325+0000", 2024-09-15T03:48:03.114 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6588000000000001e-05 2024-09-15T03:48:03.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.114 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.114 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849332+0000", 2024-09-15T03:48:03.114 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.464e-06 2024-09-15T03:48:03.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.114 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.114 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849379+0000", 2024-09-15T03:48:03.114 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6498000000000001e-05 2024-09-15T03:48:03.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.115 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.115 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849391+0000", 2024-09-15T03:48:03.115 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1926000000000001e-05 2024-09-15T03:48:03.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.115 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.115 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849497+0000", 2024-09-15T03:48:03.115 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010634200000000001 2024-09-15T03:48:03.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.115 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.115 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849516+0000", 2024-09-15T03:48:03.116 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8995999999999999e-05 2024-09-15T03:48:03.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.116 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.116 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849565+0000", 2024-09-15T03:48:03.116 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8735e-05 2024-09-15T03:48:03.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.116 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.116 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849577+0000", 2024-09-15T03:48:03.116 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2585999999999999e-05 2024-09-15T03:48:03.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.116 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.117 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845847+0000", 2024-09-15T03:48:03.117 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9962694860000001 2024-09-15T03:48:03.117 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.117 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.117 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.117 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.117 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpRead(3.7s5 49/31 ECSubRead(tid=85, to_read={3:e40d1694:::smithi00132172-50:head=0,4096,0}, subchunks={3:e40d1694:::smithi00132172-50:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:48:03.117 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.849814+0000", 2024-09-15T03:48:03.117 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.224898908, 2024-09-15T03:48:03.117 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9961114690000001, 2024-09-15T03:48:03.117 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.117 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.117 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.118 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.118 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849814+0000", 2024-09-15T03:48:03.118 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.118 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.118 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849814+0000", 2024-09-15T03:48:03.118 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.118 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.118 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849815+0000", 2024-09-15T03:48:03.118 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6499999999999999e-07 2024-09-15T03:48:03.119 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.119 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.119 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849826+0000", 2024-09-15T03:48:03.119 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0635000000000001e-05 2024-09-15T03:48:03.119 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.119 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.119 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.850154+0000", 2024-09-15T03:48:03.119 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00032866900000000002 2024-09-15T03:48:03.119 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.119 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.119 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.850169+0000", 2024-09-15T03:48:03.120 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5051e-05 2024-09-15T03:48:03.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.120 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.120 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.120 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.850226+0000", 2024-09-15T03:48:03.120 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6926999999999999e-05 2024-09-15T03:48:03.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.120 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.120 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.120 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845926+0000", 2024-09-15T03:48:03.120 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9956995219999998 2024-09-15T03:48:03.120 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.120 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.121 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.121 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.121 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.121 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=32, reqid=client.4521.0:329, at_version=49'13, trim_to=0'0, roll_forward_to=49'12))", 2024-09-15T03:48:03.121 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.853281+0000", 2024-09-15T03:48:03.121 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.221432422000007, 2024-09-15T03:48:03.121 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9927023569999998, 2024-09-15T03:48:03.121 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.121 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.121 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.121 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.121 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.121 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853281+0000", 2024-09-15T03:48:03.121 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.122 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.122 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.122 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853281+0000", 2024-09-15T03:48:03.122 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.122 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.122 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.122 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853282+0000", 2024-09-15T03:48:03.122 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.347e-06 2024-09-15T03:48:03.122 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.122 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.122 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853401+0000", 2024-09-15T03:48:03.123 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118711 2024-09-15T03:48:03.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.123 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.123 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.123 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853441+0000", 2024-09-15T03:48:03.123 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0059999999999999e-05 2024-09-15T03:48:03.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.123 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.123 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.123 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853451+0000", 2024-09-15T03:48:03.123 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5850000000000006e-06 2024-09-15T03:48:03.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.123 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.124 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.124 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853617+0000", 2024-09-15T03:48:03.124 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016655899999999999 2024-09-15T03:48:03.124 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.124 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.124 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.124 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853632+0000", 2024-09-15T03:48:03.124 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4605e-05 2024-09-15T03:48:03.124 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.124 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.124 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.124 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853842+0000", 2024-09-15T03:48:03.124 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000210523 2024-09-15T03:48:03.124 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.125 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.125 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.125 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853871+0000", 2024-09-15T03:48:03.125 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8441000000000001e-05 2024-09-15T03:48:03.125 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.125 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.125 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.125 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854188+0000", 2024-09-15T03:48:03.125 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031700800000000003 2024-09-15T03:48:03.125 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.125 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.125 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.125 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854209+0000", 2024-09-15T03:48:03.126 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0795999999999999e-05 2024-09-15T03:48:03.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.126 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.126 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.126 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845983+0000", 2024-09-15T03:48:03.126 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9917747220000002 2024-09-15T03:48:03.126 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.126 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.126 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.126 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.126 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=33, reqid=client.4521.0:329, at_version=0'0, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:48:03.126 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.854592+0000", 2024-09-15T03:48:03.126 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.220121410999994, 2024-09-15T03:48:03.127 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9914290499999998, 2024-09-15T03:48:03.127 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.127 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.127 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.127 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.127 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854592+0000", 2024-09-15T03:48:03.127 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.127 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.127 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854592+0000", 2024-09-15T03:48:03.127 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.128 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.128 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854593+0000", 2024-09-15T03:48:03.128 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1200000000000002e-07 2024-09-15T03:48:03.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.128 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.129 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854605+0000", 2024-09-15T03:48:03.129 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1946999999999999e-05 2024-09-15T03:48:03.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.129 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.129 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854624+0000", 2024-09-15T03:48:03.129 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9502000000000001e-05 2024-09-15T03:48:03.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.129 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.129 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854631+0000", 2024-09-15T03:48:03.129 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2490000000000004e-06 2024-09-15T03:48:03.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.130 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.130 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854683+0000", 2024-09-15T03:48:03.130 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1643e-05 2024-09-15T03:48:03.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.130 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.130 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854693+0000", 2024-09-15T03:48:03.130 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8199999999999992e-06 2024-09-15T03:48:03.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.130 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.130 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854768+0000", 2024-09-15T03:48:03.130 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5104999999999993e-05 2024-09-15T03:48:03.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.131 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.131 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854830+0000", 2024-09-15T03:48:03.131 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2191000000000001e-05 2024-09-15T03:48:03.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.131 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.131 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854934+0000", 2024-09-15T03:48:03.131 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010424099999999999 2024-09-15T03:48:03.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.132 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.132 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854952+0000", 2024-09-15T03:48:03.132 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7521e-05 2024-09-15T03:48:03.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.132 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.132 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.846021+0000", 2024-09-15T03:48:03.132 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9910690190000002 2024-09-15T03:48:03.132 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.132 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.132 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.133 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=34, reqid=client.4521.0:330, at_version=49'14, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:48:03.133 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.857132+0000", 2024-09-15T03:48:03.133 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.217581030000005, 2024-09-15T03:48:03.133 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9890264310000001, 2024-09-15T03:48:03.133 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.133 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.133 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.133 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.133 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857132+0000", 2024-09-15T03:48:03.133 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.134 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.134 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857132+0000", 2024-09-15T03:48:03.134 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.134 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.134 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857133+0000", 2024-09-15T03:48:03.134 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.61e-07 2024-09-15T03:48:03.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.134 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.134 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857223+0000", 2024-09-15T03:48:03.134 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0476999999999994e-05 2024-09-15T03:48:03.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.135 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.135 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857245+0000", 2024-09-15T03:48:03.135 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1415000000000001e-05 2024-09-15T03:48:03.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.135 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.135 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857254+0000", 2024-09-15T03:48:03.135 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.329e-06 2024-09-15T03:48:03.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.136 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.136 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857312+0000", 2024-09-15T03:48:03.136 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7865000000000002e-05 2024-09-15T03:48:03.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.136 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.136 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857323+0000", 2024-09-15T03:48:03.136 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0655999999999999e-05 2024-09-15T03:48:03.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.137 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.137 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857817+0000", 2024-09-15T03:48:03.137 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00049391300000000001 2024-09-15T03:48:03.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.137 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.137 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857946+0000", 2024-09-15T03:48:03.137 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012926699999999999 2024-09-15T03:48:03.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.137 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.137 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858076+0000", 2024-09-15T03:48:03.137 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012957600000000001 2024-09-15T03:48:03.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.138 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.138 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.138 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858094+0000", 2024-09-15T03:48:03.138 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8715e-05 2024-09-15T03:48:03.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.138 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.138 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.138 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.846159+0000", 2024-09-15T03:48:03.138 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9880644570000001 2024-09-15T03:48:03.138 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.138 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.138 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.139 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=86, reqid=client.4521.0:334, at_version=49'20, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:48:03.139 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.857463+0000", 2024-09-15T03:48:03.139 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.217250711999995, 2024-09-15T03:48:03.139 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0025860940000002, 2024-09-15T03:48:03.139 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.139 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.139 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.139 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.139 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857463+0000", 2024-09-15T03:48:03.139 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.140 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857463+0000", 2024-09-15T03:48:03.140 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.140 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857464+0000", 2024-09-15T03:48:03.140 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0210000000000001e-06 2024-09-15T03:48:03.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.140 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857489+0000", 2024-09-15T03:48:03.140 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5500999999999999e-05 2024-09-15T03:48:03.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.141 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.141 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857508+0000", 2024-09-15T03:48:03.141 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9025000000000002e-05 2024-09-15T03:48:03.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.141 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.141 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857515+0000", 2024-09-15T03:48:03.141 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9759999999999998e-06 2024-09-15T03:48:03.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.141 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.141 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845961+0000", 2024-09-15T03:48:03.142 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9884457740000001 2024-09-15T03:48:03.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.142 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.142 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845966+0000", 2024-09-15T03:48:03.142 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3750000000000002e-06 2024-09-15T03:48:03.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.142 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.142 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.846127+0000", 2024-09-15T03:48:03.142 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000160462 2024-09-15T03:48:03.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.142 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.143 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.846136+0000", 2024-09-15T03:48:03.143 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6260000000000007e-06 2024-09-15T03:48:03.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.144 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.144 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.846191+0000", 2024-09-15T03:48:03.144 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4274999999999998e-05 2024-09-15T03:48:03.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.144 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.144 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.853397+0000", 2024-09-15T03:48:03.144 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0072068519999999997 2024-09-15T03:48:03.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.860049+0000", 2024-09-15T03:48:03.145 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0066512070000000001 2024-09-15T03:48:03.145 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.145 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.145 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.145 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.cs3 49/28 ECSubWrite(tid=15, reqid=client.4521.0:337, at_version=49'10, trim_to=0'0, roll_forward_to=43'8))", 2024-09-15T03:48:03.145 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.857712+0000", 2024-09-15T03:48:03.146 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.217001393000004, 2024-09-15T03:48:03.146 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9893153209999999, 2024-09-15T03:48:03.146 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.146 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.146 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.146 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.146 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857712+0000", 2024-09-15T03:48:03.146 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.146 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.147 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857712+0000", 2024-09-15T03:48:03.147 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.147 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.147 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857712+0000", 2024-09-15T03:48:03.147 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8000000000000002e-07 2024-09-15T03:48:03.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.148 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.148 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.148 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857755+0000", 2024-09-15T03:48:03.148 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2960999999999997e-05 2024-09-15T03:48:03.148 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.148 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.149 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.149 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857773+0000", 2024-09-15T03:48:03.149 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7635999999999999e-05 2024-09-15T03:48:03.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.149 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.149 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857780+0000", 2024-09-15T03:48:03.149 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.024e-06 2024-09-15T03:48:03.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.149 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.149 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857883+0000", 2024-09-15T03:48:03.149 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103224 2024-09-15T03:48:03.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.150 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.150 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857893+0000", 2024-09-15T03:48:03.150 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0265000000000001e-05 2024-09-15T03:48:03.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.150 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.150 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858180+0000", 2024-09-15T03:48:03.150 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000286602 2024-09-15T03:48:03.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.150 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.150 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858193+0000", 2024-09-15T03:48:03.151 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3342e-05 2024-09-15T03:48:03.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.151 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858233+0000", 2024-09-15T03:48:03.151 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9811999999999998e-05 2024-09-15T03:48:03.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.151 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858253+0000", 2024-09-15T03:48:03.151 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0259e-05 2024-09-15T03:48:03.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.152 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.152 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.847027+0000", 2024-09-15T03:48:03.152 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.988773916 2024-09-15T03:48:03.152 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.152 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.152 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.152 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=87, reqid=client.4521.0:335, at_version=49'21, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:48:03.152 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.858481+0000", 2024-09-15T03:48:03.152 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.216232243999997, 2024-09-15T03:48:03.152 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.004544015, 2024-09-15T03:48:03.152 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.152 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.153 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.153 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.153 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858481+0000", 2024-09-15T03:48:03.153 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.153 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.153 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858481+0000", 2024-09-15T03:48:03.153 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.153 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.154 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858482+0000", 2024-09-15T03:48:03.154 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.09e-07 2024-09-15T03:48:03.154 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.154 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.154 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858505+0000", 2024-09-15T03:48:03.154 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3133e-05 2024-09-15T03:48:03.154 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.154 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.154 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858524+0000", 2024-09-15T03:48:03.154 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8706e-05 2024-09-15T03:48:03.154 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.155 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.155 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858532+0000", 2024-09-15T03:48:03.155 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5539999999999998e-06 2024-09-15T03:48:03.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.155 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.155 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851213+0000", 2024-09-15T03:48:03.155 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.992680526 2024-09-15T03:48:03.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.155 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.155 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851221+0000", 2024-09-15T03:48:03.155 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5380000000000002e-06 2024-09-15T03:48:03.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.156 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.156 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851309+0000", 2024-09-15T03:48:03.156 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8069e-05 2024-09-15T03:48:03.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.156 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.156 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851495+0000", 2024-09-15T03:48:03.156 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018516999999999999 2024-09-15T03:48:03.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.156 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.157 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851557+0000", 2024-09-15T03:48:03.157 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2031000000000003e-05 2024-09-15T03:48:03.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.157 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.157 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.853665+0000", 2024-09-15T03:48:03.157 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021079499999999999 2024-09-15T03:48:03.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.157 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.157 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.863025+0000", 2024-09-15T03:48:03.157 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0093605289999999994 2024-09-15T03:48:03.157 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.158 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.158 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.158 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.cs3 49/28 ECSubWrite(tid=16, reqid=client.4521.0:337, at_version=0'0, trim_to=0'0, roll_forward_to=49'10))", 2024-09-15T03:48:03.158 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.859219+0000", 2024-09-15T03:48:03.158 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.215494273000004, 2024-09-15T03:48:03.158 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9879305710000001, 2024-09-15T03:48:03.158 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.158 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.158 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.159 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.159 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859219+0000", 2024-09-15T03:48:03.159 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.159 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.159 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859219+0000", 2024-09-15T03:48:03.159 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.160 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.160 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859220+0000", 2024-09-15T03:48:03.160 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3300000000000001e-07 2024-09-15T03:48:03.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.160 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.160 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859233+0000", 2024-09-15T03:48:03.160 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2781e-05 2024-09-15T03:48:03.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.161 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859251+0000", 2024-09-15T03:48:03.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8770000000000002e-05 2024-09-15T03:48:03.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.161 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859259+0000", 2024-09-15T03:48:03.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7309999999999993e-06 2024-09-15T03:48:03.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.162 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859342+0000", 2024-09-15T03:48:03.162 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2909000000000002e-05 2024-09-15T03:48:03.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.162 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.162 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859357+0000", 2024-09-15T03:48:03.162 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5245000000000001e-05 2024-09-15T03:48:03.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.163 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.163 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859453+0000", 2024-09-15T03:48:03.163 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5878999999999995e-05 2024-09-15T03:48:03.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.163 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.163 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859475+0000", 2024-09-15T03:48:03.163 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1591999999999999e-05 2024-09-15T03:48:03.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.163 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.163 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859518+0000", 2024-09-15T03:48:03.164 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3021e-05 2024-09-15T03:48:03.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.164 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.164 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859536+0000", 2024-09-15T03:48:03.164 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8420999999999999e-05 2024-09-15T03:48:03.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.164 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.164 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.847150+0000", 2024-09-15T03:48:03.164 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9876134890000001 2024-09-15T03:48:03.164 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.164 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.164 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.165 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=88, reqid=client.4521.0:336, at_version=49'22, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:48:03.165 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.859709+0000", 2024-09-15T03:48:03.165 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.215004496000006, 2024-09-15T03:48:03.165 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0037862020000001, 2024-09-15T03:48:03.165 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.165 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.165 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.165 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.165 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859709+0000", 2024-09-15T03:48:03.165 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.166 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.166 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859709+0000", 2024-09-15T03:48:03.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.166 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.166 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859710+0000", 2024-09-15T03:48:03.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.052e-06 2024-09-15T03:48:03.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.166 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.167 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859729+0000", 2024-09-15T03:48:03.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9409999999999999e-05 2024-09-15T03:48:03.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.167 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.167 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859759+0000", 2024-09-15T03:48:03.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9998e-05 2024-09-15T03:48:03.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.167 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.167 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859767+0000", 2024-09-15T03:48:03.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.14e-06 2024-09-15T03:48:03.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.168 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.168 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851350+0000", 2024-09-15T03:48:03.168 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.99158261 2024-09-15T03:48:03.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.168 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.168 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851358+0000", 2024-09-15T03:48:03.168 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1070000000000005e-06 2024-09-15T03:48:03.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.168 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.169 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851443+0000", 2024-09-15T03:48:03.169 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4760999999999997e-05 2024-09-15T03:48:03.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.169 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.169 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851495+0000", 2024-09-15T03:48:03.169 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2070999999999997e-05 2024-09-15T03:48:03.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.169 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.169 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851557+0000", 2024-09-15T03:48:03.169 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1829999999999996e-05 2024-09-15T03:48:03.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.170 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.170 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.853682+0000", 2024-09-15T03:48:03.170 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021249110000000002 2024-09-15T03:48:03.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.170 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.170 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.863495+0000", 2024-09-15T03:48:03.170 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0098133119999999994 2024-09-15T03:48:03.170 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.170 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.170 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.171 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=35, reqid=client.4521.0:331, at_version=49'15, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:48:03.171 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.860778+0000", 2024-09-15T03:48:03.171 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.213935601000003, 2024-09-15T03:48:03.171 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9863441239999999, 2024-09-15T03:48:03.171 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.171 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.171 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.171 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.171 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860778+0000", 2024-09-15T03:48:03.171 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.172 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.172 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860778+0000", 2024-09-15T03:48:03.172 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.172 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.172 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860779+0000", 2024-09-15T03:48:03.172 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0109999999999999e-06 2024-09-15T03:48:03.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.172 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.172 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860922+0000", 2024-09-15T03:48:03.173 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014357399999999999 2024-09-15T03:48:03.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.173 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.173 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860944+0000", 2024-09-15T03:48:03.173 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1424e-05 2024-09-15T03:48:03.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.173 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.173 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860951+0000", 2024-09-15T03:48:03.173 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5610000000000001e-06 2024-09-15T03:48:03.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.174 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.174 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861022+0000", 2024-09-15T03:48:03.174 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1126999999999998e-05 2024-09-15T03:48:03.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.174 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.174 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861034+0000", 2024-09-15T03:48:03.174 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1810999999999999e-05 2024-09-15T03:48:03.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.174 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.174 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861164+0000", 2024-09-15T03:48:03.174 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000129727 2024-09-15T03:48:03.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.175 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.175 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861196+0000", 2024-09-15T03:48:03.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2240000000000003e-05 2024-09-15T03:48:03.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.175 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.175 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861259+0000", 2024-09-15T03:48:03.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2454999999999997e-05 2024-09-15T03:48:03.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.175 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.176 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861274+0000", 2024-09-15T03:48:03.176 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5588000000000001e-05 2024-09-15T03:48:03.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.176 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.176 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.847122+0000", 2024-09-15T03:48:03.176 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9858476060000001 2024-09-15T03:48:03.176 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.176 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.176 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.176 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=36, reqid=client.4521.0:332, at_version=49'16, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:48:03.177 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.861537+0000", 2024-09-15T03:48:03.177 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.213176038, 2024-09-15T03:48:03.177 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.985662638, 2024-09-15T03:48:03.177 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.177 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.177 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.177 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.177 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861537+0000", 2024-09-15T03:48:03.177 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.177 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.178 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861537+0000", 2024-09-15T03:48:03.178 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.178 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.178 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861538+0000", 2024-09-15T03:48:03.178 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7000000000000003e-07 2024-09-15T03:48:03.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.178 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.178 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861571+0000", 2024-09-15T03:48:03.178 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2493000000000002e-05 2024-09-15T03:48:03.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.179 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.179 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861588+0000", 2024-09-15T03:48:03.179 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7076e-05 2024-09-15T03:48:03.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.179 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.179 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861595+0000", 2024-09-15T03:48:03.179 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3440000000000003e-06 2024-09-15T03:48:03.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.179 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.179 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861665+0000", 2024-09-15T03:48:03.179 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9814999999999995e-05 2024-09-15T03:48:03.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.180 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.180 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861674+0000", 2024-09-15T03:48:03.180 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9740000000000008e-06 2024-09-15T03:48:03.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.180 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.180 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861767+0000", 2024-09-15T03:48:03.180 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3572999999999993e-05 2024-09-15T03:48:03.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.180 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.180 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861778+0000", 2024-09-15T03:48:03.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1083e-05 2024-09-15T03:48:03.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.181 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861888+0000", 2024-09-15T03:48:03.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010937700000000001 2024-09-15T03:48:03.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.181 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861906+0000", 2024-09-15T03:48:03.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.863e-05 2024-09-15T03:48:03.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.182 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.847200+0000", 2024-09-15T03:48:03.182 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.985293403 2024-09-15T03:48:03.182 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.182 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.182 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.182 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=37, reqid=client.4521.0:332, at_version=0'0, trim_to=0'0, roll_forward_to=49'16))", 2024-09-15T03:48:03.182 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.864457+0000", 2024-09-15T03:48:03.182 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.210256349000005, 2024-09-15T03:48:03.182 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9827806690000003, 2024-09-15T03:48:03.182 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.182 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.182 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.183 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.183 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864457+0000", 2024-09-15T03:48:03.183 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.183 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.183 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864457+0000", 2024-09-15T03:48:03.183 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.183 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.183 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864458+0000", 2024-09-15T03:48:03.184 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.029e-06 2024-09-15T03:48:03.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.184 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.184 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864472+0000", 2024-09-15T03:48:03.184 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4287000000000001e-05 2024-09-15T03:48:03.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.184 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.184 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864504+0000", 2024-09-15T03:48:03.184 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1435e-05 2024-09-15T03:48:03.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864511+0000", 2024-09-15T03:48:03.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4900000000000003e-06 2024-09-15T03:48:03.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864583+0000", 2024-09-15T03:48:03.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1828000000000004e-05 2024-09-15T03:48:03.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864593+0000", 2024-09-15T03:48:03.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0427e-05 2024-09-15T03:48:03.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864701+0000", 2024-09-15T03:48:03.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000107969 2024-09-15T03:48:03.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864721+0000", 2024-09-15T03:48:03.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9422000000000001e-05 2024-09-15T03:48:03.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864798+0000", 2024-09-15T03:48:03.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6731999999999996e-05 2024-09-15T03:48:03.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864815+0000", 2024-09-15T03:48:03.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7286999999999999e-05 2024-09-15T03:48:03.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.847238+0000", 2024-09-15T03:48:03.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9824227629999998 2024-09-15T03:48:03.187 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.187 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.187 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.188 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.bs4 49/31 ECSubWrite(tid=75, reqid=client.4521.0:338, at_version=49'58, trim_to=0'0, roll_forward_to=47'56))", 2024-09-15T03:48:03.188 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.870036+0000", 2024-09-15T03:48:03.188 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 67.204677724999996, 2024-09-15T03:48:03.188 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.978548666, 2024-09-15T03:48:03.188 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.188 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.188 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.188 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.188 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870036+0000", 2024-09-15T03:48:03.188 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.189 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.189 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870036+0000", 2024-09-15T03:48:03.189 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.189 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.189 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870036+0000", 2024-09-15T03:48:03.189 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.61e-07 2024-09-15T03:48:03.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.189 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.189 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870051+0000", 2024-09-15T03:48:03.189 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5074e-05 2024-09-15T03:48:03.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.190 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.190 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870061+0000", 2024-09-15T03:48:03.190 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.066e-05 2024-09-15T03:48:03.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.190 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.190 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870067+0000", 2024-09-15T03:48:03.190 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1580000000000004e-06 2024-09-15T03:48:03.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.190 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.190 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870121+0000", 2024-09-15T03:48:03.191 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4605999999999999e-05 2024-09-15T03:48:03.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.191 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.191 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870134+0000", 2024-09-15T03:48:03.191 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2748999999999999e-05 2024-09-15T03:48:03.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.191 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.191 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870450+0000", 2024-09-15T03:48:03.191 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031624399999999998 2024-09-15T03:48:03.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.191 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.192 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870459+0000", 2024-09-15T03:48:03.192 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0499999999999997e-06 2024-09-15T03:48:03.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.192 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.192 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870496+0000", 2024-09-15T03:48:03.192 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6548999999999999e-05 2024-09-15T03:48:03.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.192 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.192 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870509+0000", 2024-09-15T03:48:03.192 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3216e-05 2024-09-15T03:48:03.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.193 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.193 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.848584+0000", 2024-09-15T03:48:03.193 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9780751990000001 2024-09-15T03:48:03.193 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.193 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.193 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.193 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.193 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.193 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:03.195 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:48:03.195 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:48:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:48:03.196 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:48:03.196 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:48:03.196 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:48:03.197 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:48:03.197 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:48:03.197 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:48:03.197 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:48:03.198 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:48:03.198 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:48:03.199 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:48:03.199 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:48:03.194+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:48:03.296 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:48:03.295+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:48:03.338 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:03.338 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:48:03.338 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-15T03:48:03.338 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:03.346 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:48:03.397 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:48:03.396+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:48:03.446 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:03.446 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:48:03.446 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-15T03:48:03.446 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-15T03:48:03.447 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:03.455 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:48:03.498 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:48:03.497+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:48:03.557 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:03.557 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-15T03:48:03.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-15T03:48:03.557 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-15T03:48:03.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.557 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:195 3.es0 3:7f0816e1:::smithi00132172-31:head [copy-from ver 4] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:48:03.557 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:41.112078+0000", 2024-09-15T03:48:03.557 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 82.442961335000007, 2024-09-15T03:48:03.557 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012535605, 2024-09-15T03:48:03.557 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 195 2024-09-15T03:48:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112078+0000", 2024-09-15T03:48:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.558 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112078+0000", 2024-09-15T03:48:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112080+0000", 2024-09-15T03:48:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4399999999999999e-06 2024-09-15T03:48:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112090+0000", 2024-09-15T03:48:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7310000000000005e-06 2024-09-15T03:48:03.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112107+0000", 2024-09-15T03:48:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7557000000000001e-05 2024-09-15T03:48:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112113+0000", 2024-09-15T03:48:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6910000000000002e-06 2024-09-15T03:48:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.560 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112176+0000", 2024-09-15T03:48:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3014000000000003e-05 2024-09-15T03:48:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112308+0000", 2024-09-15T03:48:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000131402 2024-09-15T03:48:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118088+0000", 2024-09-15T03:48:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0057801570000000002 2024-09-15T03:48:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118560+0000", 2024-09-15T03:48:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00047206199999999998 2024-09-15T03:48:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118573+0000", 2024-09-15T03:48:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3521e-05 2024-09-15T03:48:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118667+0000", 2024-09-15T03:48:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.323e-05 2024-09-15T03:48:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.119046+0000", 2024-09-15T03:48:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00037924699999999998 2024-09-15T03:48:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.124533+0000", 2024-09-15T03:48:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0054869849999999998 2024-09-15T03:48:03.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.124613+0000", 2024-09-15T03:48:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0568000000000005e-05 2024-09-15T03:48:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:384 3.es0 3:77a7da34:::smithi00132172-4:head [read 0~1146880,checksum,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e56)", 2024-09-15T03:48:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:02.864202+0000", 2024-09-15T03:48:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 60.690837555999998, 2024-09-15T03:48:03.564 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012715746999999999, 2024-09-15T03:48:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 384 2024-09-15T03:48:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864202+0000", 2024-09-15T03:48:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864202+0000", 2024-09-15T03:48:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864202+0000", 2024-09-15T03:48:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9400000000000004e-07 2024-09-15T03:48:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864206+0000", 2024-09-15T03:48:03.566 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9770000000000002e-06 2024-09-15T03:48:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864215+0000", 2024-09-15T03:48:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.337e-06 2024-09-15T03:48:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864221+0000", 2024-09-15T03:48:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4570000000000002e-06 2024-09-15T03:48:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.567 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864255+0000", 2024-09-15T03:48:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3725e-05 2024-09-15T03:48:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:48:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864260+0000", 2024-09-15T03:48:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.395e-06 2024-09-15T03:48:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.871305+0000", 2024-09-15T03:48:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0070446800000000002 2024-09-15T03:48:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.568 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.871413+0000", 2024-09-15T03:48:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000107731 2024-09-15T03:48:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.876917+0000", 2024-09-15T03:48:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0055046510000000002 2024-09-15T03:48:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:385 3.es0 3:77a7da34:::smithi00132172-4:head [read 0~1146880,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e56)", 2024-09-15T03:48:03.569 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:02.864230+0000", 2024-09-15T03:48:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 60.690808625000003, 2024-09-15T03:48:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.014027420000000001, 2024-09-15T03:48:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 385 2024-09-15T03:48:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864230+0000", 2024-09-15T03:48:03.570 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864230+0000", 2024-09-15T03:48:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864231+0000", 2024-09-15T03:48:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7700000000000005e-07 2024-09-15T03:48:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864234+0000", 2024-09-15T03:48:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0369999999999999e-06 2024-09-15T03:48:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864242+0000", 2024-09-15T03:48:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4200000000000007e-06 2024-09-15T03:48:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864247+0000", 2024-09-15T03:48:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4789999999999998e-06 2024-09-15T03:48:03.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864290+0000", 2024-09-15T03:48:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3594000000000003e-05 2024-09-15T03:48:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:48:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864297+0000", 2024-09-15T03:48:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3579999999999999e-06 2024-09-15T03:48:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.573 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.872828+0000", 2024-09-15T03:48:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0085315449999999998 2024-09-15T03:48:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.872956+0000", 2024-09-15T03:48:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000127818 2024-09-15T03:48:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.878258+0000", 2024-09-15T03:48:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0053016920000000002 2024-09-15T03:48:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.574 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:386 3.es0 3:77a7da34:::smithi00132172-4:head [read 0~1146880,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e56)", 2024-09-15T03:48:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:02.864259+0000", 2024-09-15T03:48:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 60.690779831, 2024-09-15T03:48:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.014933481, 2024-09-15T03:48:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 386 2024-09-15T03:48:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.575 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864259+0000", 2024-09-15T03:48:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864259+0000", 2024-09-15T03:48:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.576 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864260+0000", 2024-09-15T03:48:03.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3799999999999998e-07 2024-09-15T03:48:03.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864262+0000", 2024-09-15T03:48:03.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7209999999999999e-06 2024-09-15T03:48:03.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.577 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864270+0000", 2024-09-15T03:48:03.577 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6240000000000002e-06 2024-09-15T03:48:03.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.577 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864274+0000", 2024-09-15T03:48:03.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1699999999999999e-06 2024-09-15T03:48:03.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864316+0000", 2024-09-15T03:48:03.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1489999999999997e-05 2024-09-15T03:48:03.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.578 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:48:03.578 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864321+0000", 2024-09-15T03:48:03.578 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1100000000000002e-06 2024-09-15T03:48:03.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.873888+0000", 2024-09-15T03:48:03.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0095676790000000008 2024-09-15T03:48:03.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:03.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.874019+0000", 2024-09-15T03:48:03.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013085199999999999 2024-09-15T03:48:03.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.579 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.579 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.879193+0000", 2024-09-15T03:48:03.579 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0051733969999999997 2024-09-15T03:48:03.579 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.580 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.580 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.580 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 60/31 ECSubWrite(tid=70, reqid=client.4521.0:421, at_version=60'22, trim_to=0'0, roll_forward_to=47'20))", 2024-09-15T03:48:03.580 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.837585+0000", 2024-09-15T03:48:03.580 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 56.717454539999999, 2024-09-15T03:48:03.580 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011527037, 2024-09-15T03:48:03.580 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.580 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.580 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.580 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837585+0000", 2024-09-15T03:48:03.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.581 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837585+0000", 2024-09-15T03:48:03.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.581 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.581 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.581 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837585+0000", 2024-09-15T03:48:03.581 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1199999999999999e-07 2024-09-15T03:48:03.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837609+0000", 2024-09-15T03:48:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4647999999999998e-05 2024-09-15T03:48:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837631+0000", 2024-09-15T03:48:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1840000000000001e-05 2024-09-15T03:48:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837641+0000", 2024-09-15T03:48:03.582 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0263e-05 2024-09-15T03:48:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.838036+0000", 2024-09-15T03:48:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00039492500000000002 2024-09-15T03:48:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.838048+0000", 2024-09-15T03:48:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1599e-05 2024-09-15T03:48:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.583 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.838445+0000", 2024-09-15T03:48:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000397313 2024-09-15T03:48:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.838479+0000", 2024-09-15T03:48:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3368000000000003e-05 2024-09-15T03:48:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.838525+0000", 2024-09-15T03:48:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6100999999999998e-05 2024-09-15T03:48:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.841589+0000", 2024-09-15T03:48:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003063957 2024-09-15T03:48:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.849112+0000", 2024-09-15T03:48:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0075228109999999999 2024-09-15T03:48:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:446 3.es0 3:75485f06:::smithi00132172-19:head [copy-from ver 38] snapc 1a=[1a,18,17,16,15,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:48:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.951492+0000", 2024-09-15T03:48:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 56.603547444999997, 2024-09-15T03:48:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013140193, 2024-09-15T03:48:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 446 2024-09-15T03:48:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951492+0000", 2024-09-15T03:48:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951492+0000", 2024-09-15T03:48:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951493+0000", 2024-09-15T03:48:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.525e-06 2024-09-15T03:48:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951500+0000", 2024-09-15T03:48:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6649999999999997e-06 2024-09-15T03:48:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951514+0000", 2024-09-15T03:48:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3713999999999999e-05 2024-09-15T03:48:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951518+0000", 2024-09-15T03:48:03.588 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.921e-06 2024-09-15T03:48:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951569+0000", 2024-09-15T03:48:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0281999999999997e-05 2024-09-15T03:48:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951676+0000", 2024-09-15T03:48:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000107527 2024-09-15T03:48:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.589 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.961625+0000", 2024-09-15T03:48:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0099492209999999994 2024-09-15T03:48:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.962100+0000", 2024-09-15T03:48:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00047458100000000002 2024-09-15T03:48:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.962112+0000", 2024-09-15T03:48:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2069000000000001e-05 2024-09-15T03:48:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.962343+0000", 2024-09-15T03:48:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000230962 2024-09-15T03:48:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.962561+0000", 2024-09-15T03:48:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000217899 2024-09-15T03:48:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.964176+0000", 2024-09-15T03:48:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001615109 2024-09-15T03:48:03.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.964632+0000", 2024-09-15T03:48:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00045571799999999998 2024-09-15T03:48:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 65/64 ECSubWrite(tid=106, reqid=client.4521.0:464, at_version=65'25, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:48:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.852924+0000", 2024-09-15T03:48:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 51.702114917999999, 2024-09-15T03:48:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013666606, 2024-09-15T03:48:03.592 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852924+0000", 2024-09-15T03:48:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852924+0000", 2024-09-15T03:48:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852925+0000", 2024-09-15T03:48:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9999999999999997e-07 2024-09-15T03:48:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852945+0000", 2024-09-15T03:48:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9952000000000001e-05 2024-09-15T03:48:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852957+0000", 2024-09-15T03:48:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2532e-05 2024-09-15T03:48:03.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852965+0000", 2024-09-15T03:48:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6359999999999994e-06 2024-09-15T03:48:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.853100+0000", 2024-09-15T03:48:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013520900000000001 2024-09-15T03:48:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.853111+0000", 2024-09-15T03:48:03.595 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0508e-05 2024-09-15T03:48:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.853220+0000", 2024-09-15T03:48:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000109053 2024-09-15T03:48:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.853247+0000", 2024-09-15T03:48:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7025999999999998e-05 2024-09-15T03:48:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.596 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854114+0000", 2024-09-15T03:48:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00086709299999999999 2024-09-15T03:48:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854129+0000", 2024-09-15T03:48:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5602000000000001e-05 2024-09-15T03:48:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866591+0000", 2024-09-15T03:48:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012461395 2024-09-15T03:48:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.597 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 65/64 ECSubWrite(tid=107, reqid=client.4521.0:465, at_version=65'26, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:48:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.854248+0000", 2024-09-15T03:48:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 51.700790652000002, 2024-09-15T03:48:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012733421999999999, 2024-09-15T03:48:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.598 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854248+0000", 2024-09-15T03:48:03.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.599 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.599 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854248+0000", 2024-09-15T03:48:03.599 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.599 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.599 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.600 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854249+0000", 2024-09-15T03:48:03.600 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8999999999999997e-07 2024-09-15T03:48:03.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.600 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.600 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854274+0000", 2024-09-15T03:48:03.600 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4678e-05 2024-09-15T03:48:03.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.600 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.600 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.600 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854289+0000", 2024-09-15T03:48:03.600 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5622000000000001e-05 2024-09-15T03:48:03.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.601 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854296+0000", 2024-09-15T03:48:03.601 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2869999999999999e-06 2024-09-15T03:48:03.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.601 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859614+0000", 2024-09-15T03:48:03.601 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0053176869999999998 2024-09-15T03:48:03.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.601 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.601 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859625+0000", 2024-09-15T03:48:03.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0475000000000001e-05 2024-09-15T03:48:03.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.602 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.602 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859728+0000", 2024-09-15T03:48:03.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010383 2024-09-15T03:48:03.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.602 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.602 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859769+0000", 2024-09-15T03:48:03.602 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0132999999999997e-05 2024-09-15T03:48:03.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.602 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.603 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859855+0000", 2024-09-15T03:48:03.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6188999999999998e-05 2024-09-15T03:48:03.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.603 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.603 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860468+0000", 2024-09-15T03:48:03.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00061328800000000003 2024-09-15T03:48:03.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.603 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.603 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866982+0000", 2024-09-15T03:48:03.603 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0065137429999999998 2024-09-15T03:48:03.603 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.603 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.604 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.604 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 65/64 ECSubWrite(tid=108, reqid=client.4521.0:466, at_version=65'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:48:03.604 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.858803+0000", 2024-09-15T03:48:03.604 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 51.696236231, 2024-09-15T03:48:03.604 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012489828999999999, 2024-09-15T03:48:03.604 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.604 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.604 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.604 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.604 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.604 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858803+0000", 2024-09-15T03:48:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858803+0000", 2024-09-15T03:48:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858803+0000", 2024-09-15T03:48:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1699999999999999e-07 2024-09-15T03:48:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.605 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.606 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858970+0000", 2024-09-15T03:48:03.606 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000166844 2024-09-15T03:48:03.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.606 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858982+0000", 2024-09-15T03:48:03.606 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2092999999999999e-05 2024-09-15T03:48:03.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.606 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.606 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858988+0000", 2024-09-15T03:48:03.606 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1999999999999999e-06 2024-09-15T03:48:03.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.607 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.607 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859762+0000", 2024-09-15T03:48:03.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00077351199999999996 2024-09-15T03:48:03.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.607 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.607 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859770+0000", 2024-09-15T03:48:03.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0360000000000006e-06 2024-09-15T03:48:03.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.607 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.607 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860086+0000", 2024-09-15T03:48:03.607 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031557999999999998 2024-09-15T03:48:03.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.608 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.608 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860099+0000", 2024-09-15T03:48:03.608 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3329999999999999e-05 2024-09-15T03:48:03.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.608 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.608 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860225+0000", 2024-09-15T03:48:03.608 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012638300000000001 2024-09-15T03:48:03.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.608 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.608 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861591+0000", 2024-09-15T03:48:03.608 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0013660059999999999 2024-09-15T03:48:03.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.609 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.609 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871293+0000", 2024-09-15T03:48:03.609 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0097014279999999998 2024-09-15T03:48:03.609 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.609 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.609 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.609 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 65/64 ECSubWrite(tid=111, reqid=client.4521.0:467, at_version=65'29, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:48:03.609 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.858996+0000", 2024-09-15T03:48:03.609 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 51.696043500999998, 2024-09-15T03:48:03.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012595857, 2024-09-15T03:48:03.610 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.610 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.610 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.610 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.610 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858996+0000", 2024-09-15T03:48:03.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.610 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.610 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858996+0000", 2024-09-15T03:48:03.610 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.611 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.611 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858996+0000", 2024-09-15T03:48:03.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9900000000000001e-07 2024-09-15T03:48:03.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.611 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.611 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859088+0000", 2024-09-15T03:48:03.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2473999999999996e-05 2024-09-15T03:48:03.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.611 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.611 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859102+0000", 2024-09-15T03:48:03.611 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3933e-05 2024-09-15T03:48:03.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.612 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.612 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859114+0000", 2024-09-15T03:48:03.612 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1262e-05 2024-09-15T03:48:03.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.612 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.612 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860136+0000", 2024-09-15T03:48:03.612 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010221169999999999 2024-09-15T03:48:03.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.612 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.612 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860144+0000", 2024-09-15T03:48:03.613 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6270000000000006e-06 2024-09-15T03:48:03.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.613 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.613 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860264+0000", 2024-09-15T03:48:03.613 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000119954 2024-09-15T03:48:03.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860274+0000", 2024-09-15T03:48:03.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8959999999999998e-06 2024-09-15T03:48:03.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860365+0000", 2024-09-15T03:48:03.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1195999999999999e-05 2024-09-15T03:48:03.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.616 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.616 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.616 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861615+0000", 2024-09-15T03:48:03.616 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012495690000000001 2024-09-15T03:48:03.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.617 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.617 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871591+0000", 2024-09-15T03:48:03.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0099764299999999997 2024-09-15T03:48:03.617 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.617 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.617 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.617 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.617 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 65/64 ECSubWrite(tid=112, reqid=client.4521.0:468, at_version=65'30, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:48:03.617 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.859121+0000", 2024-09-15T03:48:03.617 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 51.695918239000001, 2024-09-15T03:48:03.617 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012988807, 2024-09-15T03:48:03.618 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.618 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.618 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859121+0000", 2024-09-15T03:48:03.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.618 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.618 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.618 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859121+0000", 2024-09-15T03:48:03.618 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859121+0000", 2024-09-15T03:48:03.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4499999999999998e-07 2024-09-15T03:48:03.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859170+0000", 2024-09-15T03:48:03.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9008999999999998e-05 2024-09-15T03:48:03.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.619 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.619 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.619 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859179+0000", 2024-09-15T03:48:03.619 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5809999999999997e-06 2024-09-15T03:48:03.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859185+0000", 2024-09-15T03:48:03.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7350000000000001e-06 2024-09-15T03:48:03.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860314+0000", 2024-09-15T03:48:03.620 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001129403 2024-09-15T03:48:03.620 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.620 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.620 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.620 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860324+0000", 2024-09-15T03:48:03.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0274e-05 2024-09-15T03:48:03.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.621 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.621 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860440+0000", 2024-09-15T03:48:03.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011603 2024-09-15T03:48:03.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.621 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.621 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860450+0000", 2024-09-15T03:48:03.621 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0243e-05 2024-09-15T03:48:03.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.621 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.621 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860518+0000", 2024-09-15T03:48:03.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7780000000000005e-05 2024-09-15T03:48:03.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861625+0000", 2024-09-15T03:48:03.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0011072460000000001 2024-09-15T03:48:03.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.622 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.622 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.622 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.872110+0000", 2024-09-15T03:48:03.622 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010484161000000001 2024-09-15T03:48:03.622 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.623 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.623 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.623 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 65/64 ECSubWrite(tid=113, reqid=client.4521.0:469, at_version=65'31, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:48:03.623 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.860777+0000", 2024-09-15T03:48:03.623 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 51.694261656999998, 2024-09-15T03:48:03.623 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012054540000000001, 2024-09-15T03:48:03.623 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.623 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.623 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.623 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.623 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860777+0000", 2024-09-15T03:48:03.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860777+0000", 2024-09-15T03:48:03.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.624 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860778+0000", 2024-09-15T03:48:03.624 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3399999999999999e-07 2024-09-15T03:48:03.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.624 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860795+0000", 2024-09-15T03:48:03.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6691e-05 2024-09-15T03:48:03.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.625 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860807+0000", 2024-09-15T03:48:03.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2149e-05 2024-09-15T03:48:03.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.625 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.625 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860814+0000", 2024-09-15T03:48:03.625 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.255e-06 2024-09-15T03:48:03.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861681+0000", 2024-09-15T03:48:03.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00086670300000000003 2024-09-15T03:48:03.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861690+0000", 2024-09-15T03:48:03.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3039999999999994e-06 2024-09-15T03:48:03.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.626 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.626 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861789+0000", 2024-09-15T03:48:03.626 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8851999999999995e-05 2024-09-15T03:48:03.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861819+0000", 2024-09-15T03:48:03.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9646e-05 2024-09-15T03:48:03.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.627 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861852+0000", 2024-09-15T03:48:03.627 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3436000000000003e-05 2024-09-15T03:48:03.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.627 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861871+0000", 2024-09-15T03:48:03.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9202e-05 2024-09-15T03:48:03.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.628 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.628 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.872832+0000", 2024-09-15T03:48:03.628 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010960668 2024-09-15T03:48:03.628 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.628 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.628 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.628 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 65/64 ECSubWrite(tid=116, reqid=client.4521.0:469, at_version=0'0, trim_to=0'0, roll_forward_to=65'31))", 2024-09-15T03:48:03.629 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.866498+0000", 2024-09-15T03:48:03.629 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 51.688540947999996, 2024-09-15T03:48:03.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013459561, 2024-09-15T03:48:03.629 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.629 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:03.629 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866498+0000", 2024-09-15T03:48:03.629 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.629 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.629 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866498+0000", 2024-09-15T03:48:03.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866499+0000", 2024-09-15T03:48:03.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9800000000000003e-07 2024-09-15T03:48:03.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.630 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.630 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866509+0000", 2024-09-15T03:48:03.630 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8980000000000008e-06 2024-09-15T03:48:03.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866520+0000", 2024-09-15T03:48:03.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0927000000000001e-05 2024-09-15T03:48:03.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866528+0000", 2024-09-15T03:48:03.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.388e-06 2024-09-15T03:48:03.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.631 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.631 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870788+0000", 2024-09-15T03:48:03.631 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.004259657 2024-09-15T03:48:03.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870798+0000", 2024-09-15T03:48:03.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0156999999999999e-05 2024-09-15T03:48:03.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871002+0000", 2024-09-15T03:48:03.632 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000204174 2024-09-15T03:48:03.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.632 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.632 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871015+0000", 2024-09-15T03:48:03.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3422e-05 2024-09-15T03:48:03.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871057+0000", 2024-09-15T03:48:03.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1270999999999998e-05 2024-09-15T03:48:03.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.633 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.633 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.872441+0000", 2024-09-15T03:48:03.633 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0013840980000000001 2024-09-15T03:48:03.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.634 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.634 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.879958+0000", 2024-09-15T03:48:03.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0075169709999999999 2024-09-15T03:48:03.634 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.634 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.634 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.634 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:488 3.es0 3:75485f06:::smithi00132172-19:head [rollback 12] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-15T03:48:03.634 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.867857+0000", 2024-09-15T03:48:03.634 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 51.687181703, 2024-09-15T03:48:03.634 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011714672000000001, 2024-09-15T03:48:03.634 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.635 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:03.635 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:03.635 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:03.635 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:03.635 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 488 2024-09-15T03:48:03.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.635 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.635 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867857+0000", 2024-09-15T03:48:03.635 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.635 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867857+0000", 2024-09-15T03:48:03.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867859+0000", 2024-09-15T03:48:03.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.562e-06 2024-09-15T03:48:03.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.636 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.636 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867874+0000", 2024-09-15T03:48:03.636 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5170999999999999e-05 2024-09-15T03:48:03.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867886+0000", 2024-09-15T03:48:03.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1819e-05 2024-09-15T03:48:03.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867892+0000", 2024-09-15T03:48:03.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0449999999999997e-06 2024-09-15T03:48:03.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.637 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.637 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868553+0000", 2024-09-15T03:48:03.637 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00066058700000000002 2024-09-15T03:48:03.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:03.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868660+0000", 2024-09-15T03:48:03.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000107878 2024-09-15T03:48:03.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869303+0000", 2024-09-15T03:48:03.638 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00064280399999999997 2024-09-15T03:48:03.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.638 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.638 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869419+0000", 2024-09-15T03:48:03.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011619400000000001 2024-09-15T03:48:03.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869431+0000", 2024-09-15T03:48:03.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1941e-05 2024-09-15T03:48:03.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.639 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869461+0000", 2024-09-15T03:48:03.639 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9478000000000001e-05 2024-09-15T03:48:03.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.639 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.639 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870370+0000", 2024-09-15T03:48:03.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00090957899999999999 2024-09-15T03:48:03.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:03.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.873494+0000", 2024-09-15T03:48:03.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0031240679999999998 2024-09-15T03:48:03.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.640 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.640 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.879572+0000", 2024-09-15T03:48:03.640 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0060775459999999996 2024-09-15T03:48:03.640 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.640 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.641 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.641 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:489 3.es0 3:75485f06:::smithi00132172-19:head [append 0~0] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-15T03:48:03.641 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.867898+0000", 2024-09-15T03:48:03.641 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 51.687141093999998, 2024-09-15T03:48:03.641 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012220945, 2024-09-15T03:48:03.641 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.641 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:03.641 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:03.641 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:03.641 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:03.641 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 489 2024-09-15T03:48:03.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.642 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867898+0000", 2024-09-15T03:48:03.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.642 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867898+0000", 2024-09-15T03:48:03.642 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.642 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867899+0000", 2024-09-15T03:48:03.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0900000000000001e-07 2024-09-15T03:48:03.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867901+0000", 2024-09-15T03:48:03.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.65e-06 2024-09-15T03:48:03.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.643 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.643 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867908+0000", 2024-09-15T03:48:03.643 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2790000000000004e-06 2024-09-15T03:48:03.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867911+0000", 2024-09-15T03:48:03.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9700000000000001e-06 2024-09-15T03:48:03.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869480+0000", 2024-09-15T03:48:03.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001568051 2024-09-15T03:48:03.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.644 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:03.644 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869565+0000", 2024-09-15T03:48:03.644 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5500999999999993e-05 2024-09-15T03:48:03.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870027+0000", 2024-09-15T03:48:03.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00046226900000000002 2024-09-15T03:48:03.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870140+0000", 2024-09-15T03:48:03.645 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112223 2024-09-15T03:48:03.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.645 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.645 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870168+0000", 2024-09-15T03:48:03.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8031e-05 2024-09-15T03:48:03.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870202+0000", 2024-09-15T03:48:03.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4598999999999999e-05 2024-09-15T03:48:03.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.646 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.646 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870372+0000", 2024-09-15T03:48:03.646 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016990199999999999 2024-09-15T03:48:03.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:03.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.879822+0000", 2024-09-15T03:48:03.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0094500739999999993 2024-09-15T03:48:03.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.647 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.647 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.880119+0000", 2024-09-15T03:48:03.647 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029678700000000002 2024-09-15T03:48:03.647 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.647 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.648 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.648 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:830 3.es0 3:79526987:::smithi00132172-28:head [copy-from ver 107] snapc 3d=[3d,3b,3a,38,37,35,34,32,2d,2b,25,24,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-15T03:48:03.648 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:43.912967+0000", 2024-09-15T03:48:03.648 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 19.642071791999999, 2024-09-15T03:48:03.648 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.026454678999999998, 2024-09-15T03:48:03.648 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.648 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:03.648 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:03.648 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:03.648 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:03.648 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 830 2024-09-15T03:48:03.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.649 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.912967+0000", 2024-09-15T03:48:03.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.649 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.912967+0000", 2024-09-15T03:48:03.649 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.649 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.912969+0000", 2024-09-15T03:48:03.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.939e-06 2024-09-15T03:48:03.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.912980+0000", 2024-09-15T03:48:03.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.04e-05 2024-09-15T03:48:03.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.650 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.650 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.912999+0000", 2024-09-15T03:48:03.650 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9391000000000001e-05 2024-09-15T03:48:03.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913015+0000", 2024-09-15T03:48:03.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5568000000000001e-05 2024-09-15T03:48:03.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913068+0000", 2024-09-15T03:48:03.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3270999999999999e-05 2024-09-15T03:48:03.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.651 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:03.651 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913152+0000", 2024-09-15T03:48:03.651 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4155999999999999e-05 2024-09-15T03:48:03.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:03.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.934430+0000", 2024-09-15T03:48:03.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.021278372 2024-09-15T03:48:03.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:03.652 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.934773+0000", 2024-09-15T03:48:03.652 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00034307700000000001 2024-09-15T03:48:03.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.652 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:03.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.934792+0000", 2024-09-15T03:48:03.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8640000000000001e-05 2024-09-15T03:48:03.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:03.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.935203+0000", 2024-09-15T03:48:03.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00041061100000000002 2024-09-15T03:48:03.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.653 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:03.653 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.935232+0000", 2024-09-15T03:48:03.653 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9093000000000001e-05 2024-09-15T03:48:03.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:03.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.939341+0000", 2024-09-15T03:48:03.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041089849999999999 2024-09-15T03:48:03.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.654 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.654 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.939422+0000", 2024-09-15T03:48:03.654 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1175999999999997e-05 2024-09-15T03:48:03.654 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.654 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.654 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.655 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:831 3.es0 3:79526987:::smithi00132172-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2024-09-15T03:48:03.655 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:43.913039+0000", 2024-09-15T03:48:03.655 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 19.641999824999999, 2024-09-15T03:48:03.655 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.023987538999999999, 2024-09-15T03:48:03.655 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.655 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:03.655 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:03.655 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:03.655 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:03.655 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 831 2024-09-15T03:48:03.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.655 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.655 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913039+0000", 2024-09-15T03:48:03.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913039+0000", 2024-09-15T03:48:03.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.656 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.656 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913040+0000", 2024-09-15T03:48:03.656 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4900000000000005e-07 2024-09-15T03:48:03.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913044+0000", 2024-09-15T03:48:03.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0550000000000001e-06 2024-09-15T03:48:03.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913053+0000", 2024-09-15T03:48:03.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4150000000000005e-06 2024-09-15T03:48:03.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.657 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.657 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913058+0000", 2024-09-15T03:48:03.657 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8609999999999997e-06 2024-09-15T03:48:03.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913284+0000", 2024-09-15T03:48:03.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022525400000000001 2024-09-15T03:48:03.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for blocked object", 2024-09-15T03:48:03.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913347+0000", 2024-09-15T03:48:03.658 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3400999999999998e-05 2024-09-15T03:48:03.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.658 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.658 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.936662+0000", 2024-09-15T03:48:03.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.023315361 2024-09-15T03:48:03.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:03.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.936840+0000", 2024-09-15T03:48:03.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017759699999999999 2024-09-15T03:48:03.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.659 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.659 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.937027+0000", 2024-09-15T03:48:03.659 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018684600000000001 2024-09-15T03:48:03.659 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.659 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.659 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.660 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:1016 3.es0 3:7e6c26d0:::smithi00132172-14:head [read 0~4571136,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112)", 2024-09-15T03:48:03.660 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:58.980413+0000", 2024-09-15T03:48:03.660 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 4.5746261480000001, 2024-09-15T03:48:03.660 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012968637999999999, 2024-09-15T03:48:03.660 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.660 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:03.660 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:03.660 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:03.660 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:03.660 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 1016 2024-09-15T03:48:03.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.661 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980413+0000", 2024-09-15T03:48:03.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980413+0000", 2024-09-15T03:48:03.661 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.661 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.661 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980416+0000", 2024-09-15T03:48:03.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.79e-06 2024-09-15T03:48:03.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980428+0000", 2024-09-15T03:48:03.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1942e-05 2024-09-15T03:48:03.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.662 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.662 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980449+0000", 2024-09-15T03:48:03.662 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1399999999999998e-05 2024-09-15T03:48:03.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.663 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.663 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980457+0000", 2024-09-15T03:48:03.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8709999999999997e-06 2024-09-15T03:48:03.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.663 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.663 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980543+0000", 2024-09-15T03:48:03.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5870000000000003e-05 2024-09-15T03:48:03.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.663 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:03.663 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980678+0000", 2024-09-15T03:48:03.663 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013535399999999999 2024-09-15T03:48:03.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.664 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.664 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.993382+0000", 2024-09-15T03:48:03.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012703411 2024-09-15T03:48:03.664 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.664 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.664 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.664 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:1017 3.es0 3:7e6c26d0:::smithi00132172-14:head [read 0~4571136,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112)", 2024-09-15T03:48:03.664 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:58.980469+0000", 2024-09-15T03:48:03.664 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 4.5745704790000001, 2024-09-15T03:48:03.664 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.016319225999999999, 2024-09-15T03:48:03.664 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 1017 2024-09-15T03:48:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980469+0000", 2024-09-15T03:48:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980469+0000", 2024-09-15T03:48:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980469+0000", 2024-09-15T03:48:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5200000000000002e-07 2024-09-15T03:48:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980473+0000", 2024-09-15T03:48:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7019999999999999e-06 2024-09-15T03:48:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980478+0000", 2024-09-15T03:48:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4770000000000002e-06 2024-09-15T03:48:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980481+0000", 2024-09-15T03:48:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.543e-06 2024-09-15T03:48:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.981078+0000", 2024-09-15T03:48:03.667 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00059686400000000001 2024-09-15T03:48:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.981163+0000", 2024-09-15T03:48:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5185999999999994e-05 2024-09-15T03:48:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.996788+0000", 2024-09-15T03:48:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.015625001999999999 2024-09-15T03:48:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:1018 3.es0 3:7e6c26d0:::smithi00132172-14:head [read 0~4571136,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112)", 2024-09-15T03:48:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:58.980510+0000", 2024-09-15T03:48:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 4.5745295109999997, 2024-09-15T03:48:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.019757963, 2024-09-15T03:48:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 1018 2024-09-15T03:48:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:03.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980510+0000", 2024-09-15T03:48:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980510+0000", 2024-09-15T03:48:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980510+0000", 2024-09-15T03:48:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7599999999999996e-07 2024-09-15T03:48:03.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980518+0000", 2024-09-15T03:48:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3180000000000001e-06 2024-09-15T03:48:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980529+0000", 2024-09-15T03:48:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1158000000000001e-05 2024-09-15T03:48:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980535+0000", 2024-09-15T03:48:03.671 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2029999999999997e-06 2024-09-15T03:48:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.983134+0000", 2024-09-15T03:48:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00259903 2024-09-15T03:48:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.983239+0000", 2024-09-15T03:48:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010511800000000001 2024-09-15T03:48:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:03.672 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:03.673 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.000268+0000", 2024-09-15T03:48:03.673 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.01702826 2024-09-15T03:48:03.673 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.673 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.673 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.673 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:03.673 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:03.673 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:03.675 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:48:03.675 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:48:03.674+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:48:03.775 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:48:03.775+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:48:03.818 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:03.818 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:48:03.818 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-15T03:48:03.818 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:03.826 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:48:03.876 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:48:03.875+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:48:03.929 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:03.929 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:48:03.929 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-15T03:48:03.929 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-15T03:48:03.929 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:03.937 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:48:03.962 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.9 to 1 (10000) 2024-09-15T03:48:03.976 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-15T03:48:03.977 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:48:03.977+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:48:04.067 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:04.068 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-15T03:48:04.068 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-15T03:48:04.068 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-15T03:48:04.068 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.068 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:333 3.9s0 3:94e9e019:::smithi00132172-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:48:04.068 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.856848+0000", 2024-09-15T03:48:04.068 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 68.208859373999999, 2024-09-15T03:48:04.068 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012151039000000001, 2024-09-15T03:48:04.068 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:04.068 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:04.069 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:04.069 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:04.069 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.069 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 333 2024-09-15T03:48:04.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.069 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:04.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.069 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:04.069 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856848+0000", 2024-09-15T03:48:04.069 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.069 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:04.069 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856848+0000", 2024-09-15T03:48:04.070 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.070 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:04.070 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856849+0000", 2024-09-15T03:48:04.070 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5499999999999999e-07 2024-09-15T03:48:04.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.070 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:04.070 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:48:04.070 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.378e-06 2024-09-15T03:48:04.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.070 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:04.071 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856854+0000", 2024-09-15T03:48:04.071 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.653e-06 2024-09-15T03:48:04.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.071 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.071 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856856+0000", 2024-09-15T03:48:04.071 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4059999999999999e-06 2024-09-15T03:48:04.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.071 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.071 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.862404+0000", 2024-09-15T03:48:04.071 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0055481310000000004 2024-09-15T03:48:04.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.072 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for rw locks", 2024-09-15T03:48:04.072 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.862519+0000", 2024-09-15T03:48:04.072 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011449400000000001 2024-09-15T03:48:04.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.072 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.072 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864592+0000", 2024-09-15T03:48:04.072 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020730200000000001 2024-09-15T03:48:04.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.072 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:04.073 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864687+0000", 2024-09-15T03:48:04.073 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5531000000000004e-05 2024-09-15T03:48:04.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.073 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:04.073 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.868999+0000", 2024-09-15T03:48:04.073 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0043121710000000001 2024-09-15T03:48:04.073 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.073 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:04.073 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.074 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:458 3.9s0 3:97177042:::smithi00132172-18:head [append 0~753664] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:48:04.074 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.841113+0000", 2024-09-15T03:48:04.074 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 53.224594969000002, 2024-09-15T03:48:04.074 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0334256719999999, 2024-09-15T03:48:04.074 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:04.074 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.074 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:04.074 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:04.074 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.074 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 458 2024-09-15T03:48:04.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.075 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:04.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.075 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:04.076 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841113+0000", 2024-09-15T03:48:04.076 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.076 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.077 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:04.077 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841113+0000", 2024-09-15T03:48:04.077 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.077 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:04.077 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841114+0000", 2024-09-15T03:48:04.077 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.491e-06 2024-09-15T03:48:04.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.077 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.077 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:04.078 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841577+0000", 2024-09-15T03:48:04.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000462571 2024-09-15T03:48:04.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.078 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:04.078 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841589+0000", 2024-09-15T03:48:04.078 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2119999999999999e-05 2024-09-15T03:48:04.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.078 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.078 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.079 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841596+0000", 2024-09-15T03:48:04.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0740000000000004e-06 2024-09-15T03:48:04.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.079 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.079 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841708+0000", 2024-09-15T03:48:04.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112126 2024-09-15T03:48:04.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.079 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.079 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:48:04.079 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841712+0000", 2024-09-15T03:48:04.079 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3639999999999999e-06 2024-09-15T03:48:04.079 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.856485+0000", 2024-09-15T03:48:04.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0147736949999999 2024-09-15T03:48:04.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:04.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.856699+0000", 2024-09-15T03:48:04.080 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000213372 2024-09-15T03:48:04.080 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.080 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.080 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.080 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.862261+0000", 2024-09-15T03:48:04.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0055625450000000003 2024-09-15T03:48:04.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.862781+0000", 2024-09-15T03:48:04.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00052013500000000002 2024-09-15T03:48:04.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.081 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.862885+0000", 2024-09-15T03:48:04.081 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103455 2024-09-15T03:48:04.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.081 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.081 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.863028+0000", 2024-09-15T03:48:04.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014313999999999999 2024-09-15T03:48:04.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.864079+0000", 2024-09-15T03:48:04.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010511229999999999 2024-09-15T03:48:04.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.082 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:04.082 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874458+0000", 2024-09-15T03:48:04.082 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010379019999999999 2024-09-15T03:48:04.082 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.082 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.083 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:04.083 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874538+0000", 2024-09-15T03:48:04.083 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0440999999999998e-05 2024-09-15T03:48:04.083 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.083 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:04.083 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.083 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:459 3.9s0 3:97177042:::smithi00132172-18:head [append 0~720896] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:48:04.083 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.846766+0000", 2024-09-15T03:48:04.084 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 53.218941346000001, 2024-09-15T03:48:04.084 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0281560110000001, 2024-09-15T03:48:04.084 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:04.084 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.084 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:04.084 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:04.084 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.084 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 459 2024-09-15T03:48:04.084 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.084 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:04.084 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.084 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:04.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.846766+0000", 2024-09-15T03:48:04.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:04.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.846766+0000", 2024-09-15T03:48:04.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.085 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:04.085 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.846769+0000", 2024-09-15T03:48:04.085 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.323e-06 2024-09-15T03:48:04.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.085 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.086 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:04.086 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847323+0000", 2024-09-15T03:48:04.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000554267 2024-09-15T03:48:04.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.086 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:04.086 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847338+0000", 2024-09-15T03:48:04.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.465e-05 2024-09-15T03:48:04.086 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.086 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.086 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.086 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847345+0000", 2024-09-15T03:48:04.086 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6029999999999999e-06 2024-09-15T03:48:04.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.087 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847494+0000", 2024-09-15T03:48:04.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000148541 2024-09-15T03:48:04.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:48:04.087 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847498+0000", 2024-09-15T03:48:04.087 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7629999999999998e-06 2024-09-15T03:48:04.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.087 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.087 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.864944+0000", 2024-09-15T03:48:04.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0174461050000001 2024-09-15T03:48:04.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:04.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.865122+0000", 2024-09-15T03:48:04.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000178123 2024-09-15T03:48:04.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.088 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.088 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.088 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867632+0000", 2024-09-15T03:48:04.088 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002510249 2024-09-15T03:48:04.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867775+0000", 2024-09-15T03:48:04.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014326000000000001 2024-09-15T03:48:04.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868663+0000", 2024-09-15T03:48:04.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00088783799999999995 2024-09-15T03:48:04.089 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.089 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.089 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.089 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868802+0000", 2024-09-15T03:48:04.089 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000139173 2024-09-15T03:48:04.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874167+0000", 2024-09-15T03:48:04.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.005365117 2024-09-15T03:48:04.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:04.090 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874842+0000", 2024-09-15T03:48:04.090 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00067414199999999997 2024-09-15T03:48:04.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.090 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.090 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:04.091 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874922+0000", 2024-09-15T03:48:04.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0857000000000002e-05 2024-09-15T03:48:04.091 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.091 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:04.091 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.091 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.091 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:460 3.9s0 3:97177042:::smithi00132172-18:head [append 0~49152] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:48:04.091 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.847359+0000", 2024-09-15T03:48:04.091 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 53.218348972000001, 2024-09-15T03:48:04.091 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0278644889999999, 2024-09-15T03:48:04.091 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:04.091 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.091 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:04.092 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:04.092 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.092 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 460 2024-09-15T03:48:04.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.092 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:04.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.092 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:04.092 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847359+0000", 2024-09-15T03:48:04.092 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.092 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:04.092 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847359+0000", 2024-09-15T03:48:04.093 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.093 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.093 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:04.093 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847360+0000", 2024-09-15T03:48:04.093 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.26e-06 2024-09-15T03:48:04.093 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.093 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:04.093 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847390+0000", 2024-09-15T03:48:04.093 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9533999999999999e-05 2024-09-15T03:48:04.093 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.093 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.093 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847406+0000", 2024-09-15T03:48:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6725999999999999e-05 2024-09-15T03:48:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847412+0000", 2024-09-15T03:48:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4149999999999998e-06 2024-09-15T03:48:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847522+0000", 2024-09-15T03:48:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011041500000000001 2024-09-15T03:48:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.095 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:48:04.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847523+0000", 2024-09-15T03:48:04.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.09e-07 2024-09-15T03:48:04.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.095 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868928+0000", 2024-09-15T03:48:04.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0214053089999999 2024-09-15T03:48:04.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.095 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:04.095 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869043+0000", 2024-09-15T03:48:04.095 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000115021 2024-09-15T03:48:04.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869635+0000", 2024-09-15T03:48:04.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00059164199999999997 2024-09-15T03:48:04.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869755+0000", 2024-09-15T03:48:04.096 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0001205 2024-09-15T03:48:04.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.096 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.096 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870274+0000", 2024-09-15T03:48:04.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00051842900000000005 2024-09-15T03:48:04.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870322+0000", 2024-09-15T03:48:04.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8164000000000002e-05 2024-09-15T03:48:04.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.097 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.097 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874941+0000", 2024-09-15T03:48:04.097 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0046186140000000001 2024-09-15T03:48:04.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.098 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:04.098 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875140+0000", 2024-09-15T03:48:04.098 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019890700000000001 2024-09-15T03:48:04.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.098 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:04.098 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875223+0000", 2024-09-15T03:48:04.098 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3644000000000001e-05 2024-09-15T03:48:04.098 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.098 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:04.098 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.099 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:461 3.9s0 3:97177042:::smithi00132172-18:head [setxattr _header (57) in=7b] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:48:04.099 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.847417+0000", 2024-09-15T03:48:04.099 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 53.218290867999997, 2024-09-15T03:48:04.099 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0332682499999999, 2024-09-15T03:48:04.099 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:04.099 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.099 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:04.099 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:04.099 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.099 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 461 2024-09-15T03:48:04.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.099 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:04.099 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.099 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:04.100 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847417+0000", 2024-09-15T03:48:04.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.100 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:04.100 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847417+0000", 2024-09-15T03:48:04.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.100 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:04.100 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847418+0000", 2024-09-15T03:48:04.100 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8999999999999995e-07 2024-09-15T03:48:04.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.101 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:04.101 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847421+0000", 2024-09-15T03:48:04.101 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1769999999999998e-06 2024-09-15T03:48:04.101 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.101 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:04.101 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847428+0000", 2024-09-15T03:48:04.101 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6969999999999996e-06 2024-09-15T03:48:04.101 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.101 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.101 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847432+0000", 2024-09-15T03:48:04.102 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3209999999999998e-06 2024-09-15T03:48:04.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.102 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847542+0000", 2024-09-15T03:48:04.102 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011035999999999999 2024-09-15T03:48:04.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.102 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.102 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:48:04.103 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847543+0000", 2024-09-15T03:48:04.103 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1200000000000002e-07 2024-09-15T03:48:04.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.103 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.103 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.103 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875612+0000", 2024-09-15T03:48:04.103 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0280692330000001 2024-09-15T03:48:04.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.103 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.103 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:04.103 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875735+0000", 2024-09-15T03:48:04.103 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012276600000000001 2024-09-15T03:48:04.103 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.104 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.104 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.104 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876307+0000", 2024-09-15T03:48:04.104 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00057198600000000002 2024-09-15T03:48:04.104 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.104 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.104 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.104 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876408+0000", 2024-09-15T03:48:04.104 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010148400000000001 2024-09-15T03:48:04.104 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.104 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.104 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.104 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876508+0000", 2024-09-15T03:48:04.104 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9328999999999997e-05 2024-09-15T03:48:04.105 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.105 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.105 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.105 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876546+0000", 2024-09-15T03:48:04.105 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8361000000000001e-05 2024-09-15T03:48:04.105 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.105 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.105 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.105 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876602+0000", 2024-09-15T03:48:04.105 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6412999999999997e-05 2024-09-15T03:48:04.105 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.105 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.105 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:04.106 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.880589+0000", 2024-09-15T03:48:04.106 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0039866190000000003 2024-09-15T03:48:04.106 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.106 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.106 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:04.106 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.880685+0000", 2024-09-15T03:48:04.106 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6001999999999993e-05 2024-09-15T03:48:04.106 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.106 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:04.106 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.106 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.106 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.106 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:462 3.9s0 3:97177042:::smithi00132172-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:48:04.106 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.847436+0000", 2024-09-15T03:48:04.107 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 53.218271909999999, 2024-09-15T03:48:04.107 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.03734576, 2024-09-15T03:48:04.107 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:04.107 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:04.107 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:04.107 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:04.107 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.107 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 462 2024-09-15T03:48:04.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.107 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:04.107 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.107 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:04.107 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847436+0000", 2024-09-15T03:48:04.107 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.108 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:04.108 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847436+0000", 2024-09-15T03:48:04.108 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.108 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:04.108 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847436+0000", 2024-09-15T03:48:04.108 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3099999999999998e-07 2024-09-15T03:48:04.108 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.108 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.108 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:04.109 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847438+0000", 2024-09-15T03:48:04.109 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9570000000000001e-06 2024-09-15T03:48:04.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.109 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:04.109 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847445+0000", 2024-09-15T03:48:04.109 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.238e-06 2024-09-15T03:48:04.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.109 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.109 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847454+0000", 2024-09-15T03:48:04.109 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5930000000000006e-06 2024-09-15T03:48:04.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.109 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.110 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.110 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847562+0000", 2024-09-15T03:48:04.110 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108448 2024-09-15T03:48:04.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.110 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:48:04.110 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847563+0000", 2024-09-15T03:48:04.110 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0200000000000002e-07 2024-09-15T03:48:04.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.110 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.110 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.110 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876482+0000", 2024-09-15T03:48:04.110 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.028918658 2024-09-15T03:48:04.110 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.111 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for rw locks", 2024-09-15T03:48:04.111 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876593+0000", 2024-09-15T03:48:04.111 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011145300000000001 2024-09-15T03:48:04.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.111 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.111 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.881189+0000", 2024-09-15T03:48:04.111 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0045956139999999996 2024-09-15T03:48:04.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.111 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:04.111 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.881293+0000", 2024-09-15T03:48:04.111 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103875 2024-09-15T03:48:04.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.112 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:04.112 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.884782+0000", 2024-09-15T03:48:04.112 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003488891 2024-09-15T03:48:04.112 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.112 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:04.112 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.112 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.112 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:561 3.9s0 3:9cd57211:::smithi00132172-16:head [append 0~557056] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:48:04.112 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:20.935324+0000", 2024-09-15T03:48:04.112 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 43.130384208999999, 2024-09-15T03:48:04.113 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012459909999999999, 2024-09-15T03:48:04.113 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:04.113 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.113 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:04.113 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:04.113 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.113 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 561 2024-09-15T03:48:04.113 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.113 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:04.113 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.113 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:04.113 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935324+0000", 2024-09-15T03:48:04.113 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.114 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:04.114 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935324+0000", 2024-09-15T03:48:04.114 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.114 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:04.114 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935326+0000", 2024-09-15T03:48:04.114 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8909999999999999e-06 2024-09-15T03:48:04.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.114 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.114 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:04.114 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935587+0000", 2024-09-15T03:48:04.115 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026019699999999999 2024-09-15T03:48:04.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.115 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:04.115 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935610+0000", 2024-09-15T03:48:04.115 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2949e-05 2024-09-15T03:48:04.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.115 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.115 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935618+0000", 2024-09-15T03:48:04.115 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6289999999999999e-06 2024-09-15T03:48:04.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.115 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.116 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935688+0000", 2024-09-15T03:48:04.116 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9801999999999993e-05 2024-09-15T03:48:04.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.116 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-15T03:48:04.116 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935698+0000", 2024-09-15T03:48:04.116 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0103e-05 2024-09-15T03:48:04.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.116 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.116 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.116 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939626+0000", 2024-09-15T03:48:04.116 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0039276759999999997 2024-09-15T03:48:04.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.117 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.117 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:04.117 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939844+0000", 2024-09-15T03:48:04.117 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000217741 2024-09-15T03:48:04.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.117 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.117 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.117 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941127+0000", 2024-09-15T03:48:04.117 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001283719 2024-09-15T03:48:04.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.117 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.117 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.117 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941395+0000", 2024-09-15T03:48:04.117 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026823800000000002 2024-09-15T03:48:04.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.118 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.118 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941448+0000", 2024-09-15T03:48:04.118 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2751999999999999e-05 2024-09-15T03:48:04.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.118 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.118 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941646+0000", 2024-09-15T03:48:04.118 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019804100000000001 2024-09-15T03:48:04.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.118 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.118 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.119 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943664+0000", 2024-09-15T03:48:04.119 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020181090000000001 2024-09-15T03:48:04.119 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.119 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:04.119 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.947709+0000", 2024-09-15T03:48:04.119 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0040445059999999998 2024-09-15T03:48:04.119 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.119 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.119 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:04.119 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.947783+0000", 2024-09-15T03:48:04.119 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4556999999999998e-05 2024-09-15T03:48:04.119 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.119 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:04.120 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.120 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.120 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:562 3.9s0 3:9cd57211:::smithi00132172-16:head [append 0~638976] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:48:04.120 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:20.938590+0000", 2024-09-15T03:48:04.120 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 43.127117425000002, 2024-09-15T03:48:04.120 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011368382, 2024-09-15T03:48:04.120 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:04.120 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.120 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:04.120 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:04.120 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.120 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 562 2024-09-15T03:48:04.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.121 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:04.121 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.121 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:04.121 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.938590+0000", 2024-09-15T03:48:04.121 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.121 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.121 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.121 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:04.121 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.938590+0000", 2024-09-15T03:48:04.121 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.121 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.121 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.121 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:04.122 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.938592+0000", 2024-09-15T03:48:04.122 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.623e-06 2024-09-15T03:48:04.122 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.122 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:04.122 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939253+0000", 2024-09-15T03:48:04.122 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00066056899999999996 2024-09-15T03:48:04.122 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.122 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:04.122 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939271+0000", 2024-09-15T03:48:04.122 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8482e-05 2024-09-15T03:48:04.122 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.123 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.123 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939280+0000", 2024-09-15T03:48:04.123 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5569999999999996e-06 2024-09-15T03:48:04.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.123 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.123 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.123 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941736+0000", 2024-09-15T03:48:04.123 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024565640000000001 2024-09-15T03:48:04.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.123 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.123 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:04.123 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941861+0000", 2024-09-15T03:48:04.123 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012484 2024-09-15T03:48:04.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.124 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.124 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.124 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943223+0000", 2024-09-15T03:48:04.124 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0013619260000000001 2024-09-15T03:48:04.124 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.124 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.124 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.124 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943354+0000", 2024-09-15T03:48:04.124 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013100399999999999 2024-09-15T03:48:04.124 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.124 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.124 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.124 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943376+0000", 2024-09-15T03:48:04.124 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1829999999999999e-05 2024-09-15T03:48:04.125 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.125 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.125 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.125 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943579+0000", 2024-09-15T03:48:04.125 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020374400000000001 2024-09-15T03:48:04.125 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.125 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.125 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.125 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943904+0000", 2024-09-15T03:48:04.125 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000324846 2024-09-15T03:48:04.125 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.125 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.125 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:04.126 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949897+0000", 2024-09-15T03:48:04.126 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0059927879999999998 2024-09-15T03:48:04.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.126 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.126 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:04.126 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949959+0000", 2024-09-15T03:48:04.126 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1608999999999999e-05 2024-09-15T03:48:04.126 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.126 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:04.126 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.126 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.126 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:565 3.9s0 3:9cd57211:::smithi00132172-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:48:04.126 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:20.943038+0000", 2024-09-15T03:48:04.127 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 43.122670081000003, 2024-09-15T03:48:04.127 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011190904, 2024-09-15T03:48:04.127 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:04.127 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:04.127 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:04.127 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:04.127 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.127 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 565 2024-09-15T03:48:04.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.127 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:04.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.127 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:04.127 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943038+0000", 2024-09-15T03:48:04.128 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.128 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:04.128 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943038+0000", 2024-09-15T03:48:04.128 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.128 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:04.128 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943038+0000", 2024-09-15T03:48:04.128 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6800000000000001e-07 2024-09-15T03:48:04.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.128 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:04.129 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943040+0000", 2024-09-15T03:48:04.129 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3700000000000002e-06 2024-09-15T03:48:04.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.129 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:04.129 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943047+0000", 2024-09-15T03:48:04.129 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1299999999999998e-06 2024-09-15T03:48:04.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.129 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.129 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943050+0000", 2024-09-15T03:48:04.129 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6229999999999999e-06 2024-09-15T03:48:04.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.130 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.130 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.947990+0000", 2024-09-15T03:48:04.130 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0049398690000000004 2024-09-15T03:48:04.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.130 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for rw locks", 2024-09-15T03:48:04.130 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.948080+0000", 2024-09-15T03:48:04.130 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9721000000000004e-05 2024-09-15T03:48:04.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.130 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.130 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.952263+0000", 2024-09-15T03:48:04.130 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.004183417 2024-09-15T03:48:04.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.131 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:04.131 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.952374+0000", 2024-09-15T03:48:04.131 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110435 2024-09-15T03:48:04.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.131 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:04.131 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.954229+0000", 2024-09-15T03:48:04.131 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018548709999999999 2024-09-15T03:48:04.131 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.131 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:04.131 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.132 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:668 3.4s0 3:2d1c787c:::smithi00132172-45:head [read 0~1703936,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:48:04.132 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:29.978875+0000", 2024-09-15T03:48:04.132 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 34.086832364000003, 2024-09-15T03:48:04.132 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011314753, 2024-09-15T03:48:04.132 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:04.132 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:04.132 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:04.132 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:04.132 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.132 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 668 2024-09-15T03:48:04.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.132 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:04.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.133 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:04.133 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978875+0000", 2024-09-15T03:48:04.133 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.133 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:04.133 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978875+0000", 2024-09-15T03:48:04.133 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.133 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:04.133 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978876+0000", 2024-09-15T03:48:04.134 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.46e-07 2024-09-15T03:48:04.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.134 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:04.134 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978878+0000", 2024-09-15T03:48:04.134 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4399999999999999e-06 2024-09-15T03:48:04.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.134 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:04.134 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978884+0000", 2024-09-15T03:48:04.134 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8930000000000002e-06 2024-09-15T03:48:04.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.134 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.135 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978887+0000", 2024-09-15T03:48:04.135 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8040000000000002e-06 2024-09-15T03:48:04.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.135 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.135 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.979013+0000", 2024-09-15T03:48:04.135 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012650899999999999 2024-09-15T03:48:04.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.135 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:48:04.135 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.979020+0000", 2024-09-15T03:48:04.135 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8159999999999996e-06 2024-09-15T03:48:04.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.136 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.136 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.981562+0000", 2024-09-15T03:48:04.136 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002542267 2024-09-15T03:48:04.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.136 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:04.136 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.981653+0000", 2024-09-15T03:48:04.136 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0509999999999999e-05 2024-09-15T03:48:04.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.136 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:04.136 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.990190+0000", 2024-09-15T03:48:04.136 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0085371679999999995 2024-09-15T03:48:04.136 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.137 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:04.137 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.137 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:692 3.9s0 3:97177042:::smithi00132172-18:head [copy-from ver 4] snapc 2f=[2d,2b,29,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:48:04.137 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.088835+0000", 2024-09-15T03:48:04.137 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 33.976873058000002, 2024-09-15T03:48:04.137 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.016647229999999999, 2024-09-15T03:48:04.137 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:04.137 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.137 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:04.137 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:04.137 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.138 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 692 2024-09-15T03:48:04.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.138 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:04.138 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.138 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:04.138 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088835+0000", 2024-09-15T03:48:04.138 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.138 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.138 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:04.138 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088835+0000", 2024-09-15T03:48:04.138 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.138 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.138 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:04.139 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088837+0000", 2024-09-15T03:48:04.139 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8670000000000001e-06 2024-09-15T03:48:04.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.139 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:04.139 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088843+0000", 2024-09-15T03:48:04.139 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8179999999999998e-06 2024-09-15T03:48:04.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.139 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:04.139 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088860+0000", 2024-09-15T03:48:04.139 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6566999999999999e-05 2024-09-15T03:48:04.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.140 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088881+0000", 2024-09-15T03:48:04.140 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0619999999999999e-05 2024-09-15T03:48:04.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.140 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088963+0000", 2024-09-15T03:48:04.140 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2650000000000003e-05 2024-09-15T03:48:04.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.140 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:04.140 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089325+0000", 2024-09-15T03:48:04.140 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00036168700000000002 2024-09-15T03:48:04.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.141 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.141 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102185+0000", 2024-09-15T03:48:04.141 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012859786 2024-09-15T03:48:04.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.141 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.141 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102633+0000", 2024-09-15T03:48:04.141 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00044844699999999998 2024-09-15T03:48:04.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.141 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.141 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102713+0000", 2024-09-15T03:48:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9616999999999993e-05 2024-09-15T03:48:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102771+0000", 2024-09-15T03:48:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7797999999999998e-05 2024-09-15T03:48:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102793+0000", 2024-09-15T03:48:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2827999999999999e-05 2024-09-15T03:48:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.104072+0000", 2024-09-15T03:48:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012787759999999999 2024-09-15T03:48:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.105482+0000", 2024-09-15T03:48:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001409769 2024-09-15T03:48:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.143 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:693 3.9s0 3:97177042:::smithi00132172-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:48:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.089498+0000", 2024-09-15T03:48:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 33.976210123000001, 2024-09-15T03:48:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013753056, 2024-09-15T03:48:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 693 2024-09-15T03:48:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.144 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089498+0000", 2024-09-15T03:48:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089498+0000", 2024-09-15T03:48:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089499+0000", 2024-09-15T03:48:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.074e-06 2024-09-15T03:48:04.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089503+0000", 2024-09-15T03:48:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1470000000000003e-06 2024-09-15T03:48:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089513+0000", 2024-09-15T03:48:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0248000000000001e-05 2024-09-15T03:48:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089519+0000", 2024-09-15T03:48:04.146 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2149999999999998e-06 2024-09-15T03:48:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089719+0000", 2024-09-15T03:48:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019960400000000001 2024-09-15T03:48:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for blocked object", 2024-09-15T03:48:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089776+0000", 2024-09-15T03:48:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7123000000000002e-05 2024-09-15T03:48:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.147 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102827+0000", 2024-09-15T03:48:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013050858 2024-09-15T03:48:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102976+0000", 2024-09-15T03:48:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014900800000000001 2024-09-15T03:48:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.148 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.103251+0000", 2024-09-15T03:48:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00027477900000000001 2024-09-15T03:48:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:705 3.8s0 3:1af4eaf3:::smithi00132172-20:head [append 0~0] snapc 2f=[2d,2b,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-15T03:48:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.982870+0000", 2024-09-15T03:48:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 33.082837712, 2024-09-15T03:48:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.01140421, 2024-09-15T03:48:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.149 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 705 2024-09-15T03:48:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982870+0000", 2024-09-15T03:48:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982870+0000", 2024-09-15T03:48:04.150 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.151 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:04.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982871+0000", 2024-09-15T03:48:04.151 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2900000000000004e-07 2024-09-15T03:48:04.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.151 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:04.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982874+0000", 2024-09-15T03:48:04.151 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9610000000000001e-06 2024-09-15T03:48:04.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.151 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:04.151 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982878+0000", 2024-09-15T03:48:04.152 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7620000000000003e-06 2024-09-15T03:48:04.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.152 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.152 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982881+0000", 2024-09-15T03:48:04.152 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.112e-06 2024-09-15T03:48:04.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.152 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.152 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982995+0000", 2024-09-15T03:48:04.152 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000113544 2024-09-15T03:48:04.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.153 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:48:04.153 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982999+0000", 2024-09-15T03:48:04.153 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9339999999999999e-06 2024-09-15T03:48:04.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.153 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.153 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986479+0000", 2024-09-15T03:48:04.153 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0034799399999999999 2024-09-15T03:48:04.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.153 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:04.153 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986599+0000", 2024-09-15T03:48:04.153 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000120364 2024-09-15T03:48:04.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.154 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.154 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987178+0000", 2024-09-15T03:48:04.154 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00057913799999999998 2024-09-15T03:48:04.154 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.154 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.154 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987321+0000", 2024-09-15T03:48:04.154 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000142208 2024-09-15T03:48:04.154 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.154 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.154 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987335+0000", 2024-09-15T03:48:04.155 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4578e-05 2024-09-15T03:48:04.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.155 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.155 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987690+0000", 2024-09-15T03:48:04.155 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00035494600000000003 2024-09-15T03:48:04.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.155 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.155 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987799+0000", 2024-09-15T03:48:04.155 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000109379 2024-09-15T03:48:04.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.155 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:04.156 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.991063+0000", 2024-09-15T03:48:04.156 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032640249999999998 2024-09-15T03:48:04.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.156 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:04.156 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.994274+0000", 2024-09-15T03:48:04.156 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032107899999999998 2024-09-15T03:48:04.156 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.156 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:04.156 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.156 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:708 3.4s0 3:2d1c787c:::smithi00132172-45:head [copy-from ver 28] snapc 32=[32,2d,2b,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-15T03:48:04.157 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:31.983317+0000", 2024-09-15T03:48:04.157 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 32.082391178999998, 2024-09-15T03:48:04.157 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.021360130000000001, 2024-09-15T03:48:04.157 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:04.157 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.157 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:04.157 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:04.157 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.157 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 708 2024-09-15T03:48:04.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.157 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:04.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.157 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:04.158 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983317+0000", 2024-09-15T03:48:04.158 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.158 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:04.158 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983317+0000", 2024-09-15T03:48:04.158 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.158 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:04.158 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983318+0000", 2024-09-15T03:48:04.158 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2780000000000001e-06 2024-09-15T03:48:04.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.159 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:04.159 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983331+0000", 2024-09-15T03:48:04.159 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2821e-05 2024-09-15T03:48:04.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.159 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:04.159 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983349+0000", 2024-09-15T03:48:04.159 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8544999999999999e-05 2024-09-15T03:48:04.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.159 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.159 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983359+0000", 2024-09-15T03:48:04.159 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8570000000000007e-06 2024-09-15T03:48:04.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.160 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.160 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983434+0000", 2024-09-15T03:48:04.160 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4556999999999998e-05 2024-09-15T03:48:04.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.160 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-15T03:48:04.160 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983452+0000", 2024-09-15T03:48:04.160 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8439000000000001e-05 2024-09-15T03:48:04.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.160 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.160 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.985636+0000", 2024-09-15T03:48:04.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021841320000000001 2024-09-15T03:48:04.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:04.161 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.985755+0000", 2024-09-15T03:48:04.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118363 2024-09-15T03:48:04.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.161 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.998959+0000", 2024-09-15T03:48:04.161 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013204053 2024-09-15T03:48:04.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.161 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.162 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.999917+0000", 2024-09-15T03:48:04.162 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095851300000000003 2024-09-15T03:48:04.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.162 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.162 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.999931+0000", 2024-09-15T03:48:04.162 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3631999999999999e-05 2024-09-15T03:48:04.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.162 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.162 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000553+0000", 2024-09-15T03:48:04.162 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00062246600000000001 2024-09-15T03:48:04.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.163 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.163 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000695+0000", 2024-09-15T03:48:04.163 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014162499999999999 2024-09-15T03:48:04.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.163 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:04.163 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.004291+0000", 2024-09-15T03:48:04.163 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0035959289999999999 2024-09-15T03:48:04.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.164 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:04.164 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.004677+0000", 2024-09-15T03:48:04.164 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00038591999999999999 2024-09-15T03:48:04.164 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.164 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:04.164 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.165 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:709 3.4s0 3:2d1c787c:::smithi00132172-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-15T03:48:04.165 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:31.983373+0000", 2024-09-15T03:48:04.165 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 32.082334338000003, 2024-09-15T03:48:04.165 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.017317583000000001, 2024-09-15T03:48:04.165 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:04.165 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:04.165 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:04.165 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:04.165 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.165 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 709 2024-09-15T03:48:04.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.165 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:04.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.165 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:04.166 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983373+0000", 2024-09-15T03:48:04.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.166 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:04.166 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983373+0000", 2024-09-15T03:48:04.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.166 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:04.166 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983374+0000", 2024-09-15T03:48:04.166 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8599999999999998e-07 2024-09-15T03:48:04.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.167 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:04.167 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983377+0000", 2024-09-15T03:48:04.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0139999999999999e-06 2024-09-15T03:48:04.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.167 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:04.167 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983388+0000", 2024-09-15T03:48:04.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0988999999999999e-05 2024-09-15T03:48:04.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.167 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.167 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983392+0000", 2024-09-15T03:48:04.167 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5059999999999997e-06 2024-09-15T03:48:04.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.168 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.168 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983542+0000", 2024-09-15T03:48:04.168 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000149887 2024-09-15T03:48:04.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.168 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:48:04.168 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983550+0000", 2024-09-15T03:48:04.168 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9589999999999995e-06 2024-09-15T03:48:04.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.168 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.168 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.985938+0000", 2024-09-15T03:48:04.169 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023879930000000001 2024-09-15T03:48:04.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.169 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for blocked object", 2024-09-15T03:48:04.169 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.986005+0000", 2024-09-15T03:48:04.169 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6926000000000003e-05 2024-09-15T03:48:04.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.169 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.169 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000370+0000", 2024-09-15T03:48:04.169 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.014364656999999999 2024-09-15T03:48:04.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.169 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:04.170 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000485+0000", 2024-09-15T03:48:04.170 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011524899999999999 2024-09-15T03:48:04.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.170 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:04.170 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000691+0000", 2024-09-15T03:48:04.170 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020581700000000001 2024-09-15T03:48:04.170 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.170 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:04.170 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.170 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:761 3.9s0 3:9cd57211:::smithi00132172-16:head [append 0~507904] snapc 32=[32,2d,2b,26,25,24,23,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T03:48:04.171 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:34.955885+0000", 2024-09-15T03:48:04.171 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 29.109822394999998, 2024-09-15T03:48:04.171 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011160715, 2024-09-15T03:48:04.171 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:04.171 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.171 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:04.171 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:04.171 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.171 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 761 2024-09-15T03:48:04.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.171 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:04.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.171 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:04.171 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.955885+0000", 2024-09-15T03:48:04.172 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.172 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:04.172 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.955885+0000", 2024-09-15T03:48:04.172 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.172 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:04.172 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.955887+0000", 2024-09-15T03:48:04.172 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8169999999999999e-06 2024-09-15T03:48:04.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.173 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:04.173 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956332+0000", 2024-09-15T03:48:04.173 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00044459600000000001 2024-09-15T03:48:04.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.173 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:04.173 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956347+0000", 2024-09-15T03:48:04.173 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5685999999999999e-05 2024-09-15T03:48:04.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.173 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.173 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956354+0000", 2024-09-15T03:48:04.173 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6810000000000001e-06 2024-09-15T03:48:04.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.174 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.174 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956805+0000", 2024-09-15T03:48:04.174 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00045055300000000002 2024-09-15T03:48:04.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.174 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:04.174 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956981+0000", 2024-09-15T03:48:04.174 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017590100000000001 2024-09-15T03:48:04.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.174 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.174 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.958362+0000", 2024-09-15T03:48:04.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001381415 2024-09-15T03:48:04.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.175 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.175 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.958524+0000", 2024-09-15T03:48:04.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016167800000000001 2024-09-15T03:48:04.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.175 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.175 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.958797+0000", 2024-09-15T03:48:04.175 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000273274 2024-09-15T03:48:04.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.175 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.176 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.959021+0000", 2024-09-15T03:48:04.176 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022373099999999999 2024-09-15T03:48:04.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.176 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.176 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.961002+0000", 2024-09-15T03:48:04.176 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019813069999999999 2024-09-15T03:48:04.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.176 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:04.176 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.966647+0000", 2024-09-15T03:48:04.176 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00564541 2024-09-15T03:48:04.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.177 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:04.177 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.967046+0000", 2024-09-15T03:48:04.177 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00039866599999999999 2024-09-15T03:48:04.177 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.177 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:04.177 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.177 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.177 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(osd.9.17:1 3.9s0 3:94e9e019:::smithi00132172-30:head [assert-version v16,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e107)", 2024-09-15T03:48:04.177 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:53.919800+0000", 2024-09-15T03:48:04.177 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 10.145907671, 2024-09-15T03:48:04.177 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27396664300000001, 2024-09-15T03:48:04.178 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:04.178 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:04.178 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:04.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.178 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:04.178 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.919800+0000", 2024-09-15T03:48:04.178 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.178 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.178 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:04.178 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.919800+0000", 2024-09-15T03:48:04.178 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.179 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:04.179 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.919802+0000", 2024-09-15T03:48:04.179 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.787e-06 2024-09-15T03:48:04.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.179 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:04.179 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.919807+0000", 2024-09-15T03:48:04.179 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4519999999999996e-06 2024-09-15T03:48:04.179 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.179 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.179 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:04.179 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.919823+0000", 2024-09-15T03:48:04.179 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5349000000000002e-05 2024-09-15T03:48:04.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.180 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.180 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.919829+0000", 2024-09-15T03:48:04.180 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.562e-06 2024-09-15T03:48:04.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.180 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.180 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.920022+0000", 2024-09-15T03:48:04.180 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000192504 2024-09-15T03:48:04.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.180 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.180 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:48:04.181 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.920025+0000", 2024-09-15T03:48:04.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5609999999999999e-06 2024-09-15T03:48:04.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.181 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:54.183165+0000", 2024-09-15T03:48:04.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26313930800000002 2024-09-15T03:48:04.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.181 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.181 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:04.181 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:54.183525+0000", 2024-09-15T03:48:04.181 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00036087700000000001 2024-09-15T03:48:04.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.182 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:04.182 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:54.193767+0000", 2024-09-15T03:48:04.182 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010241243000000001 2024-09-15T03:48:04.182 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.182 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:04.182 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.182 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.182 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.0s4 113/108 ECSubWrite(tid=157, reqid=osd.1.0:157, at_version=113'56, trim_to=0'0, roll_forward_to=111'54))", 2024-09-15T03:48:04.182 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:59.946531+0000", 2024-09-15T03:48:04.182 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 4.1191769450000004, 2024-09-15T03:48:04.182 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013185570000000001, 2024-09-15T03:48:04.182 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:04.183 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:04.183 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:04.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.183 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:04.183 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.946531+0000", 2024-09-15T03:48:04.183 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.183 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:04.183 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.946531+0000", 2024-09-15T03:48:04.183 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.183 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.183 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:04.184 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.946531+0000", 2024-09-15T03:48:04.184 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.03e-07 2024-09-15T03:48:04.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.184 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:04.184 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.946550+0000", 2024-09-15T03:48:04.184 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8725000000000001e-05 2024-09-15T03:48:04.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.184 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:04.184 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.946558+0000", 2024-09-15T03:48:04.184 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7760000000000003e-06 2024-09-15T03:48:04.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.946564+0000", 2024-09-15T03:48:04.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1950000000000002e-06 2024-09-15T03:48:04.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.185 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.947093+0000", 2024-09-15T03:48:04.185 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00052902899999999998 2024-09-15T03:48:04.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.185 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.947103+0000", 2024-09-15T03:48:04.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0023e-05 2024-09-15T03:48:04.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.947384+0000", 2024-09-15T03:48:04.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000281048 2024-09-15T03:48:04.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.186 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.186 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.947414+0000", 2024-09-15T03:48:04.186 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9745000000000001e-05 2024-09-15T03:48:04.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.947557+0000", 2024-09-15T03:48:04.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014375999999999999 2024-09-15T03:48:04.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.947578+0000", 2024-09-15T03:48:04.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0310999999999999e-05 2024-09-15T03:48:04.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.187 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:04.187 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.959716+0000", 2024-09-15T03:48:04.187 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012138754999999999 2024-09-15T03:48:04.188 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.188 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:04.188 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.188 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.1s2 113/108 ECSubWrite(tid=204, reqid=osd.6.0:202, at_version=0'0, trim_to=0'0, roll_forward_to=113'29))", 2024-09-15T03:48:04.188 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:59.950561+0000", 2024-09-15T03:48:04.188 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 4.1151465820000004, 2024-09-15T03:48:04.188 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013346731000000001, 2024-09-15T03:48:04.188 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:04.188 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:04.188 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:04.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.189 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:04.189 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.950561+0000", 2024-09-15T03:48:04.189 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.189 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:04.189 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.950561+0000", 2024-09-15T03:48:04.189 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.189 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:04.189 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.950562+0000", 2024-09-15T03:48:04.189 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8599999999999999e-07 2024-09-15T03:48:04.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.190 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:04.190 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.950570+0000", 2024-09-15T03:48:04.190 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.901e-06 2024-09-15T03:48:04.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.190 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:04.190 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.950581+0000", 2024-09-15T03:48:04.190 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0180000000000001e-05 2024-09-15T03:48:04.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.190 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.190 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.950588+0000", 2024-09-15T03:48:04.191 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8970000000000002e-06 2024-09-15T03:48:04.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.191 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.191 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.958313+0000", 2024-09-15T03:48:04.191 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0077258099999999996 2024-09-15T03:48:04.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.191 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.191 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.958323+0000", 2024-09-15T03:48:04.191 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2040000000000003e-06 2024-09-15T03:48:04.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.192 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.192 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.958428+0000", 2024-09-15T03:48:04.192 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000105474 2024-09-15T03:48:04.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.192 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.192 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.958445+0000", 2024-09-15T03:48:04.192 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6878e-05 2024-09-15T03:48:04.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.192 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.192 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.958476+0000", 2024-09-15T03:48:04.192 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0750000000000002e-05 2024-09-15T03:48:04.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.193 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.193 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.959323+0000", 2024-09-15T03:48:04.193 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00084734999999999997 2024-09-15T03:48:04.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.193 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:04.193 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.963908+0000", 2024-09-15T03:48:04.193 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0045849009999999997 2024-09-15T03:48:04.193 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.193 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:04.193 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.bs7 113/108 ECSubWrite(tid=356, reqid=osd.7.0:356, at_version=113'209, trim_to=0'0, roll_forward_to=113'205))", 2024-09-15T03:48:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:59.954002+0000", 2024-09-15T03:48:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 4.1117057949999998, 2024-09-15T03:48:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012200717, 2024-09-15T03:48:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.954002+0000", 2024-09-15T03:48:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.194 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.954002+0000", 2024-09-15T03:48:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.954003+0000", 2024-09-15T03:48:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5099999999999998e-07 2024-09-15T03:48:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.954026+0000", 2024-09-15T03:48:04.195 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3249000000000001e-05 2024-09-15T03:48:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.954041+0000", 2024-09-15T03:48:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5322999999999999e-05 2024-09-15T03:48:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.954049+0000", 2024-09-15T03:48:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0199999999999994e-06 2024-09-15T03:48:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.196 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.954124+0000", 2024-09-15T03:48:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5029000000000003e-05 2024-09-15T03:48:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.954137+0000", 2024-09-15T03:48:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2979999999999999e-05 2024-09-15T03:48:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.954434+0000", 2024-09-15T03:48:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029692800000000001 2024-09-15T03:48:04.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.955135+0000", 2024-09-15T03:48:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00070042300000000004 2024-09-15T03:48:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.955205+0000", 2024-09-15T03:48:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0036000000000005e-05 2024-09-15T03:48:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.198 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.955314+0000", 2024-09-15T03:48:04.199 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000109701 2024-09-15T03:48:04.199 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:04.199 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:04.199 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:04.199 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.966203+0000", 2024-09-15T03:48:04.199 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010888177000000001 2024-09-15T03:48:04.199 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.199 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:04.199 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.199 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:04.199 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:04.199 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:04.201 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:48:04.201 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:48:04.200+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:48:04.302 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:48:04.301+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:48:04.314 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:48:04.314 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:48:04.314 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:48:04.314 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:48:04.323 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:48:04.402 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:48:04.402+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:48:04.425 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:48:04.425 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:48:04.425 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:48:04.425 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:48:04.425 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:48:04.433 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:48:04.504 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:48:04.503+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:48:04.535 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:48:04.536 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:48:04.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:48:04.536 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:48:04.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.536 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:324 3.7s0 3:e40d1694:::smithi00132172-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:48:04.536 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:48:04.536 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.689288431999998, 2024-09-15T03:48:04.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0031668960000002, 2024-09-15T03:48:04.536 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:04.536 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:48:04.536 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:04.537 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:04.537 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.537 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 324 2024-09-15T03:48:04.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.537 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:04.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:04.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:48:04.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:04.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:48:04.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:04.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:48:04.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:48:04.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:04.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844125+0000", 2024-09-15T03:48:04.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.142e-06 2024-09-15T03:48:04.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:04.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844132+0000", 2024-09-15T03:48:04.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0439999999999998e-06 2024-09-15T03:48:04.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844136+0000", 2024-09-15T03:48:04.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.126e-06 2024-09-15T03:48:04.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.847618+0000", 2024-09-15T03:48:04.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0034813790000000002 2024-09-15T03:48:04.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:48:04.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.847709+0000", 2024-09-15T03:48:04.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0835999999999996e-05 2024-09-15T03:48:04.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.849483+0000", 2024-09-15T03:48:04.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001774929 2024-09-15T03:48:04.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:04.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.849574+0000", 2024-09-15T03:48:04.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0847999999999995e-05 2024-09-15T03:48:04.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:04.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.847290+0000", 2024-09-15T03:48:04.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9977151819999999 2024-09-15T03:48:04.542 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.542 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:04.542 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.542 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:334 3.7s0 3:ece1472c:::smithi00132172-43:head [append 0~0] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:48:04.543 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:48:04.543 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.676561344000007, 2024-09-15T03:48:04.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.997592021, 2024-09-15T03:48:04.543 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:04.543 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.543 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:04.543 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:04.543 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.543 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 334 2024-09-15T03:48:04.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.543 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:04.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:04.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:48:04.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:04.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:48:04.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:04.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856851+0000", 2024-09-15T03:48:04.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.238e-06 2024-09-15T03:48:04.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:04.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856855+0000", 2024-09-15T03:48:04.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3640000000000001e-06 2024-09-15T03:48:04.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:04.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856868+0000", 2024-09-15T03:48:04.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2588999999999999e-05 2024-09-15T03:48:04.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856873+0000", 2024-09-15T03:48:04.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5350000000000004e-06 2024-09-15T03:48:04.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856916+0000", 2024-09-15T03:48:04.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2889000000000002e-05 2024-09-15T03:48:04.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:04.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856983+0000", 2024-09-15T03:48:04.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6746000000000001e-05 2024-09-15T03:48:04.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857448+0000", 2024-09-15T03:48:04.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046498500000000001 2024-09-15T03:48:04.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857720+0000", 2024-09-15T03:48:04.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027227600000000001 2024-09-15T03:48:04.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857741+0000", 2024-09-15T03:48:04.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1149999999999999e-05 2024-09-15T03:48:04.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857795+0000", 2024-09-15T03:48:04.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3919999999999999e-05 2024-09-15T03:48:04.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859582+0000", 2024-09-15T03:48:04.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017862640000000001 2024-09-15T03:48:04.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:04.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854347+0000", 2024-09-15T03:48:04.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9947654780000001 2024-09-15T03:48:04.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:04.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854442+0000", 2024-09-15T03:48:04.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4586999999999994e-05 2024-09-15T03:48:04.550 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.550 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:04.550 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.551 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:335 3.7s0 3:ece1472c:::smithi00132172-43:head [rollback 12] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:48:04.551 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:48:04.551 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.676531327000006, 2024-09-15T03:48:04.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9977859200000001, 2024-09-15T03:48:04.551 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:04.551 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.551 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:04.551 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:04.551 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.551 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 335 2024-09-15T03:48:04.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.552 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:04.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:04.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:48:04.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:04.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:48:04.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:04.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:48:04.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8099999999999999e-07 2024-09-15T03:48:04.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:04.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856882+0000", 2024-09-15T03:48:04.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.649e-06 2024-09-15T03:48:04.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:04.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856885+0000", 2024-09-15T03:48:04.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.664e-06 2024-09-15T03:48:04.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856888+0000", 2024-09-15T03:48:04.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3910000000000001e-06 2024-09-15T03:48:04.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857826+0000", 2024-09-15T03:48:04.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00093817900000000003 2024-09-15T03:48:04.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:04.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857940+0000", 2024-09-15T03:48:04.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114424 2024-09-15T03:48:04.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858504+0000", 2024-09-15T03:48:04.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056400899999999997 2024-09-15T03:48:04.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858593+0000", 2024-09-15T03:48:04.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8463999999999999e-05 2024-09-15T03:48:04.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858604+0000", 2024-09-15T03:48:04.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1567e-05 2024-09-15T03:48:04.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858642+0000", 2024-09-15T03:48:04.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7369000000000001e-05 2024-09-15T03:48:04.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859588+0000", 2024-09-15T03:48:04.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00094661799999999998 2024-09-15T03:48:04.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:04.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854594+0000", 2024-09-15T03:48:04.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9950055550000001 2024-09-15T03:48:04.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:04.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854666+0000", 2024-09-15T03:48:04.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1749999999999996e-05 2024-09-15T03:48:04.558 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.558 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:04.558 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.558 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:336 3.7s0 3:ece1472c:::smithi00132172-43:head [append 0~0] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:48:04.558 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:48:04.558 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.676500277000002, 2024-09-15T03:48:04.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9980017820000002, 2024-09-15T03:48:04.558 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:04.558 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.558 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:04.558 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:04.559 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.559 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 336 2024-09-15T03:48:04.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.559 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:04.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:04.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:48:04.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:04.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:48:04.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:04.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:48:04.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8000000000000002e-07 2024-09-15T03:48:04.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:04.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856913+0000", 2024-09-15T03:48:04.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5400000000000001e-06 2024-09-15T03:48:04.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:04.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856918+0000", 2024-09-15T03:48:04.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1540000000000001e-06 2024-09-15T03:48:04.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856921+0000", 2024-09-15T03:48:04.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2550000000000002e-06 2024-09-15T03:48:04.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858655+0000", 2024-09-15T03:48:04.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017336070000000001 2024-09-15T03:48:04.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:04.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858741+0000", 2024-09-15T03:48:04.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6305999999999997e-05 2024-09-15T03:48:04.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859193+0000", 2024-09-15T03:48:04.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045221200000000001 2024-09-15T03:48:04.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859319+0000", 2024-09-15T03:48:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012542900000000001 2024-09-15T03:48:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859331+0000", 2024-09-15T03:48:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2028000000000001e-05 2024-09-15T03:48:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859369+0000", 2024-09-15T03:48:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8522999999999997e-05 2024-09-15T03:48:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859649+0000", 2024-09-15T03:48:04.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028008200000000002 2024-09-15T03:48:04.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:04.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854800+0000", 2024-09-15T03:48:04.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9951506540000001 2024-09-15T03:48:04.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:04.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854913+0000", 2024-09-15T03:48:04.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112712 2024-09-15T03:48:04.565 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.565 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:04.565 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.565 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:346 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:48:04.565 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:48:04.565 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.695003907, 2024-09-15T03:48:04.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005631948, 2024-09-15T03:48:04.565 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:04.565 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.566 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:04.566 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:04.566 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.566 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 346 2024-09-15T03:48:04.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.566 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:04.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:04.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:48:04.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:04.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:48:04.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:04.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838410+0000", 2024-09-15T03:48:04.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4430000000000002e-06 2024-09-15T03:48:04.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:04.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838419+0000", 2024-09-15T03:48:04.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1730000000000001e-06 2024-09-15T03:48:04.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:04.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838444+0000", 2024-09-15T03:48:04.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5111000000000001e-05 2024-09-15T03:48:04.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838455+0000", 2024-09-15T03:48:04.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0839000000000001e-05 2024-09-15T03:48:04.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:48:04.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3415000000000002e-05 2024-09-15T03:48:04.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:48:04.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838527+0000", 2024-09-15T03:48:04.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0550000000000005e-06 2024-09-15T03:48:04.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840638+0000", 2024-09-15T03:48:04.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021105379999999999 2024-09-15T03:48:04.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:04.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840729+0000", 2024-09-15T03:48:04.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0977999999999995e-05 2024-09-15T03:48:04.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.841834+0000", 2024-09-15T03:48:04.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011055640000000001 2024-09-15T03:48:04.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842223+0000", 2024-09-15T03:48:04.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038899900000000001 2024-09-15T03:48:04.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842241+0000", 2024-09-15T03:48:04.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7317e-05 2024-09-15T03:48:04.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842298+0000", 2024-09-15T03:48:04.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7825000000000003e-05 2024-09-15T03:48:04.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844866+0000", 2024-09-15T03:48:04.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025675289999999998 2024-09-15T03:48:04.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:04.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843973+0000", 2024-09-15T03:48:04.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99910710599999997 2024-09-15T03:48:04.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:04.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844039+0000", 2024-09-15T03:48:04.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6056000000000006e-05 2024-09-15T03:48:04.573 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.573 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:04.573 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.573 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:347 3.2s0 3:48ceb02f:::smithi00132172-5:head [rollback c] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:48:04.574 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:48:04.574 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.694919111000004, 2024-09-15T03:48:04.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005723841, 2024-09-15T03:48:04.574 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:04.574 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.574 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:04.574 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:04.574 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.574 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 347 2024-09-15T03:48:04.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.574 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:04.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:04.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:48:04.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:04.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:48:04.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:04.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838493+0000", 2024-09-15T03:48:04.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7300000000000005e-07 2024-09-15T03:48:04.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:04.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838498+0000", 2024-09-15T03:48:04.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9300000000000002e-06 2024-09-15T03:48:04.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:04.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838508+0000", 2024-09-15T03:48:04.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9790000000000005e-06 2024-09-15T03:48:04.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838513+0000", 2024-09-15T03:48:04.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8890000000000001e-06 2024-09-15T03:48:04.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838583+0000", 2024-09-15T03:48:04.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0696000000000002e-05 2024-09-15T03:48:04.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:48:04.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838590+0000", 2024-09-15T03:48:04.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2990000000000002e-06 2024-09-15T03:48:04.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842313+0000", 2024-09-15T03:48:04.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037238810000000001 2024-09-15T03:48:04.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:04.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842432+0000", 2024-09-15T03:48:04.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118474 2024-09-15T03:48:04.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843108+0000", 2024-09-15T03:48:04.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00067639299999999996 2024-09-15T03:48:04.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843227+0000", 2024-09-15T03:48:04.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119092 2024-09-15T03:48:04.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843242+0000", 2024-09-15T03:48:04.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4216000000000001e-05 2024-09-15T03:48:04.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843292+0000", 2024-09-15T03:48:04.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0800999999999997e-05 2024-09-15T03:48:04.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844878+0000", 2024-09-15T03:48:04.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015852749999999999 2024-09-15T03:48:04.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:04.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844147+0000", 2024-09-15T03:48:04.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99926949200000004 2024-09-15T03:48:04.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:04.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844216+0000", 2024-09-15T03:48:04.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8551000000000001e-05 2024-09-15T03:48:04.581 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.581 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:04.581 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.581 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:348 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:48:04.581 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:48:04.581 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.694893167999993, 2024-09-15T03:48:04.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.006019768, 2024-09-15T03:48:04.581 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:04.581 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.581 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:04.582 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:04.582 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.582 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 348 2024-09-15T03:48:04.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.582 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:04.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:04.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:48:04.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:04.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:48:04.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:04.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:48:04.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8700000000000001e-07 2024-09-15T03:48:04.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:04.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838521+0000", 2024-09-15T03:48:04.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3740000000000001e-06 2024-09-15T03:48:04.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:04.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838527+0000", 2024-09-15T03:48:04.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1999999999999999e-06 2024-09-15T03:48:04.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838532+0000", 2024-09-15T03:48:04.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2059999999999998e-06 2024-09-15T03:48:04.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838609+0000", 2024-09-15T03:48:04.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7193000000000004e-05 2024-09-15T03:48:04.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:48:04.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838615+0000", 2024-09-15T03:48:04.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7980000000000002e-06 2024-09-15T03:48:04.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843986+0000", 2024-09-15T03:48:04.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053706450000000003 2024-09-15T03:48:04.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:04.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844106+0000", 2024-09-15T03:48:04.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120079 2024-09-15T03:48:04.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844669+0000", 2024-09-15T03:48:04.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056290300000000001 2024-09-15T03:48:04.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844804+0000", 2024-09-15T03:48:04.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013575600000000001 2024-09-15T03:48:04.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844820+0000", 2024-09-15T03:48:04.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5145e-05 2024-09-15T03:48:04.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844893+0000", 2024-09-15T03:48:04.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3270999999999997e-05 2024-09-15T03:48:04.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845043+0000", 2024-09-15T03:48:04.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150144 2024-09-15T03:48:04.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:04.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844473+0000", 2024-09-15T03:48:04.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99943048700000003 2024-09-15T03:48:04.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:04.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844538+0000", 2024-09-15T03:48:04.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4179999999999999e-05 2024-09-15T03:48:04.589 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.589 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:04.589 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.589 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:352 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~557056] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:48:04.589 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:48:04.589 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.688695523999996, 2024-09-15T03:48:04.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.016265065, 2024-09-15T03:48:04.589 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:04.589 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.590 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:04.590 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:04.590 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.590 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 352 2024-09-15T03:48:04.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.590 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:04.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:04.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:48:04.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:04.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:48:04.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:04.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844718+0000", 2024-09-15T03:48:04.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2340000000000001e-06 2024-09-15T03:48:04.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:04.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845237+0000", 2024-09-15T03:48:04.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051953399999999999 2024-09-15T03:48:04.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:04.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845273+0000", 2024-09-15T03:48:04.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5683999999999999e-05 2024-09-15T03:48:04.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845281+0000", 2024-09-15T03:48:04.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8439999999999998e-06 2024-09-15T03:48:04.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845365+0000", 2024-09-15T03:48:04.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4651e-05 2024-09-15T03:48:04.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:04.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845486+0000", 2024-09-15T03:48:04.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012009499999999999 2024-09-15T03:48:04.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847101+0000", 2024-09-15T03:48:04.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016157350000000001 2024-09-15T03:48:04.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847602+0000", 2024-09-15T03:48:04.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050066200000000003 2024-09-15T03:48:04.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847622+0000", 2024-09-15T03:48:04.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9749999999999999e-05 2024-09-15T03:48:04.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848616+0000", 2024-09-15T03:48:04.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099390799999999994 2024-09-15T03:48:04.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848709+0000", 2024-09-15T03:48:04.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2874000000000006e-05 2024-09-15T03:48:04.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:04.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.860888+0000", 2024-09-15T03:48:04.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.012179707 2024-09-15T03:48:04.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:04.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.860981+0000", 2024-09-15T03:48:04.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2386999999999995e-05 2024-09-15T03:48:04.596 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.596 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:04.596 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.596 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:353 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~671744] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:48:04.596 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:48:04.596 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.685133331000003, 2024-09-15T03:48:04.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.012912206, 2024-09-15T03:48:04.597 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:04.597 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.597 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:04.597 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:04.597 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.597 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 353 2024-09-15T03:48:04.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.597 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:04.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:04.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:48:04.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:04.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:48:04.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:04.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848279+0000", 2024-09-15T03:48:04.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.362e-06 2024-09-15T03:48:04.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:04.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849024+0000", 2024-09-15T03:48:04.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00074495500000000005 2024-09-15T03:48:04.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:04.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849039+0000", 2024-09-15T03:48:04.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5119e-05 2024-09-15T03:48:04.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849046+0000", 2024-09-15T03:48:04.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6359999999999997e-06 2024-09-15T03:48:04.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849115+0000", 2024-09-15T03:48:04.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8998999999999994e-05 2024-09-15T03:48:04.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:04.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849251+0000", 2024-09-15T03:48:04.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013589799999999999 2024-09-15T03:48:04.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850605+0000", 2024-09-15T03:48:04.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013544659999999999 2024-09-15T03:48:04.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850703+0000", 2024-09-15T03:48:04.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7576000000000002e-05 2024-09-15T03:48:04.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850719+0000", 2024-09-15T03:48:04.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5767e-05 2024-09-15T03:48:04.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851562+0000", 2024-09-15T03:48:04.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084361899999999999 2024-09-15T03:48:04.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851705+0000", 2024-09-15T03:48:04.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000142893 2024-09-15T03:48:04.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:04.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861112+0000", 2024-09-15T03:48:04.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.009407349 2024-09-15T03:48:04.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:04.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861190+0000", 2024-09-15T03:48:04.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7566999999999998e-05 2024-09-15T03:48:04.603 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.603 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:04.603 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.604 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:354 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~98304] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:48:04.604 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:48:04.604 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.684330895000002, 2024-09-15T03:48:04.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0177595669999999, 2024-09-15T03:48:04.604 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:04.604 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.604 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:04.604 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:04.604 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.604 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 354 2024-09-15T03:48:04.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.605 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:04.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:04.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:48:04.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:04.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:48:04.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:04.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849081+0000", 2024-09-15T03:48:04.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0610000000000001e-06 2024-09-15T03:48:04.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:04.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849157+0000", 2024-09-15T03:48:04.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5804999999999997e-05 2024-09-15T03:48:04.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:04.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849168+0000", 2024-09-15T03:48:04.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0730000000000001e-05 2024-09-15T03:48:04.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849174+0000", 2024-09-15T03:48:04.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3609999999999997e-06 2024-09-15T03:48:04.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850814+0000", 2024-09-15T03:48:04.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001640124 2024-09-15T03:48:04.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:04.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850894+0000", 2024-09-15T03:48:04.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0167999999999995e-05 2024-09-15T03:48:04.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851406+0000", 2024-09-15T03:48:04.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051203200000000005 2024-09-15T03:48:04.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851506+0000", 2024-09-15T03:48:04.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9457999999999995e-05 2024-09-15T03:48:04.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851574+0000", 2024-09-15T03:48:04.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8145000000000006e-05 2024-09-15T03:48:04.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851632+0000", 2024-09-15T03:48:04.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8108e-05 2024-09-15T03:48:04.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851712+0000", 2024-09-15T03:48:04.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9380000000000002e-05 2024-09-15T03:48:04.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:04.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.866753+0000", 2024-09-15T03:48:04.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0150417979999999 2024-09-15T03:48:04.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:04.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.866840+0000", 2024-09-15T03:48:04.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6397000000000006e-05 2024-09-15T03:48:04.611 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.611 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:04.611 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.612 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:355 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [setxattr _header (57) in=7b] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:48:04.612 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:48:04.612 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.684201582, 2024-09-15T03:48:04.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.018024966, 2024-09-15T03:48:04.612 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:04.612 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.612 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:04.612 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:04.612 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.612 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 355 2024-09-15T03:48:04.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.612 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:04.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:04.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:48:04.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:04.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:48:04.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:04.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849210+0000", 2024-09-15T03:48:04.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.33e-07 2024-09-15T03:48:04.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:04.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849215+0000", 2024-09-15T03:48:04.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4379999999999997e-06 2024-09-15T03:48:04.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:04.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849234+0000", 2024-09-15T03:48:04.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9409999999999999e-05 2024-09-15T03:48:04.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849241+0000", 2024-09-15T03:48:04.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6370000000000002e-06 2024-09-15T03:48:04.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851823+0000", 2024-09-15T03:48:04.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025817689999999998 2024-09-15T03:48:04.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:04.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851934+0000", 2024-09-15T03:48:04.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111357 2024-09-15T03:48:04.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852466+0000", 2024-09-15T03:48:04.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053248100000000001 2024-09-15T03:48:04.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852580+0000", 2024-09-15T03:48:04.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011379699999999999 2024-09-15T03:48:04.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852607+0000", 2024-09-15T03:48:04.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279e-05 2024-09-15T03:48:04.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852693+0000", 2024-09-15T03:48:04.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6792999999999994e-05 2024-09-15T03:48:04.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852869+0000", 2024-09-15T03:48:04.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017537499999999999 2024-09-15T03:48:04.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:04.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867006+0000", 2024-09-15T03:48:04.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.014136996 2024-09-15T03:48:04.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:04.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867234+0000", 2024-09-15T03:48:04.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022870099999999999 2024-09-15T03:48:04.618 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.618 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:04.618 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.619 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:356 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:48:04.619 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:48:04.619 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.684164819000003, 2024-09-15T03:48:04.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.020403521, 2024-09-15T03:48:04.619 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:04.619 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:48:04.619 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:04.619 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:04.619 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.619 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 356 2024-09-15T03:48:04.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.620 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:04.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:04.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:48:04.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:04.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:48:04.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:04.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849247+0000", 2024-09-15T03:48:04.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2699999999999999e-07 2024-09-15T03:48:04.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:04.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849250+0000", 2024-09-15T03:48:04.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2610000000000002e-06 2024-09-15T03:48:04.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:04.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849257+0000", 2024-09-15T03:48:04.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.703e-06 2024-09-15T03:48:04.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849261+0000", 2024-09-15T03:48:04.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7910000000000002e-06 2024-09-15T03:48:04.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852672+0000", 2024-09-15T03:48:04.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003411627 2024-09-15T03:48:04.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:48:04.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852801+0000", 2024-09-15T03:48:04.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000128918 2024-09-15T03:48:04.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867347+0000", 2024-09-15T03:48:04.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0145458380000001 2024-09-15T03:48:04.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:04.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867476+0000", 2024-09-15T03:48:04.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012915100000000001 2024-09-15T03:48:04.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:04.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.869650+0000", 2024-09-15T03:48:04.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021737050000000002 2024-09-15T03:48:04.624 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.624 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:04.624 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.624 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:527 3.7s0 3:e4b8e19d:::smithi00132172-41:head [copy-from ver 51] snapc 23=[23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:48:04.624 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:48:04.624 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 48.584595856999996, 2024-09-15T03:48:04.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030362388000000001, 2024-09-15T03:48:04.625 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:04.625 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.625 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:04.625 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:04.625 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.625 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 527 2024-09-15T03:48:04.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.625 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:04.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:04.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:48:04.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:04.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:48:04.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:04.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948817+0000", 2024-09-15T03:48:04.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3230000000000001e-06 2024-09-15T03:48:04.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:04.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948825+0000", 2024-09-15T03:48:04.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5900000000000008e-06 2024-09-15T03:48:04.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:04.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948838+0000", 2024-09-15T03:48:04.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2712e-05 2024-09-15T03:48:04.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948842+0000", 2024-09-15T03:48:04.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4170000000000002e-06 2024-09-15T03:48:04.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948923+0000", 2024-09-15T03:48:04.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0686000000000006e-05 2024-09-15T03:48:04.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:48:04.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949020+0000", 2024-09-15T03:48:04.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6984999999999994e-05 2024-09-15T03:48:04.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954390+0000", 2024-09-15T03:48:04.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053704790000000001 2024-09-15T03:48:04.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:04.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954558+0000", 2024-09-15T03:48:04.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000167202 2024-09-15T03:48:04.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.973998+0000", 2024-09-15T03:48:04.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019440316999999999 2024-09-15T03:48:04.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974527+0000", 2024-09-15T03:48:04.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052943800000000002 2024-09-15T03:48:04.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974541+0000", 2024-09-15T03:48:04.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3774999999999999e-05 2024-09-15T03:48:04.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974841+0000", 2024-09-15T03:48:04.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029942899999999998 2024-09-15T03:48:04.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.976067+0000", 2024-09-15T03:48:04.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012265399999999999 2024-09-15T03:48:04.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:04.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.978952+0000", 2024-09-15T03:48:04.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028848839999999999 2024-09-15T03:48:04.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:04.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.979178+0000", 2024-09-15T03:48:04.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022561099999999999 2024-09-15T03:48:04.632 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.632 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:04.632 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.632 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:528 3.7s0 3:e4b8e19d:::smithi00132172-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:48:04.632 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:48:04.632 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 48.584561448000002, 2024-09-15T03:48:04.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026927036000000001, 2024-09-15T03:48:04.632 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:04.633 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:48:04.633 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:04.633 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:04.633 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.633 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 528 2024-09-15T03:48:04.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.633 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:04.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:04.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:48:04.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:04.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:48:04.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:04.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:48:04.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9700000000000003e-07 2024-09-15T03:48:04.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:04.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948852+0000", 2024-09-15T03:48:04.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.604e-06 2024-09-15T03:48:04.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:04.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948855+0000", 2024-09-15T03:48:04.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9369999999999997e-06 2024-09-15T03:48:04.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948858+0000", 2024-09-15T03:48:04.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2709999999999999e-06 2024-09-15T03:48:04.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949080+0000", 2024-09-15T03:48:04.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000222148 2024-09-15T03:48:04.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:48:04.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949156+0000", 2024-09-15T03:48:04.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6215000000000001e-05 2024-09-15T03:48:04.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954754+0000", 2024-09-15T03:48:04.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055978060000000003 2024-09-15T03:48:04.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:48:04.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954858+0000", 2024-09-15T03:48:04.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010419199999999999 2024-09-15T03:48:04.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974861+0000", 2024-09-15T03:48:04.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020002671999999999 2024-09-15T03:48:04.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:04.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974987+0000", 2024-09-15T03:48:04.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012638699999999999 2024-09-15T03:48:04.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:04.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.975777+0000", 2024-09-15T03:48:04.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00078950699999999995 2024-09-15T03:48:04.638 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.638 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:04.638 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.638 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:924 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~737280] snapc 45=[45,44,43,42,41,40,3e,3d,3b,3a,38,37,35,34,2d,2b,25,24,22,21,20,16,12,11,3] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-15T03:48:04.638 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:53.925245+0000", 2024-09-15T03:48:04.638 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.608165796, 2024-09-15T03:48:04.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.29827357799999998, 2024-09-15T03:48:04.639 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:04.639 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.639 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:04.639 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:04.639 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.639 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 924 2024-09-15T03:48:04.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.639 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:04.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:04.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.925245+0000", 2024-09-15T03:48:04.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:04.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.925245+0000", 2024-09-15T03:48:04.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:04.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.925247+0000", 2024-09-15T03:48:04.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9999999999999999e-06 2024-09-15T03:48:04.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:04.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.926127+0000", 2024-09-15T03:48:04.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000879957 2024-09-15T03:48:04.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:04.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.926146+0000", 2024-09-15T03:48:04.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8349999999999999e-05 2024-09-15T03:48:04.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.926153+0000", 2024-09-15T03:48:04.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3889999999999999e-06 2024-09-15T03:48:04.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.926220+0000", 2024-09-15T03:48:04.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7490000000000006e-05 2024-09-15T03:48:04.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:48:04.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.926224+0000", 2024-09-15T03:48:04.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5590000000000001e-06 2024-09-15T03:48:04.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.209778+0000", 2024-09-15T03:48:04.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.28355351899999998 2024-09-15T03:48:04.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:04.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.209967+0000", 2024-09-15T03:48:04.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018960400000000001 2024-09-15T03:48:04.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.211917+0000", 2024-09-15T03:48:04.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019494320000000001 2024-09-15T03:48:04.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.212779+0000", 2024-09-15T03:48:04.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086249099999999995 2024-09-15T03:48:04.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.212798+0000", 2024-09-15T03:48:04.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9015e-05 2024-09-15T03:48:04.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.212925+0000", 2024-09-15T03:48:04.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012736099999999999 2024-09-15T03:48:04.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.212946+0000", 2024-09-15T03:48:04.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0919000000000001e-05 2024-09-15T03:48:04.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:04.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.215488+0000", 2024-09-15T03:48:04.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025418350000000001 2024-09-15T03:48:04.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:04.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.223519+0000", 2024-09-15T03:48:04.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0080306570000000001 2024-09-15T03:48:04.647 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.647 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:04.647 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.648 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:925 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~770048] snapc 45=[45,44,43,42,41,40,3e,3d,3b,3a,38,37,35,34,2d,2b,25,24,22,21,20,16,12,11,3] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-15T03:48:04.648 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:53.930327+0000", 2024-09-15T03:48:04.648 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.603084082000001, 2024-09-15T03:48:04.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.29345302099999998, 2024-09-15T03:48:04.648 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:04.648 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.648 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:04.648 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:04.648 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.648 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 925 2024-09-15T03:48:04.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.648 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:04.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:04.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.930327+0000", 2024-09-15T03:48:04.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:04.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.930327+0000", 2024-09-15T03:48:04.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:04.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.930328+0000", 2024-09-15T03:48:04.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.15e-06 2024-09-15T03:48:04.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:04.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.930988+0000", 2024-09-15T03:48:04.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000659594 2024-09-15T03:48:04.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:04.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.931004+0000", 2024-09-15T03:48:04.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6101000000000001e-05 2024-09-15T03:48:04.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.931011+0000", 2024-09-15T03:48:04.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8940000000000004e-06 2024-09-15T03:48:04.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.931050+0000", 2024-09-15T03:48:04.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9085000000000003e-05 2024-09-15T03:48:04.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:48:04.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.931052+0000", 2024-09-15T03:48:04.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1789999999999998e-06 2024-09-15T03:48:04.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.213039+0000", 2024-09-15T03:48:04.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.281986827 2024-09-15T03:48:04.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:04.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.213228+0000", 2024-09-15T03:48:04.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000189159 2024-09-15T03:48:04.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.214530+0000", 2024-09-15T03:48:04.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001302147 2024-09-15T03:48:04.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.214723+0000", 2024-09-15T03:48:04.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000192998 2024-09-15T03:48:04.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.214741+0000", 2024-09-15T03:48:04.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7762e-05 2024-09-15T03:48:04.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.215136+0000", 2024-09-15T03:48:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039510400000000002 2024-09-15T03:48:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.215269+0000", 2024-09-15T03:48:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013283300000000001 2024-09-15T03:48:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.216320+0000", 2024-09-15T03:48:04.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001050888 2024-09-15T03:48:04.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:04.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.223780+0000", 2024-09-15T03:48:04.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074602999999999996 2024-09-15T03:48:04.655 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.655 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:04.655 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.655 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:926 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~540672] snapc 45=[45,44,43,42,41,40,3e,3d,3b,3a,38,37,35,34,2d,2b,25,24,22,21,20,16,12,11,3] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-15T03:48:04.655 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:53.933251+0000", 2024-09-15T03:48:04.655 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.600160421, 2024-09-15T03:48:04.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.29574198400000001, 2024-09-15T03:48:04.656 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:04.656 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.656 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:04.656 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:04.656 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.656 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 926 2024-09-15T03:48:04.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.656 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:04.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:04.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933251+0000", 2024-09-15T03:48:04.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:04.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933251+0000", 2024-09-15T03:48:04.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:04.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933252+0000", 2024-09-15T03:48:04.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.569999999999999e-07 2024-09-15T03:48:04.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:04.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933849+0000", 2024-09-15T03:48:04.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059778100000000003 2024-09-15T03:48:04.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:04.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933865+0000", 2024-09-15T03:48:04.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5170999999999999e-05 2024-09-15T03:48:04.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933871+0000", 2024-09-15T03:48:04.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2899999999999999e-06 2024-09-15T03:48:04.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933904+0000", 2024-09-15T03:48:04.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3519000000000002e-05 2024-09-15T03:48:04.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:48:04.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933905+0000", 2024-09-15T03:48:04.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.096e-06 2024-09-15T03:48:04.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.217183+0000", 2024-09-15T03:48:04.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.28327724799999998 2024-09-15T03:48:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.217399+0000", 2024-09-15T03:48:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000216429 2024-09-15T03:48:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.224559+0000", 2024-09-15T03:48:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071598429999999999 2024-09-15T03:48:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.224765+0000", 2024-09-15T03:48:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020648500000000001 2024-09-15T03:48:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.224794+0000", 2024-09-15T03:48:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8645000000000001e-05 2024-09-15T03:48:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.224949+0000", 2024-09-15T03:48:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015473200000000001 2024-09-15T03:48:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.227877+0000", 2024-09-15T03:48:04.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029284110000000001 2024-09-15T03:48:04.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:04.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.228692+0000", 2024-09-15T03:48:04.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081477700000000004 2024-09-15T03:48:04.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:04.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.228993+0000", 2024-09-15T03:48:04.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030059999999999999 2024-09-15T03:48:04.663 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.663 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:04.663 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.663 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:927 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~245760] snapc 45=[45,44,43,42,41,40,3e,3d,3b,3a,38,37,35,34,2d,2b,25,24,22,21,20,16,12,11,3] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-15T03:48:04.663 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:53.934611+0000", 2024-09-15T03:48:04.663 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.598799571000001, 2024-09-15T03:48:04.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.29515997300000002, 2024-09-15T03:48:04.663 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:04.663 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.663 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:04.664 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:04.664 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.664 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 927 2024-09-15T03:48:04.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.664 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:04.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:04.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934611+0000", 2024-09-15T03:48:04.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:04.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934611+0000", 2024-09-15T03:48:04.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:04.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934612+0000", 2024-09-15T03:48:04.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3099999999999996e-07 2024-09-15T03:48:04.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:04.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934841+0000", 2024-09-15T03:48:04.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022889699999999999 2024-09-15T03:48:04.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:04.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934855+0000", 2024-09-15T03:48:04.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3998999999999999e-05 2024-09-15T03:48:04.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934862+0000", 2024-09-15T03:48:04.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3300000000000004e-06 2024-09-15T03:48:04.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934894+0000", 2024-09-15T03:48:04.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1995000000000001e-05 2024-09-15T03:48:04.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:48:04.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934894+0000", 2024-09-15T03:48:04.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6199999999999998e-07 2024-09-15T03:48:04.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.224939+0000", 2024-09-15T03:48:04.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.29004481300000001 2024-09-15T03:48:04.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:04.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.225469+0000", 2024-09-15T03:48:04.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052985499999999997 2024-09-15T03:48:04.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.226886+0000", 2024-09-15T03:48:04.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001416785 2024-09-15T03:48:04.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.227699+0000", 2024-09-15T03:48:04.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000813363 2024-09-15T03:48:04.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.227716+0000", 2024-09-15T03:48:04.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7017e-05 2024-09-15T03:48:04.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.227783+0000", 2024-09-15T03:48:04.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7057999999999994e-05 2024-09-15T03:48:04.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.227879+0000", 2024-09-15T03:48:04.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6417999999999997e-05 2024-09-15T03:48:04.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:04.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.229372+0000", 2024-09-15T03:48:04.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014929019999999999 2024-09-15T03:48:04.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:04.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.229771+0000", 2024-09-15T03:48:04.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039914799999999999 2024-09-15T03:48:04.671 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.671 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:04.671 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.671 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:928 3.2s0 3:48ceb02f:::smithi00132172-5:head [setxattr _header (56) in=7b] snapc 45=[45,44,43,42,41,40,3e,3d,3b,3a,38,37,35,34,2d,2b,25,24,22,21,20,16,12,11,3] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-15T03:48:04.671 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:53.934871+0000", 2024-09-15T03:48:04.672 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.598539657, 2024-09-15T03:48:04.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.298571425, 2024-09-15T03:48:04.672 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:04.672 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:04.672 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:04.672 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:04.672 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.672 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 928 2024-09-15T03:48:04.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.672 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:04.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:04.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934871+0000", 2024-09-15T03:48:04.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:04.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934871+0000", 2024-09-15T03:48:04.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:04.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934872+0000", 2024-09-15T03:48:04.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9100000000000003e-07 2024-09-15T03:48:04.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:04.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934876+0000", 2024-09-15T03:48:04.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.687e-06 2024-09-15T03:48:04.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:04.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934884+0000", 2024-09-15T03:48:04.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7049999999999999e-06 2024-09-15T03:48:04.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934888+0000", 2024-09-15T03:48:04.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.639e-06 2024-09-15T03:48:04.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934911+0000", 2024-09-15T03:48:04.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3224999999999999e-05 2024-09-15T03:48:04.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:48:04.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934912+0000", 2024-09-15T03:48:04.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9999999999999999e-07 2024-09-15T03:48:04.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.230182+0000", 2024-09-15T03:48:04.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.29527004699999998 2024-09-15T03:48:04.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:04.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.230320+0000", 2024-09-15T03:48:04.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013794499999999999 2024-09-15T03:48:04.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:04.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.231057+0000", 2024-09-15T03:48:04.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00073703400000000002 2024-09-15T03:48:04.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:04.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.231199+0000", 2024-09-15T03:48:04.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014196099999999999 2024-09-15T03:48:04.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:04.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.231309+0000", 2024-09-15T03:48:04.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011001399999999999 2024-09-15T03:48:04.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:04.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.231349+0000", 2024-09-15T03:48:04.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0068999999999999e-05 2024-09-15T03:48:04.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:04.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.231658+0000", 2024-09-15T03:48:04.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030921799999999999 2024-09-15T03:48:04.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:04.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.232928+0000", 2024-09-15T03:48:04.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012703409999999999 2024-09-15T03:48:04.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:04.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.233443+0000", 2024-09-15T03:48:04.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051444900000000005 2024-09-15T03:48:04.679 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.679 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:04.679 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.679 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:929 3.2s0 3:48ceb02f:::smithi00132172-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2024-09-15T03:48:04.679 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:53.934895+0000", 2024-09-15T03:48:04.679 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.598516215, 2024-09-15T03:48:04.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.30125257599999999, 2024-09-15T03:48:04.680 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:04.680 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:48:04.680 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:04.680 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:04.680 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:04.680 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 929 2024-09-15T03:48:04.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.680 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:04.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:04.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934895+0000", 2024-09-15T03:48:04.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:04.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934895+0000", 2024-09-15T03:48:04.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:04.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:04.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934895+0000", 2024-09-15T03:48:04.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2699999999999999e-07 2024-09-15T03:48:04.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:04.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934898+0000", 2024-09-15T03:48:04.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3549999999999999e-06 2024-09-15T03:48:04.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:04.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934905+0000", 2024-09-15T03:48:04.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0060000000000003e-06 2024-09-15T03:48:04.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:04.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934910+0000", 2024-09-15T03:48:04.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1499999999999998e-06 2024-09-15T03:48:04.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934935+0000", 2024-09-15T03:48:04.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4641e-05 2024-09-15T03:48:04.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:48:04.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934935+0000", 2024-09-15T03:48:04.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7099999999999997e-07 2024-09-15T03:48:04.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.231303+0000", 2024-09-15T03:48:04.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.29636852000000002 2024-09-15T03:48:04.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:48:04.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.231460+0000", 2024-09-15T03:48:04.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015613700000000001 2024-09-15T03:48:04.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:04.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.233845+0000", 2024-09-15T03:48:04.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023851660000000002 2024-09-15T03:48:04.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:04.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.233998+0000", 2024-09-15T03:48:04.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015345699999999999 2024-09-15T03:48:04.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:04.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:04.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:04.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.236147+0000", 2024-09-15T03:48:04.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021492460000000001 2024-09-15T03:48:04.685 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.686 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:04.686 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.686 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:04.686 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:04.686 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:48:04.688 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:48:04.688 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:48:04.687+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:48:04.788 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:48:04.788+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:48:04.829 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:48:04.829 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:48:04.829 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:48:04.829 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:48:04.837 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:48:04.843 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.11 to 1 (10000) 2024-09-15T03:48:04.855 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-15T03:48:04.897 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:48:04.896+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:48:04.939 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:48:04.939 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:48:04.939 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:48:04.940 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:48:04.940 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:48:04.948 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:48:04.979 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T03:48:04.990 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:48:04.989+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:48:05.051 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:48:05.052 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:48:05.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:48:05.052 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:48:05.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.052 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 38/28 ECSubWrite(tid=33, reqid=client.4521.0:222, at_version=38'14, trim_to=0'0, roll_forward_to=35'12))", 2024-09-15T03:48:05.052 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:48:05.052 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.919072022999998, 2024-09-15T03:48:05.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9949187450000001, 2024-09-15T03:48:05.052 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:05.052 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:05.052 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:05.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:05.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:48:05.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:05.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130992+0000", 2024-09-15T03:48:05.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:05.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.130996+0000", 2024-09-15T03:48:05.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8809999999999998e-06 2024-09-15T03:48:05.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:05.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131020+0000", 2024-09-15T03:48:05.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4190999999999999e-05 2024-09-15T03:48:05.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:05.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131039+0000", 2024-09-15T03:48:05.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9228999999999998e-05 2024-09-15T03:48:05.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:05.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131048+0000", 2024-09-15T03:48:05.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2380000000000003e-06 2024-09-15T03:48:05.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:05.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131122+0000", 2024-09-15T03:48:05.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3146999999999996e-05 2024-09-15T03:48:05.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:05.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131132+0000", 2024-09-15T03:48:05.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.08e-05 2024-09-15T03:48:05.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:05.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131381+0000", 2024-09-15T03:48:05.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000248936 2024-09-15T03:48:05.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:05.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131392+0000", 2024-09-15T03:48:05.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0987e-05 2024-09-15T03:48:05.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:05.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131444+0000", 2024-09-15T03:48:05.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1585000000000002e-05 2024-09-15T03:48:05.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:05.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.131465+0000", 2024-09-15T03:48:05.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1365999999999998e-05 2024-09-15T03:48:05.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:05.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.125911+0000", 2024-09-15T03:48:05.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9944453850000001 2024-09-15T03:48:05.059 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.059 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:05.059 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.059 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 38/28 ECSubWrite(tid=34, reqid=client.4521.0:222, at_version=0'0, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T03:48:05.060 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:48:05.060 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.917672924000001, 2024-09-15T03:48:05.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.002752697, 2024-09-15T03:48:05.060 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:05.060 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:05.060 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:05.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:05.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:48:05.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:05.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132391+0000", 2024-09-15T03:48:05.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:05.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132392+0000", 2024-09-15T03:48:05.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5899999999999995e-07 2024-09-15T03:48:05.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:05.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132415+0000", 2024-09-15T03:48:05.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3561e-05 2024-09-15T03:48:05.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:05.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132436+0000", 2024-09-15T03:48:05.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0879000000000001e-05 2024-09-15T03:48:05.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:05.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132444+0000", 2024-09-15T03:48:05.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9200000000000004e-06 2024-09-15T03:48:05.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:05.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132497+0000", 2024-09-15T03:48:05.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2299000000000002e-05 2024-09-15T03:48:05.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:05.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132508+0000", 2024-09-15T03:48:05.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1100999999999999e-05 2024-09-15T03:48:05.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:05.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132615+0000", 2024-09-15T03:48:05.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107546 2024-09-15T03:48:05.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:05.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132635+0000", 2024-09-15T03:48:05.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9374000000000001e-05 2024-09-15T03:48:05.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:05.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132677+0000", 2024-09-15T03:48:05.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2456999999999997e-05 2024-09-15T03:48:05.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:05.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.132711+0000", 2024-09-15T03:48:05.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4118000000000001e-05 2024-09-15T03:48:05.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:05.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135144+0000", 2024-09-15T03:48:05.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.002432583 2024-09-15T03:48:05.065 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.065 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:05.065 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.065 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=45, reqid=client.4521.0:227, at_version=38'24, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:48:05.065 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:48:05.065 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.894746381999994, 2024-09-15T03:48:05.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.974765339, 2024-09-15T03:48:05.065 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:05.065 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:05.065 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:05.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:05.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:48:05.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:05.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:48:05.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:05.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.155318+0000", 2024-09-15T03:48:05.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.43e-07 2024-09-15T03:48:05.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:05.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156110+0000", 2024-09-15T03:48:05.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000792409 2024-09-15T03:48:05.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:05.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156139+0000", 2024-09-15T03:48:05.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9207e-05 2024-09-15T03:48:05.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:05.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156148+0000", 2024-09-15T03:48:05.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1269999999999992e-06 2024-09-15T03:48:05.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:05.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156248+0000", 2024-09-15T03:48:05.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9421999999999998e-05 2024-09-15T03:48:05.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:05.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156258+0000", 2024-09-15T03:48:05.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6989999999999998e-06 2024-09-15T03:48:05.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:05.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156674+0000", 2024-09-15T03:48:05.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041682600000000002 2024-09-15T03:48:05.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:05.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156705+0000", 2024-09-15T03:48:05.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0525000000000003e-05 2024-09-15T03:48:05.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:05.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157742+0000", 2024-09-15T03:48:05.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010374 2024-09-15T03:48:05.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:05.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157777+0000", 2024-09-15T03:48:05.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4878e-05 2024-09-15T03:48:05.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:05.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.130083+0000", 2024-09-15T03:48:05.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972305703 2024-09-15T03:48:05.070 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.070 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:05.070 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.070 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=37, reqid=client.4521.0:236, at_version=38'5, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:48:05.071 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:48:05.071 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.893132926000007, 2024-09-15T03:48:05.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9740836640000001, 2024-09-15T03:48:05.071 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:05.071 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:05.071 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:05.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:05.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:48:05.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:05.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:48:05.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:05.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156931+0000", 2024-09-15T03:48:05.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1300000000000001e-07 2024-09-15T03:48:05.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:05.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156953+0000", 2024-09-15T03:48:05.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1585000000000001e-05 2024-09-15T03:48:05.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:05.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156974+0000", 2024-09-15T03:48:05.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0764e-05 2024-09-15T03:48:05.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:05.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156981+0000", 2024-09-15T03:48:05.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2760000000000003e-06 2024-09-15T03:48:05.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:05.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157061+0000", 2024-09-15T03:48:05.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9583e-05 2024-09-15T03:48:05.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:05.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157075+0000", 2024-09-15T03:48:05.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4019000000000001e-05 2024-09-15T03:48:05.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:05.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157351+0000", 2024-09-15T03:48:05.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027641100000000001 2024-09-15T03:48:05.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:05.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157746+0000", 2024-09-15T03:48:05.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039553199999999998 2024-09-15T03:48:05.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:05.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157838+0000", 2024-09-15T03:48:05.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1632000000000006e-05 2024-09-15T03:48:05.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:05.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157856+0000", 2024-09-15T03:48:05.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7873999999999999e-05 2024-09-15T03:48:05.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:05.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.131015+0000", 2024-09-15T03:48:05.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9731586750000001 2024-09-15T03:48:05.076 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.076 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:05.076 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.076 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=46, reqid=client.4521.0:228, at_version=38'25, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:48:05.076 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:48:05.076 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.893074143999996, 2024-09-15T03:48:05.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9732838510000001, 2024-09-15T03:48:05.077 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:05.077 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:05.077 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:05.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:05.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:48:05.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:05.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:48:05.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:05.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.156990+0000", 2024-09-15T03:48:05.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0999999999999998e-07 2024-09-15T03:48:05.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:05.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157149+0000", 2024-09-15T03:48:05.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000158763 2024-09-15T03:48:05.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:05.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157165+0000", 2024-09-15T03:48:05.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5883999999999999e-05 2024-09-15T03:48:05.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:05.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157175+0000", 2024-09-15T03:48:05.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4949999999999997e-06 2024-09-15T03:48:05.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:05.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157219+0000", 2024-09-15T03:48:05.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4069000000000001e-05 2024-09-15T03:48:05.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:05.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157228+0000", 2024-09-15T03:48:05.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7829999999999993e-06 2024-09-15T03:48:05.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:05.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157345+0000", 2024-09-15T03:48:05.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116655 2024-09-15T03:48:05.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:05.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157746+0000", 2024-09-15T03:48:05.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040045100000000002 2024-09-15T03:48:05.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:05.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157838+0000", 2024-09-15T03:48:05.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1955999999999998e-05 2024-09-15T03:48:05.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:05.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.157888+0000", 2024-09-15T03:48:05.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0943000000000003e-05 2024-09-15T03:48:05.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:05.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.130274+0000", 2024-09-15T03:48:05.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972385142 2024-09-15T03:48:05.081 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.082 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:05.082 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.082 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=38, reqid=client.4521.0:237, at_version=38'6, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:48:05.082 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:48:05.082 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.891731574000005, 2024-09-15T03:48:05.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9764011529999999, 2024-09-15T03:48:05.082 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:05.082 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:05.082 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:05.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:05.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:48:05.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:05.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158332+0000", 2024-09-15T03:48:05.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:05.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158333+0000", 2024-09-15T03:48:05.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8599999999999997e-07 2024-09-15T03:48:05.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:05.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158353+0000", 2024-09-15T03:48:05.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9504000000000002e-05 2024-09-15T03:48:05.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:05.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158368+0000", 2024-09-15T03:48:05.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4966999999999999e-05 2024-09-15T03:48:05.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:05.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158374+0000", 2024-09-15T03:48:05.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.798e-06 2024-09-15T03:48:05.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:05.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158430+0000", 2024-09-15T03:48:05.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5744e-05 2024-09-15T03:48:05.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:05.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158440+0000", 2024-09-15T03:48:05.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4560000000000006e-06 2024-09-15T03:48:05.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:05.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158529+0000", 2024-09-15T03:48:05.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9172999999999995e-05 2024-09-15T03:48:05.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:05.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158543+0000", 2024-09-15T03:48:05.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.395e-05 2024-09-15T03:48:05.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:05.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158654+0000", 2024-09-15T03:48:05.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011110700000000001 2024-09-15T03:48:05.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:05.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158671+0000", 2024-09-15T03:48:05.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7068999999999999e-05 2024-09-15T03:48:05.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:05.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134733+0000", 2024-09-15T03:48:05.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.976062599 2024-09-15T03:48:05.087 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.087 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:05.087 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.087 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=47, reqid=client.4521.0:229, at_version=38'26, trim_to=0'0, roll_forward_to=35'22))", 2024-09-15T03:48:05.088 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:48:05.088 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.891727611999997, 2024-09-15T03:48:05.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9727810969999999, 2024-09-15T03:48:05.088 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:05.088 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:05.088 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:05.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:05.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:48:05.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:05.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158336+0000", 2024-09-15T03:48:05.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:05.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158337+0000", 2024-09-15T03:48:05.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.15e-07 2024-09-15T03:48:05.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:05.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158351+0000", 2024-09-15T03:48:05.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3871e-05 2024-09-15T03:48:05.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:05.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158372+0000", 2024-09-15T03:48:05.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1058999999999999e-05 2024-09-15T03:48:05.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:05.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158376+0000", 2024-09-15T03:48:05.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5249999999999999e-06 2024-09-15T03:48:05.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:05.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158420+0000", 2024-09-15T03:48:05.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3918000000000002e-05 2024-09-15T03:48:05.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:05.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158429+0000", 2024-09-15T03:48:05.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5150000000000006e-06 2024-09-15T03:48:05.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:05.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158514+0000", 2024-09-15T03:48:05.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5252000000000003e-05 2024-09-15T03:48:05.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:05.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158542+0000", 2024-09-15T03:48:05.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.798e-05 2024-09-15T03:48:05.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:05.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158653+0000", 2024-09-15T03:48:05.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111315 2024-09-15T03:48:05.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:05.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.158701+0000", 2024-09-15T03:48:05.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8241000000000001e-05 2024-09-15T03:48:05.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:05.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.131117+0000", 2024-09-15T03:48:05.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.972416006 2024-09-15T03:48:05.094 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.094 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:05.094 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.094 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=39, reqid=client.4521.0:238, at_version=38'7, trim_to=0'0, roll_forward_to=32'3))", 2024-09-15T03:48:05.094 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:48:05.095 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.891040455999999, 2024-09-15T03:48:05.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9758214509999998, 2024-09-15T03:48:05.095 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:05.095 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:05.095 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:05.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:05.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:48:05.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:05.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159023+0000", 2024-09-15T03:48:05.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:05.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159024+0000", 2024-09-15T03:48:05.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6700000000000003e-07 2024-09-15T03:48:05.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:05.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159096+0000", 2024-09-15T03:48:05.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1386999999999999e-05 2024-09-15T03:48:05.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:05.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159113+0000", 2024-09-15T03:48:05.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7020999999999998e-05 2024-09-15T03:48:05.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:05.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159122+0000", 2024-09-15T03:48:05.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4720000000000001e-06 2024-09-15T03:48:05.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:05.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159168+0000", 2024-09-15T03:48:05.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5766000000000002e-05 2024-09-15T03:48:05.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:05.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159176+0000", 2024-09-15T03:48:05.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3070000000000003e-06 2024-09-15T03:48:05.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:05.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159258+0000", 2024-09-15T03:48:05.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1398999999999998e-05 2024-09-15T03:48:05.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:05.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159292+0000", 2024-09-15T03:48:05.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4480000000000002e-05 2024-09-15T03:48:05.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:05.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159342+0000", 2024-09-15T03:48:05.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9450999999999998e-05 2024-09-15T03:48:05.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:05.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.159362+0000", 2024-09-15T03:48:05.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0633e-05 2024-09-15T03:48:05.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:05.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134845+0000", 2024-09-15T03:48:05.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.975482768 2024-09-15T03:48:05.100 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.100 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:05.100 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.100 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 38/28 ECSubWrite(tid=48, reqid=client.4521.0:229, at_version=0'0, trim_to=0'0, roll_forward_to=38'26))", 2024-09-15T03:48:05.100 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:48:05.100 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.88967366, 2024-09-15T03:48:05.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.974485472, 2024-09-15T03:48:05.100 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:05.100 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:05.100 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:05.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:05.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:48:05.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:05.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160390+0000", 2024-09-15T03:48:05.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:05.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160391+0000", 2024-09-15T03:48:05.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.136e-06 2024-09-15T03:48:05.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:05.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160405+0000", 2024-09-15T03:48:05.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3613e-05 2024-09-15T03:48:05.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:05.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160426+0000", 2024-09-15T03:48:05.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1234999999999999e-05 2024-09-15T03:48:05.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:05.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160434+0000", 2024-09-15T03:48:05.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.61e-06 2024-09-15T03:48:05.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:05.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160496+0000", 2024-09-15T03:48:05.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2384000000000004e-05 2024-09-15T03:48:05.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:05.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160507+0000", 2024-09-15T03:48:05.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1005000000000001e-05 2024-09-15T03:48:05.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:05.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160632+0000", 2024-09-15T03:48:05.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124566 2024-09-15T03:48:05.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:05.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160653+0000", 2024-09-15T03:48:05.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0707999999999999e-05 2024-09-15T03:48:05.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:05.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160701+0000", 2024-09-15T03:48:05.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8751999999999997e-05 2024-09-15T03:48:05.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:05.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.160739+0000", 2024-09-15T03:48:05.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7486e-05 2024-09-15T03:48:05.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:05.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.134876+0000", 2024-09-15T03:48:05.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9741369770000001 2024-09-15T03:48:05.105 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.105 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:05.105 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.106 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/31 ECSubWrite(tid=40, reqid=client.4521.0:238, at_version=0'0, trim_to=0'0, roll_forward_to=38'7))", 2024-09-15T03:48:05.106 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:48:05.106 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.887827239000003, 2024-09-15T03:48:05.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9727862730000001, 2024-09-15T03:48:05.106 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:05.106 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:05.106 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:05.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:05.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:48:05.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:05.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162237+0000", 2024-09-15T03:48:05.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:05.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162238+0000", 2024-09-15T03:48:05.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.672e-06 2024-09-15T03:48:05.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:05.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162253+0000", 2024-09-15T03:48:05.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4932e-05 2024-09-15T03:48:05.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:05.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162284+0000", 2024-09-15T03:48:05.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1211000000000003e-05 2024-09-15T03:48:05.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:05.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162296+0000", 2024-09-15T03:48:05.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1229999999999999e-05 2024-09-15T03:48:05.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:05.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162373+0000", 2024-09-15T03:48:05.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7146999999999998e-05 2024-09-15T03:48:05.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:05.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162387+0000", 2024-09-15T03:48:05.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4290999999999999e-05 2024-09-15T03:48:05.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:05.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162502+0000", 2024-09-15T03:48:05.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115054 2024-09-15T03:48:05.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:05.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162516+0000", 2024-09-15T03:48:05.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3597999999999999e-05 2024-09-15T03:48:05.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:05.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162558+0000", 2024-09-15T03:48:05.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2562999999999999e-05 2024-09-15T03:48:05.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:05.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:45.162584+0000", 2024-09-15T03:48:05.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5738e-05 2024-09-15T03:48:05.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:05.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135023+0000", 2024-09-15T03:48:05.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9724388369999999 2024-09-15T03:48:05.111 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.111 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:05.111 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.111 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 39/28 ECSubWrite(tid=38, reqid=osd.11.0:38, at_version=39'16, trim_to=0'0, roll_forward_to=38'14))", 2024-09-15T03:48:05.111 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:48:05.111 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 78.906444938999996, 2024-09-15T03:48:05.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99210653800000004, 2024-09-15T03:48:05.111 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:05.111 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:05.112 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:05.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:05.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:48:05.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:05.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:48:05.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:05.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143619+0000", 2024-09-15T03:48:05.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1800000000000002e-07 2024-09-15T03:48:05.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:05.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143648+0000", 2024-09-15T03:48:05.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.849e-05 2024-09-15T03:48:05.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:05.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143670+0000", 2024-09-15T03:48:05.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2036e-05 2024-09-15T03:48:05.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:05.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143680+0000", 2024-09-15T03:48:05.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8320000000000001e-06 2024-09-15T03:48:05.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:05.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143783+0000", 2024-09-15T03:48:05.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103569 2024-09-15T03:48:05.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:05.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.143801+0000", 2024-09-15T03:48:05.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8077999999999999e-05 2024-09-15T03:48:05.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:05.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144077+0000", 2024-09-15T03:48:05.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027569500000000001 2024-09-15T03:48:05.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:05.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144459+0000", 2024-09-15T03:48:05.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038227299999999998 2024-09-15T03:48:05.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:05.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144519+0000", 2024-09-15T03:48:05.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9907000000000001e-05 2024-09-15T03:48:05.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:05.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144628+0000", 2024-09-15T03:48:05.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108958 2024-09-15T03:48:05.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:05.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135726+0000", 2024-09-15T03:48:05.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99109738199999997 2024-09-15T03:48:05.117 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.117 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:05.117 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.117 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 39/31 ECSubWrite(tid=41, reqid=osd.6.0:41, at_version=39'9, trim_to=0'0, roll_forward_to=38'7))", 2024-09-15T03:48:05.117 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:48:05.117 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 78.905794684, 2024-09-15T03:48:05.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99106653099999997, 2024-09-15T03:48:05.117 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:05.117 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:05.117 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:05.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:05.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:48:05.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:05.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:48:05.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:05.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144269+0000", 2024-09-15T03:48:05.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.97e-07 2024-09-15T03:48:05.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:05.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144297+0000", 2024-09-15T03:48:05.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7815000000000001e-05 2024-09-15T03:48:05.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:05.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144314+0000", 2024-09-15T03:48:05.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6269e-05 2024-09-15T03:48:05.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:05.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144322+0000", 2024-09-15T03:48:05.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1880000000000002e-06 2024-09-15T03:48:05.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:05.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144767+0000", 2024-09-15T03:48:05.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044555300000000001 2024-09-15T03:48:05.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:05.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144777+0000", 2024-09-15T03:48:05.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5950000000000005e-06 2024-09-15T03:48:05.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:05.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144992+0000", 2024-09-15T03:48:05.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000215479 2024-09-15T03:48:05.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:05.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145142+0000", 2024-09-15T03:48:05.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149302 2024-09-15T03:48:05.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:05.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145249+0000", 2024-09-15T03:48:05.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106971 2024-09-15T03:48:05.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:05.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145299+0000", 2024-09-15T03:48:05.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0413e-05 2024-09-15T03:48:05.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:05.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135336+0000", 2024-09-15T03:48:05.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99003674900000005 2024-09-15T03:48:05.124 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.124 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:05.124 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.124 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 39/28 ECSubWrite(tid=50, reqid=osd.2.0:50, at_version=39'28, trim_to=0'0, roll_forward_to=38'26))", 2024-09-15T03:48:05.124 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:48:05.124 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 78.905497202000006, 2024-09-15T03:48:05.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99095576299999999, 2024-09-15T03:48:05.124 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:05.125 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:05.125 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:05.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:05.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:48:05.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:05.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:48:05.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:05.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144567+0000", 2024-09-15T03:48:05.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.03e-07 2024-09-15T03:48:05.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:05.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144593+0000", 2024-09-15T03:48:05.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6520999999999999e-05 2024-09-15T03:48:05.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:05.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144611+0000", 2024-09-15T03:48:05.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7963000000000001e-05 2024-09-15T03:48:05.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:05.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144619+0000", 2024-09-15T03:48:05.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7409999999999992e-06 2024-09-15T03:48:05.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:05.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144821+0000", 2024-09-15T03:48:05.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000201652 2024-09-15T03:48:05.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:05.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.144832+0000", 2024-09-15T03:48:05.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0827e-05 2024-09-15T03:48:05.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:05.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145099+0000", 2024-09-15T03:48:05.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026688299999999999 2024-09-15T03:48:05.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:05.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145143+0000", 2024-09-15T03:48:05.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.481e-05 2024-09-15T03:48:05.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:05.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145249+0000", 2024-09-15T03:48:05.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010618799999999999 2024-09-15T03:48:05.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:05.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.145340+0000", 2024-09-15T03:48:05.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0340000000000006e-05 2024-09-15T03:48:05.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:05.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.135522+0000", 2024-09-15T03:48:05.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99018263500000003 2024-09-15T03:48:05.129 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.129 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:05.130 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.130 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=43, reqid=client.4521.0:240, at_version=39'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:48:05.130 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:48:05.130 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 78.902156481999995, 2024-09-15T03:48:05.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99104975500000003, 2024-09-15T03:48:05.130 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:05.130 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:05.130 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:05.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:05.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:48:05.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:05.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147907+0000", 2024-09-15T03:48:05.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:05.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147908+0000", 2024-09-15T03:48:05.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.05e-07 2024-09-15T03:48:05.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:05.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147931+0000", 2024-09-15T03:48:05.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3621e-05 2024-09-15T03:48:05.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:05.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147945+0000", 2024-09-15T03:48:05.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3648999999999999e-05 2024-09-15T03:48:05.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:05.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.147951+0000", 2024-09-15T03:48:05.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0510000000000002e-06 2024-09-15T03:48:05.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:05.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148055+0000", 2024-09-15T03:48:05.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104421 2024-09-15T03:48:05.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:05.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148068+0000", 2024-09-15T03:48:05.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3019e-05 2024-09-15T03:48:05.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:05.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148375+0000", 2024-09-15T03:48:05.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030683400000000002 2024-09-15T03:48:05.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:05.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148399+0000", 2024-09-15T03:48:05.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3760999999999998e-05 2024-09-15T03:48:05.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:05.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148453+0000", 2024-09-15T03:48:05.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3851999999999999e-05 2024-09-15T03:48:05.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:05.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148481+0000", 2024-09-15T03:48:05.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8637999999999999e-05 2024-09-15T03:48:05.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:05.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.138957+0000", 2024-09-15T03:48:05.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99047580400000002 2024-09-15T03:48:05.135 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.135 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:05.135 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.135 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.as7 39/28 ECSubWrite(tid=39, reqid=osd.11.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=39'16))", 2024-09-15T03:48:05.136 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:48:05.136 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 78.902023993, 2024-09-15T03:48:05.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99452509, 2024-09-15T03:48:05.136 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:05.136 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:05.136 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:05.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:05.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:48:05.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:05.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148040+0000", 2024-09-15T03:48:05.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:05.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148041+0000", 2024-09-15T03:48:05.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0999999999999998e-07 2024-09-15T03:48:05.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:05.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148052+0000", 2024-09-15T03:48:05.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.129e-05 2024-09-15T03:48:05.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:05.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148064+0000", 2024-09-15T03:48:05.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173e-05 2024-09-15T03:48:05.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:05.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148068+0000", 2024-09-15T03:48:05.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5680000000000001e-06 2024-09-15T03:48:05.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:05.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148136+0000", 2024-09-15T03:48:05.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7508999999999993e-05 2024-09-15T03:48:05.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:05.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148148+0000", 2024-09-15T03:48:05.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2002e-05 2024-09-15T03:48:05.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:05.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148265+0000", 2024-09-15T03:48:05.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117669 2024-09-15T03:48:05.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:05.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148289+0000", 2024-09-15T03:48:05.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3672e-05 2024-09-15T03:48:05.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:05.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148348+0000", 2024-09-15T03:48:05.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8838000000000001e-05 2024-09-15T03:48:05.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:05.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.148368+0000", 2024-09-15T03:48:05.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9942999999999999e-05 2024-09-15T03:48:05.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:05.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142565+0000", 2024-09-15T03:48:05.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99419715900000005 2024-09-15T03:48:05.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.141 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:05.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.142 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=45, reqid=client.4521.0:241, at_version=39'17, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:48:05.142 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:48:05.142 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 78.899020969999995, 2024-09-15T03:48:05.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99118202200000005, 2024-09-15T03:48:05.142 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:05.142 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:05.142 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:05.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:05.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:48:05.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:05.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151043+0000", 2024-09-15T03:48:05.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:05.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151044+0000", 2024-09-15T03:48:05.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1340000000000001e-06 2024-09-15T03:48:05.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:05.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151073+0000", 2024-09-15T03:48:05.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8894000000000002e-05 2024-09-15T03:48:05.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:05.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151095+0000", 2024-09-15T03:48:05.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1889e-05 2024-09-15T03:48:05.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:05.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151107+0000", 2024-09-15T03:48:05.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2136999999999999e-05 2024-09-15T03:48:05.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:05.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151189+0000", 2024-09-15T03:48:05.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1883000000000002e-05 2024-09-15T03:48:05.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:05.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151203+0000", 2024-09-15T03:48:05.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3825999999999999e-05 2024-09-15T03:48:05.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:05.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151305+0000", 2024-09-15T03:48:05.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101816 2024-09-15T03:48:05.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:05.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151337+0000", 2024-09-15T03:48:05.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2703000000000002e-05 2024-09-15T03:48:05.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:05.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151387+0000", 2024-09-15T03:48:05.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9366000000000001e-05 2024-09-15T03:48:05.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:05.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151405+0000", 2024-09-15T03:48:05.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8748000000000001e-05 2024-09-15T03:48:05.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:05.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142225+0000", 2024-09-15T03:48:05.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99081962599999995 2024-09-15T03:48:05.147 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.147 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:05.147 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.147 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.4s2 39/31 ECSubWrite(tid=42, reqid=osd.6.0:41, at_version=0'0, trim_to=0'0, roll_forward_to=39'9))", 2024-09-15T03:48:05.147 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:48:05.147 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 78.898724928999997, 2024-09-15T03:48:05.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99092639999999999, 2024-09-15T03:48:05.147 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:05.147 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:05.148 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:05.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:05.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:48:05.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:05.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151339+0000", 2024-09-15T03:48:05.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:05.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151340+0000", 2024-09-15T03:48:05.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8299999999999996e-07 2024-09-15T03:48:05.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:05.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151350+0000", 2024-09-15T03:48:05.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0675e-05 2024-09-15T03:48:05.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:05.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151362+0000", 2024-09-15T03:48:05.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2048999999999999e-05 2024-09-15T03:48:05.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:05.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151369+0000", 2024-09-15T03:48:05.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3230000000000003e-06 2024-09-15T03:48:05.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:05.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151425+0000", 2024-09-15T03:48:05.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5964000000000002e-05 2024-09-15T03:48:05.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:05.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151435+0000", 2024-09-15T03:48:05.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9240000000000002e-06 2024-09-15T03:48:05.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:05.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151530+0000", 2024-09-15T03:48:05.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5649000000000005e-05 2024-09-15T03:48:05.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:05.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151558+0000", 2024-09-15T03:48:05.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7976000000000001e-05 2024-09-15T03:48:05.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:05.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151599+0000", 2024-09-15T03:48:05.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1152000000000002e-05 2024-09-15T03:48:05.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:05.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151617+0000", 2024-09-15T03:48:05.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7676000000000002e-05 2024-09-15T03:48:05.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:05.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142265+0000", 2024-09-15T03:48:05.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99064822900000005 2024-09-15T03:48:05.153 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.153 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:05.153 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.154 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=46, reqid=client.4521.0:242, at_version=39'18, trim_to=0'0, roll_forward_to=35'14))", 2024-09-15T03:48:05.154 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:48:05.154 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 78.898699292999993, 2024-09-15T03:48:05.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.991000204, 2024-09-15T03:48:05.154 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:05.154 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:05.154 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:05.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:05.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:48:05.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:05.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:48:05.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:05.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151365+0000", 2024-09-15T03:48:05.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3399999999999999e-07 2024-09-15T03:48:05.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:05.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151393+0000", 2024-09-15T03:48:05.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8149000000000001e-05 2024-09-15T03:48:05.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:05.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151405+0000", 2024-09-15T03:48:05.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2063e-05 2024-09-15T03:48:05.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:05.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151415+0000", 2024-09-15T03:48:05.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4579999999999999e-06 2024-09-15T03:48:05.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:05.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151472+0000", 2024-09-15T03:48:05.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7018999999999997e-05 2024-09-15T03:48:05.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:05.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151480+0000", 2024-09-15T03:48:05.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1410000000000005e-06 2024-09-15T03:48:05.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:05.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151571+0000", 2024-09-15T03:48:05.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1243999999999996e-05 2024-09-15T03:48:05.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:05.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151602+0000", 2024-09-15T03:48:05.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0834000000000003e-05 2024-09-15T03:48:05.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:05.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151632+0000", 2024-09-15T03:48:05.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9527e-05 2024-09-15T03:48:05.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:05.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.151685+0000", 2024-09-15T03:48:05.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3599e-05 2024-09-15T03:48:05.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:05.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142365+0000", 2024-09-15T03:48:05.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99067953600000003 2024-09-15T03:48:05.159 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.159 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:05.160 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.160 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.es6 39/28 ECSubWrite(tid=51, reqid=osd.2.0:50, at_version=0'0, trim_to=0'0, roll_forward_to=39'28))", 2024-09-15T03:48:05.160 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:48:05.160 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 78.895335854999999, 2024-09-15T03:48:05.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98766307399999997, 2024-09-15T03:48:05.160 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:05.160 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:05.160 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:05.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:05.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:48:05.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:05.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154728+0000", 2024-09-15T03:48:05.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:05.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154729+0000", 2024-09-15T03:48:05.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3799999999999996e-07 2024-09-15T03:48:05.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:05.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154742+0000", 2024-09-15T03:48:05.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2909e-05 2024-09-15T03:48:05.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:05.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154755+0000", 2024-09-15T03:48:05.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3251999999999999e-05 2024-09-15T03:48:05.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:05.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154761+0000", 2024-09-15T03:48:05.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6559999999999997e-06 2024-09-15T03:48:05.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:05.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154829+0000", 2024-09-15T03:48:05.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8549999999999999e-05 2024-09-15T03:48:05.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:05.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154840+0000", 2024-09-15T03:48:05.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0159999999999999e-05 2024-09-15T03:48:05.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:05.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154943+0000", 2024-09-15T03:48:05.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010347400000000001 2024-09-15T03:48:05.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:05.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.154955+0000", 2024-09-15T03:48:05.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1935e-05 2024-09-15T03:48:05.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:05.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156442+0000", 2024-09-15T03:48:05.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014867859999999999 2024-09-15T03:48:05.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:05.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156466+0000", 2024-09-15T03:48:05.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4077999999999999e-05 2024-09-15T03:48:05.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:05.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142391+0000", 2024-09-15T03:48:05.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98592533599999999 2024-09-15T03:48:05.165 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.165 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:05.165 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.165 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDECSubOpWrite(3.0s7 39/31 ECSubWrite(tid=48, reqid=client.4521.0:242, at_version=0'0, trim_to=0'0, roll_forward_to=39'18))", 2024-09-15T03:48:05.165 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:48:05.165 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 78.893448836000005, 2024-09-15T03:48:05.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98592495099999999, 2024-09-15T03:48:05.166 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:05.166 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-15T03:48:05.166 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:05.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:05.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:48:05.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:05.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156615+0000", 2024-09-15T03:48:05.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:05.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:05.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156616+0000", 2024-09-15T03:48:05.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1999999999999998e-07 2024-09-15T03:48:05.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:05.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156629+0000", 2024-09-15T03:48:05.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2870999999999999e-05 2024-09-15T03:48:05.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:05.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156649+0000", 2024-09-15T03:48:05.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9786e-05 2024-09-15T03:48:05.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:05.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156657+0000", 2024-09-15T03:48:05.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5110000000000003e-06 2024-09-15T03:48:05.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:05.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156723+0000", 2024-09-15T03:48:05.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6209000000000003e-05 2024-09-15T03:48:05.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:05.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156734+0000", 2024-09-15T03:48:05.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0475999999999999e-05 2024-09-15T03:48:05.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:05.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156868+0000", 2024-09-15T03:48:05.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013402999999999999 2024-09-15T03:48:05.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:05.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156889+0000", 2024-09-15T03:48:05.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1146e-05 2024-09-15T03:48:05.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:05.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156937+0000", 2024-09-15T03:48:05.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8031999999999997e-05 2024-09-15T03:48:05.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:05.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:46.156970+0000", 2024-09-15T03:48:05.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2462999999999997e-05 2024-09-15T03:48:05.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:05.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:05.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:05.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:47.142540+0000", 2024-09-15T03:48:05.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98557050700000004 2024-09-15T03:48:05.170 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.170 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:05.171 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.171 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:05.171 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:05.171 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:48:05.172 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:48:05.173 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:48:05.171+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:48:05.200 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:48:05.273 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:48:05.272+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:48:05.316 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:48:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:48:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:48:05.316 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:48:05.318 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:48:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:48:05.324 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:48:05.332 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:48:05.332 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:48:05.347 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:48:05.347 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:48:05.361 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:48:05.361 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:48:05.373 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:48:05.373+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:48:05.375 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:48:05.375 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:48:05.390 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:48:05.390 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:48:05.404 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:48:05.404 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:48:05.418 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:48:05.418 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:48:05.425 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:05.425 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:05.433 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:48:05.434 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:48:05.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:48:05.448 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:48:05.449 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:48:05.464 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:48:05.464 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:48:05.477 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:48:05.477+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:48:05.479 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:48:05.479 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:48:05.534 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:05.535 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:05.543 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:48:05.575 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:48:05.575+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:48:05.676 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:48:05.676+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:48:05.685 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:05.686 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:05.694 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:48:05.777 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:48:05.776+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:48:05.795 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:05.795 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:05.804 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:48:05.878 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:48:05.878+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:48:05.904 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:05.905 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:05.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:48:05.975 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.10 to 1 (10000) 2024-09-15T03:48:05.975 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-15T03:48:05.979 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:48:05.978+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:48:05.992 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-15T03:48:06.023 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:06.023 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:06.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:48:06.080 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:48:06.080+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:48:06.130 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:06.130 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:06.139 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:48:06.181 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:48:06.181+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:48:06.238 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:06.238 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:06.246 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:48:06.289 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:48:06.289+0000 7fd62abf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33736) UID: 0 2024-09-15T03:48:06.342 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:06.342 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:06.351 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:48:06.383 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:48:06.382+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:48:06.483 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:48:06.483+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:48:06.485 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:06.486 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:06.494 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:48:06.584 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:48:06.584+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:48:06.589 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:06.589 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:06.598 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:48:06.685 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:48:06.685+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:48:06.734 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:06.735 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:06.743 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:48:06.786 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:48:06.786+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:48:06.838 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:06.838 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:06.847 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:48:06.892 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:48:06.891+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:48:06.944 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:06.944 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:06.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:48:06.981 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.6 to 1 (10000) 2024-09-15T03:48:06.988 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:48:06.987+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:48:07.000 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-15T03:48:07.000 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:48:07.089 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:48:07.088+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:48:07.092 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:07.092 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:07.101 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:48:07.189 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:48:07.189+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:48:07.239 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:07.239 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:07.247 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:48:07.291 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:48:07.290+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:48:07.324 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:48:07.325 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-15T03:48:07.335 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-09-15T03:48:06.981453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":183697971,"num_objects":193,"num_object_clones":139,"num_object_copies":1520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":9,"num_read":533,"num_read_kb":239247,"num_write":1029,"num_write_kb":173944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":267,"num_bytes_recovered":285786150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1589,"ondisk_log_size":1589,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":12,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":1124155392,"kb_used":10221720,"kb_used_data":1113935212,"kb_used_omap":55676,"kb_used_meta":0,"kb_avail":1113933672,"statfs":{"total":1151135121408,"available":1140668080128,"internally_reserved":1576960,"allocated":1140669657088,"data_stored":1140669657088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57012943,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.018648"},"pg_stats":[{"pgid":"3.f","version":"113'71","reported_seq":461,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.190992+0000","last_change":"2024-09-15T03:48:03.620987+0000","last_active":"2024-09-15T03:48:05.190992+0000","last_peered":"2024-09-15T03:48:05.190992+0000","last_clean":"2024-09-15T03:48:05.190992+0000","last_became_active":"2024-09-15T03:47:55.850051+0000","last_became_peered":"2024-09-15T03:47:55.850051+0000","last_unstale":"2024-09-15T03:48:05.190992+0000","last_undegraded":"2024-09-15T03:48:05.190992+0000","last_fullsized":"2024-09-15T03:48:05.190992+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'71","last_scrub_stamp":"2024-09-15T03:48:03.620910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:03.620910+0000","objects_scrubbed":8,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:13.277978+0000","scrub_duration":0.0050823509999999997,"objects_trimmed":0,"snaptrim_duration":0.0077598160000000001,"stat_sum":{"num_bytes":8028160,"num_objects":8,"num_object_clones":5,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":32,"num_read_kb":42506,"num_write":26,"num_write_kb":7840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0,7,2,8,4,10],"acting":[11,1,0,7,2,8,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.e","version":"113'210","reported_seq":442,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.974944+0000","last_change":"2024-09-15T03:47:59.982417+0000","last_active":"2024-09-15T03:48:04.974944+0000","last_peered":"2024-09-15T03:48:04.974944+0000","last_clean":"2024-09-15T03:48:04.974944+0000","last_became_active":"2024-09-15T03:46:36.170536+0000","last_became_peered":"2024-09-15T03:46:36.170536+0000","last_unstale":"2024-09-15T03:48:04.974944+0000","last_undegraded":"2024-09-15T03:48:04.974944+0000","last_fullsized":"2024-09-15T03:48:04.974944+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'173","last_scrub_stamp":"2024-09-15T03:47:54.108780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:47:54.108780+0000","objects_scrubbed":21,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:11.038305+0000","scrub_duration":0.0088561600000000001,"objects_trimmed":0,"snaptrim_duration":0.037652644999999998,"stat_sum":{"num_bytes":17612800,"num_objects":22,"num_object_clones":15,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":2,"num_read":68,"num_read_kb":34008,"num_write":51,"num_write_kb":15520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,11,7,8,5,9],"acting":[2,0,1,11,7,8,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.d","version":"113'75","reported_seq":399,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.191773+0000","last_change":"2024-09-15T03:47:59.958101+0000","last_active":"2024-09-15T03:48:05.191773+0000","last_peered":"2024-09-15T03:48:05.191773+0000","last_clean":"2024-09-15T03:48:05.191773+0000","last_became_active":"2024-09-15T03:47:55.850104+0000","last_became_peered":"2024-09-15T03:47:55.850104+0000","last_unstale":"2024-09-15T03:48:05.191773+0000","last_undegraded":"2024-09-15T03:48:05.191773+0000","last_fullsized":"2024-09-15T03:48:05.191773+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'62","last_scrub_stamp":"2024-09-15T03:47:56.721765+0000","last_deep_scrub":"109'62","last_deep_scrub_stamp":"2024-09-15T03:47:56.721765+0000","last_clean_scrub_stamp":"2024-09-15T03:47:56.721765+0000","objects_scrubbed":10,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:15.880649+0000","scrub_duration":0.00827518,"objects_trimmed":0,"snaptrim_duration":0.013785528999999999,"stat_sum":{"num_bytes":7651328,"num_objects":10,"num_object_clones":8,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":20,"num_read_kb":8406,"num_write":43,"num_write_kb":7472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":10780672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,1,3,10,7,9,4],"acting":[8,11,1,3,10,7,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.c","version":"113'28","reported_seq":383,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.840818+0000","last_change":"2024-09-15T03:47:59.955875+0000","last_active":"2024-09-15T03:48:04.840818+0000","last_peered":"2024-09-15T03:48:04.840818+0000","last_clean":"2024-09-15T03:48:04.840818+0000","last_became_active":"2024-09-15T03:47:36.848722+0000","last_became_peered":"2024-09-15T03:47:36.848722+0000","last_unstale":"2024-09-15T03:48:04.840818+0000","last_undegraded":"2024-09-15T03:48:04.840818+0000","last_fullsized":"2024-09-15T03:48:04.840818+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'16","last_scrub_stamp":"2024-09-15T03:47:41.493521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:47:41.493521+0000","objects_scrubbed":3,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:09.644586+0000","scrub_duration":0.0039888800000000002,"objects_trimmed":0,"snaptrim_duration":0.012831169999999999,"stat_sum":{"num_bytes":2965504,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":18,"num_read_kb":8502,"num_write":10,"num_write_kb":3440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,11,1,0,6,5,2],"acting":[9,7,11,1,0,6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.b","version":"113'212","reported_seq":766,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.843752+0000","last_change":"2024-09-15T03:48:03.010113+0000","last_active":"2024-09-15T03:48:04.843752+0000","last_peered":"2024-09-15T03:48:04.843752+0000","last_clean":"2024-09-15T03:48:04.843752+0000","last_became_active":"2024-09-15T03:47:55.850099+0000","last_became_peered":"2024-09-15T03:47:55.850099+0000","last_unstale":"2024-09-15T03:48:04.843752+0000","last_undegraded":"2024-09-15T03:48:04.843752+0000","last_fullsized":"2024-09-15T03:48:04.843752+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'212","last_scrub_stamp":"2024-09-15T03:48:03.010035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:03.010035+0000","objects_scrubbed":27,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:22.035341+0000","scrub_duration":0.022865639,"objects_trimmed":0,"snaptrim_duration":0.025351973,"stat_sum":{"num_bytes":24117248,"num_objects":27,"num_object_clones":20,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":2,"num_read":65,"num_read_kb":36889,"num_write":170,"num_write_kb":22832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":74006528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,8,4,1,10,11,3],"acting":[7,6,8,4,1,10,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.a","version":"113'84","reported_seq":240,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.896901+0000","last_change":"2024-09-15T03:48:03.286753+0000","last_active":"2024-09-15T03:48:04.896901+0000","last_peered":"2024-09-15T03:48:04.896901+0000","last_clean":"2024-09-15T03:48:04.896901+0000","last_became_active":"2024-09-15T03:47:55.854826+0000","last_became_peered":"2024-09-15T03:47:55.854826+0000","last_unstale":"2024-09-15T03:48:04.896901+0000","last_undegraded":"2024-09-15T03:48:04.896901+0000","last_fullsized":"2024-09-15T03:48:04.896901+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'84","last_scrub_stamp":"2024-09-15T03:48:03.286675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:03.286675+0000","objects_scrubbed":8,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:03.509620+0000","scrub_duration":0.0063142770000000001,"objects_trimmed":0,"snaptrim_duration":0.013016882,"stat_sum":{"num_bytes":6995968,"num_objects":8,"num_object_clones":6,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":16,"num_read_kb":6728,"num_write":30,"num_write_kb":6832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":30703616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8,9,5,6,2,0],"acting":[4,10,8,9,5,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.9","version":"113'80","reported_seq":492,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.979274+0000","last_change":"2024-09-15T03:47:59.971224+0000","last_active":"2024-09-15T03:48:04.979274+0000","last_peered":"2024-09-15T03:48:04.979274+0000","last_clean":"2024-09-15T03:48:04.979274+0000","last_became_active":"2024-09-15T03:47:55.851394+0000","last_became_peered":"2024-09-15T03:47:55.851394+0000","last_unstale":"2024-09-15T03:48:04.979274+0000","last_undegraded":"2024-09-15T03:48:04.979274+0000","last_fullsized":"2024-09-15T03:48:04.979274+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012050854,"stat_sum":{"num_bytes":12025856,"num_objects":13,"num_object_clones":10,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":15,"num_read_kb":8,"num_write":59,"num_write_kb":11744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":27590656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,10,1,9,6,8,11],"acting":[3,2,10,1,9,6,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.8","version":"113'52","reported_seq":377,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.979298+0000","last_change":"2024-09-15T03:47:59.962623+0000","last_active":"2024-09-15T03:48:04.979298+0000","last_peered":"2024-09-15T03:48:04.979298+0000","last_clean":"2024-09-15T03:48:04.979298+0000","last_became_active":"2024-09-15T03:47:55.848776+0000","last_became_peered":"2024-09-15T03:47:55.848776+0000","last_unstale":"2024-09-15T03:48:04.979298+0000","last_undegraded":"2024-09-15T03:48:04.979298+0000","last_fullsized":"2024-09-15T03:48:04.979298+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01649231,"stat_sum":{"num_bytes":4079616,"num_objects":6,"num_object_clones":4,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":23,"num_read_kb":4616,"num_write":54,"num_write_kb":3984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":11223040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,11,4,1,0,10],"acting":[3,7,2,11,4,1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"1.0","version":"20'92","reported_seq":226,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.059683+0000","last_change":"2024-09-15T03:47:39.774326+0000","last_active":"2024-09-15T03:48:04.059683+0000","last_peered":"2024-09-15T03:48:04.059683+0000","last_clean":"2024-09-15T03:48:04.059683+0000","last_became_active":"2024-09-15T03:46:18.011130+0000","last_became_peered":"2024-09-15T03:46:18.011130+0000","last_unstale":"2024-09-15T03:48:04.059683+0000","last_undegraded":"2024-09-15T03:48:04.059683+0000","last_fullsized":"2024-09-15T03:48:04.059683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-15T03:47:39.774262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_clean_scrub_stamp":"2024-09-15T03:47:39.774262+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:43.691013+0000","scrub_duration":0.0027413759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"22'1","reported_seq":118,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:03.962737+0000","last_change":"2024-09-15T03:47:52.292975+0000","last_active":"2024-09-15T03:48:03.962737+0000","last_peered":"2024-09-15T03:48:03.962737+0000","last_clean":"2024-09-15T03:48:03.962737+0000","last_became_active":"2024-09-15T03:46:24.038633+0000","last_became_peered":"2024-09-15T03:46:24.038633+0000","last_unstale":"2024-09-15T03:48:03.962737+0000","last_undegraded":"2024-09-15T03:48:03.962737+0000","last_fullsized":"2024-09-15T03:48:03.962737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'1","last_scrub_stamp":"2024-09-15T03:47:52.292926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:52.292926+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:56.362597+0000","scrub_duration":0.001409429,"objects_trimmed":0,"snaptrim_duration":0.00030178200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"113'80","reported_seq":513,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.896581+0000","last_change":"2024-09-15T03:47:59.966281+0000","last_active":"2024-09-15T03:48:04.896581+0000","last_peered":"2024-09-15T03:48:04.896581+0000","last_clean":"2024-09-15T03:48:04.896581+0000","last_became_active":"2024-09-15T03:47:55.854075+0000","last_became_peered":"2024-09-15T03:47:55.854075+0000","last_unstale":"2024-09-15T03:48:04.896581+0000","last_undegraded":"2024-09-15T03:48:04.896581+0000","last_fullsized":"2024-09-15T03:48:04.896581+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021707911999999999,"stat_sum":{"num_bytes":13320192,"num_objects":10,"num_object_clones":7,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":37,"num_read_kb":16719,"num_write":39,"num_write_kb":13088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":18628608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,7,8,6,11,1],"acting":[4,2,10,7,8,6,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":115,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.059319+0000","last_change":"2024-09-15T03:47:27.779257+0000","last_active":"2024-09-15T03:48:04.059319+0000","last_peered":"2024-09-15T03:48:04.059319+0000","last_clean":"2024-09-15T03:48:04.059319+0000","last_became_active":"2024-09-15T03:46:24.063106+0000","last_became_peered":"2024-09-15T03:46:24.063106+0000","last_unstale":"2024-09-15T03:48:04.059319+0000","last_undegraded":"2024-09-15T03:48:04.059319+0000","last_fullsized":"2024-09-15T03:48:04.059319+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.779210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.779210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:42.717969+0000","scrub_duration":0.0021030710000000002,"objects_trimmed":0,"snaptrim_duration":0.000393075,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"113'29","reported_seq":174,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.687310+0000","last_change":"2024-09-15T03:48:05.687193+0000","last_active":"2024-09-15T03:48:05.687310+0000","last_peered":"2024-09-15T03:48:05.687310+0000","last_clean":"2024-09-15T03:48:05.687310+0000","last_became_active":"2024-09-15T03:47:55.848033+0000","last_became_peered":"2024-09-15T03:47:55.848033+0000","last_unstale":"2024-09-15T03:48:05.687310+0000","last_undegraded":"2024-09-15T03:48:05.687310+0000","last_fullsized":"2024-09-15T03:48:05.687310+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'29","last_scrub_stamp":"2024-09-15T03:48:05.687085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:05.687085+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:32.245260+0000","scrub_duration":0.0044105530000000002,"objects_trimmed":0,"snaptrim_duration":0.0090284240000000002,"stat_sum":{"num_bytes":4653056,"num_objects":5,"num_object_clones":4,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":3075,"num_write":12,"num_write_kb":4032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":6045696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3,10,5,2,9,7],"acting":[6,11,3,10,5,2,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":120,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.843432+0000","last_change":"2024-09-15T03:47:42.997012+0000","last_active":"2024-09-15T03:48:04.843432+0000","last_peered":"2024-09-15T03:48:04.843432+0000","last_clean":"2024-09-15T03:48:04.843432+0000","last_became_active":"2024-09-15T03:46:24.037471+0000","last_became_peered":"2024-09-15T03:46:24.037471+0000","last_unstale":"2024-09-15T03:48:04.843432+0000","last_undegraded":"2024-09-15T03:48:04.843432+0000","last_fullsized":"2024-09-15T03:48:04.843432+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:42.996957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:42.996957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:53.443125+0000","scrub_duration":0.0014164309999999999,"objects_trimmed":0,"snaptrim_duration":0.00026490299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"113'56","reported_seq":456,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-15T03:48:03.778938+0000","last_change":"2024-09-15T03:48:03.778794+0000","last_active":"2024-09-15T03:48:03.778938+0000","last_peered":"2024-09-15T03:48:03.778938+0000","last_clean":"2024-09-15T03:48:03.778938+0000","last_became_active":"2024-09-15T03:47:55.848512+0000","last_became_peered":"2024-09-15T03:47:55.848512+0000","last_unstale":"2024-09-15T03:48:03.778938+0000","last_undegraded":"2024-09-15T03:48:03.778938+0000","last_fullsized":"2024-09-15T03:48:03.778938+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'56","last_scrub_stamp":"2024-09-15T03:48:03.778652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:03.778652+0000","objects_scrubbed":8,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:05.162717+0000","scrub_duration":0.0067734900000000001,"objects_trimmed":0,"snaptrim_duration":0.012362052,"stat_sum":{"num_bytes":4997120,"num_objects":8,"num_object_clones":5,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":20,"num_read_kb":7496,"num_write":43,"num_write_kb":4880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":16449536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,9,4,3,6,0,10],"acting":[1,2,9,4,3,6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"c","length":"1"},{"start":"12","length":"1"},{"start":"17","length":"2"},{"start":"20","length":"1"},{"start":"23","length":"2"},{"start":"26","length":"1"},{"start":"29","length":"1"},{"start":"2f","length":"1"},{"start":"32","length":"1"}]},{"pgid":"2.2","version":"24'2","reported_seq":161,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.781140+0000","last_change":"2024-09-15T03:48:04.781084+0000","last_active":"2024-09-15T03:48:04.781140+0000","last_peered":"2024-09-15T03:48:04.781140+0000","last_clean":"2024-09-15T03:48:04.781140+0000","last_became_active":"2024-09-15T03:47:36.848076+0000","last_became_peered":"2024-09-15T03:47:36.848076+0000","last_unstale":"2024-09-15T03:48:04.781140+0000","last_undegraded":"2024-09-15T03:48:04.781140+0000","last_fullsized":"2024-09-15T03:48:04.781140+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'2","last_scrub_stamp":"2024-09-15T03:48:04.781025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:48:04.781025+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:13.832081+0000","scrub_duration":0.0016375750000000001,"objects_trimmed":0,"snaptrim_duration":0.000944677,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"113'196","reported_seq":461,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.094888+0000","last_change":"2024-09-15T03:47:59.976831+0000","last_active":"2024-09-15T03:48:05.094888+0000","last_peered":"2024-09-15T03:48:05.094888+0000","last_clean":"2024-09-15T03:48:05.094888+0000","last_became_active":"2024-09-15T03:47:55.855061+0000","last_became_peered":"2024-09-15T03:47:55.855061+0000","last_unstale":"2024-09-15T03:48:05.094888+0000","last_undegraded":"2024-09-15T03:48:05.094888+0000","last_fullsized":"2024-09-15T03:48:05.094888+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'156","last_scrub_stamp":"2024-09-15T03:47:56.178879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:47:56.178879+0000","objects_scrubbed":22,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:09.356437+0000","scrub_duration":0.014803648000000001,"objects_trimmed":0,"snaptrim_duration":0.031600082000000002,"stat_sum":{"num_bytes":20545536,"num_objects":25,"num_object_clones":19,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":27,"num_read_kb":352,"num_write":173,"num_write_kb":21440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,4,7,1,2,9,0],"acting":[10,8,4,7,1,2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.5","version":"113'79","reported_seq":384,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.094605+0000","last_change":"2024-09-15T03:48:04.063851+0000","last_active":"2024-09-15T03:48:05.094605+0000","last_peered":"2024-09-15T03:48:05.094605+0000","last_clean":"2024-09-15T03:48:05.094605+0000","last_became_active":"2024-09-15T03:47:55.860659+0000","last_became_peered":"2024-09-15T03:47:55.860659+0000","last_unstale":"2024-09-15T03:48:05.094605+0000","last_undegraded":"2024-09-15T03:48:05.094605+0000","last_fullsized":"2024-09-15T03:48:05.094605+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'79","last_scrub_stamp":"2024-09-15T03:48:04.063762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:04.063762+0000","objects_scrubbed":9,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:30.399619+0000","scrub_duration":0.0069124379999999999,"objects_trimmed":0,"snaptrim_duration":0.013330699,"stat_sum":{"num_bytes":24330240,"num_objects":9,"num_object_clones":8,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":14,"num_read_kb":11048,"num_write":34,"num_write_kb":13376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":23789568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4,9,8,7,3,0],"acting":[10,6,4,9,8,7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":138,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.840281+0000","last_change":"2024-09-15T03:47:39.930675+0000","last_active":"2024-09-15T03:48:04.840281+0000","last_peered":"2024-09-15T03:48:04.840281+0000","last_clean":"2024-09-15T03:48:04.840281+0000","last_became_active":"2024-09-15T03:47:33.998802+0000","last_became_peered":"2024-09-15T03:47:33.998802+0000","last_unstale":"2024-09-15T03:48:04.840281+0000","last_undegraded":"2024-09-15T03:48:04.840281+0000","last_fullsized":"2024-09-15T03:48:04.840281+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:39.930628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:39.930628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:03.535396+0000","scrub_duration":0.001146333,"objects_trimmed":0,"snaptrim_duration":0.00034123999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"113'42","reported_seq":399,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.191327+0000","last_change":"2024-09-15T03:48:03.723173+0000","last_active":"2024-09-15T03:48:05.191327+0000","last_peered":"2024-09-15T03:48:05.191327+0000","last_clean":"2024-09-15T03:48:05.191327+0000","last_became_active":"2024-09-15T03:47:55.846248+0000","last_became_peered":"2024-09-15T03:47:55.846248+0000","last_unstale":"2024-09-15T03:48:05.191327+0000","last_undegraded":"2024-09-15T03:48:05.191327+0000","last_fullsized":"2024-09-15T03:48:05.191327+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'42","last_scrub_stamp":"2024-09-15T03:48:03.723102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:03.723102+0000","objects_scrubbed":5,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:25.602306+0000","scrub_duration":0.0052425570000000001,"objects_trimmed":0,"snaptrim_duration":0.011769172,"stat_sum":{"num_bytes":7389184,"num_objects":5,"num_object_clones":4,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":6052,"num_write":11,"num_write_kb":7952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":9814016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5,8,0,4,7,10],"acting":[6,2,5,8,0,4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":117,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.060875+0000","last_change":"2024-09-15T03:47:30.025153+0000","last_active":"2024-09-15T03:48:04.060875+0000","last_peered":"2024-09-15T03:48:04.060875+0000","last_clean":"2024-09-15T03:48:04.060875+0000","last_became_active":"2024-09-15T03:46:24.038945+0000","last_became_peered":"2024-09-15T03:46:24.038945+0000","last_unstale":"2024-09-15T03:48:04.060875+0000","last_undegraded":"2024-09-15T03:48:04.060875+0000","last_fullsized":"2024-09-15T03:48:04.060875+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:30.025107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:30.025107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:36.231260+0000","scrub_duration":0.0013039429999999999,"objects_trimmed":0,"snaptrim_duration":0.00032787499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"113'153","reported_seq":520,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.258880+0000","last_change":"2024-09-15T03:48:05.258804+0000","last_active":"2024-09-15T03:48:05.258880+0000","last_peered":"2024-09-15T03:48:05.258880+0000","last_clean":"2024-09-15T03:48:05.258880+0000","last_became_active":"2024-09-15T03:47:55.848466+0000","last_became_peered":"2024-09-15T03:47:55.848466+0000","last_unstale":"2024-09-15T03:48:05.258880+0000","last_undegraded":"2024-09-15T03:48:05.258880+0000","last_fullsized":"2024-09-15T03:48:05.258880+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'153","last_scrub_stamp":"2024-09-15T03:48:05.258733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:05.258733+0000","objects_scrubbed":22,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:30.071424+0000","scrub_duration":0.012102717000000001,"objects_trimmed":0,"snaptrim_duration":0.036439371999999998,"stat_sum":{"num_bytes":19546112,"num_objects":22,"num_object_clones":17,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":1,"num_read":64,"num_read_kb":52263,"num_write":118,"num_write_kb":18976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":27639808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,7,2,1,11,10],"acting":[4,3,9,7,2,1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":118,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.060192+0000","last_change":"2024-09-15T03:47:27.297139+0000","last_active":"2024-09-15T03:48:04.060192+0000","last_peered":"2024-09-15T03:48:04.060192+0000","last_clean":"2024-09-15T03:48:04.060192+0000","last_became_active":"2024-09-15T03:46:24.063380+0000","last_became_peered":"2024-09-15T03:46:24.063380+0000","last_unstale":"2024-09-15T03:48:04.060192+0000","last_undegraded":"2024-09-15T03:48:04.060192+0000","last_fullsized":"2024-09-15T03:48:04.060192+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.297083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.297083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:48.875998+0000","scrub_duration":0.0018328789999999999,"objects_trimmed":0,"snaptrim_duration":0.00034490600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"113'47","reported_seq":487,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.896781+0000","last_change":"2024-09-15T03:47:59.953939+0000","last_active":"2024-09-15T03:48:04.896781+0000","last_peered":"2024-09-15T03:48:04.896781+0000","last_clean":"2024-09-15T03:48:04.896781+0000","last_became_active":"2024-09-15T03:47:55.847868+0000","last_became_peered":"2024-09-15T03:47:55.847868+0000","last_unstale":"2024-09-15T03:48:04.896781+0000","last_undegraded":"2024-09-15T03:48:04.896781+0000","last_fullsized":"2024-09-15T03:48:04.896781+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010037447,"stat_sum":{"num_bytes":4849664,"num_objects":7,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":13,"num_read_kb":515,"num_write":31,"num_write_kb":8384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":7176192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,9,8,3,11,1,10],"acting":[4,7,9,8,3,11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":115,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.059914+0000","last_change":"2024-09-15T03:47:42.675933+0000","last_active":"2024-09-15T03:48:04.059914+0000","last_peered":"2024-09-15T03:48:04.059914+0000","last_clean":"2024-09-15T03:48:04.059914+0000","last_became_active":"2024-09-15T03:46:24.063335+0000","last_became_peered":"2024-09-15T03:46:24.063335+0000","last_unstale":"2024-09-15T03:48:04.059914+0000","last_undegraded":"2024-09-15T03:48:04.059914+0000","last_fullsized":"2024-09-15T03:48:04.059914+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:42.675882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:42.675882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:02.697036+0000","scrub_duration":0.001445478,"objects_trimmed":0,"snaptrim_duration":0.00065928000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":183107584,"num_objects":189,"num_object_clones":139,"num_object_copies":1512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":189,"num_whiteouts":9,"num_read":457,"num_read_kb":239183,"num_write":904,"num_write_kb":171792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":265,"num_bytes_recovered":285786112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1494,"ondisk_log_size":1494,"up":128,"acting":128,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":11,"up_from":17,"seq":73014444056,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":846004,"kb_used_data":92833684,"kb_used_omap":4805,"kb_used_meta":0,"kb_avail":92833612,"statfs":{"total":95927926784,"available":95061618688,"internally_reserved":73728,"allocated":95061692416,"data_stored":95061692416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4920982,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.46600000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":121.575,"5min":121.575,"15min":121.575},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2060.797,"5min":2060.797,"15min":2060.797},"last":0.753},{"interface":"front","average":{"1min":121.56,"5min":121.56,"15min":121.56},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2060.7159999999999,"5min":2060.7159999999999,"15min":2060.7159999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.61499999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.79300000000000004}]},{"osd":4,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.67300000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.68400000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.70299999999999996}]},{"osd":7,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.63},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.74299999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.59999999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.84299999999999997}]},{"osd":10,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.82199999999999995}]}]},{"osd":10,"up_from":105,"seq":450971566084,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":871700,"kb_used_data":92807916,"kb_used_omap":3648,"kb_used_meta":0,"kb_avail":92807916,"statfs":{"total":95927926784,"available":95035305984,"internally_reserved":0,"allocated":95035305984,"data_stored":95035305984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3736470,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]}]},{"osd":9,"up_from":17,"seq":73014444056,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":855500,"kb_used_data":92824200,"kb_used_omap":5160,"kb_used_meta":0,"kb_avail":92824116,"statfs":{"total":95927926784,"available":95051894784,"internally_reserved":86016,"allocated":95051980800,"data_stored":95051980800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5284527,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.499},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.61499999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.51400000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.66100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.56299999999999994}]},{"osd":5,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.52500000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.46700000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.55000000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.625}]},{"osd":10,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.54100000000000004}]},{"osd":11,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.57399999999999995}]}]},{"osd":8,"up_from":17,"seq":73014444057,"num_pgs":15,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":861168,"kb_used_data":92818540,"kb_used_omap":5191,"kb_used_meta":0,"kb_avail":92818448,"statfs":{"total":95927926784,"available":95046090752,"internally_reserved":94208,"allocated":95046184960,"data_stored":95046184960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5316028,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.81699999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":42.381,"5min":42.381,"15min":42.381},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":963.98400000000004,"5min":963.98400000000004,"15min":963.98400000000004},"last":0.63100000000000001},{"interface":"front","average":{"1min":42.447000000000003,"5min":42.447000000000003,"15min":42.447000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":964.79600000000005,"5min":964.79600000000005,"15min":964.79600000000005},"last":0.56499999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.752}]},{"osd":4,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.83299999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:47:25 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.79200000000000004}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.73099999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.73999999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.437},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.60899999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.80400000000000005}]}]},{"osd":7,"up_from":17,"seq":73014444056,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":864892,"kb_used_data":92814832,"kb_used_omap":5646,"kb_used_meta":0,"kb_avail":92814724,"statfs":{"total":95927926784,"available":95042277376,"internally_reserved":110592,"allocated":95042387968,"data_stored":95042387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5781943,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.58199999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.41699999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.46700000000000003}]},{"osd":4,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.56000000000000005}]},{"osd":5,"last update":"Sun Sep 15 03:47:27 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.59999999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.71399999999999997}]},{"osd":8,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.626},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.66500000000000004}]},{"osd":9,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.75900000000000001}]},{"osd":10,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.746}]},{"osd":11,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.78300000000000003}]}]},{"osd":6,"up_from":17,"seq":73014444056,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":838972,"kb_used_data":92840728,"kb_used_omap":3843,"kb_used_meta":0,"kb_avail":92840644,"statfs":{"total":95927926784,"available":95068819456,"internally_reserved":86016,"allocated":95068905472,"data_stored":95068905472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3935639,"internal_metadata":0},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.52000000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:47:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.56299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.57399999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.55200000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.67500000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.66000000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.502}]},{"osd":8,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.78400000000000003}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.77700000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":11,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.83299999999999996}]}]},{"osd":5,"up_from":17,"seq":73014444056,"num_pgs":7,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":814900,"kb_used_data":92864772,"kb_used_omap":1826,"kb_used_meta":0,"kb_avail":92864716,"statfs":{"total":95927926784,"available":95093469184,"internally_reserved":57344,"allocated":95093526528,"data_stored":95093526528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1870318,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.29799999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":113.717,"5min":113.717,"15min":113.717},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2037.425,"5min":2037.425,"15min":2037.425},"last":0.34000000000000002},{"interface":"front","average":{"1min":113.752,"5min":113.752,"15min":113.752},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2037.3230000000001,"5min":2037.3230000000001,"15min":2037.3230000000001},"last":0.51400000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.379},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.53700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":4,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.439}]},{"osd":6,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.42499999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.497}]},{"osd":8,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.61799999999999999}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.63},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.65500000000000003}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.83399999999999996}]},{"osd":11,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.58899999999999997}]}]},{"osd":4,"up_from":17,"seq":73014444056,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":863984,"kb_used_data":92815728,"kb_used_omap":5620,"kb_used_meta":0,"kb_avail":92815632,"statfs":{"total":95927926784,"available":95043207168,"internally_reserved":98304,"allocated":95043305472,"data_stored":95043305472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5755199,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.47299999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":84.314999999999998,"5min":84.314999999999998,"15min":84.314999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1758.9780000000001,"5min":1758.9780000000001,"15min":1758.9780000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":84.364000000000004,"5min":84.364000000000004,"15min":84.364000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1758.671,"5min":1758.671,"15min":1758.671},"last":0.42699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.48999999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.33700000000000002}]},{"osd":5,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.375},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.52500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.55700000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.61299999999999999}]},{"osd":9,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":0.623}]},{"osd":10,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.53800000000000003}]},{"osd":11,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.58399999999999996}]}]},{"osd":3,"up_from":17,"seq":73014444056,"num_pgs":10,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":842612,"kb_used_data":92837084,"kb_used_omap":4588,"kb_used_meta":0,"kb_avail":92837004,"statfs":{"total":95927926784,"available":95065092096,"internally_reserved":81920,"allocated":95065174016,"data_stored":95065174016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4698419,"internal_metadata":0},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.27300000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:47:38 2024","interfaces":[{"interface":"back","average":{"1min":35.600000000000001,"5min":35.600000000000001,"15min":35.600000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":564.05100000000004,"5min":564.05100000000004,"15min":564.05100000000004},"last":0.309},{"interface":"front","average":{"1min":35.677,"5min":35.677,"15min":35.677},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":564.57000000000005,"5min":564.57000000000005,"15min":564.57000000000005},"last":0.40500000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.49399999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.61399999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":2.3940000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":5.2830000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.59699999999999998}]},{"osd":7,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.443},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.64400000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.89100000000000001}]},{"osd":9,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.84199999999999997}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.752}]}]},{"osd":2,"up_from":17,"seq":73014444056,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":855316,"kb_used_data":92824420,"kb_used_omap":5155,"kb_used_meta":0,"kb_avail":92824300,"statfs":{"total":95927926784,"available":95052083200,"internally_reserved":122880,"allocated":95052206080,"data_stored":95052206080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5279060,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:37 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.438}]},{"osd":1,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":34.457999999999998,"5min":34.457999999999998,"15min":34.457999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":647.92399999999998,"5min":647.92399999999998,"15min":647.92399999999998},"last":0.28100000000000003},{"interface":"front","average":{"1min":34.475000000000001,"5min":34.475000000000001,"15min":34.475000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":648.02700000000004,"5min":648.02700000000004,"15min":648.02700000000004},"last":0.34999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.46400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.54500000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.55400000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.503},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.56599999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.343,"15min":1.343},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":14.721,"5min":14.721,"15min":14.721},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.371,"5min":1.371,"15min":1.371},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":14.970000000000001,"5min":14.970000000000001,"15min":14.970000000000001},"last":0.77800000000000002}]},{"osd":9,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.377,"15min":1.377},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":14.888,"5min":14.888,"15min":14.888},"last":0.69799999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":14.654999999999999,"5min":14.654999999999999,"15min":14.654999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":1.409,"5min":1.409,"15min":1.409},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":15.096,"5min":15.096,"15min":15.096},"last":0.60499999999999998}]}]},{"osd":1,"up_from":17,"seq":73014444056,"num_pgs":13,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":862556,"kb_used_data":92817764,"kb_used_omap":5933,"kb_used_meta":0,"kb_avail":92817060,"statfs":{"total":95927926784,"available":95044669440,"internally_reserved":720896,"allocated":95045390336,"data_stored":95045390336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6076292,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.313}]},{"osd":2,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":0.437},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.55300000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.41099999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":2.6920000000000002,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":0.53600000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.7440000000000002,"5min":2.7440000000000002,"15min":2.7440000000000002},"last":0.58299999999999996}]},{"osd":6,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":3.278,"5min":3.278,"15min":3.278},"last":0.52000000000000002}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.81399999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":119.93899999999999,"5min":119.93899999999999,"15min":119.93899999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2861.7190000000001,"5min":2861.7190000000001,"15min":2861.7190000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.946,"5min":1.946,"15min":1.946},"last":0.60499999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":119.937,"5min":119.937,"15min":119.937},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":2861.6010000000001,"5min":2861.6010000000001,"15min":2861.6010000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.48399999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":119.96599999999999,"5min":119.96599999999999,"15min":119.96599999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2861.5639999999999,"5min":2861.5639999999999,"15min":2861.5639999999999},"last":0.627}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]}]},{"osd":0,"up_from":17,"seq":73014444056,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":844116,"kb_used_data":92835544,"kb_used_omap":4255,"kb_used_meta":0,"kb_avail":92835500,"statfs":{"total":95927926784,"available":95063552000,"internally_reserved":45056,"allocated":95063597056,"data_stored":95063597056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4358066,"internal_metadata":0},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.46999999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.804,"5min":1.804,"15min":1.804},"last":0.379}]},{"osd":3,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.52,"5min":1.52,"15min":1.52},"last":0.376},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.35399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":5,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.502}]},{"osd":6,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.48999999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.68700000000000006}]},{"osd":8,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.44400000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":10,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.58899999999999997}]},{"osd":11,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.56499999999999995}]}]}],"pool_statfs":[]}} 2024-09-15T03:48:07.339 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:48:07.384 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:07.384 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:07.392 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:48:07.391+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:48:07.393 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:48:07.490 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:48:07.493 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:48:07.492+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:48:07.496 INFO:teuthology.orchestra.run.smithi001.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:07.496 INFO:teuthology.orchestra.run.smithi001.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:07.506 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:48:07.593 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:48:07.593+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:48:07.626 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:48:07.626 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:48:07.637 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:48:07.638 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-15T03:48:07.641 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:48:07.641 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:48:07.649 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-09-15T03:48:06.981453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":183697971,"num_objects":193,"num_object_clones":139,"num_object_copies":1520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":9,"num_read":533,"num_read_kb":239247,"num_write":1029,"num_write_kb":173944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":267,"num_bytes_recovered":285786150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1589,"ondisk_log_size":1589,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":12,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":1124155392,"kb_used":10221720,"kb_used_data":1113935212,"kb_used_omap":55676,"kb_used_meta":0,"kb_avail":1113933672,"statfs":{"total":1151135121408,"available":1140668080128,"internally_reserved":1576960,"allocated":1140669657088,"data_stored":1140669657088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57012943,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.018648"},"pg_stats":[{"pgid":"3.f","version":"113'71","reported_seq":461,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.190992+0000","last_change":"2024-09-15T03:48:03.620987+0000","last_active":"2024-09-15T03:48:05.190992+0000","last_peered":"2024-09-15T03:48:05.190992+0000","last_clean":"2024-09-15T03:48:05.190992+0000","last_became_active":"2024-09-15T03:47:55.850051+0000","last_became_peered":"2024-09-15T03:47:55.850051+0000","last_unstale":"2024-09-15T03:48:05.190992+0000","last_undegraded":"2024-09-15T03:48:05.190992+0000","last_fullsized":"2024-09-15T03:48:05.190992+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'71","last_scrub_stamp":"2024-09-15T03:48:03.620910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:03.620910+0000","objects_scrubbed":8,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:13.277978+0000","scrub_duration":0.0050823509999999997,"objects_trimmed":0,"snaptrim_duration":0.0077598160000000001,"stat_sum":{"num_bytes":8028160,"num_objects":8,"num_object_clones":5,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":32,"num_read_kb":42506,"num_write":26,"num_write_kb":7840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0,7,2,8,4,10],"acting":[11,1,0,7,2,8,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.e","version":"113'210","reported_seq":442,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.974944+0000","last_change":"2024-09-15T03:47:59.982417+0000","last_active":"2024-09-15T03:48:04.974944+0000","last_peered":"2024-09-15T03:48:04.974944+0000","last_clean":"2024-09-15T03:48:04.974944+0000","last_became_active":"2024-09-15T03:46:36.170536+0000","last_became_peered":"2024-09-15T03:46:36.170536+0000","last_unstale":"2024-09-15T03:48:04.974944+0000","last_undegraded":"2024-09-15T03:48:04.974944+0000","last_fullsized":"2024-09-15T03:48:04.974944+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'173","last_scrub_stamp":"2024-09-15T03:47:54.108780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:47:54.108780+0000","objects_scrubbed":21,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:11.038305+0000","scrub_duration":0.0088561600000000001,"objects_trimmed":0,"snaptrim_duration":0.037652644999999998,"stat_sum":{"num_bytes":17612800,"num_objects":22,"num_object_clones":15,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":2,"num_read":68,"num_read_kb":34008,"num_write":51,"num_write_kb":15520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,11,7,8,5,9],"acting":[2,0,1,11,7,8,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.d","version":"113'75","reported_seq":399,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.191773+0000","last_change":"2024-09-15T03:47:59.958101+0000","last_active":"2024-09-15T03:48:05.191773+0000","last_peered":"2024-09-15T03:48:05.191773+0000","last_clean":"2024-09-15T03:48:05.191773+0000","last_became_active":"2024-09-15T03:47:55.850104+0000","last_became_peered":"2024-09-15T03:47:55.850104+0000","last_unstale":"2024-09-15T03:48:05.191773+0000","last_undegraded":"2024-09-15T03:48:05.191773+0000","last_fullsized":"2024-09-15T03:48:05.191773+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'62","last_scrub_stamp":"2024-09-15T03:47:56.721765+0000","last_deep_scrub":"109'62","last_deep_scrub_stamp":"2024-09-15T03:47:56.721765+0000","last_clean_scrub_stamp":"2024-09-15T03:47:56.721765+0000","objects_scrubbed":10,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:15.880649+0000","scrub_duration":0.00827518,"objects_trimmed":0,"snaptrim_duration":0.013785528999999999,"stat_sum":{"num_bytes":7651328,"num_objects":10,"num_object_clones":8,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":20,"num_read_kb":8406,"num_write":43,"num_write_kb":7472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":10780672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,1,3,10,7,9,4],"acting":[8,11,1,3,10,7,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.c","version":"113'28","reported_seq":383,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.840818+0000","last_change":"2024-09-15T03:47:59.955875+0000","last_active":"2024-09-15T03:48:04.840818+0000","last_peered":"2024-09-15T03:48:04.840818+0000","last_clean":"2024-09-15T03:48:04.840818+0000","last_became_active":"2024-09-15T03:47:36.848722+0000","last_became_peered":"2024-09-15T03:47:36.848722+0000","last_unstale":"2024-09-15T03:48:04.840818+0000","last_undegraded":"2024-09-15T03:48:04.840818+0000","last_fullsized":"2024-09-15T03:48:04.840818+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'16","last_scrub_stamp":"2024-09-15T03:47:41.493521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:47:41.493521+0000","objects_scrubbed":3,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:09.644586+0000","scrub_duration":0.0039888800000000002,"objects_trimmed":0,"snaptrim_duration":0.012831169999999999,"stat_sum":{"num_bytes":2965504,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":18,"num_read_kb":8502,"num_write":10,"num_write_kb":3440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,11,1,0,6,5,2],"acting":[9,7,11,1,0,6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.b","version":"113'212","reported_seq":766,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.843752+0000","last_change":"2024-09-15T03:48:03.010113+0000","last_active":"2024-09-15T03:48:04.843752+0000","last_peered":"2024-09-15T03:48:04.843752+0000","last_clean":"2024-09-15T03:48:04.843752+0000","last_became_active":"2024-09-15T03:47:55.850099+0000","last_became_peered":"2024-09-15T03:47:55.850099+0000","last_unstale":"2024-09-15T03:48:04.843752+0000","last_undegraded":"2024-09-15T03:48:04.843752+0000","last_fullsized":"2024-09-15T03:48:04.843752+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'212","last_scrub_stamp":"2024-09-15T03:48:03.010035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:03.010035+0000","objects_scrubbed":27,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:22.035341+0000","scrub_duration":0.022865639,"objects_trimmed":0,"snaptrim_duration":0.025351973,"stat_sum":{"num_bytes":24117248,"num_objects":27,"num_object_clones":20,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":2,"num_read":65,"num_read_kb":36889,"num_write":170,"num_write_kb":22832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":74006528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,8,4,1,10,11,3],"acting":[7,6,8,4,1,10,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.a","version":"113'84","reported_seq":240,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.896901+0000","last_change":"2024-09-15T03:48:03.286753+0000","last_active":"2024-09-15T03:48:04.896901+0000","last_peered":"2024-09-15T03:48:04.896901+0000","last_clean":"2024-09-15T03:48:04.896901+0000","last_became_active":"2024-09-15T03:47:55.854826+0000","last_became_peered":"2024-09-15T03:47:55.854826+0000","last_unstale":"2024-09-15T03:48:04.896901+0000","last_undegraded":"2024-09-15T03:48:04.896901+0000","last_fullsized":"2024-09-15T03:48:04.896901+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'84","last_scrub_stamp":"2024-09-15T03:48:03.286675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:03.286675+0000","objects_scrubbed":8,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:03.509620+0000","scrub_duration":0.0063142770000000001,"objects_trimmed":0,"snaptrim_duration":0.013016882,"stat_sum":{"num_bytes":6995968,"num_objects":8,"num_object_clones":6,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":16,"num_read_kb":6728,"num_write":30,"num_write_kb":6832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":30703616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8,9,5,6,2,0],"acting":[4,10,8,9,5,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.9","version":"113'80","reported_seq":492,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.979274+0000","last_change":"2024-09-15T03:47:59.971224+0000","last_active":"2024-09-15T03:48:04.979274+0000","last_peered":"2024-09-15T03:48:04.979274+0000","last_clean":"2024-09-15T03:48:04.979274+0000","last_became_active":"2024-09-15T03:47:55.851394+0000","last_became_peered":"2024-09-15T03:47:55.851394+0000","last_unstale":"2024-09-15T03:48:04.979274+0000","last_undegraded":"2024-09-15T03:48:04.979274+0000","last_fullsized":"2024-09-15T03:48:04.979274+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012050854,"stat_sum":{"num_bytes":12025856,"num_objects":13,"num_object_clones":10,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":15,"num_read_kb":8,"num_write":59,"num_write_kb":11744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":27590656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,10,1,9,6,8,11],"acting":[3,2,10,1,9,6,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.8","version":"113'52","reported_seq":377,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.979298+0000","last_change":"2024-09-15T03:47:59.962623+0000","last_active":"2024-09-15T03:48:04.979298+0000","last_peered":"2024-09-15T03:48:04.979298+0000","last_clean":"2024-09-15T03:48:04.979298+0000","last_became_active":"2024-09-15T03:47:55.848776+0000","last_became_peered":"2024-09-15T03:47:55.848776+0000","last_unstale":"2024-09-15T03:48:04.979298+0000","last_undegraded":"2024-09-15T03:48:04.979298+0000","last_fullsized":"2024-09-15T03:48:04.979298+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01649231,"stat_sum":{"num_bytes":4079616,"num_objects":6,"num_object_clones":4,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":23,"num_read_kb":4616,"num_write":54,"num_write_kb":3984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":11223040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,11,4,1,0,10],"acting":[3,7,2,11,4,1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"1.0","version":"20'92","reported_seq":226,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.059683+0000","last_change":"2024-09-15T03:47:39.774326+0000","last_active":"2024-09-15T03:48:04.059683+0000","last_peered":"2024-09-15T03:48:04.059683+0000","last_clean":"2024-09-15T03:48:04.059683+0000","last_became_active":"2024-09-15T03:46:18.011130+0000","last_became_peered":"2024-09-15T03:46:18.011130+0000","last_unstale":"2024-09-15T03:48:04.059683+0000","last_undegraded":"2024-09-15T03:48:04.059683+0000","last_fullsized":"2024-09-15T03:48:04.059683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-15T03:47:39.774262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_clean_scrub_stamp":"2024-09-15T03:47:39.774262+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:43.691013+0000","scrub_duration":0.0027413759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"22'1","reported_seq":118,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:03.962737+0000","last_change":"2024-09-15T03:47:52.292975+0000","last_active":"2024-09-15T03:48:03.962737+0000","last_peered":"2024-09-15T03:48:03.962737+0000","last_clean":"2024-09-15T03:48:03.962737+0000","last_became_active":"2024-09-15T03:46:24.038633+0000","last_became_peered":"2024-09-15T03:46:24.038633+0000","last_unstale":"2024-09-15T03:48:03.962737+0000","last_undegraded":"2024-09-15T03:48:03.962737+0000","last_fullsized":"2024-09-15T03:48:03.962737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'1","last_scrub_stamp":"2024-09-15T03:47:52.292926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:52.292926+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:56.362597+0000","scrub_duration":0.001409429,"objects_trimmed":0,"snaptrim_duration":0.00030178200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"113'80","reported_seq":513,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.896581+0000","last_change":"2024-09-15T03:47:59.966281+0000","last_active":"2024-09-15T03:48:04.896581+0000","last_peered":"2024-09-15T03:48:04.896581+0000","last_clean":"2024-09-15T03:48:04.896581+0000","last_became_active":"2024-09-15T03:47:55.854075+0000","last_became_peered":"2024-09-15T03:47:55.854075+0000","last_unstale":"2024-09-15T03:48:04.896581+0000","last_undegraded":"2024-09-15T03:48:04.896581+0000","last_fullsized":"2024-09-15T03:48:04.896581+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021707911999999999,"stat_sum":{"num_bytes":13320192,"num_objects":10,"num_object_clones":7,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":37,"num_read_kb":16719,"num_write":39,"num_write_kb":13088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":18628608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,7,8,6,11,1],"acting":[4,2,10,7,8,6,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":115,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.059319+0000","last_change":"2024-09-15T03:47:27.779257+0000","last_active":"2024-09-15T03:48:04.059319+0000","last_peered":"2024-09-15T03:48:04.059319+0000","last_clean":"2024-09-15T03:48:04.059319+0000","last_became_active":"2024-09-15T03:46:24.063106+0000","last_became_peered":"2024-09-15T03:46:24.063106+0000","last_unstale":"2024-09-15T03:48:04.059319+0000","last_undegraded":"2024-09-15T03:48:04.059319+0000","last_fullsized":"2024-09-15T03:48:04.059319+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.779210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.779210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:42.717969+0000","scrub_duration":0.0021030710000000002,"objects_trimmed":0,"snaptrim_duration":0.000393075,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"113'29","reported_seq":174,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.687310+0000","last_change":"2024-09-15T03:48:05.687193+0000","last_active":"2024-09-15T03:48:05.687310+0000","last_peered":"2024-09-15T03:48:05.687310+0000","last_clean":"2024-09-15T03:48:05.687310+0000","last_became_active":"2024-09-15T03:47:55.848033+0000","last_became_peered":"2024-09-15T03:47:55.848033+0000","last_unstale":"2024-09-15T03:48:05.687310+0000","last_undegraded":"2024-09-15T03:48:05.687310+0000","last_fullsized":"2024-09-15T03:48:05.687310+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'29","last_scrub_stamp":"2024-09-15T03:48:05.687085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:05.687085+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:32.245260+0000","scrub_duration":0.0044105530000000002,"objects_trimmed":0,"snaptrim_duration":0.0090284240000000002,"stat_sum":{"num_bytes":4653056,"num_objects":5,"num_object_clones":4,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":3075,"num_write":12,"num_write_kb":4032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":6045696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3,10,5,2,9,7],"acting":[6,11,3,10,5,2,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":120,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.843432+0000","last_change":"2024-09-15T03:47:42.997012+0000","last_active":"2024-09-15T03:48:04.843432+0000","last_peered":"2024-09-15T03:48:04.843432+0000","last_clean":"2024-09-15T03:48:04.843432+0000","last_became_active":"2024-09-15T03:46:24.037471+0000","last_became_peered":"2024-09-15T03:46:24.037471+0000","last_unstale":"2024-09-15T03:48:04.843432+0000","last_undegraded":"2024-09-15T03:48:04.843432+0000","last_fullsized":"2024-09-15T03:48:04.843432+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:42.996957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:42.996957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:53.443125+0000","scrub_duration":0.0014164309999999999,"objects_trimmed":0,"snaptrim_duration":0.00026490299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"113'56","reported_seq":456,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-15T03:48:03.778938+0000","last_change":"2024-09-15T03:48:03.778794+0000","last_active":"2024-09-15T03:48:03.778938+0000","last_peered":"2024-09-15T03:48:03.778938+0000","last_clean":"2024-09-15T03:48:03.778938+0000","last_became_active":"2024-09-15T03:47:55.848512+0000","last_became_peered":"2024-09-15T03:47:55.848512+0000","last_unstale":"2024-09-15T03:48:03.778938+0000","last_undegraded":"2024-09-15T03:48:03.778938+0000","last_fullsized":"2024-09-15T03:48:03.778938+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'56","last_scrub_stamp":"2024-09-15T03:48:03.778652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:03.778652+0000","objects_scrubbed":8,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:05.162717+0000","scrub_duration":0.0067734900000000001,"objects_trimmed":0,"snaptrim_duration":0.012362052,"stat_sum":{"num_bytes":4997120,"num_objects":8,"num_object_clones":5,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":20,"num_read_kb":7496,"num_write":43,"num_write_kb":4880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":16449536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,9,4,3,6,0,10],"acting":[1,2,9,4,3,6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"c","length":"1"},{"start":"12","length":"1"},{"start":"17","length":"2"},{"start":"20","length":"1"},{"start":"23","length":"2"},{"start":"26","length":"1"},{"start":"29","length":"1"},{"start":"2f","length":"1"},{"start":"32","length":"1"}]},{"pgid":"2.2","version":"24'2","reported_seq":161,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.781140+0000","last_change":"2024-09-15T03:48:04.781084+0000","last_active":"2024-09-15T03:48:04.781140+0000","last_peered":"2024-09-15T03:48:04.781140+0000","last_clean":"2024-09-15T03:48:04.781140+0000","last_became_active":"2024-09-15T03:47:36.848076+0000","last_became_peered":"2024-09-15T03:47:36.848076+0000","last_unstale":"2024-09-15T03:48:04.781140+0000","last_undegraded":"2024-09-15T03:48:04.781140+0000","last_fullsized":"2024-09-15T03:48:04.781140+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'2","last_scrub_stamp":"2024-09-15T03:48:04.781025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:48:04.781025+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:13.832081+0000","scrub_duration":0.0016375750000000001,"objects_trimmed":0,"snaptrim_duration":0.000944677,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"113'196","reported_seq":461,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.094888+0000","last_change":"2024-09-15T03:47:59.976831+0000","last_active":"2024-09-15T03:48:05.094888+0000","last_peered":"2024-09-15T03:48:05.094888+0000","last_clean":"2024-09-15T03:48:05.094888+0000","last_became_active":"2024-09-15T03:47:55.855061+0000","last_became_peered":"2024-09-15T03:47:55.855061+0000","last_unstale":"2024-09-15T03:48:05.094888+0000","last_undegraded":"2024-09-15T03:48:05.094888+0000","last_fullsized":"2024-09-15T03:48:05.094888+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'156","last_scrub_stamp":"2024-09-15T03:47:56.178879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:47:56.178879+0000","objects_scrubbed":22,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:09.356437+0000","scrub_duration":0.014803648000000001,"objects_trimmed":0,"snaptrim_duration":0.031600082000000002,"stat_sum":{"num_bytes":20545536,"num_objects":25,"num_object_clones":19,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":27,"num_read_kb":352,"num_write":173,"num_write_kb":21440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,4,7,1,2,9,0],"acting":[10,8,4,7,1,2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.5","version":"113'79","reported_seq":384,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.094605+0000","last_change":"2024-09-15T03:48:04.063851+0000","last_active":"2024-09-15T03:48:05.094605+0000","last_peered":"2024-09-15T03:48:05.094605+0000","last_clean":"2024-09-15T03:48:05.094605+0000","last_became_active":"2024-09-15T03:47:55.860659+0000","last_became_peered":"2024-09-15T03:47:55.860659+0000","last_unstale":"2024-09-15T03:48:05.094605+0000","last_undegraded":"2024-09-15T03:48:05.094605+0000","last_fullsized":"2024-09-15T03:48:05.094605+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'79","last_scrub_stamp":"2024-09-15T03:48:04.063762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:04.063762+0000","objects_scrubbed":9,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:30.399619+0000","scrub_duration":0.0069124379999999999,"objects_trimmed":0,"snaptrim_duration":0.013330699,"stat_sum":{"num_bytes":24330240,"num_objects":9,"num_object_clones":8,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":14,"num_read_kb":11048,"num_write":34,"num_write_kb":13376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":23789568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4,9,8,7,3,0],"acting":[10,6,4,9,8,7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":138,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.840281+0000","last_change":"2024-09-15T03:47:39.930675+0000","last_active":"2024-09-15T03:48:04.840281+0000","last_peered":"2024-09-15T03:48:04.840281+0000","last_clean":"2024-09-15T03:48:04.840281+0000","last_became_active":"2024-09-15T03:47:33.998802+0000","last_became_peered":"2024-09-15T03:47:33.998802+0000","last_unstale":"2024-09-15T03:48:04.840281+0000","last_undegraded":"2024-09-15T03:48:04.840281+0000","last_fullsized":"2024-09-15T03:48:04.840281+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:39.930628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:39.930628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:03.535396+0000","scrub_duration":0.001146333,"objects_trimmed":0,"snaptrim_duration":0.00034123999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"113'42","reported_seq":399,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.191327+0000","last_change":"2024-09-15T03:48:03.723173+0000","last_active":"2024-09-15T03:48:05.191327+0000","last_peered":"2024-09-15T03:48:05.191327+0000","last_clean":"2024-09-15T03:48:05.191327+0000","last_became_active":"2024-09-15T03:47:55.846248+0000","last_became_peered":"2024-09-15T03:47:55.846248+0000","last_unstale":"2024-09-15T03:48:05.191327+0000","last_undegraded":"2024-09-15T03:48:05.191327+0000","last_fullsized":"2024-09-15T03:48:05.191327+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'42","last_scrub_stamp":"2024-09-15T03:48:03.723102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:03.723102+0000","objects_scrubbed":5,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:25.602306+0000","scrub_duration":0.0052425570000000001,"objects_trimmed":0,"snaptrim_duration":0.011769172,"stat_sum":{"num_bytes":7389184,"num_objects":5,"num_object_clones":4,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":6052,"num_write":11,"num_write_kb":7952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":9814016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5,8,0,4,7,10],"acting":[6,2,5,8,0,4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":117,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.060875+0000","last_change":"2024-09-15T03:47:30.025153+0000","last_active":"2024-09-15T03:48:04.060875+0000","last_peered":"2024-09-15T03:48:04.060875+0000","last_clean":"2024-09-15T03:48:04.060875+0000","last_became_active":"2024-09-15T03:46:24.038945+0000","last_became_peered":"2024-09-15T03:46:24.038945+0000","last_unstale":"2024-09-15T03:48:04.060875+0000","last_undegraded":"2024-09-15T03:48:04.060875+0000","last_fullsized":"2024-09-15T03:48:04.060875+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:30.025107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:30.025107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:36.231260+0000","scrub_duration":0.0013039429999999999,"objects_trimmed":0,"snaptrim_duration":0.00032787499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"113'153","reported_seq":520,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.258880+0000","last_change":"2024-09-15T03:48:05.258804+0000","last_active":"2024-09-15T03:48:05.258880+0000","last_peered":"2024-09-15T03:48:05.258880+0000","last_clean":"2024-09-15T03:48:05.258880+0000","last_became_active":"2024-09-15T03:47:55.848466+0000","last_became_peered":"2024-09-15T03:47:55.848466+0000","last_unstale":"2024-09-15T03:48:05.258880+0000","last_undegraded":"2024-09-15T03:48:05.258880+0000","last_fullsized":"2024-09-15T03:48:05.258880+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'153","last_scrub_stamp":"2024-09-15T03:48:05.258733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:05.258733+0000","objects_scrubbed":22,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:30.071424+0000","scrub_duration":0.012102717000000001,"objects_trimmed":0,"snaptrim_duration":0.036439371999999998,"stat_sum":{"num_bytes":19546112,"num_objects":22,"num_object_clones":17,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":1,"num_read":64,"num_read_kb":52263,"num_write":118,"num_write_kb":18976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":27639808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,7,2,1,11,10],"acting":[4,3,9,7,2,1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":118,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.060192+0000","last_change":"2024-09-15T03:47:27.297139+0000","last_active":"2024-09-15T03:48:04.060192+0000","last_peered":"2024-09-15T03:48:04.060192+0000","last_clean":"2024-09-15T03:48:04.060192+0000","last_became_active":"2024-09-15T03:46:24.063380+0000","last_became_peered":"2024-09-15T03:46:24.063380+0000","last_unstale":"2024-09-15T03:48:04.060192+0000","last_undegraded":"2024-09-15T03:48:04.060192+0000","last_fullsized":"2024-09-15T03:48:04.060192+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.297083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.297083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:48.875998+0000","scrub_duration":0.0018328789999999999,"objects_trimmed":0,"snaptrim_duration":0.00034490600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"113'47","reported_seq":487,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.896781+0000","last_change":"2024-09-15T03:47:59.953939+0000","last_active":"2024-09-15T03:48:04.896781+0000","last_peered":"2024-09-15T03:48:04.896781+0000","last_clean":"2024-09-15T03:48:04.896781+0000","last_became_active":"2024-09-15T03:47:55.847868+0000","last_became_peered":"2024-09-15T03:47:55.847868+0000","last_unstale":"2024-09-15T03:48:04.896781+0000","last_undegraded":"2024-09-15T03:48:04.896781+0000","last_fullsized":"2024-09-15T03:48:04.896781+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010037447,"stat_sum":{"num_bytes":4849664,"num_objects":7,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":13,"num_read_kb":515,"num_write":31,"num_write_kb":8384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":7176192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,9,8,3,11,1,10],"acting":[4,7,9,8,3,11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":115,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.059914+0000","last_change":"2024-09-15T03:47:42.675933+0000","last_active":"2024-09-15T03:48:04.059914+0000","last_peered":"2024-09-15T03:48:04.059914+0000","last_clean":"2024-09-15T03:48:04.059914+0000","last_became_active":"2024-09-15T03:46:24.063335+0000","last_became_peered":"2024-09-15T03:46:24.063335+0000","last_unstale":"2024-09-15T03:48:04.059914+0000","last_undegraded":"2024-09-15T03:48:04.059914+0000","last_fullsized":"2024-09-15T03:48:04.059914+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:42.675882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:42.675882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:02.697036+0000","scrub_duration":0.001445478,"objects_trimmed":0,"snaptrim_duration":0.00065928000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":183107584,"num_objects":189,"num_object_clones":139,"num_object_copies":1512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":189,"num_whiteouts":9,"num_read":457,"num_read_kb":239183,"num_write":904,"num_write_kb":171792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":265,"num_bytes_recovered":285786112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1494,"ondisk_log_size":1494,"up":128,"acting":128,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":11,"up_from":17,"seq":73014444056,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":846004,"kb_used_data":92833684,"kb_used_omap":4805,"kb_used_meta":0,"kb_avail":92833612,"statfs":{"total":95927926784,"available":95061618688,"internally_reserved":73728,"allocated":95061692416,"data_stored":95061692416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4920982,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.46600000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":121.575,"5min":121.575,"15min":121.575},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2060.797,"5min":2060.797,"15min":2060.797},"last":0.753},{"interface":"front","average":{"1min":121.56,"5min":121.56,"15min":121.56},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2060.7159999999999,"5min":2060.7159999999999,"15min":2060.7159999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.61499999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.79300000000000004}]},{"osd":4,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.67300000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.68400000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.70299999999999996}]},{"osd":7,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.63},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.74299999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.59999999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.84299999999999997}]},{"osd":10,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.82199999999999995}]}]},{"osd":10,"up_from":105,"seq":450971566084,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":871700,"kb_used_data":92807916,"kb_used_omap":3648,"kb_used_meta":0,"kb_avail":92807916,"statfs":{"total":95927926784,"available":95035305984,"internally_reserved":0,"allocated":95035305984,"data_stored":95035305984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3736470,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]}]},{"osd":9,"up_from":17,"seq":73014444056,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":855500,"kb_used_data":92824200,"kb_used_omap":5160,"kb_used_meta":0,"kb_avail":92824116,"statfs":{"total":95927926784,"available":95051894784,"internally_reserved":86016,"allocated":95051980800,"data_stored":95051980800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5284527,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.499},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.61499999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.51400000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.66100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.56299999999999994}]},{"osd":5,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.52500000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.46700000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.55000000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.625}]},{"osd":10,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.54100000000000004}]},{"osd":11,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.57399999999999995}]}]},{"osd":8,"up_from":17,"seq":73014444057,"num_pgs":15,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":861168,"kb_used_data":92818540,"kb_used_omap":5191,"kb_used_meta":0,"kb_avail":92818448,"statfs":{"total":95927926784,"available":95046090752,"internally_reserved":94208,"allocated":95046184960,"data_stored":95046184960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5316028,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.81699999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":42.381,"5min":42.381,"15min":42.381},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":963.98400000000004,"5min":963.98400000000004,"15min":963.98400000000004},"last":0.63100000000000001},{"interface":"front","average":{"1min":42.447000000000003,"5min":42.447000000000003,"15min":42.447000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":964.79600000000005,"5min":964.79600000000005,"15min":964.79600000000005},"last":0.56499999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.752}]},{"osd":4,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.83299999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:47:25 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.79200000000000004}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.73099999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.73999999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.437},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.60899999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.80400000000000005}]}]},{"osd":7,"up_from":17,"seq":73014444056,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":864892,"kb_used_data":92814832,"kb_used_omap":5646,"kb_used_meta":0,"kb_avail":92814724,"statfs":{"total":95927926784,"available":95042277376,"internally_reserved":110592,"allocated":95042387968,"data_stored":95042387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5781943,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.58199999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.41699999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.46700000000000003}]},{"osd":4,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.56000000000000005}]},{"osd":5,"last update":"Sun Sep 15 03:47:27 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.59999999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.71399999999999997}]},{"osd":8,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.626},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.66500000000000004}]},{"osd":9,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.75900000000000001}]},{"osd":10,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.746}]},{"osd":11,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.78300000000000003}]}]},{"osd":6,"up_from":17,"seq":73014444056,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":838972,"kb_used_data":92840728,"kb_used_omap":3843,"kb_used_meta":0,"kb_avail":92840644,"statfs":{"total":95927926784,"available":95068819456,"internally_reserved":86016,"allocated":95068905472,"data_stored":95068905472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3935639,"internal_metadata":0},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.52000000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:47:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.56299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.57399999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.55200000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.67500000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.66000000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.502}]},{"osd":8,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.78400000000000003}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.77700000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":11,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.83299999999999996}]}]},{"osd":5,"up_from":17,"seq":73014444056,"num_pgs":7,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":814900,"kb_used_data":92864772,"kb_used_omap":1826,"kb_used_meta":0,"kb_avail":92864716,"statfs":{"total":95927926784,"available":95093469184,"internally_reserved":57344,"allocated":95093526528,"data_stored":95093526528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1870318,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.29799999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":113.717,"5min":113.717,"15min":113.717},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2037.425,"5min":2037.425,"15min":2037.425},"last":0.34000000000000002},{"interface":"front","average":{"1min":113.752,"5min":113.752,"15min":113.752},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2037.3230000000001,"5min":2037.3230000000001,"15min":2037.3230000000001},"last":0.51400000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.379},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.53700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":4,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.439}]},{"osd":6,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.42499999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.497}]},{"osd":8,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.61799999999999999}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.63},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.65500000000000003}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.83399999999999996}]},{"osd":11,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.58899999999999997}]}]},{"osd":4,"up_from":17,"seq":73014444056,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":863984,"kb_used_data":92815728,"kb_used_omap":5620,"kb_used_meta":0,"kb_avail":92815632,"statfs":{"total":95927926784,"available":95043207168,"internally_reserved":98304,"allocated":95043305472,"data_stored":95043305472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5755199,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.47299999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":84.314999999999998,"5min":84.314999999999998,"15min":84.314999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1758.9780000000001,"5min":1758.9780000000001,"15min":1758.9780000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":84.364000000000004,"5min":84.364000000000004,"15min":84.364000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1758.671,"5min":1758.671,"15min":1758.671},"last":0.42699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.48999999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.33700000000000002}]},{"osd":5,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.375},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.52500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.55700000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.61299999999999999}]},{"osd":9,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":0.623}]},{"osd":10,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.53800000000000003}]},{"osd":11,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.58399999999999996}]}]},{"osd":3,"up_from":17,"seq":73014444056,"num_pgs":10,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":842612,"kb_used_data":92837084,"kb_used_omap":4588,"kb_used_meta":0,"kb_avail":92837004,"statfs":{"total":95927926784,"available":95065092096,"internally_reserved":81920,"allocated":95065174016,"data_stored":95065174016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4698419,"internal_metadata":0},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.27300000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:47:38 2024","interfaces":[{"interface":"back","average":{"1min":35.600000000000001,"5min":35.600000000000001,"15min":35.600000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":564.05100000000004,"5min":564.05100000000004,"15min":564.05100000000004},"last":0.309},{"interface":"front","average":{"1min":35.677,"5min":35.677,"15min":35.677},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":564.57000000000005,"5min":564.57000000000005,"15min":564.57000000000005},"last":0.40500000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.49399999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.61399999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":2.3940000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":5.2830000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.59699999999999998}]},{"osd":7,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.443},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.64400000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.89100000000000001}]},{"osd":9,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.84199999999999997}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.752}]}]},{"osd":2,"up_from":17,"seq":73014444056,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":855316,"kb_used_data":92824420,"kb_used_omap":5155,"kb_used_meta":0,"kb_avail":92824300,"statfs":{"total":95927926784,"available":95052083200,"internally_reserved":122880,"allocated":95052206080,"data_stored":95052206080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5279060,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:37 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.438}]},{"osd":1,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":34.457999999999998,"5min":34.457999999999998,"15min":34.457999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":647.92399999999998,"5min":647.92399999999998,"15min":647.92399999999998},"last":0.28100000000000003},{"interface":"front","average":{"1min":34.475000000000001,"5min":34.475000000000001,"15min":34.475000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":648.02700000000004,"5min":648.02700000000004,"15min":648.02700000000004},"last":0.34999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.46400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.54500000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.55400000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.503},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.56599999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.343,"15min":1.343},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":14.721,"5min":14.721,"15min":14.721},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.371,"5min":1.371,"15min":1.371},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":14.970000000000001,"5min":14.970000000000001,"15min":14.970000000000001},"last":0.77800000000000002}]},{"osd":9,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.377,"15min":1.377},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":14.888,"5min":14.888,"15min":14.888},"last":0.69799999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":14.654999999999999,"5min":14.654999999999999,"15min":14.654999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":1.409,"5min":1.409,"15min":1.409},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":15.096,"5min":15.096,"15min":15.096},"last":0.60499999999999998}]}]},{"osd":1,"up_from":17,"seq":73014444056,"num_pgs":13,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":862556,"kb_used_data":92817764,"kb_used_omap":5933,"kb_used_meta":0,"kb_avail":92817060,"statfs":{"total":95927926784,"available":95044669440,"internally_reserved":720896,"allocated":95045390336,"data_stored":95045390336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6076292,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.313}]},{"osd":2,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":0.437},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.55300000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.41099999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":2.6920000000000002,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":0.53600000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.7440000000000002,"5min":2.7440000000000002,"15min":2.7440000000000002},"last":0.58299999999999996}]},{"osd":6,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":3.278,"5min":3.278,"15min":3.278},"last":0.52000000000000002}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.81399999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":119.93899999999999,"5min":119.93899999999999,"15min":119.93899999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2861.7190000000001,"5min":2861.7190000000001,"15min":2861.7190000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.946,"5min":1.946,"15min":1.946},"last":0.60499999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":119.937,"5min":119.937,"15min":119.937},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":2861.6010000000001,"5min":2861.6010000000001,"15min":2861.6010000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.48399999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":119.96599999999999,"5min":119.96599999999999,"15min":119.96599999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2861.5639999999999,"5min":2861.5639999999999,"15min":2861.5639999999999},"last":0.627}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]}]},{"osd":0,"up_from":17,"seq":73014444056,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":844116,"kb_used_data":92835544,"kb_used_omap":4255,"kb_used_meta":0,"kb_avail":92835500,"statfs":{"total":95927926784,"available":95063552000,"internally_reserved":45056,"allocated":95063597056,"data_stored":95063597056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4358066,"internal_metadata":0},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.46999999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.804,"5min":1.804,"15min":1.804},"last":0.379}]},{"osd":3,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.52,"5min":1.52,"15min":1.52},"last":0.376},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.35399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":5,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.502}]},{"osd":6,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.48999999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.68700000000000006}]},{"osd":8,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.44400000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":10,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.58899999999999997}]},{"osd":11,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.56499999999999995}]}]}],"pool_statfs":[]}} 2024-09-15T03:48:07.653 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-15T03:48:07.653 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T03:48:07.654 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-09-15T03:48:07.654 INFO:tasks.ceph.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 12 2024-09-15T03:48:07.654 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:48:07.655 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:07.655 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:48:07.655 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-15T03:48:07.655 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-15T03:48:07.655 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:07.657 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:48:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:48:07.661 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:48:07.673 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:48:07.674 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:48:07.690 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:48:07.691 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:48:07.695 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:48:07.694+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:48:07.709 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:48:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:48:07.724 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:48:07.725 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:48:07.739 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:48:07.740 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:48:07.754 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:48:07.755 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:48:07.771 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:48:07.771 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:48:07.784 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:07.784 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-15T03:48:07.784 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-15T03:48:07.784 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-15T03:48:07.784 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.784 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs2 60/31 ECSubWrite(tid=70, reqid=client.4521.0:421, at_version=60'22, trim_to=0'0, roll_forward_to=47'20))", 2024-09-15T03:48:07.785 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.837529+0000", 2024-09-15T03:48:07.785 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 60.944070222999997, 2024-09-15T03:48:07.785 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0088963150000000001, 2024-09-15T03:48:07.785 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:07.785 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:07.785 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:07.785 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.785 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:07.785 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837529+0000", 2024-09-15T03:48:07.785 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.785 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.785 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.785 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:07.786 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837529+0000", 2024-09-15T03:48:07.786 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.786 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.786 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.786 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:07.786 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837529+0000", 2024-09-15T03:48:07.786 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8199999999999999e-07 2024-09-15T03:48:07.786 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.786 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.786 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:07.786 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837562+0000", 2024-09-15T03:48:07.786 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3080000000000002e-05 2024-09-15T03:48:07.786 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.787 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.787 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:07.787 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837572+0000", 2024-09-15T03:48:07.787 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7219999999999994e-06 2024-09-15T03:48:07.787 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.787 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.787 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:07.787 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837580+0000", 2024-09-15T03:48:07.787 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4470000000000006e-06 2024-09-15T03:48:07.787 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.787 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.787 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:07.787 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837727+0000", 2024-09-15T03:48:07.787 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014676199999999999 2024-09-15T03:48:07.788 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.788 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.788 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:07.788 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837738+0000", 2024-09-15T03:48:07.788 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0946e-05 2024-09-15T03:48:07.788 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.788 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.788 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:07.788 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.838091+0000", 2024-09-15T03:48:07.788 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00035338600000000002 2024-09-15T03:48:07.788 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.788 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.789 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:07.789 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.839733+0000", 2024-09-15T03:48:07.789 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0016420409999999999 2024-09-15T03:48:07.789 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.789 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.789 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:07.789 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.839815+0000", 2024-09-15T03:48:07.789 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1433000000000005e-05 2024-09-15T03:48:07.789 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.790 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.790 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:07.790 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.839841+0000", 2024-09-15T03:48:07.790 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6529e-05 2024-09-15T03:48:07.790 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.790 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.790 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:07.790 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.846425+0000", 2024-09-15T03:48:07.790 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0065837869999999998 2024-09-15T03:48:07.791 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.791 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:07.791 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.791 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.791 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.792 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.ds4 60/31 ECSubWrite(tid=122, reqid=osd.8.0:122, at_version=60'16, trim_to=0'0, roll_forward_to=59'14))", 2024-09-15T03:48:07.792 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.840427+0000", 2024-09-15T03:48:07.792 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 60.941171722999997, 2024-09-15T03:48:07.792 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0064843990000000001, 2024-09-15T03:48:07.792 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:07.792 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:07.792 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:07.792 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.792 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:07.792 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840427+0000", 2024-09-15T03:48:07.793 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.793 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.793 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.793 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:07.793 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840427+0000", 2024-09-15T03:48:07.793 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.793 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.793 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.793 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:07.793 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840428+0000", 2024-09-15T03:48:07.793 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1099999999999996e-07 2024-09-15T03:48:07.793 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.793 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.793 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:07.794 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840460+0000", 2024-09-15T03:48:07.794 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1869999999999998e-05 2024-09-15T03:48:07.794 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.794 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.794 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:07.794 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840487+0000", 2024-09-15T03:48:07.794 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7778999999999999e-05 2024-09-15T03:48:07.794 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.794 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.794 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:07.794 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840496+0000", 2024-09-15T03:48:07.794 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9660000000000002e-06 2024-09-15T03:48:07.794 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.795 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.795 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:07.795 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840565+0000", 2024-09-15T03:48:07.795 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8464e-05 2024-09-15T03:48:07.795 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.795 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.795 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:07.795 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840577+0000", 2024-09-15T03:48:07.795 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1789e-05 2024-09-15T03:48:07.795 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.795 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.795 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:07.795 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840853+0000", 2024-09-15T03:48:07.795 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00027612699999999998 2024-09-15T03:48:07.796 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.796 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.796 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:07.796 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840900+0000", 2024-09-15T03:48:07.796 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6801000000000001e-05 2024-09-15T03:48:07.796 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.796 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.796 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:07.796 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840948+0000", 2024-09-15T03:48:07.796 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8406999999999999e-05 2024-09-15T03:48:07.796 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.796 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.796 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:07.796 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.840972+0000", 2024-09-15T03:48:07.797 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4377999999999999e-05 2024-09-15T03:48:07.797 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.797 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.797 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:07.797 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.846912+0000", 2024-09-15T03:48:07.797 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0059393069999999996 2024-09-15T03:48:07.797 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.797 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:07.797 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.797 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.797 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.797 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs2 60/31 ECSubWrite(tid=71, reqid=client.4521.0:421, at_version=0'0, trim_to=0'0, roll_forward_to=60'22))", 2024-09-15T03:48:07.797 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.843064+0000", 2024-09-15T03:48:07.798 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 60.938534615000002, 2024-09-15T03:48:07.798 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0066467000000000002, 2024-09-15T03:48:07.798 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:07.798 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:07.798 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:07.798 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.798 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:07.798 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843064+0000", 2024-09-15T03:48:07.798 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.798 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.798 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.798 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:07.798 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843064+0000", 2024-09-15T03:48:07.798 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.799 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.799 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:07.799 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843065+0000", 2024-09-15T03:48:07.799 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5600000000000004e-07 2024-09-15T03:48:07.799 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.799 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:07.799 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843077+0000", 2024-09-15T03:48:07.799 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1336e-05 2024-09-15T03:48:07.799 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.799 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:07.799 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843089+0000", 2024-09-15T03:48:07.800 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2869e-05 2024-09-15T03:48:07.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.800 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:07.800 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843097+0000", 2024-09-15T03:48:07.800 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.503e-06 2024-09-15T03:48:07.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.800 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:07.800 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843157+0000", 2024-09-15T03:48:07.800 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9580999999999998e-05 2024-09-15T03:48:07.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.801 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:07.801 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843167+0000", 2024-09-15T03:48:07.801 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0406e-05 2024-09-15T03:48:07.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.801 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:07.801 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843279+0000", 2024-09-15T03:48:07.801 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112427 2024-09-15T03:48:07.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.801 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:07.801 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843339+0000", 2024-09-15T03:48:07.801 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0031000000000002e-05 2024-09-15T03:48:07.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.802 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:07.802 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843380+0000", 2024-09-15T03:48:07.802 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0683000000000003e-05 2024-09-15T03:48:07.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.802 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:07.802 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.843393+0000", 2024-09-15T03:48:07.802 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3334e-05 2024-09-15T03:48:07.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.802 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:07.802 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.849711+0000", 2024-09-15T03:48:07.803 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0063176739999999997 2024-09-15T03:48:07.803 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.803 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:07.803 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.803 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.8s6 60/31 ECSubWrite(tid=47, reqid=osd.3.0:47, at_version=60'10, trim_to=0'0, roll_forward_to=58'8))", 2024-09-15T03:48:07.803 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.844673+0000", 2024-09-15T03:48:07.803 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 60.936925784000003, 2024-09-15T03:48:07.803 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0078207390000000002, 2024-09-15T03:48:07.803 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:07.803 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:07.803 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:07.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.804 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:07.804 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844673+0000", 2024-09-15T03:48:07.804 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.804 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:07.804 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844673+0000", 2024-09-15T03:48:07.804 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.804 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:07.804 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844674+0000", 2024-09-15T03:48:07.805 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4299999999999999e-07 2024-09-15T03:48:07.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.805 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:07.805 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844699+0000", 2024-09-15T03:48:07.805 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5003000000000001e-05 2024-09-15T03:48:07.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.805 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:07.805 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844708+0000", 2024-09-15T03:48:07.805 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8109999999999997e-06 2024-09-15T03:48:07.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.805 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:07.806 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.844715+0000", 2024-09-15T03:48:07.806 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.127e-06 2024-09-15T03:48:07.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.806 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:07.806 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.848363+0000", 2024-09-15T03:48:07.806 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0036482390000000002 2024-09-15T03:48:07.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.806 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:07.806 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.848374+0000", 2024-09-15T03:48:07.806 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1137e-05 2024-09-15T03:48:07.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.807 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:07.807 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.848615+0000", 2024-09-15T03:48:07.807 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00024119899999999999 2024-09-15T03:48:07.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.807 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:07.807 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.851231+0000", 2024-09-15T03:48:07.807 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0026161079999999998 2024-09-15T03:48:07.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.807 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:07.807 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.851513+0000", 2024-09-15T03:48:07.808 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000282102 2024-09-15T03:48:07.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.808 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.808 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:07.808 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.851556+0000", 2024-09-15T03:48:07.808 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2960999999999997e-05 2024-09-15T03:48:07.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.808 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.808 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:07.808 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.852494+0000", 2024-09-15T03:48:07.808 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000937709 2024-09-15T03:48:07.808 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.808 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:07.809 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.809 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs2 65/64 ECSubWrite(tid=103, reqid=client.4521.0:463, at_version=65'24, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:48:07.809 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.848275+0000", 2024-09-15T03:48:07.809 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.933324474000003, 2024-09-15T03:48:07.809 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0067122830000000003, 2024-09-15T03:48:07.809 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:07.809 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:07.809 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:07.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.809 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:07.809 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848275+0000", 2024-09-15T03:48:07.809 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.810 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:07.810 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848275+0000", 2024-09-15T03:48:07.810 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.810 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:07.810 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848275+0000", 2024-09-15T03:48:07.810 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7899999999999999e-07 2024-09-15T03:48:07.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.810 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:07.811 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848319+0000", 2024-09-15T03:48:07.811 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4360000000000002e-05 2024-09-15T03:48:07.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.811 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:07.811 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848336+0000", 2024-09-15T03:48:07.811 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6606e-05 2024-09-15T03:48:07.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.811 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:07.811 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848344+0000", 2024-09-15T03:48:07.811 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0930000000000003e-06 2024-09-15T03:48:07.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.812 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:07.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848719+0000", 2024-09-15T03:48:07.812 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00037497000000000002 2024-09-15T03:48:07.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.812 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:07.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.848729+0000", 2024-09-15T03:48:07.812 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9860000000000006e-06 2024-09-15T03:48:07.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.812 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:07.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.849013+0000", 2024-09-15T03:48:07.812 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00028383400000000001 2024-09-15T03:48:07.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.813 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:07.813 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.849048+0000", 2024-09-15T03:48:07.813 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5474e-05 2024-09-15T03:48:07.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.813 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:07.813 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.850459+0000", 2024-09-15T03:48:07.813 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001410325 2024-09-15T03:48:07.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.813 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:07.813 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.850515+0000", 2024-09-15T03:48:07.814 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6801e-05 2024-09-15T03:48:07.814 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.814 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:07.814 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854987+0000", 2024-09-15T03:48:07.814 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0044713549999999998 2024-09-15T03:48:07.814 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.814 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:07.814 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.814 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.814 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs2 65/64 ECSubWrite(tid=107, reqid=client.4521.0:465, at_version=65'26, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:48:07.814 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.851183+0000", 2024-09-15T03:48:07.815 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.930416325000003, 2024-09-15T03:48:07.815 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0066932240000000002, 2024-09-15T03:48:07.815 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:07.815 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:07.815 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:07.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.815 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:07.815 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851183+0000", 2024-09-15T03:48:07.815 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.815 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:07.815 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851183+0000", 2024-09-15T03:48:07.816 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.816 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:07.816 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851183+0000", 2024-09-15T03:48:07.816 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5900000000000002e-07 2024-09-15T03:48:07.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.816 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:07.816 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851210+0000", 2024-09-15T03:48:07.816 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6398e-05 2024-09-15T03:48:07.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.817 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.817 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:07.817 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851219+0000", 2024-09-15T03:48:07.817 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3789999999999995e-06 2024-09-15T03:48:07.817 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.817 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.817 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:07.817 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851225+0000", 2024-09-15T03:48:07.817 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.472e-06 2024-09-15T03:48:07.817 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.817 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.817 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:07.817 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851369+0000", 2024-09-15T03:48:07.817 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000143559 2024-09-15T03:48:07.818 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.818 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.818 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:07.818 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851378+0000", 2024-09-15T03:48:07.818 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0310000000000004e-06 2024-09-15T03:48:07.818 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.818 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.818 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:07.818 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851467+0000", 2024-09-15T03:48:07.819 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9300000000000002e-05 2024-09-15T03:48:07.819 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.819 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.819 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:07.819 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851896+0000", 2024-09-15T03:48:07.819 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00042909399999999998 2024-09-15T03:48:07.819 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.819 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.819 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:07.819 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.851990+0000", 2024-09-15T03:48:07.819 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3428000000000001e-05 2024-09-15T03:48:07.819 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.819 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.820 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:07.820 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852035+0000", 2024-09-15T03:48:07.820 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4960000000000003e-05 2024-09-15T03:48:07.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.820 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.820 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:07.820 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.857876+0000", 2024-09-15T03:48:07.820 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0058411440000000004 2024-09-15T03:48:07.820 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.820 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:07.820 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.821 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.821 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s2 65/64 ECSubWrite(tid=69, reqid=client.4521.0:458, at_version=65'22, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:48:07.821 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.863447+0000", 2024-09-15T03:48:07.821 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.918151885, 2024-09-15T03:48:07.821 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0078021109999999996, 2024-09-15T03:48:07.821 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:07.821 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:07.821 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:07.821 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.821 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:07.821 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.863447+0000", 2024-09-15T03:48:07.821 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.821 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.822 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:07.822 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.863447+0000", 2024-09-15T03:48:07.822 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.822 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:07.822 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.863448+0000", 2024-09-15T03:48:07.822 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7399999999999999e-07 2024-09-15T03:48:07.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.822 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:07.822 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.863537+0000", 2024-09-15T03:48:07.822 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9034000000000002e-05 2024-09-15T03:48:07.823 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.823 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.823 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:07.823 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.863549+0000", 2024-09-15T03:48:07.823 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2554e-05 2024-09-15T03:48:07.823 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.823 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.823 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:07.823 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.863556+0000", 2024-09-15T03:48:07.823 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4049999999999997e-06 2024-09-15T03:48:07.823 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.823 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.823 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:07.824 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.863611+0000", 2024-09-15T03:48:07.824 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5494000000000001e-05 2024-09-15T03:48:07.824 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.824 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.824 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:07.824 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.863623+0000", 2024-09-15T03:48:07.824 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2265e-05 2024-09-15T03:48:07.824 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.824 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.824 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:07.824 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.864398+0000", 2024-09-15T03:48:07.824 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00077489300000000002 2024-09-15T03:48:07.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.825 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:07.825 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.864411+0000", 2024-09-15T03:48:07.825 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2529e-05 2024-09-15T03:48:07.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.825 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:07.825 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.864573+0000", 2024-09-15T03:48:07.825 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016181600000000001 2024-09-15T03:48:07.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.825 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:07.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.864588+0000", 2024-09-15T03:48:07.826 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5526999999999999e-05 2024-09-15T03:48:07.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.826 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:07.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871249+0000", 2024-09-15T03:48:07.826 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0066612199999999998 2024-09-15T03:48:07.826 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.826 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:07.826 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.826 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.es1 65/28 ECSubWrite(tid=104, reqid=client.4521.0:488, at_version=65'68, trim_to=0'0, roll_forward_to=64'65))", 2024-09-15T03:48:07.826 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.869959+0000", 2024-09-15T03:48:07.827 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.911640519999999, 2024-09-15T03:48:07.827 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0085038979999999993, 2024-09-15T03:48:07.827 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:07.827 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:07.827 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:07.827 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.827 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:07.827 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869959+0000", 2024-09-15T03:48:07.827 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.827 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.827 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:07.827 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869959+0000", 2024-09-15T03:48:07.828 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.828 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.828 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:07.828 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869959+0000", 2024-09-15T03:48:07.828 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4900000000000001e-07 2024-09-15T03:48:07.828 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.828 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:07.828 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869989+0000", 2024-09-15T03:48:07.828 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0179e-05 2024-09-15T03:48:07.828 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.828 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:07.829 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870000+0000", 2024-09-15T03:48:07.829 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0414000000000001e-05 2024-09-15T03:48:07.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.829 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:07.829 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870005+0000", 2024-09-15T03:48:07.829 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3870000000000003e-06 2024-09-15T03:48:07.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.829 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:07.829 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871808+0000", 2024-09-15T03:48:07.829 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001803453 2024-09-15T03:48:07.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.830 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.830 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:07.830 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871817+0000", 2024-09-15T03:48:07.830 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9309999999999997e-06 2024-09-15T03:48:07.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.830 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.830 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:07.830 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871919+0000", 2024-09-15T03:48:07.830 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000101195 2024-09-15T03:48:07.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.830 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.830 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:07.830 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871941+0000", 2024-09-15T03:48:07.830 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2693000000000001e-05 2024-09-15T03:48:07.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.831 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:07.831 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.872611+0000", 2024-09-15T03:48:07.831 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00067015100000000001 2024-09-15T03:48:07.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.831 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:07.831 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.872799+0000", 2024-09-15T03:48:07.831 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000188116 2024-09-15T03:48:07.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.831 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:07.831 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.878462+0000", 2024-09-15T03:48:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00566293 2024-09-15T03:48:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.es1 65/28 ECSubWrite(tid=105, reqid=client.4521.0:489, at_version=65'69, trim_to=0'0, roll_forward_to=64'65))", 2024-09-15T03:48:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.870601+0000", 2024-09-15T03:48:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 55.910998337999999, 2024-09-15T03:48:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0085606629999999996, 2024-09-15T03:48:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:07.832 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870601+0000", 2024-09-15T03:48:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870601+0000", 2024-09-15T03:48:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870601+0000", 2024-09-15T03:48:07.833 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.23e-07 2024-09-15T03:48:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870622+0000", 2024-09-15T03:48:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0511000000000001e-05 2024-09-15T03:48:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870640+0000", 2024-09-15T03:48:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8070000000000001e-05 2024-09-15T03:48:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:07.834 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870647+0000", 2024-09-15T03:48:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0770000000000002e-06 2024-09-15T03:48:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875542+0000", 2024-09-15T03:48:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0048952650000000002 2024-09-15T03:48:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875553+0000", 2024-09-15T03:48:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0163000000000001e-05 2024-09-15T03:48:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875674+0000", 2024-09-15T03:48:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000121618 2024-09-15T03:48:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876678+0000", 2024-09-15T03:48:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010037189999999999 2024-09-15T03:48:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876902+0000", 2024-09-15T03:48:07.836 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000224279 2024-09-15T03:48:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.879132+0000", 2024-09-15T03:48:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0022297039999999999 2024-09-15T03:48:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.879161+0000", 2024-09-15T03:48:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9533999999999999e-05 2024-09-15T03:48:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.0s6 78/70 ECSubWrite(tid=113, reqid=osd.1.0:113, at_version=78'32, trim_to=0'0, roll_forward_to=76'30))", 2024-09-15T03:48:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:24.957145+0000", 2024-09-15T03:48:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 42.824454252000002, 2024-09-15T03:48:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0074294319999999997, 2024-09-15T03:48:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:24.957145+0000", 2024-09-15T03:48:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:24.957145+0000", 2024-09-15T03:48:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:24.957145+0000", 2024-09-15T03:48:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7599999999999998e-07 2024-09-15T03:48:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:24.957168+0000", 2024-09-15T03:48:07.839 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2390000000000001e-05 2024-09-15T03:48:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:24.957177+0000", 2024-09-15T03:48:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8709999999999992e-06 2024-09-15T03:48:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:24.957183+0000", 2024-09-15T03:48:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.93e-06 2024-09-15T03:48:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.840 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:24.957232+0000", 2024-09-15T03:48:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9113000000000002e-05 2024-09-15T03:48:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:24.957241+0000", 2024-09-15T03:48:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6379999999999993e-06 2024-09-15T03:48:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:24.957463+0000", 2024-09-15T03:48:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022216799999999999 2024-09-15T03:48:07.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:24.957488+0000", 2024-09-15T03:48:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4660000000000001e-05 2024-09-15T03:48:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:24.957640+0000", 2024-09-15T03:48:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000152304 2024-09-15T03:48:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:07.842 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:24.963677+0000", 2024-09-15T03:48:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0060365330000000002 2024-09-15T03:48:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:24.964574+0000", 2024-09-15T03:48:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00089744900000000003 2024-09-15T03:48:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:675 3.5s1 3:af1dd10e:::smithi00132172-27:head [copy-from ver 51] snapc 2f=[2d,2b,29,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:48:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.087096+0000", 2024-09-15T03:48:07.843 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 37.694502642000003, 2024-09-15T03:48:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.043658777000000003, 2024-09-15T03:48:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 675 2024-09-15T03:48:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:07.844 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.087096+0000", 2024-09-15T03:48:07.845 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.845 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:07.845 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.087096+0000", 2024-09-15T03:48:07.845 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.845 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:07.845 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.087098+0000", 2024-09-15T03:48:07.845 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.694e-06 2024-09-15T03:48:07.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.846 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:07.846 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.087106+0000", 2024-09-15T03:48:07.846 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5209999999999997e-06 2024-09-15T03:48:07.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.846 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:07.846 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.087124+0000", 2024-09-15T03:48:07.846 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8393000000000002e-05 2024-09-15T03:48:07.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.846 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:07.846 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.087132+0000", 2024-09-15T03:48:07.846 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4700000000000002e-06 2024-09-15T03:48:07.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.847 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:07.847 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.087198+0000", 2024-09-15T03:48:07.847 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5063999999999999e-05 2024-09-15T03:48:07.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.847 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:07.847 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.087363+0000", 2024-09-15T03:48:07.847 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016577399999999999 2024-09-15T03:48:07.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.847 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:07.848 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.126124+0000", 2024-09-15T03:48:07.848 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.038761130999999997 2024-09-15T03:48:07.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.848 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.848 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:07.848 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.126654+0000", 2024-09-15T03:48:07.848 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00052913199999999997 2024-09-15T03:48:07.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.848 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.848 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:07.848 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.126698+0000", 2024-09-15T03:48:07.848 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4261999999999997e-05 2024-09-15T03:48:07.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.849 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:07.849 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.127096+0000", 2024-09-15T03:48:07.849 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00039841799999999999 2024-09-15T03:48:07.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.849 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:07.849 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.127184+0000", 2024-09-15T03:48:07.849 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8158000000000005e-05 2024-09-15T03:48:07.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.849 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:07.849 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.130193+0000", 2024-09-15T03:48:07.850 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003008967 2024-09-15T03:48:07.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.850 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.850 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:07.850 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.130755+0000", 2024-09-15T03:48:07.850 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00056179299999999999 2024-09-15T03:48:07.850 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.850 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:07.850 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.850 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.850 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:676 3.5s1 3:af1dd10e:::smithi00132172-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:48:07.850 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.087144+0000", 2024-09-15T03:48:07.851 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 37.694455054999999, 2024-09-15T03:48:07.851 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.040030812999999998, 2024-09-15T03:48:07.851 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:07.851 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:07.851 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:07.851 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:07.851 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:07.851 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 676 2024-09-15T03:48:07.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.851 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:07.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.851 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:07.851 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.087144+0000", 2024-09-15T03:48:07.852 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.852 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:07.852 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.087144+0000", 2024-09-15T03:48:07.852 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.852 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:07.852 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.087145+0000", 2024-09-15T03:48:07.852 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1600000000000001e-07 2024-09-15T03:48:07.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.853 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:07.853 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.087147+0000", 2024-09-15T03:48:07.853 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5790000000000002e-06 2024-09-15T03:48:07.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.853 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:07.853 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.087155+0000", 2024-09-15T03:48:07.853 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8720000000000002e-06 2024-09-15T03:48:07.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.853 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:07.853 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.087160+0000", 2024-09-15T03:48:07.853 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0549999999999999e-06 2024-09-15T03:48:07.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.854 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.854 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:07.854 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.087487+0000", 2024-09-15T03:48:07.854 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000327005 2024-09-15T03:48:07.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.854 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.854 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for blocked object", 2024-09-15T03:48:07.854 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.087569+0000", 2024-09-15T03:48:07.854 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1879000000000007e-05 2024-09-15T03:48:07.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.854 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.854 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:07.855 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.126818+0000", 2024-09-15T03:48:07.855 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.039249460999999999 2024-09-15T03:48:07.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.855 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:07.855 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.126915+0000", 2024-09-15T03:48:07.855 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6575000000000003e-05 2024-09-15T03:48:07.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.855 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:07.855 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.127175+0000", 2024-09-15T03:48:07.855 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00025987100000000001 2024-09-15T03:48:07.855 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.855 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:07.856 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.856 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.8s6 84/80 ECSubWrite(tid=121, reqid=client.4521.0:703, at_version=84'26, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:48:07.856 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.984882+0000", 2024-09-15T03:48:07.856 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 36.796716605, 2024-09-15T03:48:07.856 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0076072040000000002, 2024-09-15T03:48:07.856 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:07.856 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:07.856 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:07.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.856 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:07.856 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.984882+0000", 2024-09-15T03:48:07.857 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.857 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:07.857 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.984882+0000", 2024-09-15T03:48:07.857 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.857 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:07.857 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.984883+0000", 2024-09-15T03:48:07.857 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.53e-07 2024-09-15T03:48:07.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.857 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:07.858 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.984933+0000", 2024-09-15T03:48:07.858 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0419e-05 2024-09-15T03:48:07.858 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.858 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.858 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:07.858 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.984950+0000", 2024-09-15T03:48:07.858 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6419999999999999e-05 2024-09-15T03:48:07.858 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.858 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.858 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:07.858 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.984959+0000", 2024-09-15T03:48:07.858 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2560000000000008e-06 2024-09-15T03:48:07.858 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.859 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:07.859 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.985365+0000", 2024-09-15T03:48:07.859 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00040627799999999999 2024-09-15T03:48:07.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.859 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:07.859 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.985377+0000", 2024-09-15T03:48:07.859 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.189e-05 2024-09-15T03:48:07.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.859 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.859 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:07.859 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.985676+0000", 2024-09-15T03:48:07.859 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029855700000000002 2024-09-15T03:48:07.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.860 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.860 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:07.860 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986722+0000", 2024-09-15T03:48:07.860 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010468179999999999 2024-09-15T03:48:07.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.860 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.860 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:07.860 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986809+0000", 2024-09-15T03:48:07.860 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6218999999999996e-05 2024-09-15T03:48:07.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.860 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.860 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:07.861 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987570+0000", 2024-09-15T03:48:07.861 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000761939 2024-09-15T03:48:07.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.861 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:07.861 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.992490+0000", 2024-09-15T03:48:07.861 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0049191549999999997 2024-09-15T03:48:07.861 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.861 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:07.861 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.861 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.862 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.8s6 84/80 ECSubWrite(tid=122, reqid=client.4521.0:704, at_version=84'27, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:48:07.862 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.987197+0000", 2024-09-15T03:48:07.862 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 36.794401647999997, 2024-09-15T03:48:07.862 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0095503240000000007, 2024-09-15T03:48:07.862 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:07.862 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:07.862 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:07.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.862 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:07.862 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987197+0000", 2024-09-15T03:48:07.862 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.862 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.862 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:07.863 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987197+0000", 2024-09-15T03:48:07.863 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.863 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:07.863 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987199+0000", 2024-09-15T03:48:07.863 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1510000000000001e-06 2024-09-15T03:48:07.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.863 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:07.863 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987231+0000", 2024-09-15T03:48:07.863 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2357000000000002e-05 2024-09-15T03:48:07.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.864 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:07.864 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987247+0000", 2024-09-15T03:48:07.864 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5926999999999998e-05 2024-09-15T03:48:07.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.864 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:07.864 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987255+0000", 2024-09-15T03:48:07.864 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3790000000000006e-06 2024-09-15T03:48:07.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.864 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:07.864 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987308+0000", 2024-09-15T03:48:07.864 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2722000000000001e-05 2024-09-15T03:48:07.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.865 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:07.865 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987320+0000", 2024-09-15T03:48:07.865 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1748e-05 2024-09-15T03:48:07.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.865 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:07.865 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987442+0000", 2024-09-15T03:48:07.865 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012252899999999999 2024-09-15T03:48:07.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.865 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:07.865 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987473+0000", 2024-09-15T03:48:07.866 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1164000000000002e-05 2024-09-15T03:48:07.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.866 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:07.866 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987712+0000", 2024-09-15T03:48:07.866 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00023838099999999999 2024-09-15T03:48:07.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.866 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:07.866 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987732+0000", 2024-09-15T03:48:07.866 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0131000000000001e-05 2024-09-15T03:48:07.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.867 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:07.867 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.996748+0000", 2024-09-15T03:48:07.867 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0090158349999999998 2024-09-15T03:48:07.867 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.867 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:07.867 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.867 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.8s6 84/80 ECSubWrite(tid=123, reqid=client.4521.0:705, at_version=84'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:48:07.867 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.987533+0000", 2024-09-15T03:48:07.867 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 36.794066288000003, 2024-09-15T03:48:07.867 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0096813579999999993, 2024-09-15T03:48:07.867 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:07.867 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:07.868 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:07.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.868 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:07.868 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987533+0000", 2024-09-15T03:48:07.868 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.868 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:07.868 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987533+0000", 2024-09-15T03:48:07.868 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.868 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:07.869 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987534+0000", 2024-09-15T03:48:07.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5099999999999999e-07 2024-09-15T03:48:07.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.869 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:07.869 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987557+0000", 2024-09-15T03:48:07.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3370000000000002e-05 2024-09-15T03:48:07.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.869 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:07.869 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987571+0000", 2024-09-15T03:48:07.869 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3930999999999999e-05 2024-09-15T03:48:07.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:07.870 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987578+0000", 2024-09-15T03:48:07.870 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9909999999999996e-06 2024-09-15T03:48:07.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:07.870 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.988090+0000", 2024-09-15T03:48:07.870 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00051269300000000005 2024-09-15T03:48:07.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.870 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:07.870 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.988101+0000", 2024-09-15T03:48:07.870 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0121e-05 2024-09-15T03:48:07.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.871 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:07.871 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.988207+0000", 2024-09-15T03:48:07.871 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000105888 2024-09-15T03:48:07.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.871 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:07.871 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.988238+0000", 2024-09-15T03:48:07.871 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1613999999999999e-05 2024-09-15T03:48:07.871 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.871 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:07.871 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.988374+0000", 2024-09-15T03:48:07.872 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013567900000000001 2024-09-15T03:48:07.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.872 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:07.872 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.988390+0000", 2024-09-15T03:48:07.872 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5985e-05 2024-09-15T03:48:07.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.872 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:07.872 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.997214+0000", 2024-09-15T03:48:07.872 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0088243350000000009 2024-09-15T03:48:07.872 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.873 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:07.873 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.873 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.es1 84/28 ECSubWrite(tid=145, reqid=osd.2.0:145, at_version=84'102, trim_to=0'0, roll_forward_to=83'100))", 2024-09-15T03:48:07.873 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.990189+0000", 2024-09-15T03:48:07.873 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 36.791409702000003, 2024-09-15T03:48:07.873 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0068319849999999996, 2024-09-15T03:48:07.873 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:07.873 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:07.873 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:07.873 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.873 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:07.874 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.990189+0000", 2024-09-15T03:48:07.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.874 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:07.874 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.990189+0000", 2024-09-15T03:48:07.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.874 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:07.874 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.990190+0000", 2024-09-15T03:48:07.874 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7000000000000003e-07 2024-09-15T03:48:07.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.875 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.875 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:07.875 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.990216+0000", 2024-09-15T03:48:07.875 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5321e-05 2024-09-15T03:48:07.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.875 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.875 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:07.875 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.990237+0000", 2024-09-15T03:48:07.875 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1365e-05 2024-09-15T03:48:07.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.875 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.875 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:07.875 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.990261+0000", 2024-09-15T03:48:07.876 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4150000000000001e-05 2024-09-15T03:48:07.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.876 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:07.876 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.990323+0000", 2024-09-15T03:48:07.876 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1471999999999997e-05 2024-09-15T03:48:07.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.876 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:07.876 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.990334+0000", 2024-09-15T03:48:07.876 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1299e-05 2024-09-15T03:48:07.876 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.877 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:07.877 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.990639+0000", 2024-09-15T03:48:07.877 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00030495499999999998 2024-09-15T03:48:07.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.878 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:07.878 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.990653+0000", 2024-09-15T03:48:07.878 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3667e-05 2024-09-15T03:48:07.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.878 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:07.878 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.990929+0000", 2024-09-15T03:48:07.878 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00027613300000000002 2024-09-15T03:48:07.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.878 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:07.878 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.990947+0000", 2024-09-15T03:48:07.878 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7887999999999999e-05 2024-09-15T03:48:07.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.879 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:07.879 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.997021+0000", 2024-09-15T03:48:07.879 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0060747650000000002 2024-09-15T03:48:07.879 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.879 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:07.879 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.879 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.5s7 99/89 ECSubWrite(tid=134, reqid=client.4521.0:867, at_version=99'50, trim_to=0'0, roll_forward_to=96'47))", 2024-09-15T03:48:07.879 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:45.865328+0000", 2024-09-15T03:48:07.879 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.916271380000001, 2024-09-15T03:48:07.880 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0070670940000000003, 2024-09-15T03:48:07.880 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:07.880 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:07.880 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:07.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.880 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:07.880 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.865328+0000", 2024-09-15T03:48:07.880 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.880 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:07.880 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.865328+0000", 2024-09-15T03:48:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.865329+0000", 2024-09-15T03:48:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3699999999999999e-07 2024-09-15T03:48:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.865355+0000", 2024-09-15T03:48:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.6750999999999998e-05 2024-09-15T03:48:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:07.882 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.865378+0000", 2024-09-15T03:48:07.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2429000000000002e-05 2024-09-15T03:48:07.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.882 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:07.882 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.865388+0000", 2024-09-15T03:48:07.882 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9049999999999992e-06 2024-09-15T03:48:07.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.883 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:07.883 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.865447+0000", 2024-09-15T03:48:07.883 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9765000000000001e-05 2024-09-15T03:48:07.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.883 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:07.883 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.865458+0000", 2024-09-15T03:48:07.883 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0947999999999999e-05 2024-09-15T03:48:07.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.883 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:07.883 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.865586+0000", 2024-09-15T03:48:07.883 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012724500000000001 2024-09-15T03:48:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.865600+0000", 2024-09-15T03:48:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.393e-05 2024-09-15T03:48:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.865679+0000", 2024-09-15T03:48:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9583e-05 2024-09-15T03:48:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.884 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:07.885 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.865713+0000", 2024-09-15T03:48:07.885 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4044000000000001e-05 2024-09-15T03:48:07.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.885 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:07.885 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.872395+0000", 2024-09-15T03:48:07.885 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0066815570000000003 2024-09-15T03:48:07.885 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.885 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:07.885 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.885 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.5s7 99/89 ECSubWrite(tid=135, reqid=client.4521.0:868, at_version=99'51, trim_to=0'0, roll_forward_to=96'47))", 2024-09-15T03:48:07.885 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:45.866036+0000", 2024-09-15T03:48:07.886 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 21.915563410000001, 2024-09-15T03:48:07.886 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0065123400000000001, 2024-09-15T03:48:07.886 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:07.886 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:07.886 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:07.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.886 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:07.886 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.866036+0000", 2024-09-15T03:48:07.886 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.886 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.886 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:07.886 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.866036+0000", 2024-09-15T03:48:07.887 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.887 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.887 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:07.887 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.866037+0000", 2024-09-15T03:48:07.887 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0210000000000001e-06 2024-09-15T03:48:07.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.887 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.887 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:07.887 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.866060+0000", 2024-09-15T03:48:07.887 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3611000000000001e-05 2024-09-15T03:48:07.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.887 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.888 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:07.888 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.866078+0000", 2024-09-15T03:48:07.888 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7931000000000002e-05 2024-09-15T03:48:07.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.888 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:07.888 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.866087+0000", 2024-09-15T03:48:07.888 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4009999999999997e-06 2024-09-15T03:48:07.888 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.888 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.888 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:07.888 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.866170+0000", 2024-09-15T03:48:07.888 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3465000000000001e-05 2024-09-15T03:48:07.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.889 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:07.889 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.866180+0000", 2024-09-15T03:48:07.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7880000000000001e-06 2024-09-15T03:48:07.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.889 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:07.889 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.866282+0000", 2024-09-15T03:48:07.889 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000102274 2024-09-15T03:48:07.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.889 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.889 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:07.890 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.866293+0000", 2024-09-15T03:48:07.890 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1171e-05 2024-09-15T03:48:07.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.890 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:07.890 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.866566+0000", 2024-09-15T03:48:07.890 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00027255299999999998 2024-09-15T03:48:07.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.890 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:07.890 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.866585+0000", 2024-09-15T03:48:07.890 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9157999999999999e-05 2024-09-15T03:48:07.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.891 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:07.891 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:45.872548+0000", 2024-09-15T03:48:07.891 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0059629669999999996 2024-09-15T03:48:07.891 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.891 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:07.891 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.891 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.4s4 107/106 ECSubWrite(tid=141, reqid=client.4521.0:906, at_version=107'30, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:48:07.891 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:54.204132+0000", 2024-09-15T03:48:07.891 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 13.577467284000001, 2024-09-15T03:48:07.891 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0066209299999999997, 2024-09-15T03:48:07.892 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:07.892 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:07.892 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:07.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.892 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:07.892 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:54.204132+0000", 2024-09-15T03:48:07.892 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.892 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:07.892 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:54.204132+0000", 2024-09-15T03:48:07.892 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.893 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:07.893 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:54.204133+0000", 2024-09-15T03:48:07.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1000000000000001e-06 2024-09-15T03:48:07.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.893 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:07.893 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:54.205849+0000", 2024-09-15T03:48:07.893 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001716436 2024-09-15T03:48:07.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.893 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:07.893 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:54.205877+0000", 2024-09-15T03:48:07.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7881999999999998e-05 2024-09-15T03:48:07.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.894 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:07.894 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:54.205883+0000", 2024-09-15T03:48:07.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4530000000000001e-06 2024-09-15T03:48:07.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.894 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:07.894 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:54.205930+0000", 2024-09-15T03:48:07.894 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7843999999999998e-05 2024-09-15T03:48:07.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.895 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:07.895 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:54.205942+0000", 2024-09-15T03:48:07.895 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1803e-05 2024-09-15T03:48:07.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.895 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.895 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:07.895 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:54.206391+0000", 2024-09-15T03:48:07.895 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00044876099999999999 2024-09-15T03:48:07.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.895 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.895 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:07.895 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:54.206402+0000", 2024-09-15T03:48:07.895 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0669999999999999e-05 2024-09-15T03:48:07.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.896 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:07.896 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:54.206738+0000", 2024-09-15T03:48:07.896 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00033605100000000003 2024-09-15T03:48:07.896 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.896 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:07.896 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:54.206767+0000", 2024-09-15T03:48:07.896 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8968000000000001e-05 2024-09-15T03:48:07.896 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.896 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:07.896 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:54.210753+0000", 2024-09-15T03:48:07.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003985962 2024-09-15T03:48:07.897 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.897 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:07.897 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.897 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.897 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.as7 113/108 ECSubWrite(tid=532, reqid=client.4521.0:1020, at_version=113'83, trim_to=0'0, roll_forward_to=113'81))", 2024-09-15T03:48:07.897 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:59.977102+0000", 2024-09-15T03:48:07.897 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 7.8044974119999999, 2024-09-15T03:48:07.897 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0064912479999999998, 2024-09-15T03:48:07.897 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:07.897 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:07.897 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:07.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:07.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.977102+0000", 2024-09-15T03:48:07.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:07.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.977102+0000", 2024-09-15T03:48:07.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:07.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.898 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:07.898 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.977102+0000", 2024-09-15T03:48:07.898 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5700000000000002e-07 2024-09-15T03:48:07.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.899 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:07.899 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.977261+0000", 2024-09-15T03:48:07.899 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000158927 2024-09-15T03:48:07.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.899 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:07.899 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.977274+0000", 2024-09-15T03:48:07.899 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2744e-05 2024-09-15T03:48:07.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.899 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:07.900 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.977280+0000", 2024-09-15T03:48:07.900 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.5060000000000001e-06 2024-09-15T03:48:07.900 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.900 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.900 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:07.900 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.977337+0000", 2024-09-15T03:48:07.900 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6542999999999997e-05 2024-09-15T03:48:07.900 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.900 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.900 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:07.900 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.977348+0000", 2024-09-15T03:48:07.900 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1413000000000001e-05 2024-09-15T03:48:07.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.901 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:07.901 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.977640+0000", 2024-09-15T03:48:07.901 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000291105 2024-09-15T03:48:07.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.901 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:07.901 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.977649+0000", 2024-09-15T03:48:07.901 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0289999999999994e-06 2024-09-15T03:48:07.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.901 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:07.901 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.977788+0000", 2024-09-15T03:48:07.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000139908 2024-09-15T03:48:07.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.902 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:07.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.977842+0000", 2024-09-15T03:48:07.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3838000000000002e-05 2024-09-15T03:48:07.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:07.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:07.902 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:07.902 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.983593+0000", 2024-09-15T03:48:07.902 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0057505780000000001 2024-09-15T03:48:07.902 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.902 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:07.903 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.903 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:07.903 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:07.903 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:07.904 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:48:07.904 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:48:07.905 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:48:07.905 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:48:07.906 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:48:07.906 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:48:07.903+0000 7fd62abf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33736) UID: 0 2024-09-15T03:48:07.934 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:48:07.935 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-15T03:48:07.945 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-09-15T03:48:06.981453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":183697971,"num_objects":193,"num_object_clones":139,"num_object_copies":1520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":9,"num_read":533,"num_read_kb":239247,"num_write":1029,"num_write_kb":173944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":267,"num_bytes_recovered":285786150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1589,"ondisk_log_size":1589,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":12,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":1124155392,"kb_used":10221720,"kb_used_data":1113935212,"kb_used_omap":55676,"kb_used_meta":0,"kb_avail":1113933672,"statfs":{"total":1151135121408,"available":1140668080128,"internally_reserved":1576960,"allocated":1140669657088,"data_stored":1140669657088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57012943,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.018648"},"pg_stats":[{"pgid":"3.f","version":"113'71","reported_seq":461,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.190992+0000","last_change":"2024-09-15T03:48:03.620987+0000","last_active":"2024-09-15T03:48:05.190992+0000","last_peered":"2024-09-15T03:48:05.190992+0000","last_clean":"2024-09-15T03:48:05.190992+0000","last_became_active":"2024-09-15T03:47:55.850051+0000","last_became_peered":"2024-09-15T03:47:55.850051+0000","last_unstale":"2024-09-15T03:48:05.190992+0000","last_undegraded":"2024-09-15T03:48:05.190992+0000","last_fullsized":"2024-09-15T03:48:05.190992+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'71","last_scrub_stamp":"2024-09-15T03:48:03.620910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:03.620910+0000","objects_scrubbed":8,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:13.277978+0000","scrub_duration":0.0050823509999999997,"objects_trimmed":0,"snaptrim_duration":0.0077598160000000001,"stat_sum":{"num_bytes":8028160,"num_objects":8,"num_object_clones":5,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":32,"num_read_kb":42506,"num_write":26,"num_write_kb":7840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0,7,2,8,4,10],"acting":[11,1,0,7,2,8,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.e","version":"113'210","reported_seq":442,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.974944+0000","last_change":"2024-09-15T03:47:59.982417+0000","last_active":"2024-09-15T03:48:04.974944+0000","last_peered":"2024-09-15T03:48:04.974944+0000","last_clean":"2024-09-15T03:48:04.974944+0000","last_became_active":"2024-09-15T03:46:36.170536+0000","last_became_peered":"2024-09-15T03:46:36.170536+0000","last_unstale":"2024-09-15T03:48:04.974944+0000","last_undegraded":"2024-09-15T03:48:04.974944+0000","last_fullsized":"2024-09-15T03:48:04.974944+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'173","last_scrub_stamp":"2024-09-15T03:47:54.108780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:47:54.108780+0000","objects_scrubbed":21,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:11.038305+0000","scrub_duration":0.0088561600000000001,"objects_trimmed":0,"snaptrim_duration":0.037652644999999998,"stat_sum":{"num_bytes":17612800,"num_objects":22,"num_object_clones":15,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":2,"num_read":68,"num_read_kb":34008,"num_write":51,"num_write_kb":15520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,11,7,8,5,9],"acting":[2,0,1,11,7,8,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.d","version":"113'75","reported_seq":399,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.191773+0000","last_change":"2024-09-15T03:47:59.958101+0000","last_active":"2024-09-15T03:48:05.191773+0000","last_peered":"2024-09-15T03:48:05.191773+0000","last_clean":"2024-09-15T03:48:05.191773+0000","last_became_active":"2024-09-15T03:47:55.850104+0000","last_became_peered":"2024-09-15T03:47:55.850104+0000","last_unstale":"2024-09-15T03:48:05.191773+0000","last_undegraded":"2024-09-15T03:48:05.191773+0000","last_fullsized":"2024-09-15T03:48:05.191773+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'62","last_scrub_stamp":"2024-09-15T03:47:56.721765+0000","last_deep_scrub":"109'62","last_deep_scrub_stamp":"2024-09-15T03:47:56.721765+0000","last_clean_scrub_stamp":"2024-09-15T03:47:56.721765+0000","objects_scrubbed":10,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:15.880649+0000","scrub_duration":0.00827518,"objects_trimmed":0,"snaptrim_duration":0.013785528999999999,"stat_sum":{"num_bytes":7651328,"num_objects":10,"num_object_clones":8,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":20,"num_read_kb":8406,"num_write":43,"num_write_kb":7472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":10780672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,1,3,10,7,9,4],"acting":[8,11,1,3,10,7,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.c","version":"113'28","reported_seq":383,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.840818+0000","last_change":"2024-09-15T03:47:59.955875+0000","last_active":"2024-09-15T03:48:04.840818+0000","last_peered":"2024-09-15T03:48:04.840818+0000","last_clean":"2024-09-15T03:48:04.840818+0000","last_became_active":"2024-09-15T03:47:36.848722+0000","last_became_peered":"2024-09-15T03:47:36.848722+0000","last_unstale":"2024-09-15T03:48:04.840818+0000","last_undegraded":"2024-09-15T03:48:04.840818+0000","last_fullsized":"2024-09-15T03:48:04.840818+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'16","last_scrub_stamp":"2024-09-15T03:47:41.493521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:47:41.493521+0000","objects_scrubbed":3,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:09.644586+0000","scrub_duration":0.0039888800000000002,"objects_trimmed":0,"snaptrim_duration":0.012831169999999999,"stat_sum":{"num_bytes":2965504,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":18,"num_read_kb":8502,"num_write":10,"num_write_kb":3440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,11,1,0,6,5,2],"acting":[9,7,11,1,0,6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.b","version":"113'212","reported_seq":766,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.843752+0000","last_change":"2024-09-15T03:48:03.010113+0000","last_active":"2024-09-15T03:48:04.843752+0000","last_peered":"2024-09-15T03:48:04.843752+0000","last_clean":"2024-09-15T03:48:04.843752+0000","last_became_active":"2024-09-15T03:47:55.850099+0000","last_became_peered":"2024-09-15T03:47:55.850099+0000","last_unstale":"2024-09-15T03:48:04.843752+0000","last_undegraded":"2024-09-15T03:48:04.843752+0000","last_fullsized":"2024-09-15T03:48:04.843752+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'212","last_scrub_stamp":"2024-09-15T03:48:03.010035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:03.010035+0000","objects_scrubbed":27,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:22.035341+0000","scrub_duration":0.022865639,"objects_trimmed":0,"snaptrim_duration":0.025351973,"stat_sum":{"num_bytes":24117248,"num_objects":27,"num_object_clones":20,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":2,"num_read":65,"num_read_kb":36889,"num_write":170,"num_write_kb":22832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":74006528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,8,4,1,10,11,3],"acting":[7,6,8,4,1,10,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.a","version":"113'84","reported_seq":240,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.896901+0000","last_change":"2024-09-15T03:48:03.286753+0000","last_active":"2024-09-15T03:48:04.896901+0000","last_peered":"2024-09-15T03:48:04.896901+0000","last_clean":"2024-09-15T03:48:04.896901+0000","last_became_active":"2024-09-15T03:47:55.854826+0000","last_became_peered":"2024-09-15T03:47:55.854826+0000","last_unstale":"2024-09-15T03:48:04.896901+0000","last_undegraded":"2024-09-15T03:48:04.896901+0000","last_fullsized":"2024-09-15T03:48:04.896901+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'84","last_scrub_stamp":"2024-09-15T03:48:03.286675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:03.286675+0000","objects_scrubbed":8,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:03.509620+0000","scrub_duration":0.0063142770000000001,"objects_trimmed":0,"snaptrim_duration":0.013016882,"stat_sum":{"num_bytes":6995968,"num_objects":8,"num_object_clones":6,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":16,"num_read_kb":6728,"num_write":30,"num_write_kb":6832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":30703616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8,9,5,6,2,0],"acting":[4,10,8,9,5,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.9","version":"113'80","reported_seq":492,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.979274+0000","last_change":"2024-09-15T03:47:59.971224+0000","last_active":"2024-09-15T03:48:04.979274+0000","last_peered":"2024-09-15T03:48:04.979274+0000","last_clean":"2024-09-15T03:48:04.979274+0000","last_became_active":"2024-09-15T03:47:55.851394+0000","last_became_peered":"2024-09-15T03:47:55.851394+0000","last_unstale":"2024-09-15T03:48:04.979274+0000","last_undegraded":"2024-09-15T03:48:04.979274+0000","last_fullsized":"2024-09-15T03:48:04.979274+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012050854,"stat_sum":{"num_bytes":12025856,"num_objects":13,"num_object_clones":10,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":15,"num_read_kb":8,"num_write":59,"num_write_kb":11744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":27590656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,10,1,9,6,8,11],"acting":[3,2,10,1,9,6,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.8","version":"113'52","reported_seq":377,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.979298+0000","last_change":"2024-09-15T03:47:59.962623+0000","last_active":"2024-09-15T03:48:04.979298+0000","last_peered":"2024-09-15T03:48:04.979298+0000","last_clean":"2024-09-15T03:48:04.979298+0000","last_became_active":"2024-09-15T03:47:55.848776+0000","last_became_peered":"2024-09-15T03:47:55.848776+0000","last_unstale":"2024-09-15T03:48:04.979298+0000","last_undegraded":"2024-09-15T03:48:04.979298+0000","last_fullsized":"2024-09-15T03:48:04.979298+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01649231,"stat_sum":{"num_bytes":4079616,"num_objects":6,"num_object_clones":4,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":23,"num_read_kb":4616,"num_write":54,"num_write_kb":3984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":11223040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,11,4,1,0,10],"acting":[3,7,2,11,4,1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"1.0","version":"20'92","reported_seq":226,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.059683+0000","last_change":"2024-09-15T03:47:39.774326+0000","last_active":"2024-09-15T03:48:04.059683+0000","last_peered":"2024-09-15T03:48:04.059683+0000","last_clean":"2024-09-15T03:48:04.059683+0000","last_became_active":"2024-09-15T03:46:18.011130+0000","last_became_peered":"2024-09-15T03:46:18.011130+0000","last_unstale":"2024-09-15T03:48:04.059683+0000","last_undegraded":"2024-09-15T03:48:04.059683+0000","last_fullsized":"2024-09-15T03:48:04.059683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-15T03:47:39.774262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_clean_scrub_stamp":"2024-09-15T03:47:39.774262+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:43.691013+0000","scrub_duration":0.0027413759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"22'1","reported_seq":118,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:03.962737+0000","last_change":"2024-09-15T03:47:52.292975+0000","last_active":"2024-09-15T03:48:03.962737+0000","last_peered":"2024-09-15T03:48:03.962737+0000","last_clean":"2024-09-15T03:48:03.962737+0000","last_became_active":"2024-09-15T03:46:24.038633+0000","last_became_peered":"2024-09-15T03:46:24.038633+0000","last_unstale":"2024-09-15T03:48:03.962737+0000","last_undegraded":"2024-09-15T03:48:03.962737+0000","last_fullsized":"2024-09-15T03:48:03.962737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'1","last_scrub_stamp":"2024-09-15T03:47:52.292926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:52.292926+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:56.362597+0000","scrub_duration":0.001409429,"objects_trimmed":0,"snaptrim_duration":0.00030178200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"113'80","reported_seq":513,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.896581+0000","last_change":"2024-09-15T03:47:59.966281+0000","last_active":"2024-09-15T03:48:04.896581+0000","last_peered":"2024-09-15T03:48:04.896581+0000","last_clean":"2024-09-15T03:48:04.896581+0000","last_became_active":"2024-09-15T03:47:55.854075+0000","last_became_peered":"2024-09-15T03:47:55.854075+0000","last_unstale":"2024-09-15T03:48:04.896581+0000","last_undegraded":"2024-09-15T03:48:04.896581+0000","last_fullsized":"2024-09-15T03:48:04.896581+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021707911999999999,"stat_sum":{"num_bytes":13320192,"num_objects":10,"num_object_clones":7,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":37,"num_read_kb":16719,"num_write":39,"num_write_kb":13088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":18628608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,7,8,6,11,1],"acting":[4,2,10,7,8,6,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":115,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.059319+0000","last_change":"2024-09-15T03:47:27.779257+0000","last_active":"2024-09-15T03:48:04.059319+0000","last_peered":"2024-09-15T03:48:04.059319+0000","last_clean":"2024-09-15T03:48:04.059319+0000","last_became_active":"2024-09-15T03:46:24.063106+0000","last_became_peered":"2024-09-15T03:46:24.063106+0000","last_unstale":"2024-09-15T03:48:04.059319+0000","last_undegraded":"2024-09-15T03:48:04.059319+0000","last_fullsized":"2024-09-15T03:48:04.059319+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.779210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.779210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:42.717969+0000","scrub_duration":0.0021030710000000002,"objects_trimmed":0,"snaptrim_duration":0.000393075,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"113'29","reported_seq":174,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.687310+0000","last_change":"2024-09-15T03:48:05.687193+0000","last_active":"2024-09-15T03:48:05.687310+0000","last_peered":"2024-09-15T03:48:05.687310+0000","last_clean":"2024-09-15T03:48:05.687310+0000","last_became_active":"2024-09-15T03:47:55.848033+0000","last_became_peered":"2024-09-15T03:47:55.848033+0000","last_unstale":"2024-09-15T03:48:05.687310+0000","last_undegraded":"2024-09-15T03:48:05.687310+0000","last_fullsized":"2024-09-15T03:48:05.687310+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'29","last_scrub_stamp":"2024-09-15T03:48:05.687085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:05.687085+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:32.245260+0000","scrub_duration":0.0044105530000000002,"objects_trimmed":0,"snaptrim_duration":0.0090284240000000002,"stat_sum":{"num_bytes":4653056,"num_objects":5,"num_object_clones":4,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":3075,"num_write":12,"num_write_kb":4032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":6045696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3,10,5,2,9,7],"acting":[6,11,3,10,5,2,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":120,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.843432+0000","last_change":"2024-09-15T03:47:42.997012+0000","last_active":"2024-09-15T03:48:04.843432+0000","last_peered":"2024-09-15T03:48:04.843432+0000","last_clean":"2024-09-15T03:48:04.843432+0000","last_became_active":"2024-09-15T03:46:24.037471+0000","last_became_peered":"2024-09-15T03:46:24.037471+0000","last_unstale":"2024-09-15T03:48:04.843432+0000","last_undegraded":"2024-09-15T03:48:04.843432+0000","last_fullsized":"2024-09-15T03:48:04.843432+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:42.996957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:42.996957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:53.443125+0000","scrub_duration":0.0014164309999999999,"objects_trimmed":0,"snaptrim_duration":0.00026490299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"113'56","reported_seq":456,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-15T03:48:03.778938+0000","last_change":"2024-09-15T03:48:03.778794+0000","last_active":"2024-09-15T03:48:03.778938+0000","last_peered":"2024-09-15T03:48:03.778938+0000","last_clean":"2024-09-15T03:48:03.778938+0000","last_became_active":"2024-09-15T03:47:55.848512+0000","last_became_peered":"2024-09-15T03:47:55.848512+0000","last_unstale":"2024-09-15T03:48:03.778938+0000","last_undegraded":"2024-09-15T03:48:03.778938+0000","last_fullsized":"2024-09-15T03:48:03.778938+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'56","last_scrub_stamp":"2024-09-15T03:48:03.778652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:03.778652+0000","objects_scrubbed":8,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:05.162717+0000","scrub_duration":0.0067734900000000001,"objects_trimmed":0,"snaptrim_duration":0.012362052,"stat_sum":{"num_bytes":4997120,"num_objects":8,"num_object_clones":5,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":20,"num_read_kb":7496,"num_write":43,"num_write_kb":4880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":16449536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,9,4,3,6,0,10],"acting":[1,2,9,4,3,6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"c","length":"1"},{"start":"12","length":"1"},{"start":"17","length":"2"},{"start":"20","length":"1"},{"start":"23","length":"2"},{"start":"26","length":"1"},{"start":"29","length":"1"},{"start":"2f","length":"1"},{"start":"32","length":"1"}]},{"pgid":"2.2","version":"24'2","reported_seq":161,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.781140+0000","last_change":"2024-09-15T03:48:04.781084+0000","last_active":"2024-09-15T03:48:04.781140+0000","last_peered":"2024-09-15T03:48:04.781140+0000","last_clean":"2024-09-15T03:48:04.781140+0000","last_became_active":"2024-09-15T03:47:36.848076+0000","last_became_peered":"2024-09-15T03:47:36.848076+0000","last_unstale":"2024-09-15T03:48:04.781140+0000","last_undegraded":"2024-09-15T03:48:04.781140+0000","last_fullsized":"2024-09-15T03:48:04.781140+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'2","last_scrub_stamp":"2024-09-15T03:48:04.781025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:48:04.781025+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:13.832081+0000","scrub_duration":0.0016375750000000001,"objects_trimmed":0,"snaptrim_duration":0.000944677,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"113'196","reported_seq":461,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.094888+0000","last_change":"2024-09-15T03:47:59.976831+0000","last_active":"2024-09-15T03:48:05.094888+0000","last_peered":"2024-09-15T03:48:05.094888+0000","last_clean":"2024-09-15T03:48:05.094888+0000","last_became_active":"2024-09-15T03:47:55.855061+0000","last_became_peered":"2024-09-15T03:47:55.855061+0000","last_unstale":"2024-09-15T03:48:05.094888+0000","last_undegraded":"2024-09-15T03:48:05.094888+0000","last_fullsized":"2024-09-15T03:48:05.094888+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'156","last_scrub_stamp":"2024-09-15T03:47:56.178879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:47:56.178879+0000","objects_scrubbed":22,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:09.356437+0000","scrub_duration":0.014803648000000001,"objects_trimmed":0,"snaptrim_duration":0.031600082000000002,"stat_sum":{"num_bytes":20545536,"num_objects":25,"num_object_clones":19,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":27,"num_read_kb":352,"num_write":173,"num_write_kb":21440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,4,7,1,2,9,0],"acting":[10,8,4,7,1,2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.5","version":"113'79","reported_seq":384,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.094605+0000","last_change":"2024-09-15T03:48:04.063851+0000","last_active":"2024-09-15T03:48:05.094605+0000","last_peered":"2024-09-15T03:48:05.094605+0000","last_clean":"2024-09-15T03:48:05.094605+0000","last_became_active":"2024-09-15T03:47:55.860659+0000","last_became_peered":"2024-09-15T03:47:55.860659+0000","last_unstale":"2024-09-15T03:48:05.094605+0000","last_undegraded":"2024-09-15T03:48:05.094605+0000","last_fullsized":"2024-09-15T03:48:05.094605+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'79","last_scrub_stamp":"2024-09-15T03:48:04.063762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:04.063762+0000","objects_scrubbed":9,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:30.399619+0000","scrub_duration":0.0069124379999999999,"objects_trimmed":0,"snaptrim_duration":0.013330699,"stat_sum":{"num_bytes":24330240,"num_objects":9,"num_object_clones":8,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":14,"num_read_kb":11048,"num_write":34,"num_write_kb":13376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":23789568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4,9,8,7,3,0],"acting":[10,6,4,9,8,7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":138,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.840281+0000","last_change":"2024-09-15T03:47:39.930675+0000","last_active":"2024-09-15T03:48:04.840281+0000","last_peered":"2024-09-15T03:48:04.840281+0000","last_clean":"2024-09-15T03:48:04.840281+0000","last_became_active":"2024-09-15T03:47:33.998802+0000","last_became_peered":"2024-09-15T03:47:33.998802+0000","last_unstale":"2024-09-15T03:48:04.840281+0000","last_undegraded":"2024-09-15T03:48:04.840281+0000","last_fullsized":"2024-09-15T03:48:04.840281+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:39.930628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:39.930628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:03.535396+0000","scrub_duration":0.001146333,"objects_trimmed":0,"snaptrim_duration":0.00034123999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"113'42","reported_seq":399,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.191327+0000","last_change":"2024-09-15T03:48:03.723173+0000","last_active":"2024-09-15T03:48:05.191327+0000","last_peered":"2024-09-15T03:48:05.191327+0000","last_clean":"2024-09-15T03:48:05.191327+0000","last_became_active":"2024-09-15T03:47:55.846248+0000","last_became_peered":"2024-09-15T03:47:55.846248+0000","last_unstale":"2024-09-15T03:48:05.191327+0000","last_undegraded":"2024-09-15T03:48:05.191327+0000","last_fullsized":"2024-09-15T03:48:05.191327+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'42","last_scrub_stamp":"2024-09-15T03:48:03.723102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:03.723102+0000","objects_scrubbed":5,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:25.602306+0000","scrub_duration":0.0052425570000000001,"objects_trimmed":0,"snaptrim_duration":0.011769172,"stat_sum":{"num_bytes":7389184,"num_objects":5,"num_object_clones":4,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":6052,"num_write":11,"num_write_kb":7952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":9814016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5,8,0,4,7,10],"acting":[6,2,5,8,0,4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":117,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.060875+0000","last_change":"2024-09-15T03:47:30.025153+0000","last_active":"2024-09-15T03:48:04.060875+0000","last_peered":"2024-09-15T03:48:04.060875+0000","last_clean":"2024-09-15T03:48:04.060875+0000","last_became_active":"2024-09-15T03:46:24.038945+0000","last_became_peered":"2024-09-15T03:46:24.038945+0000","last_unstale":"2024-09-15T03:48:04.060875+0000","last_undegraded":"2024-09-15T03:48:04.060875+0000","last_fullsized":"2024-09-15T03:48:04.060875+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:30.025107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:30.025107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:36.231260+0000","scrub_duration":0.0013039429999999999,"objects_trimmed":0,"snaptrim_duration":0.00032787499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"113'153","reported_seq":520,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.258880+0000","last_change":"2024-09-15T03:48:05.258804+0000","last_active":"2024-09-15T03:48:05.258880+0000","last_peered":"2024-09-15T03:48:05.258880+0000","last_clean":"2024-09-15T03:48:05.258880+0000","last_became_active":"2024-09-15T03:47:55.848466+0000","last_became_peered":"2024-09-15T03:47:55.848466+0000","last_unstale":"2024-09-15T03:48:05.258880+0000","last_undegraded":"2024-09-15T03:48:05.258880+0000","last_fullsized":"2024-09-15T03:48:05.258880+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'153","last_scrub_stamp":"2024-09-15T03:48:05.258733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:05.258733+0000","objects_scrubbed":22,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:30.071424+0000","scrub_duration":0.012102717000000001,"objects_trimmed":0,"snaptrim_duration":0.036439371999999998,"stat_sum":{"num_bytes":19546112,"num_objects":22,"num_object_clones":17,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":1,"num_read":64,"num_read_kb":52263,"num_write":118,"num_write_kb":18976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":27639808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,7,2,1,11,10],"acting":[4,3,9,7,2,1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":118,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.060192+0000","last_change":"2024-09-15T03:47:27.297139+0000","last_active":"2024-09-15T03:48:04.060192+0000","last_peered":"2024-09-15T03:48:04.060192+0000","last_clean":"2024-09-15T03:48:04.060192+0000","last_became_active":"2024-09-15T03:46:24.063380+0000","last_became_peered":"2024-09-15T03:46:24.063380+0000","last_unstale":"2024-09-15T03:48:04.060192+0000","last_undegraded":"2024-09-15T03:48:04.060192+0000","last_fullsized":"2024-09-15T03:48:04.060192+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.297083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.297083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:48.875998+0000","scrub_duration":0.0018328789999999999,"objects_trimmed":0,"snaptrim_duration":0.00034490600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"113'47","reported_seq":487,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.896781+0000","last_change":"2024-09-15T03:47:59.953939+0000","last_active":"2024-09-15T03:48:04.896781+0000","last_peered":"2024-09-15T03:48:04.896781+0000","last_clean":"2024-09-15T03:48:04.896781+0000","last_became_active":"2024-09-15T03:47:55.847868+0000","last_became_peered":"2024-09-15T03:47:55.847868+0000","last_unstale":"2024-09-15T03:48:04.896781+0000","last_undegraded":"2024-09-15T03:48:04.896781+0000","last_fullsized":"2024-09-15T03:48:04.896781+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010037447,"stat_sum":{"num_bytes":4849664,"num_objects":7,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":13,"num_read_kb":515,"num_write":31,"num_write_kb":8384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":7176192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,9,8,3,11,1,10],"acting":[4,7,9,8,3,11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":115,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.059914+0000","last_change":"2024-09-15T03:47:42.675933+0000","last_active":"2024-09-15T03:48:04.059914+0000","last_peered":"2024-09-15T03:48:04.059914+0000","last_clean":"2024-09-15T03:48:04.059914+0000","last_became_active":"2024-09-15T03:46:24.063335+0000","last_became_peered":"2024-09-15T03:46:24.063335+0000","last_unstale":"2024-09-15T03:48:04.059914+0000","last_undegraded":"2024-09-15T03:48:04.059914+0000","last_fullsized":"2024-09-15T03:48:04.059914+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:42.675882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:42.675882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:02.697036+0000","scrub_duration":0.001445478,"objects_trimmed":0,"snaptrim_duration":0.00065928000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":183107584,"num_objects":189,"num_object_clones":139,"num_object_copies":1512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":189,"num_whiteouts":9,"num_read":457,"num_read_kb":239183,"num_write":904,"num_write_kb":171792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":265,"num_bytes_recovered":285786112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1494,"ondisk_log_size":1494,"up":128,"acting":128,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":11,"up_from":17,"seq":73014444056,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":846004,"kb_used_data":92833684,"kb_used_omap":4805,"kb_used_meta":0,"kb_avail":92833612,"statfs":{"total":95927926784,"available":95061618688,"internally_reserved":73728,"allocated":95061692416,"data_stored":95061692416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4920982,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.46600000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":121.575,"5min":121.575,"15min":121.575},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2060.797,"5min":2060.797,"15min":2060.797},"last":0.753},{"interface":"front","average":{"1min":121.56,"5min":121.56,"15min":121.56},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2060.7159999999999,"5min":2060.7159999999999,"15min":2060.7159999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.61499999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.79300000000000004}]},{"osd":4,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.67300000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.68400000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.70299999999999996}]},{"osd":7,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.63},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.74299999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.59999999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.84299999999999997}]},{"osd":10,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.82199999999999995}]}]},{"osd":10,"up_from":105,"seq":450971566084,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":871700,"kb_used_data":92807916,"kb_used_omap":3648,"kb_used_meta":0,"kb_avail":92807916,"statfs":{"total":95927926784,"available":95035305984,"internally_reserved":0,"allocated":95035305984,"data_stored":95035305984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3736470,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]}]},{"osd":9,"up_from":17,"seq":73014444056,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":855500,"kb_used_data":92824200,"kb_used_omap":5160,"kb_used_meta":0,"kb_avail":92824116,"statfs":{"total":95927926784,"available":95051894784,"internally_reserved":86016,"allocated":95051980800,"data_stored":95051980800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5284527,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.499},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.61499999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.51400000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.66100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.56299999999999994}]},{"osd":5,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.52500000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.46700000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.55000000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.625}]},{"osd":10,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.54100000000000004}]},{"osd":11,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.57399999999999995}]}]},{"osd":8,"up_from":17,"seq":73014444057,"num_pgs":15,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":861168,"kb_used_data":92818540,"kb_used_omap":5191,"kb_used_meta":0,"kb_avail":92818448,"statfs":{"total":95927926784,"available":95046090752,"internally_reserved":94208,"allocated":95046184960,"data_stored":95046184960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5316028,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.81699999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":42.381,"5min":42.381,"15min":42.381},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":963.98400000000004,"5min":963.98400000000004,"15min":963.98400000000004},"last":0.63100000000000001},{"interface":"front","average":{"1min":42.447000000000003,"5min":42.447000000000003,"15min":42.447000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":964.79600000000005,"5min":964.79600000000005,"15min":964.79600000000005},"last":0.56499999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.752}]},{"osd":4,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.83299999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:47:25 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.79200000000000004}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.73099999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.73999999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.437},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.60899999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.80400000000000005}]}]},{"osd":7,"up_from":17,"seq":73014444056,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":864892,"kb_used_data":92814832,"kb_used_omap":5646,"kb_used_meta":0,"kb_avail":92814724,"statfs":{"total":95927926784,"available":95042277376,"internally_reserved":110592,"allocated":95042387968,"data_stored":95042387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5781943,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.58199999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.41699999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.46700000000000003}]},{"osd":4,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.56000000000000005}]},{"osd":5,"last update":"Sun Sep 15 03:47:27 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.59999999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.71399999999999997}]},{"osd":8,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.626},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.66500000000000004}]},{"osd":9,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.75900000000000001}]},{"osd":10,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.746}]},{"osd":11,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.78300000000000003}]}]},{"osd":6,"up_from":17,"seq":73014444056,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":838972,"kb_used_data":92840728,"kb_used_omap":3843,"kb_used_meta":0,"kb_avail":92840644,"statfs":{"total":95927926784,"available":95068819456,"internally_reserved":86016,"allocated":95068905472,"data_stored":95068905472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3935639,"internal_metadata":0},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.52000000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:47:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.56299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.57399999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.55200000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.67500000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.66000000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.502}]},{"osd":8,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.78400000000000003}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.77700000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":11,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.83299999999999996}]}]},{"osd":5,"up_from":17,"seq":73014444056,"num_pgs":7,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":814900,"kb_used_data":92864772,"kb_used_omap":1826,"kb_used_meta":0,"kb_avail":92864716,"statfs":{"total":95927926784,"available":95093469184,"internally_reserved":57344,"allocated":95093526528,"data_stored":95093526528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1870318,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.29799999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":113.717,"5min":113.717,"15min":113.717},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2037.425,"5min":2037.425,"15min":2037.425},"last":0.34000000000000002},{"interface":"front","average":{"1min":113.752,"5min":113.752,"15min":113.752},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2037.3230000000001,"5min":2037.3230000000001,"15min":2037.3230000000001},"last":0.51400000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.379},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.53700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":4,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.439}]},{"osd":6,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.42499999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.497}]},{"osd":8,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.61799999999999999}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.63},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.65500000000000003}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.83399999999999996}]},{"osd":11,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.58899999999999997}]}]},{"osd":4,"up_from":17,"seq":73014444056,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":863984,"kb_used_data":92815728,"kb_used_omap":5620,"kb_used_meta":0,"kb_avail":92815632,"statfs":{"total":95927926784,"available":95043207168,"internally_reserved":98304,"allocated":95043305472,"data_stored":95043305472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5755199,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.47299999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":84.314999999999998,"5min":84.314999999999998,"15min":84.314999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1758.9780000000001,"5min":1758.9780000000001,"15min":1758.9780000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":84.364000000000004,"5min":84.364000000000004,"15min":84.364000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1758.671,"5min":1758.671,"15min":1758.671},"last":0.42699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.48999999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.33700000000000002}]},{"osd":5,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.375},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.52500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.55700000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.61299999999999999}]},{"osd":9,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":0.623}]},{"osd":10,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.53800000000000003}]},{"osd":11,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.58399999999999996}]}]},{"osd":3,"up_from":17,"seq":73014444056,"num_pgs":10,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":842612,"kb_used_data":92837084,"kb_used_omap":4588,"kb_used_meta":0,"kb_avail":92837004,"statfs":{"total":95927926784,"available":95065092096,"internally_reserved":81920,"allocated":95065174016,"data_stored":95065174016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4698419,"internal_metadata":0},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.27300000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:47:38 2024","interfaces":[{"interface":"back","average":{"1min":35.600000000000001,"5min":35.600000000000001,"15min":35.600000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":564.05100000000004,"5min":564.05100000000004,"15min":564.05100000000004},"last":0.309},{"interface":"front","average":{"1min":35.677,"5min":35.677,"15min":35.677},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":564.57000000000005,"5min":564.57000000000005,"15min":564.57000000000005},"last":0.40500000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.49399999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.61399999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":2.3940000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":5.2830000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.59699999999999998}]},{"osd":7,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.443},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.64400000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.89100000000000001}]},{"osd":9,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.84199999999999997}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.752}]}]},{"osd":2,"up_from":17,"seq":73014444056,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":855316,"kb_used_data":92824420,"kb_used_omap":5155,"kb_used_meta":0,"kb_avail":92824300,"statfs":{"total":95927926784,"available":95052083200,"internally_reserved":122880,"allocated":95052206080,"data_stored":95052206080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5279060,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:37 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.438}]},{"osd":1,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":34.457999999999998,"5min":34.457999999999998,"15min":34.457999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":647.92399999999998,"5min":647.92399999999998,"15min":647.92399999999998},"last":0.28100000000000003},{"interface":"front","average":{"1min":34.475000000000001,"5min":34.475000000000001,"15min":34.475000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":648.02700000000004,"5min":648.02700000000004,"15min":648.02700000000004},"last":0.34999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.46400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.54500000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.55400000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.503},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.56599999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.343,"15min":1.343},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":14.721,"5min":14.721,"15min":14.721},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.371,"5min":1.371,"15min":1.371},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":14.970000000000001,"5min":14.970000000000001,"15min":14.970000000000001},"last":0.77800000000000002}]},{"osd":9,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.377,"15min":1.377},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":14.888,"5min":14.888,"15min":14.888},"last":0.69799999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":14.654999999999999,"5min":14.654999999999999,"15min":14.654999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":1.409,"5min":1.409,"15min":1.409},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":15.096,"5min":15.096,"15min":15.096},"last":0.60499999999999998}]}]},{"osd":1,"up_from":17,"seq":73014444056,"num_pgs":13,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":862556,"kb_used_data":92817764,"kb_used_omap":5933,"kb_used_meta":0,"kb_avail":92817060,"statfs":{"total":95927926784,"available":95044669440,"internally_reserved":720896,"allocated":95045390336,"data_stored":95045390336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6076292,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.313}]},{"osd":2,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":0.437},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.55300000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.41099999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":2.6920000000000002,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":0.53600000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.7440000000000002,"5min":2.7440000000000002,"15min":2.7440000000000002},"last":0.58299999999999996}]},{"osd":6,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":3.278,"5min":3.278,"15min":3.278},"last":0.52000000000000002}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.81399999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":119.93899999999999,"5min":119.93899999999999,"15min":119.93899999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2861.7190000000001,"5min":2861.7190000000001,"15min":2861.7190000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.946,"5min":1.946,"15min":1.946},"last":0.60499999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":119.937,"5min":119.937,"15min":119.937},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":2861.6010000000001,"5min":2861.6010000000001,"15min":2861.6010000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.48399999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":119.96599999999999,"5min":119.96599999999999,"15min":119.96599999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2861.5639999999999,"5min":2861.5639999999999,"15min":2861.5639999999999},"last":0.627}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]}]},{"osd":0,"up_from":17,"seq":73014444056,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":844116,"kb_used_data":92835544,"kb_used_omap":4255,"kb_used_meta":0,"kb_avail":92835500,"statfs":{"total":95927926784,"available":95063552000,"internally_reserved":45056,"allocated":95063597056,"data_stored":95063597056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4358066,"internal_metadata":0},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.46999999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.804,"5min":1.804,"15min":1.804},"last":0.379}]},{"osd":3,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.52,"5min":1.52,"15min":1.52},"last":0.376},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.35399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":5,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.502}]},{"osd":6,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.48999999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.68700000000000006}]},{"osd":8,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.44400000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":10,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.58899999999999997}]},{"osd":11,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.56499999999999995}]}]}],"pool_statfs":[]}} 2024-09-15T03:48:07.949 INFO:tasks.ceph.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-15T03:48:07.993 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T03:48:08.005 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:48:08.005+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:48:08.035 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:08.036 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:48:08.036 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-15T03:48:08.036 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:08.045 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:48:08.106 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:48:08.106+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:48:08.187 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:08.188 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:48:08.188 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-15T03:48:08.188 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-15T03:48:08.188 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:08.196 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:48:08.207 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:48:08.206+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:48:08.299 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:08.299 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-15T03:48:08.299 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-15T03:48:08.299 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-15T03:48:08.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.300 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=80, reqid=client.4521.0:320, at_version=49'15, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:48:08.300 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.840885+0000", 2024-09-15T03:48:08.300 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 72.456693986999994, 2024-09-15T03:48:08.300 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0033862459999998, 2024-09-15T03:48:08.300 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.300 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.300 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.300 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.300 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.840885+0000", 2024-09-15T03:48:08.300 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.301 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.301 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.840885+0000", 2024-09-15T03:48:08.301 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.301 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.301 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.840885+0000", 2024-09-15T03:48:08.301 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1100000000000002e-07 2024-09-15T03:48:08.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.301 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.301 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841181+0000", 2024-09-15T03:48:08.302 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029618199999999998 2024-09-15T03:48:08.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.302 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.302 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841195+0000", 2024-09-15T03:48:08.302 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3424000000000001e-05 2024-09-15T03:48:08.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.302 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.302 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841204+0000", 2024-09-15T03:48:08.302 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0170000000000002e-06 2024-09-15T03:48:08.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.302 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.303 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841274+0000", 2024-09-15T03:48:08.303 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0561000000000004e-05 2024-09-15T03:48:08.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.303 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.303 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841289+0000", 2024-09-15T03:48:08.303 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4127e-05 2024-09-15T03:48:08.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.303 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.303 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841717+0000", 2024-09-15T03:48:08.303 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00042811800000000001 2024-09-15T03:48:08.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.304 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.304 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841745+0000", 2024-09-15T03:48:08.304 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8297e-05 2024-09-15T03:48:08.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.304 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.304 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.841852+0000", 2024-09-15T03:48:08.304 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010660199999999999 2024-09-15T03:48:08.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.304 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.304 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.842419+0000", 2024-09-15T03:48:08.304 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00056784499999999998 2024-09-15T03:48:08.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.305 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.305 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.844271+0000", 2024-09-15T03:48:08.305 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0018517619999998 2024-09-15T03:48:08.305 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.305 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.305 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.305 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=30, reqid=client.4521.0:325, at_version=49'12, trim_to=0'0, roll_forward_to=40'10))", 2024-09-15T03:48:08.305 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.844912+0000", 2024-09-15T03:48:08.305 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 72.452667405, 2024-09-15T03:48:08.305 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0007286940000002, 2024-09-15T03:48:08.306 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.306 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.306 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.306 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.306 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844912+0000", 2024-09-15T03:48:08.306 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.306 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.306 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.306 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844912+0000", 2024-09-15T03:48:08.306 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.306 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.307 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.307 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844912+0000", 2024-09-15T03:48:08.307 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.84e-07 2024-09-15T03:48:08.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.307 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.307 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.307 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844937+0000", 2024-09-15T03:48:08.307 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4729e-05 2024-09-15T03:48:08.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.307 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.307 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.307 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844957+0000", 2024-09-15T03:48:08.307 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0285999999999999e-05 2024-09-15T03:48:08.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.308 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.308 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.308 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.844965+0000", 2024-09-15T03:48:08.308 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5709999999999998e-06 2024-09-15T03:48:08.308 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.308 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.308 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.308 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845086+0000", 2024-09-15T03:48:08.308 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000120856 2024-09-15T03:48:08.308 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.308 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.308 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.308 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845121+0000", 2024-09-15T03:48:08.309 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4292999999999998e-05 2024-09-15T03:48:08.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.309 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.309 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.309 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845492+0000", 2024-09-15T03:48:08.309 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00037167400000000001 2024-09-15T03:48:08.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.309 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.309 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.309 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845529+0000", 2024-09-15T03:48:08.309 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7092000000000003e-05 2024-09-15T03:48:08.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.309 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.309 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.310 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845591+0000", 2024-09-15T03:48:08.310 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1824999999999998e-05 2024-09-15T03:48:08.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.310 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.310 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845612+0000", 2024-09-15T03:48:08.310 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.084e-05 2024-09-15T03:48:08.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.310 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.310 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845640+0000", 2024-09-15T03:48:08.310 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0000281439999998 2024-09-15T03:48:08.311 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.311 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.311 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.312 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.312 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=81, reqid=client.4521.0:321, at_version=49'16, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:48:08.312 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.845113+0000", 2024-09-15T03:48:08.312 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 72.452465997999994, 2024-09-15T03:48:08.312 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.999459995, 2024-09-15T03:48:08.312 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.312 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.312 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.312 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.312 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.312 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845113+0000", 2024-09-15T03:48:08.312 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.313 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.313 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.313 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845113+0000", 2024-09-15T03:48:08.313 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.313 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.313 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.313 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.313 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845114+0000", 2024-09-15T03:48:08.313 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0210000000000001e-06 2024-09-15T03:48:08.313 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.313 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.313 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.313 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845422+0000", 2024-09-15T03:48:08.313 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00030762999999999998 2024-09-15T03:48:08.314 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.314 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.314 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845436+0000", 2024-09-15T03:48:08.314 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3998999999999999e-05 2024-09-15T03:48:08.314 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.314 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.314 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845445+0000", 2024-09-15T03:48:08.314 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5799999999999998e-06 2024-09-15T03:48:08.314 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.314 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.315 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845538+0000", 2024-09-15T03:48:08.315 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2739999999999996e-05 2024-09-15T03:48:08.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.315 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.315 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845548+0000", 2024-09-15T03:48:08.315 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0548e-05 2024-09-15T03:48:08.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.315 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.315 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845803+0000", 2024-09-15T03:48:08.315 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00025495599999999998 2024-09-15T03:48:08.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.316 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.316 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.845998+0000", 2024-09-15T03:48:08.316 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019432799999999999 2024-09-15T03:48:08.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.316 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.316 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846122+0000", 2024-09-15T03:48:08.316 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012434600000000001 2024-09-15T03:48:08.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.316 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.316 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846178+0000", 2024-09-15T03:48:08.316 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.5943999999999998e-05 2024-09-15T03:48:08.317 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.317 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.317 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.317 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.844573+0000", 2024-09-15T03:48:08.317 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9983949029999999 2024-09-15T03:48:08.317 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.317 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.317 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.317 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.317 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.317 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=82, reqid=client.4521.0:322, at_version=49'17, trim_to=0'0, roll_forward_to=32'13))", 2024-09-15T03:48:08.317 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.846151+0000", 2024-09-15T03:48:08.317 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 72.451427976000005, 2024-09-15T03:48:08.318 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9989014119999999, 2024-09-15T03:48:08.318 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.318 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.318 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.318 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.318 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.318 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846151+0000", 2024-09-15T03:48:08.318 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.318 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.318 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.318 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846151+0000", 2024-09-15T03:48:08.318 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.319 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.319 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.319 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.319 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846152+0000", 2024-09-15T03:48:08.319 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0800000000000004e-07 2024-09-15T03:48:08.319 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.319 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.319 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.319 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846238+0000", 2024-09-15T03:48:08.319 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5866999999999996e-05 2024-09-15T03:48:08.319 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.319 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.319 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.319 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846255+0000", 2024-09-15T03:48:08.320 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6954999999999999e-05 2024-09-15T03:48:08.320 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.320 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.320 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846262+0000", 2024-09-15T03:48:08.320 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7680000000000008e-06 2024-09-15T03:48:08.320 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.320 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.320 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846325+0000", 2024-09-15T03:48:08.320 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2191000000000001e-05 2024-09-15T03:48:08.320 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.320 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.321 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846335+0000", 2024-09-15T03:48:08.321 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0767e-05 2024-09-15T03:48:08.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.321 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.321 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.321 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846446+0000", 2024-09-15T03:48:08.321 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110522 2024-09-15T03:48:08.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.321 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.321 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.321 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846458+0000", 2024-09-15T03:48:08.321 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2023e-05 2024-09-15T03:48:08.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.322 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.322 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.322 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846508+0000", 2024-09-15T03:48:08.322 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0479000000000002e-05 2024-09-15T03:48:08.322 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.322 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.322 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.322 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.846526+0000", 2024-09-15T03:48:08.322 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7836e-05 2024-09-15T03:48:08.322 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.322 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.322 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.322 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845052+0000", 2024-09-15T03:48:08.322 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9985261960000003 2024-09-15T03:48:08.322 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.323 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.323 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.323 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.323 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=31, reqid=client.4521.0:325, at_version=0'0, trim_to=0'0, roll_forward_to=49'12))", 2024-09-15T03:48:08.323 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.847407+0000", 2024-09-15T03:48:08.323 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 72.450171772999994, 2024-09-15T03:48:08.323 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9983494909999999, 2024-09-15T03:48:08.323 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.323 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.323 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.323 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.323 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847407+0000", 2024-09-15T03:48:08.324 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.324 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.324 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847407+0000", 2024-09-15T03:48:08.324 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.324 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.324 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847408+0000", 2024-09-15T03:48:08.324 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0070000000000001e-06 2024-09-15T03:48:08.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.324 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.325 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847421+0000", 2024-09-15T03:48:08.325 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3203999999999999e-05 2024-09-15T03:48:08.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.325 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.325 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.325 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847443+0000", 2024-09-15T03:48:08.325 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1539000000000002e-05 2024-09-15T03:48:08.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.325 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.325 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.325 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847452+0000", 2024-09-15T03:48:08.325 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6570000000000003e-06 2024-09-15T03:48:08.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.325 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.326 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.326 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847534+0000", 2024-09-15T03:48:08.326 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2309000000000001e-05 2024-09-15T03:48:08.326 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.326 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.326 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.326 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847544+0000", 2024-09-15T03:48:08.326 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0482000000000001e-05 2024-09-15T03:48:08.326 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.326 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.326 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.326 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847645+0000", 2024-09-15T03:48:08.326 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010021700000000001 2024-09-15T03:48:08.326 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.327 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.327 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847665+0000", 2024-09-15T03:48:08.327 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0271999999999998e-05 2024-09-15T03:48:08.327 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.327 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.327 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847708+0000", 2024-09-15T03:48:08.327 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2828999999999999e-05 2024-09-15T03:48:08.327 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.327 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.327 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847724+0000", 2024-09-15T03:48:08.327 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6334e-05 2024-09-15T03:48:08.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.328 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.328 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845757+0000", 2024-09-15T03:48:08.328 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.998032641 2024-09-15T03:48:08.328 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.328 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.328 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.328 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=83, reqid=client.4521.0:323, at_version=49'18, trim_to=0'0, roll_forward_to=49'15))", 2024-09-15T03:48:08.328 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.847691+0000", 2024-09-15T03:48:08.328 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 72.449887926000002, 2024-09-15T03:48:08.329 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9981305969999998, 2024-09-15T03:48:08.329 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.329 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.329 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.329 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.329 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.329 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847691+0000", 2024-09-15T03:48:08.329 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.329 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.329 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.329 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847691+0000", 2024-09-15T03:48:08.329 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.330 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.330 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847692+0000", 2024-09-15T03:48:08.330 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9899999999999999e-07 2024-09-15T03:48:08.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.330 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.330 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847715+0000", 2024-09-15T03:48:08.330 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2818000000000001e-05 2024-09-15T03:48:08.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.330 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.330 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847731+0000", 2024-09-15T03:48:08.331 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6401000000000001e-05 2024-09-15T03:48:08.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.331 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.331 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847738+0000", 2024-09-15T03:48:08.331 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3699999999999997e-06 2024-09-15T03:48:08.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.331 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.331 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847791+0000", 2024-09-15T03:48:08.331 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2584999999999999e-05 2024-09-15T03:48:08.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.331 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.332 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847801+0000", 2024-09-15T03:48:08.332 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0295999999999999e-05 2024-09-15T03:48:08.332 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.332 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.332 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847931+0000", 2024-09-15T03:48:08.332 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0001295 2024-09-15T03:48:08.332 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.332 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.332 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847946+0000", 2024-09-15T03:48:08.332 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4732000000000001e-05 2024-09-15T03:48:08.332 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.333 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.333 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.847989+0000", 2024-09-15T03:48:08.333 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3625000000000003e-05 2024-09-15T03:48:08.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.333 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.333 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.848004+0000", 2024-09-15T03:48:08.333 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4385e-05 2024-09-15T03:48:08.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.333 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.333 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845822+0000", 2024-09-15T03:48:08.333 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9978179860000003 2024-09-15T03:48:08.334 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.334 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.334 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.334 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.334 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.334 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=84, reqid=client.4521.0:323, at_version=0'0, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:48:08.334 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.849295+0000", 2024-09-15T03:48:08.334 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 72.448283845999995, 2024-09-15T03:48:08.334 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9965517450000001, 2024-09-15T03:48:08.335 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.336 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.336 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.336 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.336 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849295+0000", 2024-09-15T03:48:08.336 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.336 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.336 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849295+0000", 2024-09-15T03:48:08.336 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.336 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.337 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849296+0000", 2024-09-15T03:48:08.337 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6300000000000004e-07 2024-09-15T03:48:08.337 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.337 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.337 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.337 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849308+0000", 2024-09-15T03:48:08.337 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2261e-05 2024-09-15T03:48:08.337 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.337 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.337 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.337 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849325+0000", 2024-09-15T03:48:08.337 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6588000000000001e-05 2024-09-15T03:48:08.337 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.338 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.338 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.338 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849332+0000", 2024-09-15T03:48:08.338 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.464e-06 2024-09-15T03:48:08.338 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.338 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.338 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.338 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849379+0000", 2024-09-15T03:48:08.338 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6498000000000001e-05 2024-09-15T03:48:08.338 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.338 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.338 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.338 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849391+0000", 2024-09-15T03:48:08.338 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1926000000000001e-05 2024-09-15T03:48:08.339 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.339 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.339 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.339 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849497+0000", 2024-09-15T03:48:08.339 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010634200000000001 2024-09-15T03:48:08.339 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.339 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.339 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.339 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849516+0000", 2024-09-15T03:48:08.339 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8995999999999999e-05 2024-09-15T03:48:08.339 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.339 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.339 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.339 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849565+0000", 2024-09-15T03:48:08.339 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8735e-05 2024-09-15T03:48:08.340 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.340 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.340 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.340 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849577+0000", 2024-09-15T03:48:08.340 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2585999999999999e-05 2024-09-15T03:48:08.340 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.340 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.340 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.340 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845847+0000", 2024-09-15T03:48:08.340 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9962694860000001 2024-09-15T03:48:08.340 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.340 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.340 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.341 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.341 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.341 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpRead(3.7s5 49/31 ECSubRead(tid=85, to_read={3:e40d1694:::smithi00132172-50:head=0,4096,0}, subchunks={3:e40d1694:::smithi00132172-50:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:48:08.341 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.849814+0000", 2024-09-15T03:48:08.341 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 72.447764531999994, 2024-09-15T03:48:08.341 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9961114690000001, 2024-09-15T03:48:08.341 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.341 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.341 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.341 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.341 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.341 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849814+0000", 2024-09-15T03:48:08.341 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.342 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.342 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.342 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.342 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849814+0000", 2024-09-15T03:48:08.342 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.342 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.342 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.342 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.342 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849815+0000", 2024-09-15T03:48:08.342 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6499999999999999e-07 2024-09-15T03:48:08.342 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.342 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.342 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.342 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.849826+0000", 2024-09-15T03:48:08.343 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0635000000000001e-05 2024-09-15T03:48:08.343 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.343 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.343 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.343 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.850154+0000", 2024-09-15T03:48:08.343 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00032866900000000002 2024-09-15T03:48:08.343 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.343 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.343 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.343 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.850169+0000", 2024-09-15T03:48:08.343 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5051e-05 2024-09-15T03:48:08.343 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.343 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.344 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.344 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.850226+0000", 2024-09-15T03:48:08.344 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6926999999999999e-05 2024-09-15T03:48:08.344 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.344 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.344 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.344 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845926+0000", 2024-09-15T03:48:08.344 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9956995219999998 2024-09-15T03:48:08.344 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.344 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.344 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.344 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.344 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.344 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=32, reqid=client.4521.0:329, at_version=49'13, trim_to=0'0, roll_forward_to=49'12))", 2024-09-15T03:48:08.345 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.853281+0000", 2024-09-15T03:48:08.345 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 72.444298046, 2024-09-15T03:48:08.345 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9927023569999998, 2024-09-15T03:48:08.345 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.345 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.345 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.345 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.345 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.345 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853281+0000", 2024-09-15T03:48:08.345 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.345 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.345 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.346 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853281+0000", 2024-09-15T03:48:08.346 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.346 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.346 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.346 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.346 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853282+0000", 2024-09-15T03:48:08.346 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.347e-06 2024-09-15T03:48:08.346 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.346 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.346 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.346 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853401+0000", 2024-09-15T03:48:08.346 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118711 2024-09-15T03:48:08.346 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.346 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.347 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.347 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853441+0000", 2024-09-15T03:48:08.347 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0059999999999999e-05 2024-09-15T03:48:08.347 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.347 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.347 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.347 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853451+0000", 2024-09-15T03:48:08.347 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5850000000000006e-06 2024-09-15T03:48:08.347 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.347 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.347 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.347 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853617+0000", 2024-09-15T03:48:08.347 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016655899999999999 2024-09-15T03:48:08.347 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.348 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.348 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.348 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853632+0000", 2024-09-15T03:48:08.348 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4605e-05 2024-09-15T03:48:08.348 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.348 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.348 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.348 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853842+0000", 2024-09-15T03:48:08.348 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000210523 2024-09-15T03:48:08.349 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.349 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.349 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.349 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.853871+0000", 2024-09-15T03:48:08.349 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8441000000000001e-05 2024-09-15T03:48:08.349 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.349 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.349 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.349 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854188+0000", 2024-09-15T03:48:08.349 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031700800000000003 2024-09-15T03:48:08.349 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.349 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.349 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.350 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854209+0000", 2024-09-15T03:48:08.350 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0795999999999999e-05 2024-09-15T03:48:08.350 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.350 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.350 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.350 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845983+0000", 2024-09-15T03:48:08.350 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9917747220000002 2024-09-15T03:48:08.350 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.350 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.350 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.350 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.350 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.350 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=33, reqid=client.4521.0:329, at_version=0'0, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:48:08.351 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.854592+0000", 2024-09-15T03:48:08.351 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 72.442987035000002, 2024-09-15T03:48:08.351 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9914290499999998, 2024-09-15T03:48:08.351 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.351 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.351 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.351 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.351 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.351 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854592+0000", 2024-09-15T03:48:08.351 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.351 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.351 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.351 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.351 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854592+0000", 2024-09-15T03:48:08.352 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.352 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.352 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.352 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854593+0000", 2024-09-15T03:48:08.352 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1200000000000002e-07 2024-09-15T03:48:08.352 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.352 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.352 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854605+0000", 2024-09-15T03:48:08.352 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1946999999999999e-05 2024-09-15T03:48:08.352 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.353 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.353 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854624+0000", 2024-09-15T03:48:08.353 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9502000000000001e-05 2024-09-15T03:48:08.353 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.353 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.353 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.353 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854631+0000", 2024-09-15T03:48:08.353 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2490000000000004e-06 2024-09-15T03:48:08.353 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.353 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.353 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.353 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854683+0000", 2024-09-15T03:48:08.353 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1643e-05 2024-09-15T03:48:08.353 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.354 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.354 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854693+0000", 2024-09-15T03:48:08.354 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8199999999999992e-06 2024-09-15T03:48:08.354 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.354 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.354 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854768+0000", 2024-09-15T03:48:08.354 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5104999999999993e-05 2024-09-15T03:48:08.354 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.354 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.354 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854830+0000", 2024-09-15T03:48:08.354 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2191000000000001e-05 2024-09-15T03:48:08.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.355 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.355 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854934+0000", 2024-09-15T03:48:08.355 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010424099999999999 2024-09-15T03:48:08.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.355 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.355 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.854952+0000", 2024-09-15T03:48:08.356 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7521e-05 2024-09-15T03:48:08.356 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.356 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.356 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.846021+0000", 2024-09-15T03:48:08.356 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9910690190000002 2024-09-15T03:48:08.356 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.356 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.356 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.356 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.356 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=34, reqid=client.4521.0:330, at_version=49'14, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:48:08.356 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.857132+0000", 2024-09-15T03:48:08.356 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 72.440446653999999, 2024-09-15T03:48:08.357 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9890264310000001, 2024-09-15T03:48:08.357 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.357 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.357 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.357 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.357 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.357 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857132+0000", 2024-09-15T03:48:08.357 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.357 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.357 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.357 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857132+0000", 2024-09-15T03:48:08.357 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.358 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.358 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857133+0000", 2024-09-15T03:48:08.358 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.61e-07 2024-09-15T03:48:08.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.358 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.358 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857223+0000", 2024-09-15T03:48:08.358 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0476999999999994e-05 2024-09-15T03:48:08.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.358 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.358 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857245+0000", 2024-09-15T03:48:08.359 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1415000000000001e-05 2024-09-15T03:48:08.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.359 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.359 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857254+0000", 2024-09-15T03:48:08.359 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.329e-06 2024-09-15T03:48:08.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.359 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.359 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857312+0000", 2024-09-15T03:48:08.359 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7865000000000002e-05 2024-09-15T03:48:08.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.359 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.360 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857323+0000", 2024-09-15T03:48:08.360 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0655999999999999e-05 2024-09-15T03:48:08.360 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.360 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.360 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.360 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857817+0000", 2024-09-15T03:48:08.360 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00049391300000000001 2024-09-15T03:48:08.360 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.360 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.360 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.360 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857946+0000", 2024-09-15T03:48:08.360 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012926699999999999 2024-09-15T03:48:08.360 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.361 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.361 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.361 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858076+0000", 2024-09-15T03:48:08.361 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012957600000000001 2024-09-15T03:48:08.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.361 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.361 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.361 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858094+0000", 2024-09-15T03:48:08.361 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8715e-05 2024-09-15T03:48:08.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.361 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.361 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.361 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.846159+0000", 2024-09-15T03:48:08.361 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9880644570000001 2024-09-15T03:48:08.362 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.362 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.362 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.362 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=86, reqid=client.4521.0:334, at_version=49'20, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:48:08.362 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.857463+0000", 2024-09-15T03:48:08.362 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 72.440116336000003, 2024-09-15T03:48:08.362 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0025860940000002, 2024-09-15T03:48:08.362 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.362 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.362 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.362 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.363 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857463+0000", 2024-09-15T03:48:08.363 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.363 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.363 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857463+0000", 2024-09-15T03:48:08.363 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.363 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.363 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857464+0000", 2024-09-15T03:48:08.363 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0210000000000001e-06 2024-09-15T03:48:08.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.364 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.364 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857489+0000", 2024-09-15T03:48:08.364 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5500999999999999e-05 2024-09-15T03:48:08.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.364 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.364 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857508+0000", 2024-09-15T03:48:08.364 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9025000000000002e-05 2024-09-15T03:48:08.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.364 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.364 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857515+0000", 2024-09-15T03:48:08.364 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9759999999999998e-06 2024-09-15T03:48:08.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.365 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.365 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.365 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845961+0000", 2024-09-15T03:48:08.365 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9884457740000001 2024-09-15T03:48:08.365 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.365 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.365 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.365 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.845966+0000", 2024-09-15T03:48:08.365 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3750000000000002e-06 2024-09-15T03:48:08.365 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.365 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.365 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.365 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.846127+0000", 2024-09-15T03:48:08.365 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000160462 2024-09-15T03:48:08.366 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.366 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.366 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.366 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.846136+0000", 2024-09-15T03:48:08.366 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6260000000000007e-06 2024-09-15T03:48:08.366 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.366 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.366 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.366 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.846191+0000", 2024-09-15T03:48:08.366 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4274999999999998e-05 2024-09-15T03:48:08.366 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.366 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.366 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.366 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.853397+0000", 2024-09-15T03:48:08.367 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0072068519999999997 2024-09-15T03:48:08.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.367 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.367 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.367 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.860049+0000", 2024-09-15T03:48:08.367 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0066512070000000001 2024-09-15T03:48:08.367 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.367 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.367 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.367 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.367 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.cs3 49/28 ECSubWrite(tid=15, reqid=client.4521.0:337, at_version=49'10, trim_to=0'0, roll_forward_to=43'8))", 2024-09-15T03:48:08.367 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.857712+0000", 2024-09-15T03:48:08.367 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 72.439867016999997, 2024-09-15T03:48:08.368 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9893153209999999, 2024-09-15T03:48:08.368 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.368 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.368 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.368 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.368 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.368 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857712+0000", 2024-09-15T03:48:08.368 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.368 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.368 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.368 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.368 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857712+0000", 2024-09-15T03:48:08.368 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.368 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.369 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.369 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.369 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857712+0000", 2024-09-15T03:48:08.369 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8000000000000002e-07 2024-09-15T03:48:08.369 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.369 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.369 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.369 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857755+0000", 2024-09-15T03:48:08.369 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.2960999999999997e-05 2024-09-15T03:48:08.369 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.369 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.369 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.369 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857773+0000", 2024-09-15T03:48:08.370 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7635999999999999e-05 2024-09-15T03:48:08.370 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.370 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.370 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.370 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857780+0000", 2024-09-15T03:48:08.370 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.024e-06 2024-09-15T03:48:08.370 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.370 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.370 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.370 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857883+0000", 2024-09-15T03:48:08.370 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103224 2024-09-15T03:48:08.370 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.370 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.370 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.371 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.857893+0000", 2024-09-15T03:48:08.371 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0265000000000001e-05 2024-09-15T03:48:08.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.371 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.371 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858180+0000", 2024-09-15T03:48:08.371 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000286602 2024-09-15T03:48:08.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.371 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.371 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858193+0000", 2024-09-15T03:48:08.371 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3342e-05 2024-09-15T03:48:08.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.372 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.372 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858233+0000", 2024-09-15T03:48:08.372 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9811999999999998e-05 2024-09-15T03:48:08.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.372 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.372 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858253+0000", 2024-09-15T03:48:08.372 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0259e-05 2024-09-15T03:48:08.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.372 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.372 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.847027+0000", 2024-09-15T03:48:08.373 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.988773916 2024-09-15T03:48:08.373 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.373 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.373 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.373 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.373 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=87, reqid=client.4521.0:335, at_version=49'21, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:48:08.373 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.858481+0000", 2024-09-15T03:48:08.373 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 72.439097868000005, 2024-09-15T03:48:08.373 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.004544015, 2024-09-15T03:48:08.373 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.373 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.373 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.374 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.374 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858481+0000", 2024-09-15T03:48:08.374 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.374 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.374 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858481+0000", 2024-09-15T03:48:08.374 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.374 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.374 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858482+0000", 2024-09-15T03:48:08.375 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.09e-07 2024-09-15T03:48:08.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.375 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.375 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858505+0000", 2024-09-15T03:48:08.375 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3133e-05 2024-09-15T03:48:08.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.375 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.375 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858524+0000", 2024-09-15T03:48:08.375 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8706e-05 2024-09-15T03:48:08.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.375 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.376 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.858532+0000", 2024-09-15T03:48:08.376 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5539999999999998e-06 2024-09-15T03:48:08.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.376 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.376 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851213+0000", 2024-09-15T03:48:08.376 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.992680526 2024-09-15T03:48:08.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.376 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.376 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851221+0000", 2024-09-15T03:48:08.376 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5380000000000002e-06 2024-09-15T03:48:08.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851309+0000", 2024-09-15T03:48:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.8069e-05 2024-09-15T03:48:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851495+0000", 2024-09-15T03:48:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018516999999999999 2024-09-15T03:48:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851557+0000", 2024-09-15T03:48:08.377 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2031000000000003e-05 2024-09-15T03:48:08.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.378 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.378 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.853665+0000", 2024-09-15T03:48:08.378 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021079499999999999 2024-09-15T03:48:08.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.378 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.378 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.863025+0000", 2024-09-15T03:48:08.378 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0093605289999999994 2024-09-15T03:48:08.378 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.378 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.378 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.379 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.cs3 49/28 ECSubWrite(tid=16, reqid=client.4521.0:337, at_version=0'0, trim_to=0'0, roll_forward_to=49'10))", 2024-09-15T03:48:08.379 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.859219+0000", 2024-09-15T03:48:08.379 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 72.438359896999998, 2024-09-15T03:48:08.379 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9879305710000001, 2024-09-15T03:48:08.379 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.379 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.379 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.379 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.379 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859219+0000", 2024-09-15T03:48:08.379 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.380 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.380 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859219+0000", 2024-09-15T03:48:08.380 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.380 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.380 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859220+0000", 2024-09-15T03:48:08.380 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3300000000000001e-07 2024-09-15T03:48:08.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.380 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.380 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859233+0000", 2024-09-15T03:48:08.380 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2781e-05 2024-09-15T03:48:08.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.381 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.381 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859251+0000", 2024-09-15T03:48:08.381 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8770000000000002e-05 2024-09-15T03:48:08.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.381 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.381 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859259+0000", 2024-09-15T03:48:08.381 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.7309999999999993e-06 2024-09-15T03:48:08.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.381 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.381 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859342+0000", 2024-09-15T03:48:08.382 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2909000000000002e-05 2024-09-15T03:48:08.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.382 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.382 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859357+0000", 2024-09-15T03:48:08.382 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5245000000000001e-05 2024-09-15T03:48:08.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.382 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.382 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859453+0000", 2024-09-15T03:48:08.382 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5878999999999995e-05 2024-09-15T03:48:08.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.382 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.383 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859475+0000", 2024-09-15T03:48:08.383 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1591999999999999e-05 2024-09-15T03:48:08.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.383 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.383 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859518+0000", 2024-09-15T03:48:08.383 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3021e-05 2024-09-15T03:48:08.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.383 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.383 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859536+0000", 2024-09-15T03:48:08.383 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8420999999999999e-05 2024-09-15T03:48:08.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.383 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.384 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.847150+0000", 2024-09-15T03:48:08.384 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9876134890000001 2024-09-15T03:48:08.384 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.384 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.384 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.384 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.384 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.384 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.7s5 49/31 ECSubWrite(tid=88, reqid=client.4521.0:336, at_version=49'22, trim_to=0'0, roll_forward_to=49'18))", 2024-09-15T03:48:08.384 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.859709+0000", 2024-09-15T03:48:08.384 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 72.437870119999999, 2024-09-15T03:48:08.384 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0037862020000001, 2024-09-15T03:48:08.384 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.384 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.385 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.385 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.385 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.385 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859709+0000", 2024-09-15T03:48:08.385 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.385 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.385 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.385 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859709+0000", 2024-09-15T03:48:08.385 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.385 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.385 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.385 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859710+0000", 2024-09-15T03:48:08.385 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.052e-06 2024-09-15T03:48:08.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.386 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.386 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.386 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859729+0000", 2024-09-15T03:48:08.386 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9409999999999999e-05 2024-09-15T03:48:08.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.386 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.386 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.386 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859759+0000", 2024-09-15T03:48:08.386 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9998e-05 2024-09-15T03:48:08.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.386 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.386 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.387 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.859767+0000", 2024-09-15T03:48:08.387 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.14e-06 2024-09-15T03:48:08.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.387 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.387 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851350+0000", 2024-09-15T03:48:08.387 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.99158261 2024-09-15T03:48:08.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.387 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.387 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851358+0000", 2024-09-15T03:48:08.387 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1070000000000005e-06 2024-09-15T03:48:08.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.388 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.388 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851443+0000", 2024-09-15T03:48:08.388 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4760999999999997e-05 2024-09-15T03:48:08.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.388 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.388 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851495+0000", 2024-09-15T03:48:08.388 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2070999999999997e-05 2024-09-15T03:48:08.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.388 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.388 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.851557+0000", 2024-09-15T03:48:08.388 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1829999999999996e-05 2024-09-15T03:48:08.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.389 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.389 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.853682+0000", 2024-09-15T03:48:08.389 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021249110000000002 2024-09-15T03:48:08.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.389 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.389 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.863495+0000", 2024-09-15T03:48:08.389 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0098133119999999994 2024-09-15T03:48:08.389 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.389 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.389 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.390 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=35, reqid=client.4521.0:331, at_version=49'15, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:48:08.390 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.860778+0000", 2024-09-15T03:48:08.390 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 72.436801224999996, 2024-09-15T03:48:08.390 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9863441239999999, 2024-09-15T03:48:08.390 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.390 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.390 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.390 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.390 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860778+0000", 2024-09-15T03:48:08.390 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.390 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.391 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860778+0000", 2024-09-15T03:48:08.391 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.391 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.391 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860779+0000", 2024-09-15T03:48:08.391 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0109999999999999e-06 2024-09-15T03:48:08.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.391 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.391 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860922+0000", 2024-09-15T03:48:08.391 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014357399999999999 2024-09-15T03:48:08.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.392 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.392 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860944+0000", 2024-09-15T03:48:08.392 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1424e-05 2024-09-15T03:48:08.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.392 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.392 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.860951+0000", 2024-09-15T03:48:08.392 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5610000000000001e-06 2024-09-15T03:48:08.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.392 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.392 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861022+0000", 2024-09-15T03:48:08.392 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1126999999999998e-05 2024-09-15T03:48:08.393 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.393 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.393 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.393 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861034+0000", 2024-09-15T03:48:08.393 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1810999999999999e-05 2024-09-15T03:48:08.393 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.393 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.393 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.393 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861164+0000", 2024-09-15T03:48:08.393 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000129727 2024-09-15T03:48:08.393 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.393 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.393 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.394 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861196+0000", 2024-09-15T03:48:08.394 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2240000000000003e-05 2024-09-15T03:48:08.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.394 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.394 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861259+0000", 2024-09-15T03:48:08.394 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2454999999999997e-05 2024-09-15T03:48:08.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.394 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.394 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861274+0000", 2024-09-15T03:48:08.394 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5588000000000001e-05 2024-09-15T03:48:08.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.395 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.395 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.847122+0000", 2024-09-15T03:48:08.395 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9858476060000001 2024-09-15T03:48:08.395 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.395 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.395 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.395 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.395 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.395 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=36, reqid=client.4521.0:332, at_version=49'16, trim_to=0'0, roll_forward_to=49'13))", 2024-09-15T03:48:08.395 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.861537+0000", 2024-09-15T03:48:08.395 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 72.436041661999994, 2024-09-15T03:48:08.395 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.985662638, 2024-09-15T03:48:08.395 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.395 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.396 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.396 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.396 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861537+0000", 2024-09-15T03:48:08.396 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.396 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.396 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861537+0000", 2024-09-15T03:48:08.396 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.396 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.397 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861538+0000", 2024-09-15T03:48:08.397 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7000000000000003e-07 2024-09-15T03:48:08.397 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.397 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.397 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.397 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861571+0000", 2024-09-15T03:48:08.397 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.2493000000000002e-05 2024-09-15T03:48:08.397 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.397 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.397 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.397 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861588+0000", 2024-09-15T03:48:08.397 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7076e-05 2024-09-15T03:48:08.397 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.397 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.398 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.398 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861595+0000", 2024-09-15T03:48:08.398 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3440000000000003e-06 2024-09-15T03:48:08.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.398 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.398 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.398 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861665+0000", 2024-09-15T03:48:08.398 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9814999999999995e-05 2024-09-15T03:48:08.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.398 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.398 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.398 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861674+0000", 2024-09-15T03:48:08.398 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9740000000000008e-06 2024-09-15T03:48:08.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.399 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.399 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861767+0000", 2024-09-15T03:48:08.399 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3572999999999993e-05 2024-09-15T03:48:08.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.399 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.399 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861778+0000", 2024-09-15T03:48:08.399 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1083e-05 2024-09-15T03:48:08.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.399 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.399 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861888+0000", 2024-09-15T03:48:08.399 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010937700000000001 2024-09-15T03:48:08.400 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.400 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.400 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.400 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.861906+0000", 2024-09-15T03:48:08.400 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.863e-05 2024-09-15T03:48:08.400 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.400 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.400 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.400 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.847200+0000", 2024-09-15T03:48:08.400 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.985293403 2024-09-15T03:48:08.400 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.400 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.401 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.401 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/31 ECSubWrite(tid=37, reqid=client.4521.0:332, at_version=0'0, trim_to=0'0, roll_forward_to=49'16))", 2024-09-15T03:48:08.401 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.864457+0000", 2024-09-15T03:48:08.401 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 72.433121972999999, 2024-09-15T03:48:08.401 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9827806690000003, 2024-09-15T03:48:08.401 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.401 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.401 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.401 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.401 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864457+0000", 2024-09-15T03:48:08.401 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.402 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.402 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864457+0000", 2024-09-15T03:48:08.402 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.402 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.402 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864458+0000", 2024-09-15T03:48:08.402 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.029e-06 2024-09-15T03:48:08.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.402 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.402 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864472+0000", 2024-09-15T03:48:08.403 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4287000000000001e-05 2024-09-15T03:48:08.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.403 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.403 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864504+0000", 2024-09-15T03:48:08.403 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1435e-05 2024-09-15T03:48:08.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.403 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.403 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864511+0000", 2024-09-15T03:48:08.403 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4900000000000003e-06 2024-09-15T03:48:08.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.403 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.404 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864583+0000", 2024-09-15T03:48:08.404 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1828000000000004e-05 2024-09-15T03:48:08.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.404 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.404 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864593+0000", 2024-09-15T03:48:08.404 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0427e-05 2024-09-15T03:48:08.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.404 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.404 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864701+0000", 2024-09-15T03:48:08.404 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000107969 2024-09-15T03:48:08.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.405 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.405 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864721+0000", 2024-09-15T03:48:08.405 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9422000000000001e-05 2024-09-15T03:48:08.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.405 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.405 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864798+0000", 2024-09-15T03:48:08.405 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6731999999999996e-05 2024-09-15T03:48:08.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.405 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.405 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864815+0000", 2024-09-15T03:48:08.405 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7286999999999999e-05 2024-09-15T03:48:08.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.406 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.406 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.847238+0000", 2024-09-15T03:48:08.406 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9824227629999998 2024-09-15T03:48:08.406 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.406 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.406 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.406 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.bs4 49/31 ECSubWrite(tid=75, reqid=client.4521.0:338, at_version=49'58, trim_to=0'0, roll_forward_to=47'56))", 2024-09-15T03:48:08.406 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.870036+0000", 2024-09-15T03:48:08.406 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 72.427543349000004, 2024-09-15T03:48:08.406 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.978548666, 2024-09-15T03:48:08.407 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.407 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.407 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.407 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.407 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870036+0000", 2024-09-15T03:48:08.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.407 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.407 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870036+0000", 2024-09-15T03:48:08.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.408 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.408 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870036+0000", 2024-09-15T03:48:08.408 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.61e-07 2024-09-15T03:48:08.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.408 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.408 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870051+0000", 2024-09-15T03:48:08.408 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5074e-05 2024-09-15T03:48:08.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.408 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.408 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870061+0000", 2024-09-15T03:48:08.408 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.066e-05 2024-09-15T03:48:08.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.409 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870067+0000", 2024-09-15T03:48:08.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1580000000000004e-06 2024-09-15T03:48:08.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.409 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870121+0000", 2024-09-15T03:48:08.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4605999999999999e-05 2024-09-15T03:48:08.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.409 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870134+0000", 2024-09-15T03:48:08.410 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2748999999999999e-05 2024-09-15T03:48:08.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.410 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.410 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870450+0000", 2024-09-15T03:48:08.410 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031624399999999998 2024-09-15T03:48:08.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.410 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.410 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870459+0000", 2024-09-15T03:48:08.410 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0499999999999997e-06 2024-09-15T03:48:08.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.410 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870496+0000", 2024-09-15T03:48:08.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6548999999999999e-05 2024-09-15T03:48:08.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.870509+0000", 2024-09-15T03:48:08.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3216e-05 2024-09-15T03:48:08.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:58.848584+0000", 2024-09-15T03:48:08.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9780751990000001 2024-09-15T03:48:08.411 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.411 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.411 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.412 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.412 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.412 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:08.413 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:48:08.414 INFO:tasks.ceph.osd.11.smithi190.stderr:2024-09-15T03:48:08.412+0000 7f23403f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 29470) UID: 0 2024-09-15T03:48:08.514 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:48:08.513+0000 7fd62abf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33736) UID: 0 2024-09-15T03:48:08.514 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:08.515 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:48:08.515 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-15T03:48:08.515 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:08.523 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:48:08.615 INFO:tasks.ceph.osd.5.smithi067.stderr:2024-09-15T03:48:08.614+0000 7f678fff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28463) UID: 0 2024-09-15T03:48:08.623 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:08.623 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:48:08.624 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-15T03:48:08.624 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-15T03:48:08.624 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:08.632 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:48:08.716 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:48:08.715+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:48:08.735 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:08.735 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-15T03:48:08.735 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-15T03:48:08.736 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-15T03:48:08.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.736 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:195 3.es0 3:7f0816e1:::smithi00132172-31:head [copy-from ver 4] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:48:08.736 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:41.112078+0000", 2024-09-15T03:48:08.736 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 87.621181614999998, 2024-09-15T03:48:08.737 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012535605, 2024-09-15T03:48:08.737 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.737 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:08.737 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:08.737 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:08.737 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:08.737 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 195 2024-09-15T03:48:08.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.737 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.738 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.738 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112078+0000", 2024-09-15T03:48:08.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.738 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.738 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112078+0000", 2024-09-15T03:48:08.738 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.738 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.738 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112080+0000", 2024-09-15T03:48:08.739 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4399999999999999e-06 2024-09-15T03:48:08.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.739 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.739 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112090+0000", 2024-09-15T03:48:08.739 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.7310000000000005e-06 2024-09-15T03:48:08.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.739 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.739 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112107+0000", 2024-09-15T03:48:08.739 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.7557000000000001e-05 2024-09-15T03:48:08.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.739 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.740 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112113+0000", 2024-09-15T03:48:08.740 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6910000000000002e-06 2024-09-15T03:48:08.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.740 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.740 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112176+0000", 2024-09-15T03:48:08.740 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3014000000000003e-05 2024-09-15T03:48:08.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.741 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:08.741 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.112308+0000", 2024-09-15T03:48:08.741 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000131402 2024-09-15T03:48:08.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.741 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.741 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118088+0000", 2024-09-15T03:48:08.741 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0057801570000000002 2024-09-15T03:48:08.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.742 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.742 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118560+0000", 2024-09-15T03:48:08.742 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00047206199999999998 2024-09-15T03:48:08.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.742 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.743 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118573+0000", 2024-09-15T03:48:08.744 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3521e-05 2024-09-15T03:48:08.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.745 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.745 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.118667+0000", 2024-09-15T03:48:08.745 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.323e-05 2024-09-15T03:48:08.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.746 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.746 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.119046+0000", 2024-09-15T03:48:08.746 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00037924699999999998 2024-09-15T03:48:08.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.747 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:08.747 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.124533+0000", 2024-09-15T03:48:08.747 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0054869849999999998 2024-09-15T03:48:08.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.747 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.747 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.747 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:41.124613+0000", 2024-09-15T03:48:08.748 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0568000000000005e-05 2024-09-15T03:48:08.748 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.748 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.748 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.749 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:384 3.es0 3:77a7da34:::smithi00132172-4:head [read 0~1146880,checksum,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e56)", 2024-09-15T03:48:08.749 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:02.864202+0000", 2024-09-15T03:48:08.749 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 65.869057835999996, 2024-09-15T03:48:08.749 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012715746999999999, 2024-09-15T03:48:08.749 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.749 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:08.750 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:08.750 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:08.750 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:08.750 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 384 2024-09-15T03:48:08.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.750 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.750 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.750 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.751 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864202+0000", 2024-09-15T03:48:08.751 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.751 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.751 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864202+0000", 2024-09-15T03:48:08.751 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.751 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.752 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.752 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864202+0000", 2024-09-15T03:48:08.752 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9400000000000004e-07 2024-09-15T03:48:08.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.752 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.752 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.752 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864206+0000", 2024-09-15T03:48:08.752 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9770000000000002e-06 2024-09-15T03:48:08.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.753 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.753 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.753 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864215+0000", 2024-09-15T03:48:08.753 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.337e-06 2024-09-15T03:48:08.753 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.753 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.753 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.753 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864221+0000", 2024-09-15T03:48:08.753 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.4570000000000002e-06 2024-09-15T03:48:08.753 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.753 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.753 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.753 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864255+0000", 2024-09-15T03:48:08.753 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3725e-05 2024-09-15T03:48:08.754 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.754 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.754 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:48:08.754 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864260+0000", 2024-09-15T03:48:08.754 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.395e-06 2024-09-15T03:48:08.754 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.754 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.754 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.755 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.871305+0000", 2024-09-15T03:48:08.755 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0070446800000000002 2024-09-15T03:48:08.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.755 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.755 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:08.755 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.871413+0000", 2024-09-15T03:48:08.755 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000107731 2024-09-15T03:48:08.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.756 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.756 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.876917+0000", 2024-09-15T03:48:08.756 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0055046510000000002 2024-09-15T03:48:08.756 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.756 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.756 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.756 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.757 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:385 3.es0 3:77a7da34:::smithi00132172-4:head [read 0~1146880,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e56)", 2024-09-15T03:48:08.757 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:02.864230+0000", 2024-09-15T03:48:08.757 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 65.869028904999993, 2024-09-15T03:48:08.757 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.014027420000000001, 2024-09-15T03:48:08.757 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.757 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:08.757 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:08.757 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:08.757 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:08.757 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 385 2024-09-15T03:48:08.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.757 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.757 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.758 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.758 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864230+0000", 2024-09-15T03:48:08.758 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.758 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.758 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864230+0000", 2024-09-15T03:48:08.758 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.758 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.759 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.759 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864231+0000", 2024-09-15T03:48:08.759 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7700000000000005e-07 2024-09-15T03:48:08.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.759 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.759 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864234+0000", 2024-09-15T03:48:08.759 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0369999999999999e-06 2024-09-15T03:48:08.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.759 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.760 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864242+0000", 2024-09-15T03:48:08.760 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4200000000000007e-06 2024-09-15T03:48:08.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.760 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.760 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.760 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864247+0000", 2024-09-15T03:48:08.760 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.4789999999999998e-06 2024-09-15T03:48:08.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.761 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.761 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864290+0000", 2024-09-15T03:48:08.761 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3594000000000003e-05 2024-09-15T03:48:08.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.761 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:48:08.761 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864297+0000", 2024-09-15T03:48:08.761 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3579999999999999e-06 2024-09-15T03:48:08.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.762 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.762 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.872828+0000", 2024-09-15T03:48:08.762 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0085315449999999998 2024-09-15T03:48:08.762 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.762 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:08.762 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.872956+0000", 2024-09-15T03:48:08.762 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000127818 2024-09-15T03:48:08.762 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.762 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.762 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.878258+0000", 2024-09-15T03:48:08.763 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0053016920000000002 2024-09-15T03:48:08.763 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.763 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.763 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.763 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.763 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.763 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:386 3.es0 3:77a7da34:::smithi00132172-4:head [read 0~1146880,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e56)", 2024-09-15T03:48:08.763 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:02.864259+0000", 2024-09-15T03:48:08.763 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 65.869000111000005, 2024-09-15T03:48:08.763 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.014933481, 2024-09-15T03:48:08.764 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.764 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:08.764 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:08.764 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:08.764 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:08.764 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 386 2024-09-15T03:48:08.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.764 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.764 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.764 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.764 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864259+0000", 2024-09-15T03:48:08.764 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.765 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.765 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864259+0000", 2024-09-15T03:48:08.765 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.765 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.765 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864260+0000", 2024-09-15T03:48:08.765 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3799999999999998e-07 2024-09-15T03:48:08.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.765 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.765 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864262+0000", 2024-09-15T03:48:08.765 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7209999999999999e-06 2024-09-15T03:48:08.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.766 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.766 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864270+0000", 2024-09-15T03:48:08.766 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6240000000000002e-06 2024-09-15T03:48:08.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.766 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.766 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864274+0000", 2024-09-15T03:48:08.767 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1699999999999999e-06 2024-09-15T03:48:08.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.767 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.767 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864316+0000", 2024-09-15T03:48:08.767 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1489999999999997e-05 2024-09-15T03:48:08.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.767 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:48:08.767 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.864321+0000", 2024-09-15T03:48:08.767 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1100000000000002e-06 2024-09-15T03:48:08.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.767 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.768 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.873888+0000", 2024-09-15T03:48:08.768 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0095676790000000008 2024-09-15T03:48:08.768 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.768 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.768 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:08.768 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.874019+0000", 2024-09-15T03:48:08.768 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013085199999999999 2024-09-15T03:48:08.768 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.768 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.768 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.768 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:02.879193+0000", 2024-09-15T03:48:08.768 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0051733969999999997 2024-09-15T03:48:08.768 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.769 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.769 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.769 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.769 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.769 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 60/31 ECSubWrite(tid=70, reqid=client.4521.0:421, at_version=60'22, trim_to=0'0, roll_forward_to=47'20))", 2024-09-15T03:48:08.769 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.837585+0000", 2024-09-15T03:48:08.769 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 61.895674819999996, 2024-09-15T03:48:08.769 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011527037, 2024-09-15T03:48:08.769 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.769 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.770 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.770 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.770 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837585+0000", 2024-09-15T03:48:08.770 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.770 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.770 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837585+0000", 2024-09-15T03:48:08.770 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.771 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837585+0000", 2024-09-15T03:48:08.771 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1199999999999999e-07 2024-09-15T03:48:08.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.771 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837609+0000", 2024-09-15T03:48:08.771 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4647999999999998e-05 2024-09-15T03:48:08.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.771 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.771 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.771 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837631+0000", 2024-09-15T03:48:08.771 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1840000000000001e-05 2024-09-15T03:48:08.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.772 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.772 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.837641+0000", 2024-09-15T03:48:08.772 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0263e-05 2024-09-15T03:48:08.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.772 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.772 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.838036+0000", 2024-09-15T03:48:08.772 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00039492500000000002 2024-09-15T03:48:08.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.772 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.772 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.838048+0000", 2024-09-15T03:48:08.773 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1599e-05 2024-09-15T03:48:08.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.773 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.773 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.773 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.838445+0000", 2024-09-15T03:48:08.773 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000397313 2024-09-15T03:48:08.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.773 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.773 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.773 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.838479+0000", 2024-09-15T03:48:08.773 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3368000000000003e-05 2024-09-15T03:48:08.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.773 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.773 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.774 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.838525+0000", 2024-09-15T03:48:08.774 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6100999999999998e-05 2024-09-15T03:48:08.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.774 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.774 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.841589+0000", 2024-09-15T03:48:08.774 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003063957 2024-09-15T03:48:08.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.774 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.774 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.849112+0000", 2024-09-15T03:48:08.774 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0075228109999999999 2024-09-15T03:48:08.774 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.774 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.775 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.775 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.775 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.775 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:446 3.es0 3:75485f06:::smithi00132172-19:head [copy-from ver 38] snapc 1a=[1a,18,17,16,15,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:48:08.775 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:06.951492+0000", 2024-09-15T03:48:08.775 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 61.781767725000002, 2024-09-15T03:48:08.775 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013140193, 2024-09-15T03:48:08.775 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.775 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:08.775 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:08.775 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:08.775 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:08.776 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 446 2024-09-15T03:48:08.776 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.776 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.776 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.776 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.776 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951492+0000", 2024-09-15T03:48:08.776 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.776 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.776 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.776 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.776 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951492+0000", 2024-09-15T03:48:08.776 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.776 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.776 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.777 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.777 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951493+0000", 2024-09-15T03:48:08.777 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.525e-06 2024-09-15T03:48:08.777 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.777 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.777 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.777 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951500+0000", 2024-09-15T03:48:08.777 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6649999999999997e-06 2024-09-15T03:48:08.777 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.777 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.777 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.777 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951514+0000", 2024-09-15T03:48:08.777 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3713999999999999e-05 2024-09-15T03:48:08.777 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.778 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.778 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.778 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951518+0000", 2024-09-15T03:48:08.778 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.921e-06 2024-09-15T03:48:08.778 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.778 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.778 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.778 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951569+0000", 2024-09-15T03:48:08.778 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.0281999999999997e-05 2024-09-15T03:48:08.778 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.778 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.778 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:08.778 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.951676+0000", 2024-09-15T03:48:08.779 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000107527 2024-09-15T03:48:08.779 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.779 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.779 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.779 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.961625+0000", 2024-09-15T03:48:08.779 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0099492209999999994 2024-09-15T03:48:08.779 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.779 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.779 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.779 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.962100+0000", 2024-09-15T03:48:08.779 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00047458100000000002 2024-09-15T03:48:08.779 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.779 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.779 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.780 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.962112+0000", 2024-09-15T03:48:08.780 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2069000000000001e-05 2024-09-15T03:48:08.780 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.780 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.780 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.780 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.962343+0000", 2024-09-15T03:48:08.780 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000230962 2024-09-15T03:48:08.780 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.780 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.780 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.780 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.962561+0000", 2024-09-15T03:48:08.780 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000217899 2024-09-15T03:48:08.780 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.781 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.781 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:08.781 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.964176+0000", 2024-09-15T03:48:08.781 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001615109 2024-09-15T03:48:08.781 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.781 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.781 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.781 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:06.964632+0000", 2024-09-15T03:48:08.781 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00045571799999999998 2024-09-15T03:48:08.781 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.782 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.782 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.782 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.782 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.782 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 65/64 ECSubWrite(tid=106, reqid=client.4521.0:464, at_version=65'25, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:48:08.782 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.852924+0000", 2024-09-15T03:48:08.782 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 56.880335197999997, 2024-09-15T03:48:08.782 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013666606, 2024-09-15T03:48:08.782 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.782 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.782 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.782 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.782 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.782 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852924+0000", 2024-09-15T03:48:08.783 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.783 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.783 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.783 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.783 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852924+0000", 2024-09-15T03:48:08.783 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.783 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.783 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.783 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.783 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852925+0000", 2024-09-15T03:48:08.783 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9999999999999997e-07 2024-09-15T03:48:08.783 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.783 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.783 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.784 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852945+0000", 2024-09-15T03:48:08.784 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9952000000000001e-05 2024-09-15T03:48:08.784 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.784 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.784 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.784 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852957+0000", 2024-09-15T03:48:08.784 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2532e-05 2024-09-15T03:48:08.784 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.784 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.784 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.784 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.852965+0000", 2024-09-15T03:48:08.784 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6359999999999994e-06 2024-09-15T03:48:08.784 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.784 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.785 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.785 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.853100+0000", 2024-09-15T03:48:08.785 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013520900000000001 2024-09-15T03:48:08.785 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.785 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.785 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.785 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.853111+0000", 2024-09-15T03:48:08.785 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0508e-05 2024-09-15T03:48:08.785 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.785 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.785 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.785 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.853220+0000", 2024-09-15T03:48:08.785 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000109053 2024-09-15T03:48:08.785 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.786 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.786 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.786 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.853247+0000", 2024-09-15T03:48:08.786 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.7025999999999998e-05 2024-09-15T03:48:08.786 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.786 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.786 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.786 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854114+0000", 2024-09-15T03:48:08.786 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00086709299999999999 2024-09-15T03:48:08.786 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.786 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.786 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.786 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854129+0000", 2024-09-15T03:48:08.786 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5602000000000001e-05 2024-09-15T03:48:08.787 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.787 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.787 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.787 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866591+0000", 2024-09-15T03:48:08.787 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012461395 2024-09-15T03:48:08.787 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.787 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.787 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.787 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.787 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.787 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 65/64 ECSubWrite(tid=107, reqid=client.4521.0:465, at_version=65'26, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:48:08.787 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.854248+0000", 2024-09-15T03:48:08.787 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 56.879010932, 2024-09-15T03:48:08.787 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012733421999999999, 2024-09-15T03:48:08.788 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.788 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.788 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.788 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.788 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.788 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854248+0000", 2024-09-15T03:48:08.788 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.788 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.788 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.788 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.788 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854248+0000", 2024-09-15T03:48:08.788 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.788 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.788 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.789 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.789 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854249+0000", 2024-09-15T03:48:08.789 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8999999999999997e-07 2024-09-15T03:48:08.789 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.789 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.789 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.789 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854274+0000", 2024-09-15T03:48:08.789 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4678e-05 2024-09-15T03:48:08.789 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.789 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.789 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.789 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854289+0000", 2024-09-15T03:48:08.789 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5622000000000001e-05 2024-09-15T03:48:08.789 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.790 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.790 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.790 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.854296+0000", 2024-09-15T03:48:08.790 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.2869999999999999e-06 2024-09-15T03:48:08.790 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.790 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.790 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.790 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859614+0000", 2024-09-15T03:48:08.790 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0053176869999999998 2024-09-15T03:48:08.790 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.790 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.790 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.790 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859625+0000", 2024-09-15T03:48:08.790 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0475000000000001e-05 2024-09-15T03:48:08.791 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.791 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.791 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.791 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859728+0000", 2024-09-15T03:48:08.791 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010383 2024-09-15T03:48:08.791 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.791 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.791 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.791 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859769+0000", 2024-09-15T03:48:08.791 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0132999999999997e-05 2024-09-15T03:48:08.791 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.791 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.791 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.792 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859855+0000", 2024-09-15T03:48:08.792 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6188999999999998e-05 2024-09-15T03:48:08.792 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.792 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.792 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.792 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860468+0000", 2024-09-15T03:48:08.792 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00061328800000000003 2024-09-15T03:48:08.792 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.792 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.792 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.792 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866982+0000", 2024-09-15T03:48:08.792 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0065137429999999998 2024-09-15T03:48:08.792 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.792 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.793 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.793 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.793 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.793 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 65/64 ECSubWrite(tid=108, reqid=client.4521.0:466, at_version=65'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:48:08.793 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.858803+0000", 2024-09-15T03:48:08.793 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 56.874456510999998, 2024-09-15T03:48:08.793 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012489828999999999, 2024-09-15T03:48:08.793 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.793 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.793 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.793 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.793 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.793 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858803+0000", 2024-09-15T03:48:08.793 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.794 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.794 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.794 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.794 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858803+0000", 2024-09-15T03:48:08.794 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.794 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.794 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.794 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.794 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858803+0000", 2024-09-15T03:48:08.794 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1699999999999999e-07 2024-09-15T03:48:08.794 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.794 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.794 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.794 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858970+0000", 2024-09-15T03:48:08.795 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000166844 2024-09-15T03:48:08.795 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.795 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.795 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.795 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858982+0000", 2024-09-15T03:48:08.795 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2092999999999999e-05 2024-09-15T03:48:08.795 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.795 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.795 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.795 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858988+0000", 2024-09-15T03:48:08.795 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1999999999999999e-06 2024-09-15T03:48:08.795 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.795 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.795 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.796 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859762+0000", 2024-09-15T03:48:08.796 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00077351199999999996 2024-09-15T03:48:08.796 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.796 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.796 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.796 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859770+0000", 2024-09-15T03:48:08.796 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0360000000000006e-06 2024-09-15T03:48:08.796 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.796 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.796 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.796 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860086+0000", 2024-09-15T03:48:08.796 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00031557999999999998 2024-09-15T03:48:08.796 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.796 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.797 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.797 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860099+0000", 2024-09-15T03:48:08.797 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3329999999999999e-05 2024-09-15T03:48:08.797 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.797 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.797 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.797 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860225+0000", 2024-09-15T03:48:08.797 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012638300000000001 2024-09-15T03:48:08.797 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.797 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.797 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.797 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861591+0000", 2024-09-15T03:48:08.797 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0013660059999999999 2024-09-15T03:48:08.797 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.798 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.798 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.798 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871293+0000", 2024-09-15T03:48:08.798 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0097014279999999998 2024-09-15T03:48:08.798 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.798 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.798 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.798 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.798 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.798 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 65/64 ECSubWrite(tid=111, reqid=client.4521.0:467, at_version=65'29, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:48:08.798 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.858996+0000", 2024-09-15T03:48:08.798 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 56.874263781000003, 2024-09-15T03:48:08.798 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012595857, 2024-09-15T03:48:08.798 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.799 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.799 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.799 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.799 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858996+0000", 2024-09-15T03:48:08.799 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.799 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.799 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.799 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858996+0000", 2024-09-15T03:48:08.799 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.799 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.799 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.799 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.858996+0000", 2024-09-15T03:48:08.800 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9900000000000001e-07 2024-09-15T03:48:08.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.800 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.800 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859088+0000", 2024-09-15T03:48:08.800 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2473999999999996e-05 2024-09-15T03:48:08.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.800 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.800 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859102+0000", 2024-09-15T03:48:08.800 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3933e-05 2024-09-15T03:48:08.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.801 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.801 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859114+0000", 2024-09-15T03:48:08.801 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1262e-05 2024-09-15T03:48:08.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.801 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.801 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860136+0000", 2024-09-15T03:48:08.801 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010221169999999999 2024-09-15T03:48:08.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.801 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.801 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860144+0000", 2024-09-15T03:48:08.801 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6270000000000006e-06 2024-09-15T03:48:08.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.802 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.802 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860264+0000", 2024-09-15T03:48:08.802 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000119954 2024-09-15T03:48:08.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.802 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.802 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860274+0000", 2024-09-15T03:48:08.802 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8959999999999998e-06 2024-09-15T03:48:08.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.802 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.802 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860365+0000", 2024-09-15T03:48:08.802 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.1195999999999999e-05 2024-09-15T03:48:08.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.803 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.803 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861615+0000", 2024-09-15T03:48:08.803 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012495690000000001 2024-09-15T03:48:08.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.803 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.803 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871591+0000", 2024-09-15T03:48:08.803 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0099764299999999997 2024-09-15T03:48:08.803 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.803 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.803 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.804 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 65/64 ECSubWrite(tid=112, reqid=client.4521.0:468, at_version=65'30, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:48:08.804 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.859121+0000", 2024-09-15T03:48:08.804 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 56.874138518999999, 2024-09-15T03:48:08.804 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012988807, 2024-09-15T03:48:08.804 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.804 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.804 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.804 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.804 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859121+0000", 2024-09-15T03:48:08.804 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.805 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.805 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859121+0000", 2024-09-15T03:48:08.805 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.805 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.805 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859121+0000", 2024-09-15T03:48:08.805 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4499999999999998e-07 2024-09-15T03:48:08.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.805 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.805 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859170+0000", 2024-09-15T03:48:08.805 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.9008999999999998e-05 2024-09-15T03:48:08.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.806 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.806 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859179+0000", 2024-09-15T03:48:08.806 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5809999999999997e-06 2024-09-15T03:48:08.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.806 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.806 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.859185+0000", 2024-09-15T03:48:08.806 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7350000000000001e-06 2024-09-15T03:48:08.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.806 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.806 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860314+0000", 2024-09-15T03:48:08.807 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001129403 2024-09-15T03:48:08.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.807 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.807 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860324+0000", 2024-09-15T03:48:08.807 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0274e-05 2024-09-15T03:48:08.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.807 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.807 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860440+0000", 2024-09-15T03:48:08.807 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011603 2024-09-15T03:48:08.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.808 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.808 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860450+0000", 2024-09-15T03:48:08.808 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0243e-05 2024-09-15T03:48:08.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.808 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.808 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.808 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860518+0000", 2024-09-15T03:48:08.808 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.7780000000000005e-05 2024-09-15T03:48:08.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.808 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.808 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.808 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861625+0000", 2024-09-15T03:48:08.808 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0011072460000000001 2024-09-15T03:48:08.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.809 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.809 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.872110+0000", 2024-09-15T03:48:08.809 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010484161000000001 2024-09-15T03:48:08.809 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.809 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.809 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.809 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 65/64 ECSubWrite(tid=113, reqid=client.4521.0:469, at_version=65'31, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:48:08.809 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.860777+0000", 2024-09-15T03:48:08.809 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 56.872481937000003, 2024-09-15T03:48:08.809 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012054540000000001, 2024-09-15T03:48:08.810 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.810 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.810 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.810 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.810 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860777+0000", 2024-09-15T03:48:08.810 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.810 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.810 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860777+0000", 2024-09-15T03:48:08.810 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.811 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.811 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860778+0000", 2024-09-15T03:48:08.811 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3399999999999999e-07 2024-09-15T03:48:08.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.811 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.811 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860795+0000", 2024-09-15T03:48:08.811 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6691e-05 2024-09-15T03:48:08.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.811 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.811 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860807+0000", 2024-09-15T03:48:08.811 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2149e-05 2024-09-15T03:48:08.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.812 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.860814+0000", 2024-09-15T03:48:08.812 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.255e-06 2024-09-15T03:48:08.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.812 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861681+0000", 2024-09-15T03:48:08.812 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00086670300000000003 2024-09-15T03:48:08.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.812 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.812 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861690+0000", 2024-09-15T03:48:08.812 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.3039999999999994e-06 2024-09-15T03:48:08.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.813 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.813 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861789+0000", 2024-09-15T03:48:08.813 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8851999999999995e-05 2024-09-15T03:48:08.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.813 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.813 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861819+0000", 2024-09-15T03:48:08.813 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9646e-05 2024-09-15T03:48:08.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.813 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.813 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861852+0000", 2024-09-15T03:48:08.814 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3436000000000003e-05 2024-09-15T03:48:08.814 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.814 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.814 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.861871+0000", 2024-09-15T03:48:08.814 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9202e-05 2024-09-15T03:48:08.814 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.814 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.814 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.872832+0000", 2024-09-15T03:48:08.814 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010960668 2024-09-15T03:48:08.814 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.814 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.814 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.815 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.fs4 65/64 ECSubWrite(tid=116, reqid=client.4521.0:469, at_version=0'0, trim_to=0'0, roll_forward_to=65'31))", 2024-09-15T03:48:08.815 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.866498+0000", 2024-09-15T03:48:08.815 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 56.866761228000001, 2024-09-15T03:48:08.815 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013459561, 2024-09-15T03:48:08.815 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.815 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:08.815 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.815 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.815 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866498+0000", 2024-09-15T03:48:08.815 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.816 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.816 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866498+0000", 2024-09-15T03:48:08.816 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.816 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.816 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866499+0000", 2024-09-15T03:48:08.816 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.9800000000000003e-07 2024-09-15T03:48:08.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.816 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.816 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866509+0000", 2024-09-15T03:48:08.816 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8980000000000008e-06 2024-09-15T03:48:08.817 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.817 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.817 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.817 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866520+0000", 2024-09-15T03:48:08.817 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0927000000000001e-05 2024-09-15T03:48:08.817 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.817 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.817 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.817 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.866528+0000", 2024-09-15T03:48:08.817 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.388e-06 2024-09-15T03:48:08.817 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.817 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.817 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.818 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870788+0000", 2024-09-15T03:48:08.818 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.004259657 2024-09-15T03:48:08.818 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.818 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.818 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.818 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870798+0000", 2024-09-15T03:48:08.818 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0156999999999999e-05 2024-09-15T03:48:08.818 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.818 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.818 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.818 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871002+0000", 2024-09-15T03:48:08.818 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000204174 2024-09-15T03:48:08.818 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.818 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.819 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.819 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871015+0000", 2024-09-15T03:48:08.819 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3422e-05 2024-09-15T03:48:08.819 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.819 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.819 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.819 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.871057+0000", 2024-09-15T03:48:08.819 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1270999999999998e-05 2024-09-15T03:48:08.819 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.819 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.819 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.819 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.872441+0000", 2024-09-15T03:48:08.819 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0013840980000000001 2024-09-15T03:48:08.819 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.820 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.820 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.820 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.879958+0000", 2024-09-15T03:48:08.820 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0075169709999999999 2024-09-15T03:48:08.820 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.820 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.820 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.820 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.820 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:488 3.es0 3:75485f06:::smithi00132172-19:head [rollback 12] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-15T03:48:08.820 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.867857+0000", 2024-09-15T03:48:08.820 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 56.865401982999998, 2024-09-15T03:48:08.820 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011714672000000001, 2024-09-15T03:48:08.821 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.821 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:08.821 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:08.821 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:08.821 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:08.821 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 488 2024-09-15T03:48:08.821 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.821 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.821 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.821 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.821 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867857+0000", 2024-09-15T03:48:08.821 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.821 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.821 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.822 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.822 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867857+0000", 2024-09-15T03:48:08.822 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.822 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.822 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867859+0000", 2024-09-15T03:48:08.822 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.562e-06 2024-09-15T03:48:08.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.822 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.822 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.822 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867874+0000", 2024-09-15T03:48:08.822 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5170999999999999e-05 2024-09-15T03:48:08.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.823 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.823 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.823 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867886+0000", 2024-09-15T03:48:08.823 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1819e-05 2024-09-15T03:48:08.823 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.823 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.823 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.823 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867892+0000", 2024-09-15T03:48:08.823 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0449999999999997e-06 2024-09-15T03:48:08.823 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.823 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.823 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.823 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868553+0000", 2024-09-15T03:48:08.823 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00066058700000000002 2024-09-15T03:48:08.824 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.824 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.824 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:08.824 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868660+0000", 2024-09-15T03:48:08.824 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000107878 2024-09-15T03:48:08.824 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.824 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.824 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.824 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869303+0000", 2024-09-15T03:48:08.824 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00064280399999999997 2024-09-15T03:48:08.824 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.824 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.824 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.825 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869419+0000", 2024-09-15T03:48:08.825 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011619400000000001 2024-09-15T03:48:08.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.825 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.825 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869431+0000", 2024-09-15T03:48:08.825 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1941e-05 2024-09-15T03:48:08.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.825 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.825 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869461+0000", 2024-09-15T03:48:08.825 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9478000000000001e-05 2024-09-15T03:48:08.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.825 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.826 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870370+0000", 2024-09-15T03:48:08.826 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00090957899999999999 2024-09-15T03:48:08.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.826 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:08.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.873494+0000", 2024-09-15T03:48:08.826 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0031240679999999998 2024-09-15T03:48:08.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.826 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.826 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.879572+0000", 2024-09-15T03:48:08.826 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0060775459999999996 2024-09-15T03:48:08.826 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.826 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.827 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.827 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.827 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:489 3.es0 3:75485f06:::smithi00132172-19:head [append 0~0] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-15T03:48:08.827 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:11.867898+0000", 2024-09-15T03:48:08.827 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 56.865361374000003, 2024-09-15T03:48:08.827 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012220945, 2024-09-15T03:48:08.827 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.827 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:08.827 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:08.827 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:08.827 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:08.827 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 489 2024-09-15T03:48:08.828 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.828 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.828 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.828 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867898+0000", 2024-09-15T03:48:08.828 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.828 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.828 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.828 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867898+0000", 2024-09-15T03:48:08.828 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.828 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.828 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.828 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.829 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867899+0000", 2024-09-15T03:48:08.829 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0900000000000001e-07 2024-09-15T03:48:08.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.829 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.829 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867901+0000", 2024-09-15T03:48:08.829 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.65e-06 2024-09-15T03:48:08.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.829 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.829 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867908+0000", 2024-09-15T03:48:08.829 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2790000000000004e-06 2024-09-15T03:48:08.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.829 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.830 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.830 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867911+0000", 2024-09-15T03:48:08.830 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9700000000000001e-06 2024-09-15T03:48:08.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.830 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.830 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.830 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869480+0000", 2024-09-15T03:48:08.830 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001568051 2024-09-15T03:48:08.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.830 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.830 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:08.830 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869565+0000", 2024-09-15T03:48:08.830 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5500999999999993e-05 2024-09-15T03:48:08.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.831 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.831 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870027+0000", 2024-09-15T03:48:08.831 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00046226900000000002 2024-09-15T03:48:08.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.831 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.831 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870140+0000", 2024-09-15T03:48:08.831 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112223 2024-09-15T03:48:08.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.831 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.831 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.831 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870168+0000", 2024-09-15T03:48:08.831 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8031e-05 2024-09-15T03:48:08.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.832 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.832 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.832 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870202+0000", 2024-09-15T03:48:08.832 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.4598999999999999e-05 2024-09-15T03:48:08.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.832 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.832 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.832 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870372+0000", 2024-09-15T03:48:08.832 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016990199999999999 2024-09-15T03:48:08.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.832 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.832 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:08.832 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.879822+0000", 2024-09-15T03:48:08.833 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0094500739999999993 2024-09-15T03:48:08.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.833 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.833 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.880119+0000", 2024-09-15T03:48:08.833 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029678700000000002 2024-09-15T03:48:08.833 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.833 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.833 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.833 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:830 3.es0 3:79526987:::smithi00132172-28:head [copy-from ver 107] snapc 3d=[3d,3b,3a,38,37,35,34,32,2d,2b,25,24,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-15T03:48:08.833 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:43.912967+0000", 2024-09-15T03:48:08.834 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 24.820292072000001, 2024-09-15T03:48:08.834 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.026454678999999998, 2024-09-15T03:48:08.834 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.834 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:08.834 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:08.834 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:08.834 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:08.834 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 830 2024-09-15T03:48:08.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.834 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.834 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.834 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.912967+0000", 2024-09-15T03:48:08.834 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.835 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.835 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.912967+0000", 2024-09-15T03:48:08.835 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.835 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.835 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.912969+0000", 2024-09-15T03:48:08.835 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.939e-06 2024-09-15T03:48:08.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.835 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.835 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.912980+0000", 2024-09-15T03:48:08.836 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.04e-05 2024-09-15T03:48:08.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.836 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.836 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.912999+0000", 2024-09-15T03:48:08.836 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9391000000000001e-05 2024-09-15T03:48:08.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.836 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.836 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913015+0000", 2024-09-15T03:48:08.836 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5568000000000001e-05 2024-09-15T03:48:08.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.836 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.837 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913068+0000", 2024-09-15T03:48:08.837 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.3270999999999999e-05 2024-09-15T03:48:08.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.837 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:08.837 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913152+0000", 2024-09-15T03:48:08.837 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.4155999999999999e-05 2024-09-15T03:48:08.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.837 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:08.837 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.934430+0000", 2024-09-15T03:48:08.837 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.021278372 2024-09-15T03:48:08.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.838 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:08.838 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.934773+0000", 2024-09-15T03:48:08.838 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00034307700000000001 2024-09-15T03:48:08.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.838 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:08.838 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.934792+0000", 2024-09-15T03:48:08.838 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8640000000000001e-05 2024-09-15T03:48:08.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.838 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:08.838 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.935203+0000", 2024-09-15T03:48:08.838 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00041061100000000002 2024-09-15T03:48:08.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.839 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:08.839 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.935232+0000", 2024-09-15T03:48:08.839 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9093000000000001e-05 2024-09-15T03:48:08.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.839 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:08.839 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.939341+0000", 2024-09-15T03:48:08.839 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0041089849999999999 2024-09-15T03:48:08.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.839 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.839 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.939422+0000", 2024-09-15T03:48:08.840 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.1175999999999997e-05 2024-09-15T03:48:08.840 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.840 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.840 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.840 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:831 3.es0 3:79526987:::smithi00132172-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2024-09-15T03:48:08.840 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:43.913039+0000", 2024-09-15T03:48:08.840 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 24.820220105000001, 2024-09-15T03:48:08.840 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.023987538999999999, 2024-09-15T03:48:08.840 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.840 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:08.840 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:08.841 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:08.841 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:08.841 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 831 2024-09-15T03:48:08.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.841 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.841 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.841 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913039+0000", 2024-09-15T03:48:08.841 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.841 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.841 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913039+0000", 2024-09-15T03:48:08.842 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.842 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.842 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913040+0000", 2024-09-15T03:48:08.842 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4900000000000005e-07 2024-09-15T03:48:08.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.842 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.842 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913044+0000", 2024-09-15T03:48:08.842 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.0550000000000001e-06 2024-09-15T03:48:08.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.842 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.842 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913053+0000", 2024-09-15T03:48:08.843 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.4150000000000005e-06 2024-09-15T03:48:08.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.843 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.843 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913058+0000", 2024-09-15T03:48:08.843 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8609999999999997e-06 2024-09-15T03:48:08.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.843 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.843 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913284+0000", 2024-09-15T03:48:08.843 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022525400000000001 2024-09-15T03:48:08.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.843 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for blocked object", 2024-09-15T03:48:08.844 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.913347+0000", 2024-09-15T03:48:08.844 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.3400999999999998e-05 2024-09-15T03:48:08.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.844 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.844 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.844 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.936662+0000", 2024-09-15T03:48:08.844 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.023315361 2024-09-15T03:48:08.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.844 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.844 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:08.844 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.936840+0000", 2024-09-15T03:48:08.844 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017759699999999999 2024-09-15T03:48:08.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.845 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.845 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:43.937027+0000", 2024-09-15T03:48:08.845 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00018684600000000001 2024-09-15T03:48:08.845 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.845 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.845 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.845 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:1016 3.es0 3:7e6c26d0:::smithi00132172-14:head [read 0~4571136,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112)", 2024-09-15T03:48:08.845 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:58.980413+0000", 2024-09-15T03:48:08.845 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 9.7528464279999998, 2024-09-15T03:48:08.845 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012968637999999999, 2024-09-15T03:48:08.845 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.846 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:08.846 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:08.846 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:08.846 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:08.846 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 1016 2024-09-15T03:48:08.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.846 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.846 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.846 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980413+0000", 2024-09-15T03:48:08.846 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.846 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.846 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.847 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980413+0000", 2024-09-15T03:48:08.847 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.847 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.847 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980416+0000", 2024-09-15T03:48:08.847 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.79e-06 2024-09-15T03:48:08.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.847 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.847 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980428+0000", 2024-09-15T03:48:08.847 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1942e-05 2024-09-15T03:48:08.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.848 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.848 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980449+0000", 2024-09-15T03:48:08.848 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1399999999999998e-05 2024-09-15T03:48:08.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.848 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.848 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.848 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980457+0000", 2024-09-15T03:48:08.848 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8709999999999997e-06 2024-09-15T03:48:08.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.848 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.848 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.848 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980543+0000", 2024-09-15T03:48:08.848 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5870000000000003e-05 2024-09-15T03:48:08.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.849 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:08.849 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980678+0000", 2024-09-15T03:48:08.849 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013535399999999999 2024-09-15T03:48:08.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.849 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.849 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.993382+0000", 2024-09-15T03:48:08.849 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012703411 2024-09-15T03:48:08.849 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.849 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.849 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.850 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.850 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:1017 3.es0 3:7e6c26d0:::smithi00132172-14:head [read 0~4571136,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112)", 2024-09-15T03:48:08.850 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:58.980469+0000", 2024-09-15T03:48:08.850 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 9.7527907589999998, 2024-09-15T03:48:08.850 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.016319225999999999, 2024-09-15T03:48:08.850 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.850 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:08.850 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:08.850 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:08.850 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:08.850 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 1017 2024-09-15T03:48:08.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.850 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.850 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.851 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.851 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980469+0000", 2024-09-15T03:48:08.851 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.851 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.851 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980469+0000", 2024-09-15T03:48:08.851 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.851 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.851 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980469+0000", 2024-09-15T03:48:08.851 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5200000000000002e-07 2024-09-15T03:48:08.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.852 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.852 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980473+0000", 2024-09-15T03:48:08.852 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7019999999999999e-06 2024-09-15T03:48:08.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.852 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.852 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980478+0000", 2024-09-15T03:48:08.852 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4770000000000002e-06 2024-09-15T03:48:08.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.852 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.852 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980481+0000", 2024-09-15T03:48:08.852 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.543e-06 2024-09-15T03:48:08.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.853 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.853 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.981078+0000", 2024-09-15T03:48:08.853 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00059686400000000001 2024-09-15T03:48:08.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.853 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:08.853 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.981163+0000", 2024-09-15T03:48:08.853 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5185999999999994e-05 2024-09-15T03:48:08.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.853 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.854 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.996788+0000", 2024-09-15T03:48:08.854 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.015625001999999999 2024-09-15T03:48:08.854 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.854 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.854 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.854 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.854 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:1018 3.es0 3:7e6c26d0:::smithi00132172-14:head [read 0~4571136,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112)", 2024-09-15T03:48:08.854 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:58.980510+0000", 2024-09-15T03:48:08.854 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 9.7527497909999994, 2024-09-15T03:48:08.854 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.019757963, 2024-09-15T03:48:08.854 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:08.854 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:08.854 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:08.855 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:08.855 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:08.855 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 1018 2024-09-15T03:48:08.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.855 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:08.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.855 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:08.855 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980510+0000", 2024-09-15T03:48:08.855 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.855 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:08.855 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980510+0000", 2024-09-15T03:48:08.855 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:08.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.856 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:08.856 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980510+0000", 2024-09-15T03:48:08.856 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7599999999999996e-07 2024-09-15T03:48:08.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.856 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:08.856 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980518+0000", 2024-09-15T03:48:08.856 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.3180000000000001e-06 2024-09-15T03:48:08.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.856 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:08.856 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980529+0000", 2024-09-15T03:48:08.856 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.1158000000000001e-05 2024-09-15T03:48:08.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.857 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:08.857 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.980535+0000", 2024-09-15T03:48:08.857 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2029999999999997e-06 2024-09-15T03:48:08.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.857 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:08.857 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.983134+0000", 2024-09-15T03:48:08.857 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00259903 2024-09-15T03:48:08.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.857 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:08.857 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:58.983239+0000", 2024-09-15T03:48:08.858 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010511800000000001 2024-09-15T03:48:08.858 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:08.858 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:08.858 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:08.858 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.000268+0000", 2024-09-15T03:48:08.858 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.01702826 2024-09-15T03:48:08.858 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.858 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.858 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.858 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:08.858 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:08.858 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:08.860 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:48:08.860 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:48:08.859+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:48:08.961 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:48:08.960+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:48:08.990 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is set 2024-09-15T03:48:09.004 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:09.004 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:48:09.004 INFO:teuthology.orchestra.run.smithi001.stdout: "num_ops": 0 2024-09-15T03:48:09.004 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:09.013 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:48:09.098 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:48:09.097+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:48:09.155 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:09.155 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [], 2024-09-15T03:48:09.155 INFO:teuthology.orchestra.run.smithi001.stdout: "complaint_time": 30, 2024-09-15T03:48:09.155 INFO:teuthology.orchestra.run.smithi001.stdout: "num_blocked_ops": 0 2024-09-15T03:48:09.155 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:09.163 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:48:09.162+0000 7fd62abf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33736) UID: 0 2024-09-15T03:48:09.164 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:48:09.264 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:48:09.263+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:48:09.306 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: "size": 20, 2024-09-15T03:48:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 600, 2024-09-15T03:48:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: "ops": [ 2024-09-15T03:48:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:333 3.9s0 3:94e9e019:::smithi00132172-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:48:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:46:55.856848+0000", 2024-09-15T03:48:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 73.447735635000001, 2024-09-15T03:48:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012151039000000001, 2024-09-15T03:48:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 333 2024-09-15T03:48:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856848+0000", 2024-09-15T03:48:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856848+0000", 2024-09-15T03:48:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856849+0000", 2024-09-15T03:48:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.5499999999999999e-07 2024-09-15T03:48:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:48:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.378e-06 2024-09-15T03:48:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:09.310 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856854+0000", 2024-09-15T03:48:09.310 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.653e-06 2024-09-15T03:48:09.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.310 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.310 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.856856+0000", 2024-09-15T03:48:09.310 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4059999999999999e-06 2024-09-15T03:48:09.310 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.862404+0000", 2024-09-15T03:48:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0055481310000000004 2024-09-15T03:48:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for rw locks", 2024-09-15T03:48:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.862519+0000", 2024-09-15T03:48:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011449400000000001 2024-09-15T03:48:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864592+0000", 2024-09-15T03:48:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020730200000000001 2024-09-15T03:48:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.864687+0000", 2024-09-15T03:48:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.5531000000000004e-05 2024-09-15T03:48:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:46:55.868999+0000", 2024-09-15T03:48:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0043121710000000001 2024-09-15T03:48:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:458 3.9s0 3:97177042:::smithi00132172-18:head [append 0~753664] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:48:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.841113+0000", 2024-09-15T03:48:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 58.463471230000003, 2024-09-15T03:48:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0334256719999999, 2024-09-15T03:48:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 458 2024-09-15T03:48:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841113+0000", 2024-09-15T03:48:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841113+0000", 2024-09-15T03:48:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841114+0000", 2024-09-15T03:48:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.491e-06 2024-09-15T03:48:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841577+0000", 2024-09-15T03:48:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000462571 2024-09-15T03:48:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841589+0000", 2024-09-15T03:48:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2119999999999999e-05 2024-09-15T03:48:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841596+0000", 2024-09-15T03:48:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0740000000000004e-06 2024-09-15T03:48:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841708+0000", 2024-09-15T03:48:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000112126 2024-09-15T03:48:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.317 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:48:09.317 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.841712+0000", 2024-09-15T03:48:09.317 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.3639999999999999e-06 2024-09-15T03:48:09.317 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.317 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.317 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.318 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.856485+0000", 2024-09-15T03:48:09.318 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0147736949999999 2024-09-15T03:48:09.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.318 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.318 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:09.318 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.856699+0000", 2024-09-15T03:48:09.318 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000213372 2024-09-15T03:48:09.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.318 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.318 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.318 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.862261+0000", 2024-09-15T03:48:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0055625450000000003 2024-09-15T03:48:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.862781+0000", 2024-09-15T03:48:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00052013500000000002 2024-09-15T03:48:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.862885+0000", 2024-09-15T03:48:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103455 2024-09-15T03:48:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.320 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.320 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.863028+0000", 2024-09-15T03:48:09.320 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014313999999999999 2024-09-15T03:48:09.320 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.320 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.321 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.864079+0000", 2024-09-15T03:48:09.321 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0010511229999999999 2024-09-15T03:48:09.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.321 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.321 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:09.321 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874458+0000", 2024-09-15T03:48:09.321 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010379019999999999 2024-09-15T03:48:09.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.321 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.321 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:09.321 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874538+0000", 2024-09-15T03:48:09.321 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0440999999999998e-05 2024-09-15T03:48:09.322 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.322 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:09.322 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.322 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.322 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.322 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:459 3.9s0 3:97177042:::smithi00132172-18:head [append 0~720896] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:48:09.322 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.846766+0000", 2024-09-15T03:48:09.322 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 58.457817607000003, 2024-09-15T03:48:09.322 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0281560110000001, 2024-09-15T03:48:09.322 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:09.322 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.322 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:09.322 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:09.322 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.323 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 459 2024-09-15T03:48:09.323 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.323 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:09.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.323 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:09.323 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.846766+0000", 2024-09-15T03:48:09.323 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.323 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.323 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:09.323 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.846766+0000", 2024-09-15T03:48:09.323 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.324 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:09.324 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.846769+0000", 2024-09-15T03:48:09.324 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.323e-06 2024-09-15T03:48:09.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.324 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:09.324 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847323+0000", 2024-09-15T03:48:09.324 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000554267 2024-09-15T03:48:09.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.324 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:09.325 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847338+0000", 2024-09-15T03:48:09.325 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.465e-05 2024-09-15T03:48:09.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.325 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.325 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.325 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847345+0000", 2024-09-15T03:48:09.325 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.6029999999999999e-06 2024-09-15T03:48:09.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.325 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.325 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.326 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847494+0000", 2024-09-15T03:48:09.326 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000148541 2024-09-15T03:48:09.326 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.326 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.326 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:48:09.326 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847498+0000", 2024-09-15T03:48:09.326 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.7629999999999998e-06 2024-09-15T03:48:09.326 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.327 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.327 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.864944+0000", 2024-09-15T03:48:09.327 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0174461050000001 2024-09-15T03:48:09.327 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.327 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:09.327 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.865122+0000", 2024-09-15T03:48:09.327 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000178123 2024-09-15T03:48:09.327 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.327 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.328 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.328 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867632+0000", 2024-09-15T03:48:09.328 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002510249 2024-09-15T03:48:09.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.328 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.328 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.867775+0000", 2024-09-15T03:48:09.328 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014326000000000001 2024-09-15T03:48:09.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.328 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.329 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868663+0000", 2024-09-15T03:48:09.329 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00088783799999999995 2024-09-15T03:48:09.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.329 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.329 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.329 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868802+0000", 2024-09-15T03:48:09.329 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000139173 2024-09-15T03:48:09.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.329 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.329 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.329 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874167+0000", 2024-09-15T03:48:09.330 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.005365117 2024-09-15T03:48:09.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.330 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:09.330 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874842+0000", 2024-09-15T03:48:09.330 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00067414199999999997 2024-09-15T03:48:09.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.330 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:09.330 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874922+0000", 2024-09-15T03:48:09.331 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0857000000000002e-05 2024-09-15T03:48:09.331 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.331 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:09.331 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.331 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:460 3.9s0 3:97177042:::smithi00132172-18:head [append 0~49152] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:48:09.331 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.847359+0000", 2024-09-15T03:48:09.331 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 58.457225233000003, 2024-09-15T03:48:09.331 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0278644889999999, 2024-09-15T03:48:09.331 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:09.332 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.332 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:09.332 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:09.332 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.332 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 460 2024-09-15T03:48:09.332 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.332 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:09.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.332 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:09.332 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847359+0000", 2024-09-15T03:48:09.332 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.333 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:09.333 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847359+0000", 2024-09-15T03:48:09.333 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.333 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:09.333 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847360+0000", 2024-09-15T03:48:09.333 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.26e-06 2024-09-15T03:48:09.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.334 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:09.334 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847390+0000", 2024-09-15T03:48:09.334 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9533999999999999e-05 2024-09-15T03:48:09.334 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.334 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.334 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:09.334 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847406+0000", 2024-09-15T03:48:09.334 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6725999999999999e-05 2024-09-15T03:48:09.334 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.334 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.335 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.335 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847412+0000", 2024-09-15T03:48:09.335 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4149999999999998e-06 2024-09-15T03:48:09.335 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.335 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.335 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.335 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847522+0000", 2024-09-15T03:48:09.335 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011041500000000001 2024-09-15T03:48:09.335 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.335 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.335 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:48:09.336 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847523+0000", 2024-09-15T03:48:09.336 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.09e-07 2024-09-15T03:48:09.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.336 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.336 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.868928+0000", 2024-09-15T03:48:09.336 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0214053089999999 2024-09-15T03:48:09.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.336 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:09.336 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869043+0000", 2024-09-15T03:48:09.337 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000115021 2024-09-15T03:48:09.337 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.337 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.337 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.337 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869635+0000", 2024-09-15T03:48:09.337 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00059164199999999997 2024-09-15T03:48:09.337 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.337 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.337 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.337 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.869755+0000", 2024-09-15T03:48:09.337 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0001205 2024-09-15T03:48:09.337 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.338 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.338 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.338 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870274+0000", 2024-09-15T03:48:09.338 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00051842900000000005 2024-09-15T03:48:09.338 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.338 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.338 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.338 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.870322+0000", 2024-09-15T03:48:09.338 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.8164000000000002e-05 2024-09-15T03:48:09.338 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.338 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.339 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.339 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.874941+0000", 2024-09-15T03:48:09.339 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0046186140000000001 2024-09-15T03:48:09.339 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.339 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.339 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:09.339 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875140+0000", 2024-09-15T03:48:09.339 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019890700000000001 2024-09-15T03:48:09.339 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.339 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.339 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:09.339 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875223+0000", 2024-09-15T03:48:09.340 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.3644000000000001e-05 2024-09-15T03:48:09.340 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.340 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:09.340 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.340 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.340 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.340 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:461 3.9s0 3:97177042:::smithi00132172-18:head [setxattr _header (57) in=7b] snapc 21=[21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:48:09.340 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.847417+0000", 2024-09-15T03:48:09.340 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 58.457167128999998, 2024-09-15T03:48:09.340 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0332682499999999, 2024-09-15T03:48:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 461 2024-09-15T03:48:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847417+0000", 2024-09-15T03:48:09.341 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.342 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.342 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.342 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:09.342 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847417+0000", 2024-09-15T03:48:09.342 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.342 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.342 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.342 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:09.342 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847418+0000", 2024-09-15T03:48:09.342 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.8999999999999995e-07 2024-09-15T03:48:09.342 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.343 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.343 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:09.343 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847421+0000", 2024-09-15T03:48:09.343 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.1769999999999998e-06 2024-09-15T03:48:09.343 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.343 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.343 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:09.343 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847428+0000", 2024-09-15T03:48:09.343 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6969999999999996e-06 2024-09-15T03:48:09.343 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.343 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.343 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.344 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847432+0000", 2024-09-15T03:48:09.344 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3209999999999998e-06 2024-09-15T03:48:09.344 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.344 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.344 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.344 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847542+0000", 2024-09-15T03:48:09.344 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011035999999999999 2024-09-15T03:48:09.344 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.344 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.344 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:48:09.345 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847543+0000", 2024-09-15T03:48:09.345 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.1200000000000002e-07 2024-09-15T03:48:09.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.345 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.345 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.345 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875612+0000", 2024-09-15T03:48:09.345 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0280692330000001 2024-09-15T03:48:09.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.345 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.345 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:09.345 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.875735+0000", 2024-09-15T03:48:09.346 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012276600000000001 2024-09-15T03:48:09.346 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.346 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.346 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.346 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876307+0000", 2024-09-15T03:48:09.346 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00057198600000000002 2024-09-15T03:48:09.346 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.346 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.346 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.346 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876408+0000", 2024-09-15T03:48:09.346 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00010148400000000001 2024-09-15T03:48:09.347 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.347 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.347 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.347 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876508+0000", 2024-09-15T03:48:09.347 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.9328999999999997e-05 2024-09-15T03:48:09.347 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.347 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.347 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.347 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876546+0000", 2024-09-15T03:48:09.347 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8361000000000001e-05 2024-09-15T03:48:09.347 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.348 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.348 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.348 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876602+0000", 2024-09-15T03:48:09.348 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.6412999999999997e-05 2024-09-15T03:48:09.348 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.348 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.348 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:09.348 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.880589+0000", 2024-09-15T03:48:09.348 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0039866190000000003 2024-09-15T03:48:09.348 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.348 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.349 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:09.349 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.880685+0000", 2024-09-15T03:48:09.349 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.6001999999999993e-05 2024-09-15T03:48:09.349 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.349 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:09.349 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.349 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.349 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.349 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:462 3.9s0 3:97177042:::smithi00132172-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-15T03:48:09.349 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:10.847436+0000", 2024-09-15T03:48:09.349 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 58.457148171, 2024-09-15T03:48:09.350 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.03734576, 2024-09-15T03:48:09.350 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:09.350 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:09.350 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:09.350 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:09.350 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.350 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 462 2024-09-15T03:48:09.350 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.350 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:09.350 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.350 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:09.350 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847436+0000", 2024-09-15T03:48:09.351 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.351 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.351 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.351 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:09.351 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847436+0000", 2024-09-15T03:48:09.351 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.351 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.351 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.351 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:09.351 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847436+0000", 2024-09-15T03:48:09.351 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.3099999999999998e-07 2024-09-15T03:48:09.352 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.352 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:09.352 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847438+0000", 2024-09-15T03:48:09.352 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.9570000000000001e-06 2024-09-15T03:48:09.352 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.352 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:09.352 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847445+0000", 2024-09-15T03:48:09.352 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.238e-06 2024-09-15T03:48:09.353 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.353 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.353 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.353 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847454+0000", 2024-09-15T03:48:09.353 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5930000000000006e-06 2024-09-15T03:48:09.353 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.353 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.353 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.353 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847562+0000", 2024-09-15T03:48:09.353 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000108448 2024-09-15T03:48:09.354 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.354 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:48:09.354 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:10.847563+0000", 2024-09-15T03:48:09.354 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.0200000000000002e-07 2024-09-15T03:48:09.354 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.354 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.354 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.354 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876482+0000", 2024-09-15T03:48:09.354 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.028918658 2024-09-15T03:48:09.354 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.355 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for rw locks", 2024-09-15T03:48:09.355 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.876593+0000", 2024-09-15T03:48:09.355 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011145300000000001 2024-09-15T03:48:09.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.355 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.355 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.881189+0000", 2024-09-15T03:48:09.355 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0045956139999999996 2024-09-15T03:48:09.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.356 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:09.356 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.881293+0000", 2024-09-15T03:48:09.356 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000103875 2024-09-15T03:48:09.356 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.356 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:09.356 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:11.884782+0000", 2024-09-15T03:48:09.356 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.003488891 2024-09-15T03:48:09.356 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.356 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:09.357 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.357 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.357 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:561 3.9s0 3:9cd57211:::smithi00132172-16:head [append 0~557056] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:48:09.357 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:20.935324+0000", 2024-09-15T03:48:09.357 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 48.36926047, 2024-09-15T03:48:09.357 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012459909999999999, 2024-09-15T03:48:09.357 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:09.357 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.357 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:09.357 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:09.358 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.358 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 561 2024-09-15T03:48:09.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.358 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:09.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.358 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:09.358 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935324+0000", 2024-09-15T03:48:09.358 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.359 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:09.359 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935324+0000", 2024-09-15T03:48:09.359 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.359 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:09.359 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935326+0000", 2024-09-15T03:48:09.359 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8909999999999999e-06 2024-09-15T03:48:09.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.359 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:09.360 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935587+0000", 2024-09-15T03:48:09.360 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026019699999999999 2024-09-15T03:48:09.360 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.360 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.360 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:09.360 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935610+0000", 2024-09-15T03:48:09.360 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2949e-05 2024-09-15T03:48:09.360 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.360 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.360 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.360 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935618+0000", 2024-09-15T03:48:09.360 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.6289999999999999e-06 2024-09-15T03:48:09.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.361 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.361 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.361 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935688+0000", 2024-09-15T03:48:09.361 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.9801999999999993e-05 2024-09-15T03:48:09.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.361 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.361 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-15T03:48:09.361 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.935698+0000", 2024-09-15T03:48:09.361 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0103e-05 2024-09-15T03:48:09.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.362 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.362 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939626+0000", 2024-09-15T03:48:09.362 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0039276759999999997 2024-09-15T03:48:09.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.362 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:09.362 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939844+0000", 2024-09-15T03:48:09.362 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000217741 2024-09-15T03:48:09.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.363 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.363 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941127+0000", 2024-09-15T03:48:09.363 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001283719 2024-09-15T03:48:09.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.363 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.363 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941395+0000", 2024-09-15T03:48:09.363 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00026823800000000002 2024-09-15T03:48:09.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.364 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.364 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941448+0000", 2024-09-15T03:48:09.364 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2751999999999999e-05 2024-09-15T03:48:09.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.364 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.364 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941646+0000", 2024-09-15T03:48:09.364 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019804100000000001 2024-09-15T03:48:09.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.365 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.365 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943664+0000", 2024-09-15T03:48:09.365 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0020181090000000001 2024-09-15T03:48:09.365 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.365 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.365 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:09.365 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.947709+0000", 2024-09-15T03:48:09.365 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0040445059999999998 2024-09-15T03:48:09.365 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.365 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.365 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:09.366 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.947783+0000", 2024-09-15T03:48:09.366 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4556999999999998e-05 2024-09-15T03:48:09.366 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.366 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:09.366 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.366 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.366 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.366 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:562 3.9s0 3:9cd57211:::smithi00132172-16:head [append 0~638976] snapc 28=[28,27,26,25,24,23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:48:09.366 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:20.938590+0000", 2024-09-15T03:48:09.366 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 48.365993686000003, 2024-09-15T03:48:09.366 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011368382, 2024-09-15T03:48:09.366 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:09.367 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.367 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:09.367 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:09.367 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.367 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 562 2024-09-15T03:48:09.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.367 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:09.367 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.367 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:09.367 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.938590+0000", 2024-09-15T03:48:09.367 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.368 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.368 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:09.368 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.938590+0000", 2024-09-15T03:48:09.368 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.368 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.368 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.368 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:09.368 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.938592+0000", 2024-09-15T03:48:09.368 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.623e-06 2024-09-15T03:48:09.368 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.368 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.368 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:09.369 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939253+0000", 2024-09-15T03:48:09.369 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00066056899999999996 2024-09-15T03:48:09.369 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.369 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.369 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:09.369 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939271+0000", 2024-09-15T03:48:09.369 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8482e-05 2024-09-15T03:48:09.369 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.369 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.369 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.369 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.939280+0000", 2024-09-15T03:48:09.370 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5569999999999996e-06 2024-09-15T03:48:09.370 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.370 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.370 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.370 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941736+0000", 2024-09-15T03:48:09.370 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0024565640000000001 2024-09-15T03:48:09.370 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.370 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.370 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:09.370 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.941861+0000", 2024-09-15T03:48:09.370 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012484 2024-09-15T03:48:09.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.371 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.371 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943223+0000", 2024-09-15T03:48:09.371 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0013619260000000001 2024-09-15T03:48:09.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.371 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.371 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943354+0000", 2024-09-15T03:48:09.371 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00013100399999999999 2024-09-15T03:48:09.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.372 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.372 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943376+0000", 2024-09-15T03:48:09.372 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.1829999999999999e-05 2024-09-15T03:48:09.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.372 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.372 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943579+0000", 2024-09-15T03:48:09.372 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020374400000000001 2024-09-15T03:48:09.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.372 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.373 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943904+0000", 2024-09-15T03:48:09.373 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000324846 2024-09-15T03:48:09.373 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.373 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:09.373 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949897+0000", 2024-09-15T03:48:09.373 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0059927879999999998 2024-09-15T03:48:09.373 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.373 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:09.374 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.949959+0000", 2024-09-15T03:48:09.374 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1608999999999999e-05 2024-09-15T03:48:09.374 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.374 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:09.374 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.374 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:565 3.9s0 3:9cd57211:::smithi00132172-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-15T03:48:09.374 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:20.943038+0000", 2024-09-15T03:48:09.374 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 48.361546341999997, 2024-09-15T03:48:09.374 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011190904, 2024-09-15T03:48:09.375 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:09.375 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:09.375 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:09.375 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:09.375 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.375 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 565 2024-09-15T03:48:09.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.375 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:09.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.375 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:09.375 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943038+0000", 2024-09-15T03:48:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943038+0000", 2024-09-15T03:48:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943038+0000", 2024-09-15T03:48:09.376 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.6800000000000001e-07 2024-09-15T03:48:09.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.377 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:09.377 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943040+0000", 2024-09-15T03:48:09.377 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3700000000000002e-06 2024-09-15T03:48:09.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.377 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:09.377 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943047+0000", 2024-09-15T03:48:09.377 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.1299999999999998e-06 2024-09-15T03:48:09.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.378 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.378 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.943050+0000", 2024-09-15T03:48:09.378 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.6229999999999999e-06 2024-09-15T03:48:09.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.378 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.379 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.947990+0000", 2024-09-15T03:48:09.379 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0049398690000000004 2024-09-15T03:48:09.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.379 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for rw locks", 2024-09-15T03:48:09.379 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.948080+0000", 2024-09-15T03:48:09.379 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.9721000000000004e-05 2024-09-15T03:48:09.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.379 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.379 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.952263+0000", 2024-09-15T03:48:09.379 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.004183417 2024-09-15T03:48:09.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.380 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:09.380 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.952374+0000", 2024-09-15T03:48:09.380 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000110435 2024-09-15T03:48:09.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.380 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:09.380 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:20.954229+0000", 2024-09-15T03:48:09.380 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0018548709999999999 2024-09-15T03:48:09.380 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.381 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:09.381 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.381 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:668 3.4s0 3:2d1c787c:::smithi00132172-45:head [read 0~1703936,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:48:09.381 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:29.978875+0000", 2024-09-15T03:48:09.381 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 39.325708624999997, 2024-09-15T03:48:09.381 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011314753, 2024-09-15T03:48:09.381 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:09.381 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:09.381 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:09.382 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:09.382 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.382 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 668 2024-09-15T03:48:09.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.382 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:09.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.382 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:09.382 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978875+0000", 2024-09-15T03:48:09.382 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.383 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:09.383 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978875+0000", 2024-09-15T03:48:09.383 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.383 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:09.383 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978876+0000", 2024-09-15T03:48:09.383 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.46e-07 2024-09-15T03:48:09.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.384 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:09.384 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978878+0000", 2024-09-15T03:48:09.384 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.4399999999999999e-06 2024-09-15T03:48:09.384 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.384 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.384 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:09.384 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978884+0000", 2024-09-15T03:48:09.384 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8930000000000002e-06 2024-09-15T03:48:09.384 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.384 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.384 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.385 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.978887+0000", 2024-09-15T03:48:09.385 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.8040000000000002e-06 2024-09-15T03:48:09.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.385 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.385 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.385 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.979013+0000", 2024-09-15T03:48:09.385 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00012650899999999999 2024-09-15T03:48:09.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.385 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.386 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:48:09.386 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.979020+0000", 2024-09-15T03:48:09.386 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8159999999999996e-06 2024-09-15T03:48:09.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.386 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.386 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.386 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.981562+0000", 2024-09-15T03:48:09.386 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.002542267 2024-09-15T03:48:09.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.387 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:09.387 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.981653+0000", 2024-09-15T03:48:09.387 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.0509999999999999e-05 2024-09-15T03:48:09.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.387 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:09.387 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:29.990190+0000", 2024-09-15T03:48:09.387 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0085371679999999995 2024-09-15T03:48:09.387 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.387 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:09.388 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.388 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:692 3.9s0 3:97177042:::smithi00132172-18:head [copy-from ver 4] snapc 2f=[2d,2b,29,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:48:09.388 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.088835+0000", 2024-09-15T03:48:09.388 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 39.215749318999997, 2024-09-15T03:48:09.388 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.016647229999999999, 2024-09-15T03:48:09.388 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:09.388 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.388 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:09.389 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:09.389 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.389 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 692 2024-09-15T03:48:09.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.389 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:09.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.389 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:09.389 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088835+0000", 2024-09-15T03:48:09.389 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.390 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:09.390 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088835+0000", 2024-09-15T03:48:09.390 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.390 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:09.390 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088837+0000", 2024-09-15T03:48:09.390 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8670000000000001e-06 2024-09-15T03:48:09.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.390 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:09.391 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088843+0000", 2024-09-15T03:48:09.391 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8179999999999998e-06 2024-09-15T03:48:09.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.391 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:09.391 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088860+0000", 2024-09-15T03:48:09.391 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6566999999999999e-05 2024-09-15T03:48:09.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.392 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.392 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088881+0000", 2024-09-15T03:48:09.392 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0619999999999999e-05 2024-09-15T03:48:09.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.392 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.392 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.088963+0000", 2024-09-15T03:48:09.392 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.2650000000000003e-05 2024-09-15T03:48:09.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.393 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.393 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:09.393 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089325+0000", 2024-09-15T03:48:09.393 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00036168700000000002 2024-09-15T03:48:09.393 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.393 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.393 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.393 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102185+0000", 2024-09-15T03:48:09.393 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012859786 2024-09-15T03:48:09.393 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.394 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.394 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102633+0000", 2024-09-15T03:48:09.394 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00044844699999999998 2024-09-15T03:48:09.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.394 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.394 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102713+0000", 2024-09-15T03:48:09.394 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9616999999999993e-05 2024-09-15T03:48:09.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.394 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.395 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102771+0000", 2024-09-15T03:48:09.395 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7797999999999998e-05 2024-09-15T03:48:09.395 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.395 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.395 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.395 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102793+0000", 2024-09-15T03:48:09.395 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.2827999999999999e-05 2024-09-15T03:48:09.395 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.395 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.395 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:09.395 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.104072+0000", 2024-09-15T03:48:09.396 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0012787759999999999 2024-09-15T03:48:09.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.396 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:09.396 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.105482+0000", 2024-09-15T03:48:09.396 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001409769 2024-09-15T03:48:09.396 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.396 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:09.396 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.397 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.397 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:693 3.9s0 3:97177042:::smithi00132172-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:48:09.397 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.089498+0000", 2024-09-15T03:48:09.397 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 39.215086384000003, 2024-09-15T03:48:09.397 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013753056, 2024-09-15T03:48:09.397 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:09.397 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:09.397 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:09.397 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:09.397 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.397 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 693 2024-09-15T03:48:09.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.398 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:09.398 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.398 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:09.398 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089498+0000", 2024-09-15T03:48:09.398 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.398 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.398 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:09.398 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089498+0000", 2024-09-15T03:48:09.398 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.399 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:09.399 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089499+0000", 2024-09-15T03:48:09.399 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.074e-06 2024-09-15T03:48:09.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.399 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:09.399 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089503+0000", 2024-09-15T03:48:09.399 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.1470000000000003e-06 2024-09-15T03:48:09.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.400 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:09.400 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089513+0000", 2024-09-15T03:48:09.400 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0248000000000001e-05 2024-09-15T03:48:09.400 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.400 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.400 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.400 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089519+0000", 2024-09-15T03:48:09.400 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.2149999999999998e-06 2024-09-15T03:48:09.400 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.400 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.400 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.401 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089719+0000", 2024-09-15T03:48:09.401 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00019960400000000001 2024-09-15T03:48:09.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.401 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for blocked object", 2024-09-15T03:48:09.401 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.089776+0000", 2024-09-15T03:48:09.401 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.7123000000000002e-05 2024-09-15T03:48:09.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.401 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.401 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102827+0000", 2024-09-15T03:48:09.402 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013050858 2024-09-15T03:48:09.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.402 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:09.402 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.102976+0000", 2024-09-15T03:48:09.402 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014900800000000001 2024-09-15T03:48:09.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.402 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:09.402 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.103251+0000", 2024-09-15T03:48:09.402 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00027477900000000001 2024-09-15T03:48:09.403 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.403 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:09.403 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.403 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.403 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:705 3.8s0 3:1af4eaf3:::smithi00132172-20:head [append 0~0] snapc 2f=[2d,2b,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-15T03:48:09.403 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:30.982870+0000", 2024-09-15T03:48:09.403 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 38.321713973000001, 2024-09-15T03:48:09.403 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.01140421, 2024-09-15T03:48:09.403 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:09.403 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.403 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:09.404 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:09.404 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.404 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 705 2024-09-15T03:48:09.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.404 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:09.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.404 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:09.404 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982870+0000", 2024-09-15T03:48:09.404 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.405 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:09.405 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982870+0000", 2024-09-15T03:48:09.405 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.405 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:09.405 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982871+0000", 2024-09-15T03:48:09.405 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.2900000000000004e-07 2024-09-15T03:48:09.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.405 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:09.405 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982874+0000", 2024-09-15T03:48:09.406 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9610000000000001e-06 2024-09-15T03:48:09.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.406 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:09.406 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982878+0000", 2024-09-15T03:48:09.406 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.7620000000000003e-06 2024-09-15T03:48:09.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.406 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.406 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982881+0000", 2024-09-15T03:48:09.406 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.112e-06 2024-09-15T03:48:09.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.407 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.407 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982995+0000", 2024-09-15T03:48:09.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000113544 2024-09-15T03:48:09.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.407 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:48:09.407 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.982999+0000", 2024-09-15T03:48:09.407 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.9339999999999999e-06 2024-09-15T03:48:09.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.408 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.408 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986479+0000", 2024-09-15T03:48:09.408 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0034799399999999999 2024-09-15T03:48:09.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.408 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:09.408 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.986599+0000", 2024-09-15T03:48:09.408 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000120364 2024-09-15T03:48:09.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.408 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.408 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987178+0000", 2024-09-15T03:48:09.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00057913799999999998 2024-09-15T03:48:09.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.409 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987321+0000", 2024-09-15T03:48:09.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000142208 2024-09-15T03:48:09.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.409 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.409 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987335+0000", 2024-09-15T03:48:09.409 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.4578e-05 2024-09-15T03:48:09.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.410 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.410 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987690+0000", 2024-09-15T03:48:09.410 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00035494600000000003 2024-09-15T03:48:09.410 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.410 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.987799+0000", 2024-09-15T03:48:09.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000109379 2024-09-15T03:48:09.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:09.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.991063+0000", 2024-09-15T03:48:09.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032640249999999998 2024-09-15T03:48:09.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.411 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:09.411 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:30.994274+0000", 2024-09-15T03:48:09.411 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0032107899999999998 2024-09-15T03:48:09.412 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.412 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:09.412 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.412 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.412 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.412 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:708 3.4s0 3:2d1c787c:::smithi00132172-45:head [copy-from ver 28] snapc 32=[32,2d,2b,26,25,24,23,22,21,20,18,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-15T03:48:09.412 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:31.983317+0000", 2024-09-15T03:48:09.412 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 37.32126744, 2024-09-15T03:48:09.412 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.021360130000000001, 2024-09-15T03:48:09.412 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:09.412 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.413 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:09.413 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:09.413 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.413 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 708 2024-09-15T03:48:09.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.413 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:09.413 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.413 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:09.413 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983317+0000", 2024-09-15T03:48:09.413 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.414 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:09.414 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983317+0000", 2024-09-15T03:48:09.414 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.414 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.414 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:09.414 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983318+0000", 2024-09-15T03:48:09.414 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2780000000000001e-06 2024-09-15T03:48:09.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.415 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:09.415 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983331+0000", 2024-09-15T03:48:09.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2821e-05 2024-09-15T03:48:09.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.415 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:09.415 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983349+0000", 2024-09-15T03:48:09.415 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8544999999999999e-05 2024-09-15T03:48:09.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.416 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.416 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983359+0000", 2024-09-15T03:48:09.416 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.8570000000000007e-06 2024-09-15T03:48:09.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.416 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.416 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983434+0000", 2024-09-15T03:48:09.416 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.4556999999999998e-05 2024-09-15T03:48:09.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.416 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.416 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "op must wait for map", 2024-09-15T03:48:09.417 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983452+0000", 2024-09-15T03:48:09.417 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8439000000000001e-05 2024-09-15T03:48:09.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.417 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.417 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.985636+0000", 2024-09-15T03:48:09.417 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0021841320000000001 2024-09-15T03:48:09.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.417 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.417 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:09.417 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.985755+0000", 2024-09-15T03:48:09.418 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000118363 2024-09-15T03:48:09.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.418 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.418 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.998959+0000", 2024-09-15T03:48:09.418 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013204053 2024-09-15T03:48:09.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.418 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.418 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.999917+0000", 2024-09-15T03:48:09.418 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00095851300000000003 2024-09-15T03:48:09.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.419 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.419 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.999931+0000", 2024-09-15T03:48:09.419 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.3631999999999999e-05 2024-09-15T03:48:09.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.419 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.419 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000553+0000", 2024-09-15T03:48:09.419 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00062246600000000001 2024-09-15T03:48:09.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.420 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.420 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000695+0000", 2024-09-15T03:48:09.420 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014162499999999999 2024-09-15T03:48:09.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.420 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:09.420 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.004291+0000", 2024-09-15T03:48:09.420 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0035959289999999999 2024-09-15T03:48:09.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.421 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:09.421 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.004677+0000", 2024-09-15T03:48:09.421 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00038591999999999999 2024-09-15T03:48:09.421 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.421 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:09.421 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.421 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:709 3.4s0 3:2d1c787c:::smithi00132172-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-15T03:48:09.421 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:31.983373+0000", 2024-09-15T03:48:09.421 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 37.321210598999997, 2024-09-15T03:48:09.422 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.017317583000000001, 2024-09-15T03:48:09.422 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:09.422 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:09.422 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:09.422 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:09.422 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.422 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 709 2024-09-15T03:48:09.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.422 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:09.422 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.422 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:09.423 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983373+0000", 2024-09-15T03:48:09.423 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.423 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:09.423 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983373+0000", 2024-09-15T03:48:09.423 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.423 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:09.423 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983374+0000", 2024-09-15T03:48:09.423 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.8599999999999998e-07 2024-09-15T03:48:09.424 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.424 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:09.424 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983377+0000", 2024-09-15T03:48:09.424 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0139999999999999e-06 2024-09-15T03:48:09.424 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.424 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:09.424 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983388+0000", 2024-09-15T03:48:09.424 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0988999999999999e-05 2024-09-15T03:48:09.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.425 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.425 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983392+0000", 2024-09-15T03:48:09.425 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 4.5059999999999997e-06 2024-09-15T03:48:09.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.425 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.425 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.425 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983542+0000", 2024-09-15T03:48:09.425 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000149887 2024-09-15T03:48:09.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.426 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:48:09.426 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.983550+0000", 2024-09-15T03:48:09.426 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.9589999999999995e-06 2024-09-15T03:48:09.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.426 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.426 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.985938+0000", 2024-09-15T03:48:09.426 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0023879930000000001 2024-09-15T03:48:09.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.427 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for blocked object", 2024-09-15T03:48:09.427 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:31.986005+0000", 2024-09-15T03:48:09.427 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6926000000000003e-05 2024-09-15T03:48:09.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.427 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.427 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000370+0000", 2024-09-15T03:48:09.427 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.014364656999999999 2024-09-15T03:48:09.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.428 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:09.428 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000485+0000", 2024-09-15T03:48:09.428 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00011524899999999999 2024-09-15T03:48:09.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.428 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:09.428 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:32.000691+0000", 2024-09-15T03:48:09.428 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00020581700000000001 2024-09-15T03:48:09.428 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.429 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:09.429 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.429 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(client.4521.0:761 3.9s0 3:9cd57211:::smithi00132172-16:head [append 0~507904] snapc 32=[32,2d,2b,26,25,24,23,22,21,20,18,16,12,11,3,2] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T03:48:09.429 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:34.955885+0000", 2024-09-15T03:48:09.429 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 34.348698656000003, 2024-09-15T03:48:09.429 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.011160715, 2024-09-15T03:48:09.429 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:09.430 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.430 INFO:teuthology.orchestra.run.smithi001.stdout: "client_info": { 2024-09-15T03:48:09.430 INFO:teuthology.orchestra.run.smithi001.stdout: "client": "client.4521", 2024-09-15T03:48:09.430 INFO:teuthology.orchestra.run.smithi001.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.430 INFO:teuthology.orchestra.run.smithi001.stdout: "tid": 761 2024-09-15T03:48:09.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.430 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:09.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.430 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:09.430 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.955885+0000", 2024-09-15T03:48:09.430 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.431 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:09.431 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.955885+0000", 2024-09-15T03:48:09.431 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.431 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:09.431 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.955887+0000", 2024-09-15T03:48:09.431 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8169999999999999e-06 2024-09-15T03:48:09.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.432 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:09.432 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956332+0000", 2024-09-15T03:48:09.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00044459600000000001 2024-09-15T03:48:09.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.432 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:09.432 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956347+0000", 2024-09-15T03:48:09.432 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5685999999999999e-05 2024-09-15T03:48:09.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956354+0000", 2024-09-15T03:48:09.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.6810000000000001e-06 2024-09-15T03:48:09.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.433 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.433 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956805+0000", 2024-09-15T03:48:09.433 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00045055300000000002 2024-09-15T03:48:09.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:09.434 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.956981+0000", 2024-09-15T03:48:09.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00017590100000000001 2024-09-15T03:48:09.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.434 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.958362+0000", 2024-09-15T03:48:09.434 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.001381415 2024-09-15T03:48:09.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.434 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.435 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.958524+0000", 2024-09-15T03:48:09.435 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00016167800000000001 2024-09-15T03:48:09.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.435 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.435 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.435 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.958797+0000", 2024-09-15T03:48:09.435 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000273274 2024-09-15T03:48:09.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.435 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.435 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.435 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.959021+0000", 2024-09-15T03:48:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00022373099999999999 2024-09-15T03:48:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.961002+0000", 2024-09-15T03:48:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0019813069999999999 2024-09-15T03:48:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_sent", 2024-09-15T03:48:09.436 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.966647+0000", 2024-09-15T03:48:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00564541 2024-09-15T03:48:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:34.967046+0000", 2024-09-15T03:48:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00039866599999999999 2024-09-15T03:48:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.438 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "osd_op(osd.9.17:1 3.9s0 3:94e9e019:::smithi00132172-30:head [assert-version v16,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e107)", 2024-09-15T03:48:09.438 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:53.919800+0000", 2024-09-15T03:48:09.438 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 15.384783931999999, 2024-09-15T03:48:09.438 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.27396664300000001, 2024-09-15T03:48:09.438 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:09.438 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "started", 2024-09-15T03:48:09.438 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:09.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.438 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:09.438 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.919800+0000", 2024-09-15T03:48:09.440 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.440 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:09.440 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.919800+0000", 2024-09-15T03:48:09.440 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.440 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.919802+0000", 2024-09-15T03:48:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.787e-06 2024-09-15T03:48:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.919807+0000", 2024-09-15T03:48:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.4519999999999996e-06 2024-09-15T03:48:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.441 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:09.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.919823+0000", 2024-09-15T03:48:09.442 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5349000000000002e-05 2024-09-15T03:48:09.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.442 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.919829+0000", 2024-09-15T03:48:09.442 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.562e-06 2024-09-15T03:48:09.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.442 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.920022+0000", 2024-09-15T03:48:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000192504 2024-09-15T03:48:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "waiting for peered", 2024-09-15T03:48:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:53.920025+0000", 2024-09-15T03:48:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.5609999999999999e-06 2024-09-15T03:48:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.443 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:54.183165+0000", 2024-09-15T03:48:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.26313930800000002 2024-09-15T03:48:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "started", 2024-09-15T03:48:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:54.183525+0000", 2024-09-15T03:48:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00036087700000000001 2024-09-15T03:48:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:09.444 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:54.193767+0000", 2024-09-15T03:48:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010241243000000001 2024-09-15T03:48:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.0s4 113/108 ECSubWrite(tid=157, reqid=osd.1.0:157, at_version=113'56, trim_to=0'0, roll_forward_to=111'54))", 2024-09-15T03:48:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:59.946531+0000", 2024-09-15T03:48:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 9.3580532059999992, 2024-09-15T03:48:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013185570000000001, 2024-09-15T03:48:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:09.445 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.946531+0000", 2024-09-15T03:48:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.946531+0000", 2024-09-15T03:48:09.446 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.946531+0000", 2024-09-15T03:48:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.03e-07 2024-09-15T03:48:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.946550+0000", 2024-09-15T03:48:09.447 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.8725000000000001e-05 2024-09-15T03:48:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.946558+0000", 2024-09-15T03:48:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.7760000000000003e-06 2024-09-15T03:48:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.946564+0000", 2024-09-15T03:48:09.448 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 5.1950000000000002e-06 2024-09-15T03:48:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.947093+0000", 2024-09-15T03:48:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00052902899999999998 2024-09-15T03:48:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.947103+0000", 2024-09-15T03:48:09.449 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0023e-05 2024-09-15T03:48:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.947384+0000", 2024-09-15T03:48:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000281048 2024-09-15T03:48:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.947414+0000", 2024-09-15T03:48:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.9745000000000001e-05 2024-09-15T03:48:09.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.947557+0000", 2024-09-15T03:48:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00014375999999999999 2024-09-15T03:48:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.947578+0000", 2024-09-15T03:48:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.0310999999999999e-05 2024-09-15T03:48:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.959716+0000", 2024-09-15T03:48:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012138754999999999 2024-09-15T03:48:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.1s2 113/108 ECSubWrite(tid=204, reqid=osd.6.0:202, at_version=0'0, trim_to=0'0, roll_forward_to=113'29))", 2024-09-15T03:48:09.452 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:59.950561+0000", 2024-09-15T03:48:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 9.3540228429999992, 2024-09-15T03:48:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.013346731000000001, 2024-09-15T03:48:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.950561+0000", 2024-09-15T03:48:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.453 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.950561+0000", 2024-09-15T03:48:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.950562+0000", 2024-09-15T03:48:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.8599999999999999e-07 2024-09-15T03:48:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.454 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.950570+0000", 2024-09-15T03:48:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.901e-06 2024-09-15T03:48:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.950581+0000", 2024-09-15T03:48:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.0180000000000001e-05 2024-09-15T03:48:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.950588+0000", 2024-09-15T03:48:09.455 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 6.8970000000000002e-06 2024-09-15T03:48:09.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.456 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.456 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.958313+0000", 2024-09-15T03:48:09.456 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0077258099999999996 2024-09-15T03:48:09.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.456 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.456 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.958323+0000", 2024-09-15T03:48:09.456 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 9.2040000000000003e-06 2024-09-15T03:48:09.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.457 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.457 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.958428+0000", 2024-09-15T03:48:09.457 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000105474 2024-09-15T03:48:09.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.457 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.457 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.958445+0000", 2024-09-15T03:48:09.457 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.6878e-05 2024-09-15T03:48:09.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.457 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.458 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.958476+0000", 2024-09-15T03:48:09.458 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 3.0750000000000002e-05 2024-09-15T03:48:09.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.458 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.458 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.458 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.959323+0000", 2024-09-15T03:48:09.458 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00084734999999999997 2024-09-15T03:48:09.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.458 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.458 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:09.458 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.963908+0000", 2024-09-15T03:48:09.458 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.0045849009999999997 2024-09-15T03:48:09.459 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.459 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:09.459 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.459 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.459 INFO:teuthology.orchestra.run.smithi001.stdout: "description": "MOSDECSubOpWrite(3.bs7 113/108 ECSubWrite(tid=356, reqid=osd.7.0:356, at_version=113'209, trim_to=0'0, roll_forward_to=113'205))", 2024-09-15T03:48:09.459 INFO:teuthology.orchestra.run.smithi001.stdout: "initiated_at": "2024-09-15T03:47:59.954002+0000", 2024-09-15T03:48:09.459 INFO:teuthology.orchestra.run.smithi001.stdout: "age": 9.3505820560000004, 2024-09-15T03:48:09.459 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.012200717, 2024-09-15T03:48:09.459 INFO:teuthology.orchestra.run.smithi001.stdout: "type_data": { 2024-09-15T03:48:09.459 INFO:teuthology.orchestra.run.smithi001.stdout: "flag_point": "reached pg", 2024-09-15T03:48:09.459 INFO:teuthology.orchestra.run.smithi001.stdout: "events": [ 2024-09-15T03:48:09.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.460 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "initiated", 2024-09-15T03:48:09.460 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.954002+0000", 2024-09-15T03:48:09.460 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.460 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "header_read", 2024-09-15T03:48:09.460 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.954002+0000", 2024-09-15T03:48:09.460 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0 2024-09-15T03:48:09.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.461 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "throttled", 2024-09-15T03:48:09.461 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.954003+0000", 2024-09-15T03:48:09.461 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.5099999999999998e-07 2024-09-15T03:48:09.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.461 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "all_read", 2024-09-15T03:48:09.461 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.954026+0000", 2024-09-15T03:48:09.461 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 2.3249000000000001e-05 2024-09-15T03:48:09.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.462 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "dispatched", 2024-09-15T03:48:09.462 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.954041+0000", 2024-09-15T03:48:09.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.5322999999999999e-05 2024-09-15T03:48:09.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.462 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.462 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.954049+0000", 2024-09-15T03:48:09.462 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 8.0199999999999994e-06 2024-09-15T03:48:09.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "reached_pg", 2024-09-15T03:48:09.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.954124+0000", 2024-09-15T03:48:09.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.5029000000000003e-05 2024-09-15T03:48:09.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.463 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.463 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.954137+0000", 2024-09-15T03:48:09.463 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 1.2979999999999999e-05 2024-09-15T03:48:09.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.954434+0000", 2024-09-15T03:48:09.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00029692800000000001 2024-09-15T03:48:09.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.464 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.955135+0000", 2024-09-15T03:48:09.464 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.00070042300000000004 2024-09-15T03:48:09.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.464 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.465 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.955205+0000", 2024-09-15T03:48:09.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 7.0036000000000005e-05 2024-09-15T03:48:09.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.465 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.465 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.955314+0000", 2024-09-15T03:48:09.465 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.000109701 2024-09-15T03:48:09.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-09-15T03:48:09.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-09-15T03:48:09.465 INFO:teuthology.orchestra.run.smithi001.stdout: "event": "done", 2024-09-15T03:48:09.466 INFO:teuthology.orchestra.run.smithi001.stdout: "time": "2024-09-15T03:47:59.966203+0000", 2024-09-15T03:48:09.466 INFO:teuthology.orchestra.run.smithi001.stdout: "duration": 0.010888177000000001 2024-09-15T03:48:09.466 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.466 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:09.466 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.466 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-09-15T03:48:09.466 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-09-15T03:48:09.466 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:09.468 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:48:09.469 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:48:09.467+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:48:09.568 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:48:09.567+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:48:09.581 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:48:09.582 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:48:09.582 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:48:09.582 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:48:09.591 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:48:09.669 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:48:09.669+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:48:09.692 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:48:09.692 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:48:09.692 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-15T03:48:09.692 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-15T03:48:09.693 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:48:09.701 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:48:09.772 INFO:tasks.ceph.osd.9.smithi190.stderr:2024-09-15T03:48:09.770+0000 7f433a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 29465) UID: 0 2024-09-15T03:48:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:48:09.846 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-15T03:48:09.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-15T03:48:09.847 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-15T03:48:09.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.847 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:324 3.7s0 3:e40d1694:::smithi00132172-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:48:09.847 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:48:09.847 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.999766383999997, 2024-09-15T03:48:09.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0031668960000002, 2024-09-15T03:48:09.847 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:09.847 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:48:09.848 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:09.848 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:09.848 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.848 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 324 2024-09-15T03:48:09.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.848 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:09.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:09.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:48:09.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:09.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:48:09.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:09.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844123+0000", 2024-09-15T03:48:09.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:48:09.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:09.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844125+0000", 2024-09-15T03:48:09.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.142e-06 2024-09-15T03:48:09.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:09.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844132+0000", 2024-09-15T03:48:09.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0439999999999998e-06 2024-09-15T03:48:09.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.844136+0000", 2024-09-15T03:48:09.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.126e-06 2024-09-15T03:48:09.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.847618+0000", 2024-09-15T03:48:09.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0034813790000000002 2024-09-15T03:48:09.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:48:09.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.847709+0000", 2024-09-15T03:48:09.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0835999999999996e-05 2024-09-15T03:48:09.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.849483+0000", 2024-09-15T03:48:09.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001774929 2024-09-15T03:48:09.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:09.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.849574+0000", 2024-09-15T03:48:09.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0847999999999995e-05 2024-09-15T03:48:09.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:09.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.847290+0000", 2024-09-15T03:48:09.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9977151819999999 2024-09-15T03:48:09.852 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.852 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:09.853 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.853 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:334 3.7s0 3:ece1472c:::smithi00132172-43:head [append 0~0] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:48:09.853 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:48:09.853 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.987039296000006, 2024-09-15T03:48:09.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.997592021, 2024-09-15T03:48:09.853 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:09.853 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.853 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:09.853 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:09.853 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.854 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 334 2024-09-15T03:48:09.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.854 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:09.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:09.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:48:09.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:09.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856850+0000", 2024-09-15T03:48:09.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:09.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856851+0000", 2024-09-15T03:48:09.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.238e-06 2024-09-15T03:48:09.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:09.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856855+0000", 2024-09-15T03:48:09.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3640000000000001e-06 2024-09-15T03:48:09.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:09.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856868+0000", 2024-09-15T03:48:09.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2588999999999999e-05 2024-09-15T03:48:09.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856873+0000", 2024-09-15T03:48:09.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5350000000000004e-06 2024-09-15T03:48:09.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856916+0000", 2024-09-15T03:48:09.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2889000000000002e-05 2024-09-15T03:48:09.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:09.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856983+0000", 2024-09-15T03:48:09.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6746000000000001e-05 2024-09-15T03:48:09.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857448+0000", 2024-09-15T03:48:09.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046498500000000001 2024-09-15T03:48:09.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857720+0000", 2024-09-15T03:48:09.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027227600000000001 2024-09-15T03:48:09.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857741+0000", 2024-09-15T03:48:09.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1149999999999999e-05 2024-09-15T03:48:09.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857795+0000", 2024-09-15T03:48:09.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3919999999999999e-05 2024-09-15T03:48:09.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859582+0000", 2024-09-15T03:48:09.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017862640000000001 2024-09-15T03:48:09.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:09.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854347+0000", 2024-09-15T03:48:09.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9947654780000001 2024-09-15T03:48:09.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:09.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854442+0000", 2024-09-15T03:48:09.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4586999999999994e-05 2024-09-15T03:48:09.860 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.860 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:09.860 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.860 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:335 3.7s0 3:ece1472c:::smithi00132172-43:head [rollback 12] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:48:09.860 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:48:09.860 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.987009279000006, 2024-09-15T03:48:09.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9977859200000001, 2024-09-15T03:48:09.861 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:09.861 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.861 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:09.861 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:09.861 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.861 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 335 2024-09-15T03:48:09.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.861 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:09.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:09.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:48:09.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:09.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:48:09.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:09.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856880+0000", 2024-09-15T03:48:09.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8099999999999999e-07 2024-09-15T03:48:09.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:09.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856882+0000", 2024-09-15T03:48:09.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.649e-06 2024-09-15T03:48:09.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:09.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856885+0000", 2024-09-15T03:48:09.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.664e-06 2024-09-15T03:48:09.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856888+0000", 2024-09-15T03:48:09.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3910000000000001e-06 2024-09-15T03:48:09.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857826+0000", 2024-09-15T03:48:09.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00093817900000000003 2024-09-15T03:48:09.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:09.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.857940+0000", 2024-09-15T03:48:09.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114424 2024-09-15T03:48:09.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858504+0000", 2024-09-15T03:48:09.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056400899999999997 2024-09-15T03:48:09.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858593+0000", 2024-09-15T03:48:09.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8463999999999999e-05 2024-09-15T03:48:09.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858604+0000", 2024-09-15T03:48:09.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1567e-05 2024-09-15T03:48:09.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858642+0000", 2024-09-15T03:48:09.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7369000000000001e-05 2024-09-15T03:48:09.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859588+0000", 2024-09-15T03:48:09.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00094661799999999998 2024-09-15T03:48:09.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:09.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854594+0000", 2024-09-15T03:48:09.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9950055550000001 2024-09-15T03:48:09.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:09.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854666+0000", 2024-09-15T03:48:09.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1749999999999996e-05 2024-09-15T03:48:09.868 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.868 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:09.869 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.869 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:336 3.7s0 3:ece1472c:::smithi00132172-43:head [append 0~0] snapc 12=[12,11,e,d,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:48:09.869 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:48:09.869 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.986978229000002, 2024-09-15T03:48:09.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9980017820000002, 2024-09-15T03:48:09.869 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:09.869 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.869 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:09.869 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:09.869 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.869 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 336 2024-09-15T03:48:09.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.870 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:09.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:09.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:48:09.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:09.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:48:09.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:09.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856911+0000", 2024-09-15T03:48:09.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8000000000000002e-07 2024-09-15T03:48:09.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:09.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856913+0000", 2024-09-15T03:48:09.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5400000000000001e-06 2024-09-15T03:48:09.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:09.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856918+0000", 2024-09-15T03:48:09.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1540000000000001e-06 2024-09-15T03:48:09.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.856921+0000", 2024-09-15T03:48:09.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2550000000000002e-06 2024-09-15T03:48:09.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858655+0000", 2024-09-15T03:48:09.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017336070000000001 2024-09-15T03:48:09.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:09.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.858741+0000", 2024-09-15T03:48:09.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6305999999999997e-05 2024-09-15T03:48:09.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859193+0000", 2024-09-15T03:48:09.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045221200000000001 2024-09-15T03:48:09.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859319+0000", 2024-09-15T03:48:09.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012542900000000001 2024-09-15T03:48:09.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859331+0000", 2024-09-15T03:48:09.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2028000000000001e-05 2024-09-15T03:48:09.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859369+0000", 2024-09-15T03:48:09.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8522999999999997e-05 2024-09-15T03:48:09.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:55.859649+0000", 2024-09-15T03:48:09.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028008200000000002 2024-09-15T03:48:09.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:09.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854800+0000", 2024-09-15T03:48:09.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9951506540000001 2024-09-15T03:48:09.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:09.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.854913+0000", 2024-09-15T03:48:09.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112712 2024-09-15T03:48:09.876 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.876 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:09.876 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.876 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:346 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:48:09.876 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:48:09.876 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.005481859, 2024-09-15T03:48:09.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005631948, 2024-09-15T03:48:09.877 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:09.877 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.877 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:09.877 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:09.877 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.877 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 346 2024-09-15T03:48:09.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.877 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:09.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:09.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:48:09.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:09.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838407+0000", 2024-09-15T03:48:09.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:09.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838410+0000", 2024-09-15T03:48:09.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4430000000000002e-06 2024-09-15T03:48:09.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:09.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838419+0000", 2024-09-15T03:48:09.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1730000000000001e-06 2024-09-15T03:48:09.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:09.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838444+0000", 2024-09-15T03:48:09.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5111000000000001e-05 2024-09-15T03:48:09.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838455+0000", 2024-09-15T03:48:09.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0839000000000001e-05 2024-09-15T03:48:09.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:48:09.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3415000000000002e-05 2024-09-15T03:48:09.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-15T03:48:09.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838527+0000", 2024-09-15T03:48:09.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0550000000000005e-06 2024-09-15T03:48:09.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840638+0000", 2024-09-15T03:48:09.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021105379999999999 2024-09-15T03:48:09.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:09.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.840729+0000", 2024-09-15T03:48:09.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0977999999999995e-05 2024-09-15T03:48:09.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.841834+0000", 2024-09-15T03:48:09.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011055640000000001 2024-09-15T03:48:09.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842223+0000", 2024-09-15T03:48:09.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038899900000000001 2024-09-15T03:48:09.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842241+0000", 2024-09-15T03:48:09.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7317e-05 2024-09-15T03:48:09.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842298+0000", 2024-09-15T03:48:09.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7825000000000003e-05 2024-09-15T03:48:09.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844866+0000", 2024-09-15T03:48:09.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025675289999999998 2024-09-15T03:48:09.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:09.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.843973+0000", 2024-09-15T03:48:09.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99910710599999997 2024-09-15T03:48:09.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:09.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844039+0000", 2024-09-15T03:48:09.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6056000000000006e-05 2024-09-15T03:48:09.884 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.884 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:09.884 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.884 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:347 3.2s0 3:48ceb02f:::smithi00132172-5:head [rollback c] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:48:09.884 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:48:09.885 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.005397063000004, 2024-09-15T03:48:09.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005723841, 2024-09-15T03:48:09.885 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:09.885 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.885 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:09.885 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:09.885 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.885 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 347 2024-09-15T03:48:09.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.885 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:09.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:09.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:48:09.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:09.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838492+0000", 2024-09-15T03:48:09.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:09.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838493+0000", 2024-09-15T03:48:09.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7300000000000005e-07 2024-09-15T03:48:09.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:09.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838498+0000", 2024-09-15T03:48:09.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9300000000000002e-06 2024-09-15T03:48:09.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:09.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838508+0000", 2024-09-15T03:48:09.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9790000000000005e-06 2024-09-15T03:48:09.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838513+0000", 2024-09-15T03:48:09.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8890000000000001e-06 2024-09-15T03:48:09.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838583+0000", 2024-09-15T03:48:09.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0696000000000002e-05 2024-09-15T03:48:09.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:48:09.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838590+0000", 2024-09-15T03:48:09.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2990000000000002e-06 2024-09-15T03:48:09.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842313+0000", 2024-09-15T03:48:09.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037238810000000001 2024-09-15T03:48:09.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:09.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.842432+0000", 2024-09-15T03:48:09.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118474 2024-09-15T03:48:09.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843108+0000", 2024-09-15T03:48:09.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00067639299999999996 2024-09-15T03:48:09.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843227+0000", 2024-09-15T03:48:09.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119092 2024-09-15T03:48:09.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843242+0000", 2024-09-15T03:48:09.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4216000000000001e-05 2024-09-15T03:48:09.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843292+0000", 2024-09-15T03:48:09.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0800999999999997e-05 2024-09-15T03:48:09.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844878+0000", 2024-09-15T03:48:09.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015852749999999999 2024-09-15T03:48:09.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:09.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844147+0000", 2024-09-15T03:48:09.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99926949200000004 2024-09-15T03:48:09.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:09.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844216+0000", 2024-09-15T03:48:09.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8551000000000001e-05 2024-09-15T03:48:09.893 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.893 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:09.893 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.893 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:348 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~0] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:48:09.893 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:48:09.893 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.005371120000007, 2024-09-15T03:48:09.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.006019768, 2024-09-15T03:48:09.894 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:09.894 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.894 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:09.894 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:09.894 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.894 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 348 2024-09-15T03:48:09.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.894 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:09.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:09.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:48:09.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:09.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:48:09.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:09.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838518+0000", 2024-09-15T03:48:09.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8700000000000001e-07 2024-09-15T03:48:09.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:09.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838521+0000", 2024-09-15T03:48:09.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3740000000000001e-06 2024-09-15T03:48:09.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:09.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838527+0000", 2024-09-15T03:48:09.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1999999999999999e-06 2024-09-15T03:48:09.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838532+0000", 2024-09-15T03:48:09.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2059999999999998e-06 2024-09-15T03:48:09.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838609+0000", 2024-09-15T03:48:09.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7193000000000004e-05 2024-09-15T03:48:09.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:48:09.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.838615+0000", 2024-09-15T03:48:09.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7980000000000002e-06 2024-09-15T03:48:09.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.843986+0000", 2024-09-15T03:48:09.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053706450000000003 2024-09-15T03:48:09.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:09.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844106+0000", 2024-09-15T03:48:09.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120079 2024-09-15T03:48:09.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844669+0000", 2024-09-15T03:48:09.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056290300000000001 2024-09-15T03:48:09.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844804+0000", 2024-09-15T03:48:09.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013575600000000001 2024-09-15T03:48:09.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844820+0000", 2024-09-15T03:48:09.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5145e-05 2024-09-15T03:48:09.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844893+0000", 2024-09-15T03:48:09.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3270999999999997e-05 2024-09-15T03:48:09.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845043+0000", 2024-09-15T03:48:09.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150144 2024-09-15T03:48:09.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:09.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844473+0000", 2024-09-15T03:48:09.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99943048700000003 2024-09-15T03:48:09.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:09.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.844538+0000", 2024-09-15T03:48:09.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4179999999999999e-05 2024-09-15T03:48:09.902 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.902 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:09.902 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.902 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:352 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~557056] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:48:09.902 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:48:09.903 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.999173475999996, 2024-09-15T03:48:09.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.016265065, 2024-09-15T03:48:09.903 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:09.903 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.903 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:09.903 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:09.903 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.903 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 352 2024-09-15T03:48:09.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.903 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:09.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:09.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:48:09.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:09.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844716+0000", 2024-09-15T03:48:09.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:09.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.844718+0000", 2024-09-15T03:48:09.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2340000000000001e-06 2024-09-15T03:48:09.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:09.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845237+0000", 2024-09-15T03:48:09.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051953399999999999 2024-09-15T03:48:09.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:09.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845273+0000", 2024-09-15T03:48:09.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5683999999999999e-05 2024-09-15T03:48:09.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845281+0000", 2024-09-15T03:48:09.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8439999999999998e-06 2024-09-15T03:48:09.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845365+0000", 2024-09-15T03:48:09.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4651e-05 2024-09-15T03:48:09.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:09.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.845486+0000", 2024-09-15T03:48:09.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012009499999999999 2024-09-15T03:48:09.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847101+0000", 2024-09-15T03:48:09.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016157350000000001 2024-09-15T03:48:09.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847602+0000", 2024-09-15T03:48:09.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050066200000000003 2024-09-15T03:48:09.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.847622+0000", 2024-09-15T03:48:09.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9749999999999999e-05 2024-09-15T03:48:09.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848616+0000", 2024-09-15T03:48:09.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099390799999999994 2024-09-15T03:48:09.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848709+0000", 2024-09-15T03:48:09.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2874000000000006e-05 2024-09-15T03:48:09.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:09.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.860888+0000", 2024-09-15T03:48:09.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.012179707 2024-09-15T03:48:09.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:09.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.860981+0000", 2024-09-15T03:48:09.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2386999999999995e-05 2024-09-15T03:48:09.909 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.909 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:09.909 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.909 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:353 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~671744] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:48:09.909 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:48:09.909 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.995611283000002, 2024-09-15T03:48:09.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.012912206, 2024-09-15T03:48:09.910 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:09.910 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.910 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:09.910 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:09.910 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.910 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 353 2024-09-15T03:48:09.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.910 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:09.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:09.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:48:09.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:09.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848278+0000", 2024-09-15T03:48:09.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:09.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.848279+0000", 2024-09-15T03:48:09.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.362e-06 2024-09-15T03:48:09.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:09.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849024+0000", 2024-09-15T03:48:09.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00074495500000000005 2024-09-15T03:48:09.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:09.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849039+0000", 2024-09-15T03:48:09.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5119e-05 2024-09-15T03:48:09.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849046+0000", 2024-09-15T03:48:09.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6359999999999997e-06 2024-09-15T03:48:09.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849115+0000", 2024-09-15T03:48:09.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8998999999999994e-05 2024-09-15T03:48:09.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:09.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849251+0000", 2024-09-15T03:48:09.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013589799999999999 2024-09-15T03:48:09.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850605+0000", 2024-09-15T03:48:09.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013544659999999999 2024-09-15T03:48:09.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850703+0000", 2024-09-15T03:48:09.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7576000000000002e-05 2024-09-15T03:48:09.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850719+0000", 2024-09-15T03:48:09.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5767e-05 2024-09-15T03:48:09.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851562+0000", 2024-09-15T03:48:09.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084361899999999999 2024-09-15T03:48:09.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851705+0000", 2024-09-15T03:48:09.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000142893 2024-09-15T03:48:09.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:09.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861112+0000", 2024-09-15T03:48:09.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.009407349 2024-09-15T03:48:09.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:09.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.861190+0000", 2024-09-15T03:48:09.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7566999999999998e-05 2024-09-15T03:48:09.916 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.916 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:09.916 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.916 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:354 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [append 0~98304] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:48:09.916 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:48:09.916 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.994808847000002, 2024-09-15T03:48:09.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0177595669999999, 2024-09-15T03:48:09.917 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:09.917 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.917 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:09.917 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:09.917 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.917 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 354 2024-09-15T03:48:09.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.917 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:09.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:09.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:48:09.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:09.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849080+0000", 2024-09-15T03:48:09.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:09.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849081+0000", 2024-09-15T03:48:09.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0610000000000001e-06 2024-09-15T03:48:09.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:09.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849157+0000", 2024-09-15T03:48:09.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5804999999999997e-05 2024-09-15T03:48:09.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:09.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849168+0000", 2024-09-15T03:48:09.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0730000000000001e-05 2024-09-15T03:48:09.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849174+0000", 2024-09-15T03:48:09.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3609999999999997e-06 2024-09-15T03:48:09.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850814+0000", 2024-09-15T03:48:09.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001640124 2024-09-15T03:48:09.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:09.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.850894+0000", 2024-09-15T03:48:09.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0167999999999995e-05 2024-09-15T03:48:09.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851406+0000", 2024-09-15T03:48:09.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051203200000000005 2024-09-15T03:48:09.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851506+0000", 2024-09-15T03:48:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9457999999999995e-05 2024-09-15T03:48:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851574+0000", 2024-09-15T03:48:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8145000000000006e-05 2024-09-15T03:48:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851632+0000", 2024-09-15T03:48:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8108e-05 2024-09-15T03:48:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851712+0000", 2024-09-15T03:48:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9380000000000002e-05 2024-09-15T03:48:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.866753+0000", 2024-09-15T03:48:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0150417979999999 2024-09-15T03:48:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.866840+0000", 2024-09-15T03:48:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6397000000000006e-05 2024-09-15T03:48:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:355 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [setxattr _header (57) in=7b] snapc 14=[14,12,11,e,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:48:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:48:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.994679533999999, 2024-09-15T03:48:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.018024966, 2024-09-15T03:48:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 355 2024-09-15T03:48:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:48:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849209+0000", 2024-09-15T03:48:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849210+0000", 2024-09-15T03:48:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.33e-07 2024-09-15T03:48:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849215+0000", 2024-09-15T03:48:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4379999999999997e-06 2024-09-15T03:48:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849234+0000", 2024-09-15T03:48:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9409999999999999e-05 2024-09-15T03:48:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849241+0000", 2024-09-15T03:48:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6370000000000002e-06 2024-09-15T03:48:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851823+0000", 2024-09-15T03:48:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025817689999999998 2024-09-15T03:48:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.851934+0000", 2024-09-15T03:48:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111357 2024-09-15T03:48:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852466+0000", 2024-09-15T03:48:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053248100000000001 2024-09-15T03:48:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852580+0000", 2024-09-15T03:48:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011379699999999999 2024-09-15T03:48:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852607+0000", 2024-09-15T03:48:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279e-05 2024-09-15T03:48:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852693+0000", 2024-09-15T03:48:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6792999999999994e-05 2024-09-15T03:48:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852869+0000", 2024-09-15T03:48:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017537499999999999 2024-09-15T03:48:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867006+0000", 2024-09-15T03:48:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.014136996 2024-09-15T03:48:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867234+0000", 2024-09-15T03:48:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022870099999999999 2024-09-15T03:48:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:356 3.7s0 3:e2ffb0b4:::smithi00132172-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:48:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:48:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.994642771000002, 2024-09-15T03:48:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.020403521, 2024-09-15T03:48:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:48:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 356 2024-09-15T03:48:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:48:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849246+0000", 2024-09-15T03:48:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849247+0000", 2024-09-15T03:48:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2699999999999999e-07 2024-09-15T03:48:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849250+0000", 2024-09-15T03:48:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2610000000000002e-06 2024-09-15T03:48:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849257+0000", 2024-09-15T03:48:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.703e-06 2024-09-15T03:48:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.849261+0000", 2024-09-15T03:48:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7910000000000002e-06 2024-09-15T03:48:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852672+0000", 2024-09-15T03:48:09.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003411627 2024-09-15T03:48:09.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:48:09.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:57.852801+0000", 2024-09-15T03:48:09.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000128918 2024-09-15T03:48:09.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867347+0000", 2024-09-15T03:48:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0145458380000001 2024-09-15T03:48:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.867476+0000", 2024-09-15T03:48:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012915100000000001 2024-09-15T03:48:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:46:58.869650+0000", 2024-09-15T03:48:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021737050000000002 2024-09-15T03:48:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:527 3.7s0 3:e4b8e19d:::smithi00132172-41:head [copy-from ver 51] snapc 23=[23,22,21,20,18,17,16,12,11,c,3,2] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:48:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:48:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.895073809000003, 2024-09-15T03:48:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030362388000000001, 2024-09-15T03:48:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 527 2024-09-15T03:48:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:48:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948815+0000", 2024-09-15T03:48:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948817+0000", 2024-09-15T03:48:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3230000000000001e-06 2024-09-15T03:48:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948825+0000", 2024-09-15T03:48:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5900000000000008e-06 2024-09-15T03:48:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948838+0000", 2024-09-15T03:48:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2712e-05 2024-09-15T03:48:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948842+0000", 2024-09-15T03:48:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4170000000000002e-06 2024-09-15T03:48:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948923+0000", 2024-09-15T03:48:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0686000000000006e-05 2024-09-15T03:48:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:48:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949020+0000", 2024-09-15T03:48:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6984999999999994e-05 2024-09-15T03:48:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954390+0000", 2024-09-15T03:48:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053704790000000001 2024-09-15T03:48:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954558+0000", 2024-09-15T03:48:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000167202 2024-09-15T03:48:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.973998+0000", 2024-09-15T03:48:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019440316999999999 2024-09-15T03:48:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974527+0000", 2024-09-15T03:48:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052943800000000002 2024-09-15T03:48:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974541+0000", 2024-09-15T03:48:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3774999999999999e-05 2024-09-15T03:48:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974841+0000", 2024-09-15T03:48:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029942899999999998 2024-09-15T03:48:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.976067+0000", 2024-09-15T03:48:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012265399999999999 2024-09-15T03:48:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.978952+0000", 2024-09-15T03:48:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028848839999999999 2024-09-15T03:48:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.979178+0000", 2024-09-15T03:48:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022561099999999999 2024-09-15T03:48:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:528 3.7s0 3:e4b8e19d:::smithi00132172-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-15T03:48:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:48:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.895039400000002, 2024-09-15T03:48:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026927036000000001, 2024-09-15T03:48:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:48:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 528 2024-09-15T03:48:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:48:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:48:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948850+0000", 2024-09-15T03:48:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9700000000000003e-07 2024-09-15T03:48:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948852+0000", 2024-09-15T03:48:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.604e-06 2024-09-15T03:48:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948855+0000", 2024-09-15T03:48:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9369999999999997e-06 2024-09-15T03:48:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.948858+0000", 2024-09-15T03:48:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2709999999999999e-06 2024-09-15T03:48:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949080+0000", 2024-09-15T03:48:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000222148 2024-09-15T03:48:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-09-15T03:48:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.949156+0000", 2024-09-15T03:48:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6215000000000001e-05 2024-09-15T03:48:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954754+0000", 2024-09-15T03:48:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055978060000000003 2024-09-15T03:48:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for blocked object", 2024-09-15T03:48:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.954858+0000", 2024-09-15T03:48:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010419199999999999 2024-09-15T03:48:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974861+0000", 2024-09-15T03:48:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020002671999999999 2024-09-15T03:48:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.974987+0000", 2024-09-15T03:48:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012638699999999999 2024-09-15T03:48:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:15.975777+0000", 2024-09-15T03:48:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00078950699999999995 2024-09-15T03:48:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:924 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~737280] snapc 45=[45,44,43,42,41,40,3e,3d,3b,3a,38,37,35,34,2d,2b,25,24,22,21,20,16,12,11,3] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-15T03:48:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:53.925245+0000", 2024-09-15T03:48:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.918643748000001, 2024-09-15T03:48:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.29827357799999998, 2024-09-15T03:48:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 924 2024-09-15T03:48:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.925245+0000", 2024-09-15T03:48:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.925245+0000", 2024-09-15T03:48:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.925247+0000", 2024-09-15T03:48:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9999999999999999e-06 2024-09-15T03:48:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.926127+0000", 2024-09-15T03:48:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000879957 2024-09-15T03:48:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.926146+0000", 2024-09-15T03:48:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8349999999999999e-05 2024-09-15T03:48:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.926153+0000", 2024-09-15T03:48:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3889999999999999e-06 2024-09-15T03:48:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.926220+0000", 2024-09-15T03:48:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7490000000000006e-05 2024-09-15T03:48:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:48:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.926224+0000", 2024-09-15T03:48:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5590000000000001e-06 2024-09-15T03:48:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.209778+0000", 2024-09-15T03:48:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.28355351899999998 2024-09-15T03:48:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.209967+0000", 2024-09-15T03:48:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018960400000000001 2024-09-15T03:48:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.211917+0000", 2024-09-15T03:48:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019494320000000001 2024-09-15T03:48:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.212779+0000", 2024-09-15T03:48:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086249099999999995 2024-09-15T03:48:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.212798+0000", 2024-09-15T03:48:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9015e-05 2024-09-15T03:48:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.212925+0000", 2024-09-15T03:48:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012736099999999999 2024-09-15T03:48:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.212946+0000", 2024-09-15T03:48:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0919000000000001e-05 2024-09-15T03:48:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.215488+0000", 2024-09-15T03:48:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025418350000000001 2024-09-15T03:48:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.223519+0000", 2024-09-15T03:48:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0080306570000000001 2024-09-15T03:48:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:925 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~770048] snapc 45=[45,44,43,42,41,40,3e,3d,3b,3a,38,37,35,34,2d,2b,25,24,22,21,20,16,12,11,3] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-15T03:48:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:53.930327+0000", 2024-09-15T03:48:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.913562034, 2024-09-15T03:48:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.29345302099999998, 2024-09-15T03:48:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 925 2024-09-15T03:48:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.930327+0000", 2024-09-15T03:48:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.930327+0000", 2024-09-15T03:48:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.930328+0000", 2024-09-15T03:48:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.15e-06 2024-09-15T03:48:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.930988+0000", 2024-09-15T03:48:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000659594 2024-09-15T03:48:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.931004+0000", 2024-09-15T03:48:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6101000000000001e-05 2024-09-15T03:48:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.931011+0000", 2024-09-15T03:48:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8940000000000004e-06 2024-09-15T03:48:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.931050+0000", 2024-09-15T03:48:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9085000000000003e-05 2024-09-15T03:48:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:48:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.931052+0000", 2024-09-15T03:48:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1789999999999998e-06 2024-09-15T03:48:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.213039+0000", 2024-09-15T03:48:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.281986827 2024-09-15T03:48:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.213228+0000", 2024-09-15T03:48:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000189159 2024-09-15T03:48:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.214530+0000", 2024-09-15T03:48:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001302147 2024-09-15T03:48:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.214723+0000", 2024-09-15T03:48:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000192998 2024-09-15T03:48:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.214741+0000", 2024-09-15T03:48:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7762e-05 2024-09-15T03:48:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.215136+0000", 2024-09-15T03:48:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039510400000000002 2024-09-15T03:48:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.215269+0000", 2024-09-15T03:48:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013283300000000001 2024-09-15T03:48:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.216320+0000", 2024-09-15T03:48:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001050888 2024-09-15T03:48:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.223780+0000", 2024-09-15T03:48:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074602999999999996 2024-09-15T03:48:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:926 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~540672] snapc 45=[45,44,43,42,41,40,3e,3d,3b,3a,38,37,35,34,2d,2b,25,24,22,21,20,16,12,11,3] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-15T03:48:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:53.933251+0000", 2024-09-15T03:48:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.910638372999999, 2024-09-15T03:48:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.29574198400000001, 2024-09-15T03:48:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 926 2024-09-15T03:48:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933251+0000", 2024-09-15T03:48:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933251+0000", 2024-09-15T03:48:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933252+0000", 2024-09-15T03:48:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.569999999999999e-07 2024-09-15T03:48:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933849+0000", 2024-09-15T03:48:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059778100000000003 2024-09-15T03:48:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933865+0000", 2024-09-15T03:48:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5170999999999999e-05 2024-09-15T03:48:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933871+0000", 2024-09-15T03:48:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2899999999999999e-06 2024-09-15T03:48:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933904+0000", 2024-09-15T03:48:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3519000000000002e-05 2024-09-15T03:48:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:48:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.933905+0000", 2024-09-15T03:48:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.096e-06 2024-09-15T03:48:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.217183+0000", 2024-09-15T03:48:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.28327724799999998 2024-09-15T03:48:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.217399+0000", 2024-09-15T03:48:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000216429 2024-09-15T03:48:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.224559+0000", 2024-09-15T03:48:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071598429999999999 2024-09-15T03:48:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.224765+0000", 2024-09-15T03:48:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020648500000000001 2024-09-15T03:48:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.224794+0000", 2024-09-15T03:48:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8645000000000001e-05 2024-09-15T03:48:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.224949+0000", 2024-09-15T03:48:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015473200000000001 2024-09-15T03:48:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.227877+0000", 2024-09-15T03:48:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029284110000000001 2024-09-15T03:48:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.228692+0000", 2024-09-15T03:48:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081477700000000004 2024-09-15T03:48:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.228993+0000", 2024-09-15T03:48:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030059999999999999 2024-09-15T03:48:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:927 3.2s0 3:48ceb02f:::smithi00132172-5:head [append 0~245760] snapc 45=[45,44,43,42,41,40,3e,3d,3b,3a,38,37,35,34,2d,2b,25,24,22,21,20,16,12,11,3] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-15T03:48:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:53.934611+0000", 2024-09-15T03:48:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.909277523, 2024-09-15T03:48:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.29515997300000002, 2024-09-15T03:48:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 927 2024-09-15T03:48:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934611+0000", 2024-09-15T03:48:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934611+0000", 2024-09-15T03:48:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934612+0000", 2024-09-15T03:48:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3099999999999996e-07 2024-09-15T03:48:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934841+0000", 2024-09-15T03:48:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022889699999999999 2024-09-15T03:48:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934855+0000", 2024-09-15T03:48:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3998999999999999e-05 2024-09-15T03:48:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934862+0000", 2024-09-15T03:48:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3300000000000004e-06 2024-09-15T03:48:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934894+0000", 2024-09-15T03:48:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1995000000000001e-05 2024-09-15T03:48:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:48:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934894+0000", 2024-09-15T03:48:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6199999999999998e-07 2024-09-15T03:48:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.224939+0000", 2024-09-15T03:48:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.29004481300000001 2024-09-15T03:48:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.225469+0000", 2024-09-15T03:48:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052985499999999997 2024-09-15T03:48:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.226886+0000", 2024-09-15T03:48:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001416785 2024-09-15T03:48:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.227699+0000", 2024-09-15T03:48:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000813363 2024-09-15T03:48:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.227716+0000", 2024-09-15T03:48:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7017e-05 2024-09-15T03:48:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.227783+0000", 2024-09-15T03:48:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7057999999999994e-05 2024-09-15T03:48:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.227879+0000", 2024-09-15T03:48:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6417999999999997e-05 2024-09-15T03:48:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.229372+0000", 2024-09-15T03:48:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014929019999999999 2024-09-15T03:48:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.229771+0000", 2024-09-15T03:48:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039914799999999999 2024-09-15T03:48:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:928 3.2s0 3:48ceb02f:::smithi00132172-5:head [setxattr _header (56) in=7b] snapc 45=[45,44,43,42,41,40,3e,3d,3b,3a,38,37,35,34,2d,2b,25,24,22,21,20,16,12,11,3] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-15T03:48:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:53.934871+0000", 2024-09-15T03:48:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.909017608999999, 2024-09-15T03:48:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.298571425, 2024-09-15T03:48:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:48:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 928 2024-09-15T03:48:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:09.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934871+0000", 2024-09-15T03:48:09.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:09.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934871+0000", 2024-09-15T03:48:09.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:09.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934872+0000", 2024-09-15T03:48:09.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9100000000000003e-07 2024-09-15T03:48:09.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:09.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934876+0000", 2024-09-15T03:48:09.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.687e-06 2024-09-15T03:48:09.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:09.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934884+0000", 2024-09-15T03:48:09.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7049999999999999e-06 2024-09-15T03:48:09.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934888+0000", 2024-09-15T03:48:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.639e-06 2024-09-15T03:48:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934911+0000", 2024-09-15T03:48:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3224999999999999e-05 2024-09-15T03:48:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:48:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934912+0000", 2024-09-15T03:48:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9999999999999999e-07 2024-09-15T03:48:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.230182+0000", 2024-09-15T03:48:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.29527004699999998 2024-09-15T03:48:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.230320+0000", 2024-09-15T03:48:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013794499999999999 2024-09-15T03:48:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_started", 2024-09-15T03:48:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.231057+0000", 2024-09-15T03:48:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00073703400000000002 2024-09-15T03:48:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_queued_for_journal_write", 2024-09-15T03:48:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.231199+0000", 2024-09-15T03:48:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014196099999999999 2024-09-15T03:48:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "write_thread_in_journal_buffer", 2024-09-15T03:48:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.231309+0000", 2024-09-15T03:48:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011001399999999999 2024-09-15T03:48:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "journaled_completion_queued", 2024-09-15T03:48:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.231349+0000", 2024-09-15T03:48:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0068999999999999e-05 2024-09-15T03:48:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_committed", 2024-09-15T03:48:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.231658+0000", 2024-09-15T03:48:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030921799999999999 2024-09-15T03:48:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-15T03:48:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.232928+0000", 2024-09-15T03:48:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012703409999999999 2024-09-15T03:48:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.233443+0000", 2024-09-15T03:48:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051444900000000005 2024-09-15T03:48:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4521.0:929 3.2s0 3:48ceb02f:::smithi00132172-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2024-09-15T03:48:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-15T03:47:53.934895+0000", 2024-09-15T03:48:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.908994166999999, 2024-09-15T03:48:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.30125257599999999, 2024-09-15T03:48:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-15T03:48:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-15T03:48:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-15T03:48:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4521", 2024-09-15T03:48:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.1:0/1687530245", 2024-09-15T03:48:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 929 2024-09-15T03:48:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-15T03:48:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-15T03:48:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934895+0000", 2024-09-15T03:48:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-15T03:48:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934895+0000", 2024-09-15T03:48:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-15T03:48:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-15T03:48:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934895+0000", 2024-09-15T03:48:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2699999999999999e-07 2024-09-15T03:48:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-15T03:48:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934898+0000", 2024-09-15T03:48:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3549999999999999e-06 2024-09-15T03:48:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-15T03:48:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934905+0000", 2024-09-15T03:48:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0060000000000003e-06 2024-09-15T03:48:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-15T03:48:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934910+0000", 2024-09-15T03:48:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1499999999999998e-06 2024-09-15T03:48:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934935+0000", 2024-09-15T03:48:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4641e-05 2024-09-15T03:48:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-15T03:48:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:53.934935+0000", 2024-09-15T03:48:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7099999999999997e-07 2024-09-15T03:48:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.231303+0000", 2024-09-15T03:48:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.29636852000000002 2024-09-15T03:48:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-15T03:48:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.231460+0000", 2024-09-15T03:48:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015613700000000001 2024-09-15T03:48:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-15T03:48:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.233845+0000", 2024-09-15T03:48:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023851660000000002 2024-09-15T03:48:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-15T03:48:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.233998+0000", 2024-09-15T03:48:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015345699999999999 2024-09-15T03:48:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T03:48:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-15T03:48:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-15T03:48:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-15T03:47:54.236147+0000", 2024-09-15T03:48:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021492460000000001 2024-09-15T03:48:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T03:48:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T03:48:09.996 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:48:09.997 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:48:09.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:48:09.999 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:48:09.997+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:48:10.099 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:48:10.098+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:48:10.124 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0: {} 2024-09-15T03:48:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:48:10.139 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1: {} 2024-09-15T03:48:10.139 INFO:teuthology.orchestra.run.smithi001.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:48:10.140 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T03:48:10.140 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-15T03:48:10.140 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-15T03:48:10.140 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T03:48:10.148 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:48:10.153 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2: {} 2024-09-15T03:48:10.153 INFO:teuthology.orchestra.run.smithi001.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:48:10.168 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3: {} 2024-09-15T03:48:10.168 INFO:teuthology.orchestra.run.smithi001.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:48:10.183 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4: {} 2024-09-15T03:48:10.183 INFO:teuthology.orchestra.run.smithi001.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:48:10.198 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5: {} 2024-09-15T03:48:10.198 INFO:teuthology.orchestra.run.smithi001.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:48:10.199 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:48:10.199+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:48:10.212 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6: {} 2024-09-15T03:48:10.213 INFO:teuthology.orchestra.run.smithi001.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:48:10.227 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7: {} 2024-09-15T03:48:10.227 INFO:teuthology.orchestra.run.smithi001.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:48:10.242 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8: {} 2024-09-15T03:48:10.243 INFO:teuthology.orchestra.run.smithi001.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:48:10.250 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:10.250 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:10.258 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9: {} 2024-09-15T03:48:10.258 INFO:teuthology.orchestra.run.smithi001.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:48:10.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:48:10.274 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10: {} 2024-09-15T03:48:10.274 INFO:teuthology.orchestra.run.smithi001.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:48:10.288 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11: {} 2024-09-15T03:48:10.289 INFO:teuthology.orchestra.run.smithi001.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:48:10.300 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:48:10.299+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:48:10.400 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:10.400 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:10.401 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:48:10.400+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:48:10.408 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:48:10.501 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:48:10.500+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:48:10.509 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:10.509 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:10.517 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:48:10.602 INFO:tasks.ceph.osd.10.smithi190.stderr:2024-09-15T03:48:10.601+0000 7fd62abf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33736) UID: 0 2024-09-15T03:48:10.618 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:10.618 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:10.626 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:48:10.703 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-09-15T03:48:10.703+0000 7fcd1cdf9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28893) UID: 0 2024-09-15T03:48:10.726 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:10.727 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:10.735 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:48:10.804 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:48:10.804+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:48:10.836 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:10.836 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:10.845 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:48:10.882 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:48:10.905 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:48:10.904+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:48:10.946 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:10.946 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:10.954 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:48:11.005 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:48:11.004+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:48:11.005 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T03:48:11.095 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:11.095 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:11.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:48:11.105 INFO:tasks.ceph.osd.4.smithi067.stderr:2024-09-15T03:48:11.105+0000 7f63caff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28459) UID: 0 2024-09-15T03:48:11.169 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:48:11.170 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-15T03:48:11.181 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-09-15T03:48:10.164447+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":183697971,"num_objects":193,"num_object_clones":139,"num_object_copies":1520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":9,"num_read":533,"num_read_kb":239247,"num_write":1029,"num_write_kb":173944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":267,"num_bytes_recovered":285786150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1589,"ondisk_log_size":1589,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":12,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":1124155392,"kb_used":10218316,"kb_used_data":1113938024,"kb_used_omap":55701,"kb_used_meta":0,"kb_avail":1113937076,"statfs":{"total":1151135121408,"available":1140671565824,"internally_reserved":970752,"allocated":1140672536576,"data_stored":1140672536576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57038708,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.190789"},"pg_stats":[{"pgid":"3.f","version":"113'71","reported_seq":461,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.190992+0000","last_change":"2024-09-15T03:48:03.620987+0000","last_active":"2024-09-15T03:48:05.190992+0000","last_peered":"2024-09-15T03:48:05.190992+0000","last_clean":"2024-09-15T03:48:05.190992+0000","last_became_active":"2024-09-15T03:47:55.850051+0000","last_became_peered":"2024-09-15T03:47:55.850051+0000","last_unstale":"2024-09-15T03:48:05.190992+0000","last_undegraded":"2024-09-15T03:48:05.190992+0000","last_fullsized":"2024-09-15T03:48:05.190992+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'71","last_scrub_stamp":"2024-09-15T03:48:03.620910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:03.620910+0000","objects_scrubbed":8,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:13.277978+0000","scrub_duration":0.0050823509999999997,"objects_trimmed":0,"snaptrim_duration":0.0077598160000000001,"stat_sum":{"num_bytes":8028160,"num_objects":8,"num_object_clones":5,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":32,"num_read_kb":42506,"num_write":26,"num_write_kb":7840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0,7,2,8,4,10],"acting":[11,1,0,7,2,8,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.e","version":"113'210","reported_seq":442,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.974944+0000","last_change":"2024-09-15T03:47:59.982417+0000","last_active":"2024-09-15T03:48:04.974944+0000","last_peered":"2024-09-15T03:48:04.974944+0000","last_clean":"2024-09-15T03:48:04.974944+0000","last_became_active":"2024-09-15T03:46:36.170536+0000","last_became_peered":"2024-09-15T03:46:36.170536+0000","last_unstale":"2024-09-15T03:48:04.974944+0000","last_undegraded":"2024-09-15T03:48:04.974944+0000","last_fullsized":"2024-09-15T03:48:04.974944+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'173","last_scrub_stamp":"2024-09-15T03:47:54.108780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:47:54.108780+0000","objects_scrubbed":21,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:11.038305+0000","scrub_duration":0.0088561600000000001,"objects_trimmed":0,"snaptrim_duration":0.037652644999999998,"stat_sum":{"num_bytes":17612800,"num_objects":22,"num_object_clones":15,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":2,"num_read":68,"num_read_kb":34008,"num_write":51,"num_write_kb":15520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,11,7,8,5,9],"acting":[2,0,1,11,7,8,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.d","version":"113'75","reported_seq":399,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.191773+0000","last_change":"2024-09-15T03:47:59.958101+0000","last_active":"2024-09-15T03:48:05.191773+0000","last_peered":"2024-09-15T03:48:05.191773+0000","last_clean":"2024-09-15T03:48:05.191773+0000","last_became_active":"2024-09-15T03:47:55.850104+0000","last_became_peered":"2024-09-15T03:47:55.850104+0000","last_unstale":"2024-09-15T03:48:05.191773+0000","last_undegraded":"2024-09-15T03:48:05.191773+0000","last_fullsized":"2024-09-15T03:48:05.191773+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'62","last_scrub_stamp":"2024-09-15T03:47:56.721765+0000","last_deep_scrub":"109'62","last_deep_scrub_stamp":"2024-09-15T03:47:56.721765+0000","last_clean_scrub_stamp":"2024-09-15T03:47:56.721765+0000","objects_scrubbed":10,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:15.880649+0000","scrub_duration":0.00827518,"objects_trimmed":0,"snaptrim_duration":0.013785528999999999,"stat_sum":{"num_bytes":7651328,"num_objects":10,"num_object_clones":8,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":20,"num_read_kb":8406,"num_write":43,"num_write_kb":7472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":10780672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,1,3,10,7,9,4],"acting":[8,11,1,3,10,7,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.c","version":"113'28","reported_seq":383,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.840818+0000","last_change":"2024-09-15T03:47:59.955875+0000","last_active":"2024-09-15T03:48:04.840818+0000","last_peered":"2024-09-15T03:48:04.840818+0000","last_clean":"2024-09-15T03:48:04.840818+0000","last_became_active":"2024-09-15T03:47:36.848722+0000","last_became_peered":"2024-09-15T03:47:36.848722+0000","last_unstale":"2024-09-15T03:48:04.840818+0000","last_undegraded":"2024-09-15T03:48:04.840818+0000","last_fullsized":"2024-09-15T03:48:04.840818+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'16","last_scrub_stamp":"2024-09-15T03:47:41.493521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:47:41.493521+0000","objects_scrubbed":3,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:09.644586+0000","scrub_duration":0.0039888800000000002,"objects_trimmed":0,"snaptrim_duration":0.012831169999999999,"stat_sum":{"num_bytes":2965504,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":18,"num_read_kb":8502,"num_write":10,"num_write_kb":3440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,11,1,0,6,5,2],"acting":[9,7,11,1,0,6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.b","version":"113'212","reported_seq":766,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.843752+0000","last_change":"2024-09-15T03:48:03.010113+0000","last_active":"2024-09-15T03:48:04.843752+0000","last_peered":"2024-09-15T03:48:04.843752+0000","last_clean":"2024-09-15T03:48:04.843752+0000","last_became_active":"2024-09-15T03:47:55.850099+0000","last_became_peered":"2024-09-15T03:47:55.850099+0000","last_unstale":"2024-09-15T03:48:04.843752+0000","last_undegraded":"2024-09-15T03:48:04.843752+0000","last_fullsized":"2024-09-15T03:48:04.843752+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'212","last_scrub_stamp":"2024-09-15T03:48:03.010035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:03.010035+0000","objects_scrubbed":27,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:22.035341+0000","scrub_duration":0.022865639,"objects_trimmed":0,"snaptrim_duration":0.025351973,"stat_sum":{"num_bytes":24117248,"num_objects":27,"num_object_clones":20,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":2,"num_read":65,"num_read_kb":36889,"num_write":170,"num_write_kb":22832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":74006528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,8,4,1,10,11,3],"acting":[7,6,8,4,1,10,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.a","version":"113'84","reported_seq":240,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.896901+0000","last_change":"2024-09-15T03:48:03.286753+0000","last_active":"2024-09-15T03:48:04.896901+0000","last_peered":"2024-09-15T03:48:04.896901+0000","last_clean":"2024-09-15T03:48:04.896901+0000","last_became_active":"2024-09-15T03:47:55.854826+0000","last_became_peered":"2024-09-15T03:47:55.854826+0000","last_unstale":"2024-09-15T03:48:04.896901+0000","last_undegraded":"2024-09-15T03:48:04.896901+0000","last_fullsized":"2024-09-15T03:48:04.896901+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'84","last_scrub_stamp":"2024-09-15T03:48:03.286675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:03.286675+0000","objects_scrubbed":8,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:03.509620+0000","scrub_duration":0.0063142770000000001,"objects_trimmed":0,"snaptrim_duration":0.013016882,"stat_sum":{"num_bytes":6995968,"num_objects":8,"num_object_clones":6,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":16,"num_read_kb":6728,"num_write":30,"num_write_kb":6832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":30703616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8,9,5,6,2,0],"acting":[4,10,8,9,5,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.9","version":"113'80","reported_seq":492,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.979274+0000","last_change":"2024-09-15T03:47:59.971224+0000","last_active":"2024-09-15T03:48:04.979274+0000","last_peered":"2024-09-15T03:48:04.979274+0000","last_clean":"2024-09-15T03:48:04.979274+0000","last_became_active":"2024-09-15T03:47:55.851394+0000","last_became_peered":"2024-09-15T03:47:55.851394+0000","last_unstale":"2024-09-15T03:48:04.979274+0000","last_undegraded":"2024-09-15T03:48:04.979274+0000","last_fullsized":"2024-09-15T03:48:04.979274+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012050854,"stat_sum":{"num_bytes":12025856,"num_objects":13,"num_object_clones":10,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":15,"num_read_kb":8,"num_write":59,"num_write_kb":11744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":27590656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,10,1,9,6,8,11],"acting":[3,2,10,1,9,6,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.8","version":"113'52","reported_seq":377,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.979298+0000","last_change":"2024-09-15T03:47:59.962623+0000","last_active":"2024-09-15T03:48:04.979298+0000","last_peered":"2024-09-15T03:48:04.979298+0000","last_clean":"2024-09-15T03:48:04.979298+0000","last_became_active":"2024-09-15T03:47:55.848776+0000","last_became_peered":"2024-09-15T03:47:55.848776+0000","last_unstale":"2024-09-15T03:48:04.979298+0000","last_undegraded":"2024-09-15T03:48:04.979298+0000","last_fullsized":"2024-09-15T03:48:04.979298+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01649231,"stat_sum":{"num_bytes":4079616,"num_objects":6,"num_object_clones":4,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":23,"num_read_kb":4616,"num_write":54,"num_write_kb":3984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":11223040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,11,4,1,0,10],"acting":[3,7,2,11,4,1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"1.0","version":"20'92","reported_seq":226,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.059683+0000","last_change":"2024-09-15T03:47:39.774326+0000","last_active":"2024-09-15T03:48:04.059683+0000","last_peered":"2024-09-15T03:48:04.059683+0000","last_clean":"2024-09-15T03:48:04.059683+0000","last_became_active":"2024-09-15T03:46:18.011130+0000","last_became_peered":"2024-09-15T03:46:18.011130+0000","last_unstale":"2024-09-15T03:48:04.059683+0000","last_undegraded":"2024-09-15T03:48:04.059683+0000","last_fullsized":"2024-09-15T03:48:04.059683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-15T03:47:39.774262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_clean_scrub_stamp":"2024-09-15T03:47:39.774262+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:43.691013+0000","scrub_duration":0.0027413759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"22'1","reported_seq":118,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:03.962737+0000","last_change":"2024-09-15T03:47:52.292975+0000","last_active":"2024-09-15T03:48:03.962737+0000","last_peered":"2024-09-15T03:48:03.962737+0000","last_clean":"2024-09-15T03:48:03.962737+0000","last_became_active":"2024-09-15T03:46:24.038633+0000","last_became_peered":"2024-09-15T03:46:24.038633+0000","last_unstale":"2024-09-15T03:48:03.962737+0000","last_undegraded":"2024-09-15T03:48:03.962737+0000","last_fullsized":"2024-09-15T03:48:03.962737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'1","last_scrub_stamp":"2024-09-15T03:47:52.292926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:52.292926+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:56.362597+0000","scrub_duration":0.001409429,"objects_trimmed":0,"snaptrim_duration":0.00030178200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"113'80","reported_seq":513,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.896581+0000","last_change":"2024-09-15T03:47:59.966281+0000","last_active":"2024-09-15T03:48:04.896581+0000","last_peered":"2024-09-15T03:48:04.896581+0000","last_clean":"2024-09-15T03:48:04.896581+0000","last_became_active":"2024-09-15T03:47:55.854075+0000","last_became_peered":"2024-09-15T03:47:55.854075+0000","last_unstale":"2024-09-15T03:48:04.896581+0000","last_undegraded":"2024-09-15T03:48:04.896581+0000","last_fullsized":"2024-09-15T03:48:04.896581+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021707911999999999,"stat_sum":{"num_bytes":13320192,"num_objects":10,"num_object_clones":7,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":37,"num_read_kb":16719,"num_write":39,"num_write_kb":13088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":18628608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,7,8,6,11,1],"acting":[4,2,10,7,8,6,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":115,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.059319+0000","last_change":"2024-09-15T03:47:27.779257+0000","last_active":"2024-09-15T03:48:04.059319+0000","last_peered":"2024-09-15T03:48:04.059319+0000","last_clean":"2024-09-15T03:48:04.059319+0000","last_became_active":"2024-09-15T03:46:24.063106+0000","last_became_peered":"2024-09-15T03:46:24.063106+0000","last_unstale":"2024-09-15T03:48:04.059319+0000","last_undegraded":"2024-09-15T03:48:04.059319+0000","last_fullsized":"2024-09-15T03:48:04.059319+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.779210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.779210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:42.717969+0000","scrub_duration":0.0021030710000000002,"objects_trimmed":0,"snaptrim_duration":0.000393075,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"113'29","reported_seq":174,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.687310+0000","last_change":"2024-09-15T03:48:05.687193+0000","last_active":"2024-09-15T03:48:05.687310+0000","last_peered":"2024-09-15T03:48:05.687310+0000","last_clean":"2024-09-15T03:48:05.687310+0000","last_became_active":"2024-09-15T03:47:55.848033+0000","last_became_peered":"2024-09-15T03:47:55.848033+0000","last_unstale":"2024-09-15T03:48:05.687310+0000","last_undegraded":"2024-09-15T03:48:05.687310+0000","last_fullsized":"2024-09-15T03:48:05.687310+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'29","last_scrub_stamp":"2024-09-15T03:48:05.687085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:05.687085+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:32.245260+0000","scrub_duration":0.0044105530000000002,"objects_trimmed":0,"snaptrim_duration":0.0090284240000000002,"stat_sum":{"num_bytes":4653056,"num_objects":5,"num_object_clones":4,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":3075,"num_write":12,"num_write_kb":4032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":6045696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3,10,5,2,9,7],"acting":[6,11,3,10,5,2,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":120,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.843432+0000","last_change":"2024-09-15T03:47:42.997012+0000","last_active":"2024-09-15T03:48:04.843432+0000","last_peered":"2024-09-15T03:48:04.843432+0000","last_clean":"2024-09-15T03:48:04.843432+0000","last_became_active":"2024-09-15T03:46:24.037471+0000","last_became_peered":"2024-09-15T03:46:24.037471+0000","last_unstale":"2024-09-15T03:48:04.843432+0000","last_undegraded":"2024-09-15T03:48:04.843432+0000","last_fullsized":"2024-09-15T03:48:04.843432+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:42.996957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:42.996957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:53.443125+0000","scrub_duration":0.0014164309999999999,"objects_trimmed":0,"snaptrim_duration":0.00026490299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"113'56","reported_seq":461,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-15T03:48:06.207177+0000","last_change":"2024-09-15T03:48:03.778794+0000","last_active":"2024-09-15T03:48:06.207177+0000","last_peered":"2024-09-15T03:48:06.207177+0000","last_clean":"2024-09-15T03:48:06.207177+0000","last_became_active":"2024-09-15T03:47:55.848512+0000","last_became_peered":"2024-09-15T03:47:55.848512+0000","last_unstale":"2024-09-15T03:48:06.207177+0000","last_undegraded":"2024-09-15T03:48:06.207177+0000","last_fullsized":"2024-09-15T03:48:06.207177+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'56","last_scrub_stamp":"2024-09-15T03:48:03.778652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:03.778652+0000","objects_scrubbed":8,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:05.162717+0000","scrub_duration":0.0067734900000000001,"objects_trimmed":0,"snaptrim_duration":0.012362052,"stat_sum":{"num_bytes":4997120,"num_objects":8,"num_object_clones":5,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":20,"num_read_kb":7496,"num_write":43,"num_write_kb":4880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":16449536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,9,4,3,6,0,10],"acting":[1,2,9,4,3,6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.2","version":"24'2","reported_seq":161,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.781140+0000","last_change":"2024-09-15T03:48:04.781084+0000","last_active":"2024-09-15T03:48:04.781140+0000","last_peered":"2024-09-15T03:48:04.781140+0000","last_clean":"2024-09-15T03:48:04.781140+0000","last_became_active":"2024-09-15T03:47:36.848076+0000","last_became_peered":"2024-09-15T03:47:36.848076+0000","last_unstale":"2024-09-15T03:48:04.781140+0000","last_undegraded":"2024-09-15T03:48:04.781140+0000","last_fullsized":"2024-09-15T03:48:04.781140+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'2","last_scrub_stamp":"2024-09-15T03:48:04.781025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:48:04.781025+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:13.832081+0000","scrub_duration":0.0016375750000000001,"objects_trimmed":0,"snaptrim_duration":0.000944677,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"113'196","reported_seq":461,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.094888+0000","last_change":"2024-09-15T03:47:59.976831+0000","last_active":"2024-09-15T03:48:05.094888+0000","last_peered":"2024-09-15T03:48:05.094888+0000","last_clean":"2024-09-15T03:48:05.094888+0000","last_became_active":"2024-09-15T03:47:55.855061+0000","last_became_peered":"2024-09-15T03:47:55.855061+0000","last_unstale":"2024-09-15T03:48:05.094888+0000","last_undegraded":"2024-09-15T03:48:05.094888+0000","last_fullsized":"2024-09-15T03:48:05.094888+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'156","last_scrub_stamp":"2024-09-15T03:47:56.178879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:47:56.178879+0000","objects_scrubbed":22,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:09.356437+0000","scrub_duration":0.014803648000000001,"objects_trimmed":0,"snaptrim_duration":0.031600082000000002,"stat_sum":{"num_bytes":20545536,"num_objects":25,"num_object_clones":19,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":27,"num_read_kb":352,"num_write":173,"num_write_kb":21440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,4,7,1,2,9,0],"acting":[10,8,4,7,1,2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"3.5","version":"113'79","reported_seq":384,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.094605+0000","last_change":"2024-09-15T03:48:04.063851+0000","last_active":"2024-09-15T03:48:05.094605+0000","last_peered":"2024-09-15T03:48:05.094605+0000","last_clean":"2024-09-15T03:48:05.094605+0000","last_became_active":"2024-09-15T03:47:55.860659+0000","last_became_peered":"2024-09-15T03:47:55.860659+0000","last_unstale":"2024-09-15T03:48:05.094605+0000","last_undegraded":"2024-09-15T03:48:05.094605+0000","last_fullsized":"2024-09-15T03:48:05.094605+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'79","last_scrub_stamp":"2024-09-15T03:48:04.063762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:04.063762+0000","objects_scrubbed":9,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:30.399619+0000","scrub_duration":0.0069124379999999999,"objects_trimmed":0,"snaptrim_duration":0.013330699,"stat_sum":{"num_bytes":24330240,"num_objects":9,"num_object_clones":8,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":14,"num_read_kb":11048,"num_write":34,"num_write_kb":13376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":23789568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4,9,8,7,3,0],"acting":[10,6,4,9,8,7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":138,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.840281+0000","last_change":"2024-09-15T03:47:39.930675+0000","last_active":"2024-09-15T03:48:04.840281+0000","last_peered":"2024-09-15T03:48:04.840281+0000","last_clean":"2024-09-15T03:48:04.840281+0000","last_became_active":"2024-09-15T03:47:33.998802+0000","last_became_peered":"2024-09-15T03:47:33.998802+0000","last_unstale":"2024-09-15T03:48:04.840281+0000","last_undegraded":"2024-09-15T03:48:04.840281+0000","last_fullsized":"2024-09-15T03:48:04.840281+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:39.930628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:39.930628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:03.535396+0000","scrub_duration":0.001146333,"objects_trimmed":0,"snaptrim_duration":0.00034123999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"113'42","reported_seq":399,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.191327+0000","last_change":"2024-09-15T03:48:03.723173+0000","last_active":"2024-09-15T03:48:05.191327+0000","last_peered":"2024-09-15T03:48:05.191327+0000","last_clean":"2024-09-15T03:48:05.191327+0000","last_became_active":"2024-09-15T03:47:55.846248+0000","last_became_peered":"2024-09-15T03:47:55.846248+0000","last_unstale":"2024-09-15T03:48:05.191327+0000","last_undegraded":"2024-09-15T03:48:05.191327+0000","last_fullsized":"2024-09-15T03:48:05.191327+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'42","last_scrub_stamp":"2024-09-15T03:48:03.723102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:03.723102+0000","objects_scrubbed":5,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:25.602306+0000","scrub_duration":0.0052425570000000001,"objects_trimmed":0,"snaptrim_duration":0.011769172,"stat_sum":{"num_bytes":7389184,"num_objects":5,"num_object_clones":4,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":6052,"num_write":11,"num_write_kb":7952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":9814016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5,8,0,4,7,10],"acting":[6,2,5,8,0,4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":117,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.060875+0000","last_change":"2024-09-15T03:47:30.025153+0000","last_active":"2024-09-15T03:48:04.060875+0000","last_peered":"2024-09-15T03:48:04.060875+0000","last_clean":"2024-09-15T03:48:04.060875+0000","last_became_active":"2024-09-15T03:46:24.038945+0000","last_became_peered":"2024-09-15T03:46:24.038945+0000","last_unstale":"2024-09-15T03:48:04.060875+0000","last_undegraded":"2024-09-15T03:48:04.060875+0000","last_fullsized":"2024-09-15T03:48:04.060875+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:30.025107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:30.025107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:36.231260+0000","scrub_duration":0.0013039429999999999,"objects_trimmed":0,"snaptrim_duration":0.00032787499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"113'153","reported_seq":520,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:05.258880+0000","last_change":"2024-09-15T03:48:05.258804+0000","last_active":"2024-09-15T03:48:05.258880+0000","last_peered":"2024-09-15T03:48:05.258880+0000","last_clean":"2024-09-15T03:48:05.258880+0000","last_became_active":"2024-09-15T03:47:55.848466+0000","last_became_peered":"2024-09-15T03:47:55.848466+0000","last_unstale":"2024-09-15T03:48:05.258880+0000","last_undegraded":"2024-09-15T03:48:05.258880+0000","last_fullsized":"2024-09-15T03:48:05.258880+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'153","last_scrub_stamp":"2024-09-15T03:48:05.258733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:48:05.258733+0000","objects_scrubbed":22,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:30.071424+0000","scrub_duration":0.012102717000000001,"objects_trimmed":0,"snaptrim_duration":0.036439371999999998,"stat_sum":{"num_bytes":19546112,"num_objects":22,"num_object_clones":17,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":1,"num_read":64,"num_read_kb":52263,"num_write":118,"num_write_kb":18976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":27639808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,7,2,1,11,10],"acting":[4,3,9,7,2,1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":118,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.060192+0000","last_change":"2024-09-15T03:47:27.297139+0000","last_active":"2024-09-15T03:48:04.060192+0000","last_peered":"2024-09-15T03:48:04.060192+0000","last_clean":"2024-09-15T03:48:04.060192+0000","last_became_active":"2024-09-15T03:46:24.063380+0000","last_became_peered":"2024-09-15T03:46:24.063380+0000","last_unstale":"2024-09-15T03:48:04.060192+0000","last_undegraded":"2024-09-15T03:48:04.060192+0000","last_fullsized":"2024-09-15T03:48:04.060192+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.297083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.297083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:48.875998+0000","scrub_duration":0.0018328789999999999,"objects_trimmed":0,"snaptrim_duration":0.00034490600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"113'47","reported_seq":487,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.896781+0000","last_change":"2024-09-15T03:47:59.953939+0000","last_active":"2024-09-15T03:48:04.896781+0000","last_peered":"2024-09-15T03:48:04.896781+0000","last_clean":"2024-09-15T03:48:04.896781+0000","last_became_active":"2024-09-15T03:47:55.847868+0000","last_became_peered":"2024-09-15T03:47:55.847868+0000","last_unstale":"2024-09-15T03:48:04.896781+0000","last_undegraded":"2024-09-15T03:48:04.896781+0000","last_fullsized":"2024-09-15T03:48:04.896781+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:35.075105+0000","last_clean_scrub_stamp":"2024-09-15T03:46:35.075105+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010037447,"stat_sum":{"num_bytes":4849664,"num_objects":7,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":13,"num_read_kb":515,"num_write":31,"num_write_kb":8384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":7176192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,9,8,3,11,1,10],"acting":[4,7,9,8,3,11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":115,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-15T03:48:04.059914+0000","last_change":"2024-09-15T03:47:42.675933+0000","last_active":"2024-09-15T03:48:04.059914+0000","last_peered":"2024-09-15T03:48:04.059914+0000","last_clean":"2024-09-15T03:48:04.059914+0000","last_became_active":"2024-09-15T03:46:24.063335+0000","last_became_peered":"2024-09-15T03:46:24.063335+0000","last_unstale":"2024-09-15T03:48:04.059914+0000","last_undegraded":"2024-09-15T03:48:04.059914+0000","last_fullsized":"2024-09-15T03:48:04.059914+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:42.675882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:42.675882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:02.697036+0000","scrub_duration":0.001445478,"objects_trimmed":0,"snaptrim_duration":0.00065928000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":183107584,"num_objects":189,"num_object_clones":139,"num_object_copies":1512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":189,"num_whiteouts":9,"num_read":457,"num_read_kb":239183,"num_write":904,"num_write_kb":171792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":265,"num_bytes_recovered":285786112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1494,"ondisk_log_size":1494,"up":128,"acting":128,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":11,"up_from":17,"seq":73014444056,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":846004,"kb_used_data":92833684,"kb_used_omap":4805,"kb_used_meta":0,"kb_avail":92833612,"statfs":{"total":95927926784,"available":95061618688,"internally_reserved":73728,"allocated":95061692416,"data_stored":95061692416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4920982,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.46600000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":121.575,"5min":121.575,"15min":121.575},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2060.797,"5min":2060.797,"15min":2060.797},"last":0.753},{"interface":"front","average":{"1min":121.56,"5min":121.56,"15min":121.56},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2060.7159999999999,"5min":2060.7159999999999,"15min":2060.7159999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.61499999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.79300000000000004}]},{"osd":4,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.67300000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.68400000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.70299999999999996}]},{"osd":7,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.63},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.74299999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.59999999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.84299999999999997}]},{"osd":10,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.82199999999999995}]}]},{"osd":10,"up_from":105,"seq":450971566084,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":871700,"kb_used_data":92807916,"kb_used_omap":3648,"kb_used_meta":0,"kb_avail":92807916,"statfs":{"total":95927926784,"available":95035305984,"internally_reserved":0,"allocated":95035305984,"data_stored":95035305984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3736470,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]}]},{"osd":9,"up_from":17,"seq":73014444056,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":855500,"kb_used_data":92824200,"kb_used_omap":5160,"kb_used_meta":0,"kb_avail":92824116,"statfs":{"total":95927926784,"available":95051894784,"internally_reserved":86016,"allocated":95051980800,"data_stored":95051980800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5284527,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.499},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.61499999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.51400000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.66100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.56299999999999994}]},{"osd":5,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.52500000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.46700000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.55000000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.625}]},{"osd":10,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.54100000000000004}]},{"osd":11,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.57399999999999995}]}]},{"osd":8,"up_from":17,"seq":73014444057,"num_pgs":15,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":861168,"kb_used_data":92818540,"kb_used_omap":5191,"kb_used_meta":0,"kb_avail":92818448,"statfs":{"total":95927926784,"available":95046090752,"internally_reserved":94208,"allocated":95046184960,"data_stored":95046184960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5316028,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.81699999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":42.381,"5min":42.381,"15min":42.381},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":963.98400000000004,"5min":963.98400000000004,"15min":963.98400000000004},"last":0.63100000000000001},{"interface":"front","average":{"1min":42.447000000000003,"5min":42.447000000000003,"15min":42.447000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":964.79600000000005,"5min":964.79600000000005,"15min":964.79600000000005},"last":0.56499999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.752}]},{"osd":4,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.83299999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:47:25 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.79200000000000004}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.73099999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.73999999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.437},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.60899999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.80400000000000005}]}]},{"osd":7,"up_from":17,"seq":73014444056,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":864892,"kb_used_data":92814832,"kb_used_omap":5646,"kb_used_meta":0,"kb_avail":92814724,"statfs":{"total":95927926784,"available":95042277376,"internally_reserved":110592,"allocated":95042387968,"data_stored":95042387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5781943,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.58199999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.41699999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.46700000000000003}]},{"osd":4,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.56000000000000005}]},{"osd":5,"last update":"Sun Sep 15 03:47:27 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.59999999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.71399999999999997}]},{"osd":8,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.626},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.66500000000000004}]},{"osd":9,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.75900000000000001}]},{"osd":10,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.746}]},{"osd":11,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.78300000000000003}]}]},{"osd":6,"up_from":17,"seq":73014444056,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":838972,"kb_used_data":92840728,"kb_used_omap":3843,"kb_used_meta":0,"kb_avail":92840644,"statfs":{"total":95927926784,"available":95068819456,"internally_reserved":86016,"allocated":95068905472,"data_stored":95068905472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3935639,"internal_metadata":0},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.52000000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:47:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.56299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.57399999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.55200000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.67500000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.66000000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.502}]},{"osd":8,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.78400000000000003}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.77700000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":11,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.83299999999999996}]}]},{"osd":5,"up_from":17,"seq":73014444056,"num_pgs":7,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":814900,"kb_used_data":92864772,"kb_used_omap":1826,"kb_used_meta":0,"kb_avail":92864716,"statfs":{"total":95927926784,"available":95093469184,"internally_reserved":57344,"allocated":95093526528,"data_stored":95093526528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1870318,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.29799999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":113.717,"5min":113.717,"15min":113.717},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2037.425,"5min":2037.425,"15min":2037.425},"last":0.34000000000000002},{"interface":"front","average":{"1min":113.752,"5min":113.752,"15min":113.752},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2037.3230000000001,"5min":2037.3230000000001,"15min":2037.3230000000001},"last":0.51400000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.379},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.53700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":4,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.439}]},{"osd":6,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.42499999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.497}]},{"osd":8,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.61799999999999999}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.63},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.65500000000000003}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.83399999999999996}]},{"osd":11,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.58899999999999997}]}]},{"osd":4,"up_from":17,"seq":73014444056,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":863984,"kb_used_data":92815728,"kb_used_omap":5620,"kb_used_meta":0,"kb_avail":92815632,"statfs":{"total":95927926784,"available":95043207168,"internally_reserved":98304,"allocated":95043305472,"data_stored":95043305472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5755199,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.47299999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":84.314999999999998,"5min":84.314999999999998,"15min":84.314999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1758.9780000000001,"5min":1758.9780000000001,"15min":1758.9780000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":84.364000000000004,"5min":84.364000000000004,"15min":84.364000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1758.671,"5min":1758.671,"15min":1758.671},"last":0.42699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.48999999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.33700000000000002}]},{"osd":5,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.375},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.52500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.55700000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.61299999999999999}]},{"osd":9,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":0.623}]},{"osd":10,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.53800000000000003}]},{"osd":11,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.58399999999999996}]}]},{"osd":3,"up_from":17,"seq":73014444056,"num_pgs":10,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":842612,"kb_used_data":92837084,"kb_used_omap":4588,"kb_used_meta":0,"kb_avail":92837004,"statfs":{"total":95927926784,"available":95065092096,"internally_reserved":81920,"allocated":95065174016,"data_stored":95065174016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4698419,"internal_metadata":0},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.27300000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:47:38 2024","interfaces":[{"interface":"back","average":{"1min":35.600000000000001,"5min":35.600000000000001,"15min":35.600000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":564.05100000000004,"5min":564.05100000000004,"15min":564.05100000000004},"last":0.309},{"interface":"front","average":{"1min":35.677,"5min":35.677,"15min":35.677},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":564.57000000000005,"5min":564.57000000000005,"15min":564.57000000000005},"last":0.40500000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.49399999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.61399999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":2.3940000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":5.2830000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.59699999999999998}]},{"osd":7,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.443},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.64400000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.89100000000000001}]},{"osd":9,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.84199999999999997}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.752}]}]},{"osd":2,"up_from":17,"seq":73014444056,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":855316,"kb_used_data":92824420,"kb_used_omap":5155,"kb_used_meta":0,"kb_avail":92824300,"statfs":{"total":95927926784,"available":95052083200,"internally_reserved":122880,"allocated":95052206080,"data_stored":95052206080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5279060,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:37 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.438}]},{"osd":1,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":34.457999999999998,"5min":34.457999999999998,"15min":34.457999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":647.92399999999998,"5min":647.92399999999998,"15min":647.92399999999998},"last":0.28100000000000003},{"interface":"front","average":{"1min":34.475000000000001,"5min":34.475000000000001,"15min":34.475000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":648.02700000000004,"5min":648.02700000000004,"15min":648.02700000000004},"last":0.34999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.46400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.54500000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.55400000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.503},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.56599999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.343,"15min":1.343},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":14.721,"5min":14.721,"15min":14.721},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.371,"5min":1.371,"15min":1.371},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":14.970000000000001,"5min":14.970000000000001,"15min":14.970000000000001},"last":0.77800000000000002}]},{"osd":9,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.377,"15min":1.377},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":14.888,"5min":14.888,"15min":14.888},"last":0.69799999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":14.654999999999999,"5min":14.654999999999999,"15min":14.654999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":1.409,"5min":1.409,"15min":1.409},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":15.096,"5min":15.096,"15min":15.096},"last":0.60499999999999998}]}]},{"osd":1,"up_from":17,"seq":73014444057,"num_pgs":13,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":859152,"kb_used_data":92820576,"kb_used_omap":5959,"kb_used_meta":0,"kb_avail":92820464,"statfs":{"total":95927926784,"available":95048155136,"internally_reserved":114688,"allocated":95048269824,"data_stored":95048269824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6102057,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.46800000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.51900000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.432},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.504}]},{"osd":4,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":2.6920000000000002,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":0.65500000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.7440000000000002,"5min":2.7440000000000002,"15min":2.7440000000000002},"last":0.67700000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":3.278,"5min":3.278,"15min":3.278},"last":0.73799999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.81399999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":119.93899999999999,"5min":119.93899999999999,"15min":119.93899999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2861.7190000000001,"5min":2861.7190000000001,"15min":2861.7190000000001},"last":0.749},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.946,"5min":1.946,"15min":1.946},"last":0.622}]},{"osd":9,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":119.937,"5min":119.937,"15min":119.937},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":2861.6010000000001,"5min":2861.6010000000001,"15min":2861.6010000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.69199999999999995}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":119.96599999999999,"5min":119.96599999999999,"15min":119.96599999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2861.5639999999999,"5min":2861.5639999999999,"15min":2861.5639999999999},"last":0.70199999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]}]},{"osd":0,"up_from":17,"seq":73014444056,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":844116,"kb_used_data":92835544,"kb_used_omap":4255,"kb_used_meta":0,"kb_avail":92835500,"statfs":{"total":95927926784,"available":95063552000,"internally_reserved":45056,"allocated":95063597056,"data_stored":95063597056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4358066,"internal_metadata":0},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.46999999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.804,"5min":1.804,"15min":1.804},"last":0.379}]},{"osd":3,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.52,"5min":1.52,"15min":1.52},"last":0.376},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.35399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":5,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.502}]},{"osd":6,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.48999999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.68700000000000006}]},{"osd":8,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.44400000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":10,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.58899999999999997}]},{"osd":11,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.56499999999999995}]}]}],"pool_statfs":[]}} 2024-09-15T03:48:11.184 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-15T03:48:11.185 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-15T03:48:11.213 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:11.213 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:11.222 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:48:11.228 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:48:11.228+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:48:11.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-15T03:48:11.306+0000 7f402ffff640 0 --2- 172.21.15.1:0/1367684231 >> v2:172.21.15.67:3300/0 conn(0x7f403014c470 0x7f403014e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-15T03:48:11.307 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:48:11.306+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:48:11.321 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:11.321 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:11.330 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:48:11.408 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-09-15T03:48:11.407+0000 7f0a92df9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28885) UID: 0 2024-09-15T03:48:11.426 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:11.427 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:11.435 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:48:11.510 INFO:tasks.ceph.osd.7.smithi067.stderr:2024-09-15T03:48:11.509+0000 7f38d17f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28466) UID: 0 2024-09-15T03:48:11.531 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:11.531 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:11.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:48:11.610 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:48:11.609+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:48:11.636 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:11.636 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:11.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:48:11.729 INFO:tasks.ceph.osd.8.smithi190.stderr:2024-09-15T03:48:11.729+0000 7f52271f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 29461) UID: 0 2024-09-15T03:48:11.782 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:11.782 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:11.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:48:11.812 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-09-15T03:48:11.812+0000 7f624a5f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28881) UID: 0 2024-09-15T03:48:11.914 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-09-15T03:48:11.913+0000 7fac8dff9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28890) UID: 0 2024-09-15T03:48:11.929 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:11.929 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:11.937 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:48:11.999 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'unique_pool_0' removed 2024-09-15T03:48:12.000 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is set 2024-09-15T03:48:12.015 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-15T03:48:12.024 INFO:tasks.thrashosds:joining thrashosds 2024-09-15T03:48:12.025 INFO:tasks.ceph.osd.6.smithi067.stderr:2024-09-15T03:48:12.014+0000 7f6fa25f9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28462) UID: 0 2024-09-15T03:48:12.039 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:12.039 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:12.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:48:12.145 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:12.145 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:12.153 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:48:12.256 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:12.256 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:12.265 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:48:12.361 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:12.361 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:12.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:48:12.466 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-09-15T03:48:12.466 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:48:12.950 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-15T03:48:13.073 INFO:teuthology.orchestra.run.smithi001.stdout:mon.a: {} 2024-09-15T03:48:13.073 INFO:teuthology.orchestra.run.smithi001.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-15T03:48:13.082 INFO:teuthology.orchestra.run.smithi001.stdout:mon.b: {} 2024-09-15T03:48:13.083 INFO:teuthology.orchestra.run.smithi001.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-15T03:48:13.091 INFO:teuthology.orchestra.run.smithi001.stdout:mon.c: {} 2024-09-15T03:48:13.092 INFO:teuthology.orchestra.run.smithi001.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-15T03:48:13.103 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-15T03:48:14.006 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.0 to 1 (10000) 2024-09-15T03:48:14.021 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-15T03:48:14.025 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T03:48:15.012 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-15T03:48:15.012 INFO:teuthology.orchestra.run.smithi001.stderr:noscrub is unset 2024-09-15T03:48:15.030 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T03:48:15.031 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-15T03:48:16.018 INFO:teuthology.orchestra.run.smithi001.stderr:nodeep-scrub is unset 2024-09-15T03:48:16.019 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.1 to 1 (10000) 2024-09-15T03:48:16.041 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-15T03:48:17.020 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-15T03:48:17.037 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-15T03:48:18.030 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.2 to 1 (10000) 2024-09-15T03:48:18.046 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-15T03:48:19.034 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-15T03:48:19.049 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-15T03:48:20.035 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.3 to 1 (10000) 2024-09-15T03:48:20.051 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-15T03:48:21.044 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-15T03:48:21.058 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-15T03:48:22.046 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.4 to 1 (10000) 2024-09-15T03:48:22.062 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-15T03:48:23.052 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-15T03:48:23.068 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-15T03:48:24.060 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.5 to 1 (10000) 2024-09-15T03:48:24.076 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-15T03:48:25.067 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-15T03:48:25.083 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-15T03:48:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.6 to 1 (10000) 2024-09-15T03:48:26.094 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-15T03:48:27.082 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-15T03:48:27.099 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-15T03:48:28.090 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.7 to 1 (10000) 2024-09-15T03:48:28.107 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-15T03:48:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-15T03:48:29.111 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-15T03:48:30.105 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.8 to 1 (10000) 2024-09-15T03:48:30.122 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 1 2024-09-15T03:48:31.111 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-09-15T03:48:31.126 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-15T03:48:32.116 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.9 to 1 (10000) 2024-09-15T03:48:32.133 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 9 1 2024-09-15T03:48:33.122 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-09-15T03:48:33.137 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-15T03:48:34.127 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.11 to 1 (10000) 2024-09-15T03:48:34.144 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 1 2024-09-15T03:48:35.135 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-09-15T03:48:35.153 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-15T03:48:36.145 INFO:teuthology.orchestra.run.smithi001.stderr:reweighted osd.10 to 1 (10000) 2024-09-15T03:48:36.161 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 2024-09-15T03:48:37.152 INFO:teuthology.orchestra.run.smithi001.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-09-15T03:48:37.169 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-15T03:48:37.169 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-15T03:48:37.169 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-15T03:48:37.169 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-15T03:48:37.169 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-15T03:48:37.169 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:48:37.449 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:48:37.449 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":147,"fsid":"f27b540e-e5e3-4e95-afe8-c95ee66b0a62","created":"2024-09-15T03:46:09.843094+0000","modified":"2024-09-15T03:48:37.149253+0000","last_up_change":"2024-09-15T03:47:51.883425+0000","last_in_change":"2024-09-15T03:47:32.982084+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:46:16.185530+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:46:22.284406+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":24,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"cd433a6a-c875-4e6e-b89c-938caecbae3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":28908},{"type":"v1","addr":"172.21.15.1:6801","nonce":28908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":28908},{"type":"v1","addr":"172.21.15.1:6803","nonce":28908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":28908},{"type":"v1","addr":"172.21.15.1:6807","nonce":28908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":28908},{"type":"v1","addr":"172.21.15.1:6805","nonce":28908}]},"public_addr":"172.21.15.1:6801/28908","cluster_addr":"172.21.15.1:6803/28908","heartbeat_back_addr":"172.21.15.1:6807/28908","heartbeat_front_addr":"172.21.15.1:6805/28908","state":["exists","up"]},{"osd":1,"uuid":"789930f9-42bf-476b-9f09-a21bffe33a79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":28913},{"type":"v1","addr":"172.21.15.1:6809","nonce":28913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":28913},{"type":"v1","addr":"172.21.15.1:6811","nonce":28913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":28913},{"type":"v1","addr":"172.21.15.1:6815","nonce":28913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":28913},{"type":"v1","addr":"172.21.15.1:6813","nonce":28913}]},"public_addr":"172.21.15.1:6809/28913","cluster_addr":"172.21.15.1:6811/28913","heartbeat_back_addr":"172.21.15.1:6815/28913","heartbeat_front_addr":"172.21.15.1:6813/28913","state":["exists","up"]},{"osd":2,"uuid":"9a83fec5-0d63-461c-b516-289443b3882b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":28911},{"type":"v1","addr":"172.21.15.1:6823","nonce":28911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":28911},{"type":"v1","addr":"172.21.15.1:6827","nonce":28911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":28911},{"type":"v1","addr":"172.21.15.1:6831","nonce":28911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":28911},{"type":"v1","addr":"172.21.15.1:6829","nonce":28911}]},"public_addr":"172.21.15.1:6823/28911","cluster_addr":"172.21.15.1:6827/28911","heartbeat_back_addr":"172.21.15.1:6831/28911","heartbeat_front_addr":"172.21.15.1:6829/28911","state":["exists","up"]},{"osd":3,"uuid":"d944bf89-e183-496a-9ca3-7949741b4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":28914},{"type":"v1","addr":"172.21.15.1:6817","nonce":28914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":28914},{"type":"v1","addr":"172.21.15.1:6819","nonce":28914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":28914},{"type":"v1","addr":"172.21.15.1:6826","nonce":28914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":28914},{"type":"v1","addr":"172.21.15.1:6822","nonce":28914}]},"public_addr":"172.21.15.1:6817/28914","cluster_addr":"172.21.15.1:6819/28914","heartbeat_back_addr":"172.21.15.1:6826/28914","heartbeat_front_addr":"172.21.15.1:6822/28914","state":["exists","up"]},{"osd":4,"uuid":"15ec2dbd-d099-4aff-94b2-1bf9dbb00b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":28468},{"type":"v1","addr":"172.21.15.67:6805","nonce":28468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6807","nonce":28468},{"type":"v1","addr":"172.21.15.67:6809","nonce":28468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":28468},{"type":"v1","addr":"172.21.15.67:6815","nonce":28468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":28468},{"type":"v1","addr":"172.21.15.67:6813","nonce":28468}]},"public_addr":"172.21.15.67:6805/28468","cluster_addr":"172.21.15.67:6809/28468","heartbeat_back_addr":"172.21.15.67:6815/28468","heartbeat_front_addr":"172.21.15.67:6813/28468","state":["exists","up"]},{"osd":5,"uuid":"1e45de07-746b-4612-9a52-d05509cae9a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":28495},{"type":"v1","addr":"172.21.15.67:6825","nonce":28495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":28495},{"type":"v1","addr":"172.21.15.67:6827","nonce":28495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":28495},{"type":"v1","addr":"172.21.15.67:6831","nonce":28495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":28495},{"type":"v1","addr":"172.21.15.67:6829","nonce":28495}]},"public_addr":"172.21.15.67:6825/28495","cluster_addr":"172.21.15.67:6827/28495","heartbeat_back_addr":"172.21.15.67:6831/28495","heartbeat_front_addr":"172.21.15.67:6829/28495","state":["exists","up"]},{"osd":6,"uuid":"ccf785f8-751d-473e-b17c-3df9346be9f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":28469},{"type":"v1","addr":"172.21.15.67:6801","nonce":28469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":28469},{"type":"v1","addr":"172.21.15.67:6804","nonce":28469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":28469},{"type":"v1","addr":"172.21.15.67:6812","nonce":28469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":28469},{"type":"v1","addr":"172.21.15.67:6808","nonce":28469}]},"public_addr":"172.21.15.67:6801/28469","cluster_addr":"172.21.15.67:6804/28469","heartbeat_back_addr":"172.21.15.67:6812/28469","heartbeat_front_addr":"172.21.15.67:6808/28469","state":["exists","up"]},{"osd":7,"uuid":"f9f59987-869d-42eb-9eb6-08c39e3afe6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":28470},{"type":"v1","addr":"172.21.15.67:6817","nonce":28470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":28470},{"type":"v1","addr":"172.21.15.67:6819","nonce":28470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":28470},{"type":"v1","addr":"172.21.15.67:6823","nonce":28470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":28470},{"type":"v1","addr":"172.21.15.67:6821","nonce":28470}]},"public_addr":"172.21.15.67:6817/28470","cluster_addr":"172.21.15.67:6819/28470","heartbeat_back_addr":"172.21.15.67:6823/28470","heartbeat_front_addr":"172.21.15.67:6821/28470","state":["exists","up"]},{"osd":8,"uuid":"69ddfe52-11df-4608-9a10-e8907c4cd87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":29472},{"type":"v1","addr":"172.21.15.190:6809","nonce":29472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":29472},{"type":"v1","addr":"172.21.15.190:6811","nonce":29472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":29472},{"type":"v1","addr":"172.21.15.190:6815","nonce":29472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":29472},{"type":"v1","addr":"172.21.15.190:6813","nonce":29472}]},"public_addr":"172.21.15.190:6809/29472","cluster_addr":"172.21.15.190:6811/29472","heartbeat_back_addr":"172.21.15.190:6815/29472","heartbeat_front_addr":"172.21.15.190:6813/29472","state":["exists","up"]},{"osd":9,"uuid":"f92e1609-2a1a-4814-9ac0-44a73373d229","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":29473},{"type":"v1","addr":"172.21.15.190:6801","nonce":29473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":29473},{"type":"v1","addr":"172.21.15.190:6803","nonce":29473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":29473},{"type":"v1","addr":"172.21.15.190:6807","nonce":29473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":29473},{"type":"v1","addr":"172.21.15.190:6805","nonce":29473}]},"public_addr":"172.21.15.190:6801/29473","cluster_addr":"172.21.15.190:6803/29473","heartbeat_back_addr":"172.21.15.190:6807/29473","heartbeat_front_addr":"172.21.15.190:6805/29473","state":["exists","up"]},{"osd":10,"uuid":"5f2e8ca3-8c65-4595-8e82-27dc3768ab99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":78,"up_from":105,"up_thru":108,"down_at":80,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":33741},{"type":"v1","addr":"172.21.15.190:6817","nonce":33741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":33741},{"type":"v1","addr":"172.21.15.190:6819","nonce":33741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":33741},{"type":"v1","addr":"172.21.15.190:6823","nonce":33741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":33741},{"type":"v1","addr":"172.21.15.190:6821","nonce":33741}]},"public_addr":"172.21.15.190:6817/33741","cluster_addr":"172.21.15.190:6819/33741","heartbeat_back_addr":"172.21.15.190:6823/33741","heartbeat_front_addr":"172.21.15.190:6821/33741","state":["exists","up"]},{"osd":11,"uuid":"cedf7df4-f875-42f5-a8cf-13cb82f05cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":29475},{"type":"v1","addr":"172.21.15.190:6825","nonce":29475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":29475},{"type":"v1","addr":"172.21.15.190:6827","nonce":29475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":29475},{"type":"v1","addr":"172.21.15.190:6831","nonce":29475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":29475},{"type":"v1","addr":"172.21.15.190:6829","nonce":29475}]},"public_addr":"172.21.15.190:6825/29475","cluster_addr":"172.21.15.190:6827/29475","heartbeat_back_addr":"172.21.15.190:6831/29475","heartbeat_front_addr":"172.21.15.190:6829/29475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:37.565633+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:48:35.370424+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:25.983954+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:48:30.827938+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:48:30.042695+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:34.158155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:37.031163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:18.853134+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:29.597174+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:37.409413+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:51.125326+0000","dead_epoch":80},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:48:36.717248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lrcprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","l":"3","m":"2","name":"lrcprofile","plugin":"lrc"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:48:37.462 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-15T03:48:37.462 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:48:37.735 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:48:37.736 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":147,"fsid":"f27b540e-e5e3-4e95-afe8-c95ee66b0a62","created":"2024-09-15T03:46:09.843094+0000","modified":"2024-09-15T03:48:37.149253+0000","last_up_change":"2024-09-15T03:47:51.883425+0000","last_in_change":"2024-09-15T03:47:32.982084+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:46:16.185530+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:46:22.284406+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":24,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"cd433a6a-c875-4e6e-b89c-938caecbae3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":28908},{"type":"v1","addr":"172.21.15.1:6801","nonce":28908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":28908},{"type":"v1","addr":"172.21.15.1:6803","nonce":28908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":28908},{"type":"v1","addr":"172.21.15.1:6807","nonce":28908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":28908},{"type":"v1","addr":"172.21.15.1:6805","nonce":28908}]},"public_addr":"172.21.15.1:6801/28908","cluster_addr":"172.21.15.1:6803/28908","heartbeat_back_addr":"172.21.15.1:6807/28908","heartbeat_front_addr":"172.21.15.1:6805/28908","state":["exists","up"]},{"osd":1,"uuid":"789930f9-42bf-476b-9f09-a21bffe33a79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":28913},{"type":"v1","addr":"172.21.15.1:6809","nonce":28913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":28913},{"type":"v1","addr":"172.21.15.1:6811","nonce":28913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":28913},{"type":"v1","addr":"172.21.15.1:6815","nonce":28913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":28913},{"type":"v1","addr":"172.21.15.1:6813","nonce":28913}]},"public_addr":"172.21.15.1:6809/28913","cluster_addr":"172.21.15.1:6811/28913","heartbeat_back_addr":"172.21.15.1:6815/28913","heartbeat_front_addr":"172.21.15.1:6813/28913","state":["exists","up"]},{"osd":2,"uuid":"9a83fec5-0d63-461c-b516-289443b3882b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":28911},{"type":"v1","addr":"172.21.15.1:6823","nonce":28911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":28911},{"type":"v1","addr":"172.21.15.1:6827","nonce":28911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":28911},{"type":"v1","addr":"172.21.15.1:6831","nonce":28911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":28911},{"type":"v1","addr":"172.21.15.1:6829","nonce":28911}]},"public_addr":"172.21.15.1:6823/28911","cluster_addr":"172.21.15.1:6827/28911","heartbeat_back_addr":"172.21.15.1:6831/28911","heartbeat_front_addr":"172.21.15.1:6829/28911","state":["exists","up"]},{"osd":3,"uuid":"d944bf89-e183-496a-9ca3-7949741b4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":28914},{"type":"v1","addr":"172.21.15.1:6817","nonce":28914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":28914},{"type":"v1","addr":"172.21.15.1:6819","nonce":28914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":28914},{"type":"v1","addr":"172.21.15.1:6826","nonce":28914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":28914},{"type":"v1","addr":"172.21.15.1:6822","nonce":28914}]},"public_addr":"172.21.15.1:6817/28914","cluster_addr":"172.21.15.1:6819/28914","heartbeat_back_addr":"172.21.15.1:6826/28914","heartbeat_front_addr":"172.21.15.1:6822/28914","state":["exists","up"]},{"osd":4,"uuid":"15ec2dbd-d099-4aff-94b2-1bf9dbb00b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":28468},{"type":"v1","addr":"172.21.15.67:6805","nonce":28468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6807","nonce":28468},{"type":"v1","addr":"172.21.15.67:6809","nonce":28468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":28468},{"type":"v1","addr":"172.21.15.67:6815","nonce":28468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":28468},{"type":"v1","addr":"172.21.15.67:6813","nonce":28468}]},"public_addr":"172.21.15.67:6805/28468","cluster_addr":"172.21.15.67:6809/28468","heartbeat_back_addr":"172.21.15.67:6815/28468","heartbeat_front_addr":"172.21.15.67:6813/28468","state":["exists","up"]},{"osd":5,"uuid":"1e45de07-746b-4612-9a52-d05509cae9a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":28495},{"type":"v1","addr":"172.21.15.67:6825","nonce":28495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":28495},{"type":"v1","addr":"172.21.15.67:6827","nonce":28495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":28495},{"type":"v1","addr":"172.21.15.67:6831","nonce":28495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":28495},{"type":"v1","addr":"172.21.15.67:6829","nonce":28495}]},"public_addr":"172.21.15.67:6825/28495","cluster_addr":"172.21.15.67:6827/28495","heartbeat_back_addr":"172.21.15.67:6831/28495","heartbeat_front_addr":"172.21.15.67:6829/28495","state":["exists","up"]},{"osd":6,"uuid":"ccf785f8-751d-473e-b17c-3df9346be9f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":28469},{"type":"v1","addr":"172.21.15.67:6801","nonce":28469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":28469},{"type":"v1","addr":"172.21.15.67:6804","nonce":28469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":28469},{"type":"v1","addr":"172.21.15.67:6812","nonce":28469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":28469},{"type":"v1","addr":"172.21.15.67:6808","nonce":28469}]},"public_addr":"172.21.15.67:6801/28469","cluster_addr":"172.21.15.67:6804/28469","heartbeat_back_addr":"172.21.15.67:6812/28469","heartbeat_front_addr":"172.21.15.67:6808/28469","state":["exists","up"]},{"osd":7,"uuid":"f9f59987-869d-42eb-9eb6-08c39e3afe6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":28470},{"type":"v1","addr":"172.21.15.67:6817","nonce":28470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":28470},{"type":"v1","addr":"172.21.15.67:6819","nonce":28470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":28470},{"type":"v1","addr":"172.21.15.67:6823","nonce":28470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":28470},{"type":"v1","addr":"172.21.15.67:6821","nonce":28470}]},"public_addr":"172.21.15.67:6817/28470","cluster_addr":"172.21.15.67:6819/28470","heartbeat_back_addr":"172.21.15.67:6823/28470","heartbeat_front_addr":"172.21.15.67:6821/28470","state":["exists","up"]},{"osd":8,"uuid":"69ddfe52-11df-4608-9a10-e8907c4cd87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":29472},{"type":"v1","addr":"172.21.15.190:6809","nonce":29472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":29472},{"type":"v1","addr":"172.21.15.190:6811","nonce":29472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":29472},{"type":"v1","addr":"172.21.15.190:6815","nonce":29472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":29472},{"type":"v1","addr":"172.21.15.190:6813","nonce":29472}]},"public_addr":"172.21.15.190:6809/29472","cluster_addr":"172.21.15.190:6811/29472","heartbeat_back_addr":"172.21.15.190:6815/29472","heartbeat_front_addr":"172.21.15.190:6813/29472","state":["exists","up"]},{"osd":9,"uuid":"f92e1609-2a1a-4814-9ac0-44a73373d229","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":29473},{"type":"v1","addr":"172.21.15.190:6801","nonce":29473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":29473},{"type":"v1","addr":"172.21.15.190:6803","nonce":29473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":29473},{"type":"v1","addr":"172.21.15.190:6807","nonce":29473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":29473},{"type":"v1","addr":"172.21.15.190:6805","nonce":29473}]},"public_addr":"172.21.15.190:6801/29473","cluster_addr":"172.21.15.190:6803/29473","heartbeat_back_addr":"172.21.15.190:6807/29473","heartbeat_front_addr":"172.21.15.190:6805/29473","state":["exists","up"]},{"osd":10,"uuid":"5f2e8ca3-8c65-4595-8e82-27dc3768ab99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":78,"up_from":105,"up_thru":108,"down_at":80,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":33741},{"type":"v1","addr":"172.21.15.190:6817","nonce":33741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":33741},{"type":"v1","addr":"172.21.15.190:6819","nonce":33741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":33741},{"type":"v1","addr":"172.21.15.190:6823","nonce":33741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":33741},{"type":"v1","addr":"172.21.15.190:6821","nonce":33741}]},"public_addr":"172.21.15.190:6817/33741","cluster_addr":"172.21.15.190:6819/33741","heartbeat_back_addr":"172.21.15.190:6823/33741","heartbeat_front_addr":"172.21.15.190:6821/33741","state":["exists","up"]},{"osd":11,"uuid":"cedf7df4-f875-42f5-a8cf-13cb82f05cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":29475},{"type":"v1","addr":"172.21.15.190:6825","nonce":29475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":29475},{"type":"v1","addr":"172.21.15.190:6827","nonce":29475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":29475},{"type":"v1","addr":"172.21.15.190:6831","nonce":29475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":29475},{"type":"v1","addr":"172.21.15.190:6829","nonce":29475}]},"public_addr":"172.21.15.190:6825/29475","cluster_addr":"172.21.15.190:6827/29475","heartbeat_back_addr":"172.21.15.190:6831/29475","heartbeat_front_addr":"172.21.15.190:6829/29475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:37.565633+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:48:35.370424+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:25.983954+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:48:30.827938+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:48:30.042695+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:34.158155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:37.031163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:18.853134+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:29.597174+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:37.409413+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:51.125326+0000","dead_epoch":80},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:48:36.717248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lrcprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","l":"3","m":"2","name":"lrcprofile","plugin":"lrc"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:48:37.749 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T03:48:37.749 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-15T03:48:37.750 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T03:48:37.750 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T03:48:37.751 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-15T03:48:37.751 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-15T03:48:37.752 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-15T03:48:37.752 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-15T03:48:37.753 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-15T03:48:37.753 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-15T03:48:37.754 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-15T03:48:37.754 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-15T03:48:38.063 INFO:teuthology.orchestra.run.smithi001.stdout:73014444063 2024-09-15T03:48:38.063 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-15T03:48:38.065 INFO:teuthology.orchestra.run.smithi001.stdout:73014444063 2024-09-15T03:48:38.065 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:48:38.079 INFO:teuthology.orchestra.run.smithi001.stdout:450971566091 2024-09-15T03:48:38.080 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-15T03:48:38.088 INFO:teuthology.orchestra.run.smithi001.stdout:73014444063 2024-09-15T03:48:38.089 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-15T03:48:38.097 INFO:teuthology.orchestra.run.smithi001.stdout:73014444063 2024-09-15T03:48:38.097 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T03:48:38.103 INFO:teuthology.orchestra.run.smithi001.stdout:73014444063 2024-09-15T03:48:38.103 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-15T03:48:38.112 INFO:teuthology.orchestra.run.smithi001.stdout:73014444063 2024-09-15T03:48:38.112 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:48:38.115 INFO:teuthology.orchestra.run.smithi001.stdout:73014444063 2024-09-15T03:48:38.115 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:48:38.119 INFO:teuthology.orchestra.run.smithi001.stdout:73014444064 2024-09-15T03:48:38.119 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-15T03:48:38.122 INFO:teuthology.orchestra.run.smithi001.stdout:73014444063 2024-09-15T03:48:38.122 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-15T03:48:38.137 INFO:teuthology.orchestra.run.smithi001.stdout:73014444063 2024-09-15T03:48:38.137 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-15T03:48:38.179 INFO:teuthology.orchestra.run.smithi001.stdout:73014444063 2024-09-15T03:48:38.179 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T03:48:38.738 INFO:teuthology.orchestra.run.smithi001.stdout:450971566090 2024-09-15T03:48:38.756 INFO:tasks.ceph.ceph_manager.ceph:need seq 450971566091 got 450971566090 for osd.10 2024-09-15T03:48:38.817 INFO:teuthology.orchestra.run.smithi001.stdout:73014444062 2024-09-15T03:48:38.837 INFO:teuthology.orchestra.run.smithi001.stdout:73014444062 2024-09-15T03:48:38.857 INFO:teuthology.orchestra.run.smithi001.stdout:73014444062 2024-09-15T03:48:38.879 INFO:teuthology.orchestra.run.smithi001.stdout:73014444062 2024-09-15T03:48:38.880 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444063 got 73014444062 for osd.4 2024-09-15T03:48:38.881 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444063 got 73014444062 for osd.3 2024-09-15T03:48:38.883 INFO:teuthology.orchestra.run.smithi001.stdout:73014444062 2024-09-15T03:48:38.900 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444063 got 73014444062 for osd.0 2024-09-15T03:48:38.909 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444063 got 73014444062 for osd.5 2024-09-15T03:48:38.911 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444063 got 73014444062 for osd.6 2024-09-15T03:48:38.920 INFO:teuthology.orchestra.run.smithi001.stdout:73014444062 2024-09-15T03:48:38.926 INFO:teuthology.orchestra.run.smithi001.stdout:73014444062 2024-09-15T03:48:38.935 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444063 got 73014444062 for osd.7 2024-09-15T03:48:38.939 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444063 got 73014444062 for osd.2 2024-09-15T03:48:38.983 INFO:teuthology.orchestra.run.smithi001.stdout:73014444062 2024-09-15T03:48:39.001 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444063 got 73014444062 for osd.9 2024-09-15T03:48:39.002 INFO:teuthology.orchestra.run.smithi001.stdout:73014444062 2024-09-15T03:48:39.015 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444063 got 73014444062 for osd.11 2024-09-15T03:48:39.016 INFO:teuthology.orchestra.run.smithi001.stdout:73014444063 2024-09-15T03:48:39.032 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444064 got 73014444063 for osd.8 2024-09-15T03:48:39.044 INFO:teuthology.orchestra.run.smithi001.stdout:73014444062 2024-09-15T03:48:39.057 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444063 got 73014444062 for osd.1 2024-09-15T03:48:39.757 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-15T03:48:39.881 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T03:48:39.881 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:48:39.902 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:48:39.910 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-15T03:48:39.912 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-15T03:48:39.936 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-15T03:48:39.940 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:48:40.003 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-15T03:48:40.016 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-15T03:48:40.033 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-15T03:48:40.058 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T03:48:40.208 INFO:teuthology.orchestra.run.smithi001.stdout:450971566091 2024-09-15T03:48:40.250 INFO:tasks.ceph.ceph_manager.ceph:need seq 450971566091 got 450971566091 for osd.10 2024-09-15T03:48:40.250 DEBUG:teuthology.parallel:result is None 2024-09-15T03:48:40.436 INFO:teuthology.orchestra.run.smithi001.stdout:73014444063 2024-09-15T03:48:40.456 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444063 got 73014444063 for osd.3 2024-09-15T03:48:40.457 DEBUG:teuthology.parallel:result is None 2024-09-15T03:48:40.458 INFO:teuthology.orchestra.run.smithi001.stdout:73014444063 2024-09-15T03:48:40.484 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444063 got 73014444063 for osd.4 2024-09-15T03:48:40.484 DEBUG:teuthology.parallel:result is None 2024-09-15T03:48:40.590 INFO:teuthology.orchestra.run.smithi001.stdout:73014444063 2024-09-15T03:48:40.603 INFO:teuthology.orchestra.run.smithi001.stdout:73014444063 2024-09-15T03:48:40.611 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444063 got 73014444063 for osd.0 2024-09-15T03:48:40.611 DEBUG:teuthology.parallel:result is None 2024-09-15T03:48:40.613 INFO:teuthology.orchestra.run.smithi001.stdout:73014444063 2024-09-15T03:48:40.629 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444063 got 73014444063 for osd.5 2024-09-15T03:48:40.630 DEBUG:teuthology.parallel:result is None 2024-09-15T03:48:40.634 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444063 got 73014444063 for osd.6 2024-09-15T03:48:40.634 DEBUG:teuthology.parallel:result is None 2024-09-15T03:48:40.663 INFO:teuthology.orchestra.run.smithi001.stdout:73014444063 2024-09-15T03:48:40.680 INFO:teuthology.orchestra.run.smithi001.stdout:73014444063 2024-09-15T03:48:40.681 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444063 got 73014444063 for osd.2 2024-09-15T03:48:40.681 DEBUG:teuthology.parallel:result is None 2024-09-15T03:48:40.688 INFO:teuthology.orchestra.run.smithi001.stdout:73014444063 2024-09-15T03:48:40.693 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444063 got 73014444063 for osd.7 2024-09-15T03:48:40.693 DEBUG:teuthology.parallel:result is None 2024-09-15T03:48:40.709 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444063 got 73014444063 for osd.11 2024-09-15T03:48:40.710 DEBUG:teuthology.parallel:result is None 2024-09-15T03:48:40.711 INFO:teuthology.orchestra.run.smithi001.stdout:73014444063 2024-09-15T03:48:40.723 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444063 got 73014444063 for osd.9 2024-09-15T03:48:40.723 DEBUG:teuthology.parallel:result is None 2024-09-15T03:48:40.747 INFO:teuthology.orchestra.run.smithi001.stdout:73014444064 2024-09-15T03:48:40.760 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444063 got 73014444064 for osd.1 2024-09-15T03:48:40.760 DEBUG:teuthology.parallel:result is None 2024-09-15T03:48:40.778 INFO:teuthology.orchestra.run.smithi001.stdout:73014444064 2024-09-15T03:48:40.791 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444064 got 73014444064 for osd.8 2024-09-15T03:48:40.791 DEBUG:teuthology.parallel:result is None 2024-09-15T03:48:40.791 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-15T03:48:40.792 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:48:41.107 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:48:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-15T03:48:41.118 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-09-15T03:48:40.170557+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":125,"num_write_kb":2152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":2,"num_per_pool_omap_osds":0,"kb":1124155392,"kb_used":9578232,"kb_used_data":1114577696,"kb_used_omap":58207,"kb_used_meta":0,"kb_avail":1114577160,"statfs":{"total":1151135121408,"available":1141327011840,"internally_reserved":548864,"allocated":1141327560704,"data_stored":1141327560704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59604195,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001275"},"pg_stats":[{"pgid":"1.0","version":"20'92","reported_seq":251,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.795857+0000","last_change":"2024-09-15T03:47:39.774326+0000","last_active":"2024-09-15T03:48:36.795857+0000","last_peered":"2024-09-15T03:48:36.795857+0000","last_clean":"2024-09-15T03:48:36.795857+0000","last_became_active":"2024-09-15T03:46:18.011130+0000","last_became_peered":"2024-09-15T03:46:18.011130+0000","last_unstale":"2024-09-15T03:48:36.795857+0000","last_undegraded":"2024-09-15T03:48:36.795857+0000","last_fullsized":"2024-09-15T03:48:36.795857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-15T03:47:39.774262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_clean_scrub_stamp":"2024-09-15T03:47:39.774262+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:43.691013+0000","scrub_duration":0.0027413759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"22'1","reported_seq":145,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.144868+0000","last_change":"2024-09-15T03:47:52.292975+0000","last_active":"2024-09-15T03:48:36.144868+0000","last_peered":"2024-09-15T03:48:36.144868+0000","last_clean":"2024-09-15T03:48:36.144868+0000","last_became_active":"2024-09-15T03:46:24.038633+0000","last_became_peered":"2024-09-15T03:46:24.038633+0000","last_unstale":"2024-09-15T03:48:36.144868+0000","last_undegraded":"2024-09-15T03:48:36.144868+0000","last_fullsized":"2024-09-15T03:48:36.144868+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'1","last_scrub_stamp":"2024-09-15T03:47:52.292926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:52.292926+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:56.362597+0000","scrub_duration":0.001409429,"objects_trimmed":0,"snaptrim_duration":0.00030178200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":140,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.795838+0000","last_change":"2024-09-15T03:47:27.779257+0000","last_active":"2024-09-15T03:48:36.795838+0000","last_peered":"2024-09-15T03:48:36.795838+0000","last_clean":"2024-09-15T03:48:36.795838+0000","last_became_active":"2024-09-15T03:46:24.063106+0000","last_became_peered":"2024-09-15T03:46:24.063106+0000","last_unstale":"2024-09-15T03:48:36.795838+0000","last_undegraded":"2024-09-15T03:48:36.795838+0000","last_fullsized":"2024-09-15T03:48:36.795838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.779210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.779210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:42.717969+0000","scrub_duration":0.0021030710000000002,"objects_trimmed":0,"snaptrim_duration":0.000393075,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":148,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.995562+0000","last_change":"2024-09-15T03:47:42.997012+0000","last_active":"2024-09-15T03:48:36.995562+0000","last_peered":"2024-09-15T03:48:36.995562+0000","last_clean":"2024-09-15T03:48:36.995562+0000","last_became_active":"2024-09-15T03:46:24.037471+0000","last_became_peered":"2024-09-15T03:46:24.037471+0000","last_unstale":"2024-09-15T03:48:36.995562+0000","last_undegraded":"2024-09-15T03:48:36.995562+0000","last_fullsized":"2024-09-15T03:48:36.995562+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:42.996957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:42.996957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:53.443125+0000","scrub_duration":0.0014164309999999999,"objects_trimmed":0,"snaptrim_duration":0.00026490299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"24'2","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.795873+0000","last_change":"2024-09-15T03:48:04.781084+0000","last_active":"2024-09-15T03:48:36.795873+0000","last_peered":"2024-09-15T03:48:36.795873+0000","last_clean":"2024-09-15T03:48:36.795873+0000","last_became_active":"2024-09-15T03:47:36.848076+0000","last_became_peered":"2024-09-15T03:47:36.848076+0000","last_unstale":"2024-09-15T03:48:36.795873+0000","last_undegraded":"2024-09-15T03:48:36.795873+0000","last_fullsized":"2024-09-15T03:48:36.795873+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'2","last_scrub_stamp":"2024-09-15T03:48:04.781025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:48:04.781025+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:13.832081+0000","scrub_duration":0.0016375750000000001,"objects_trimmed":0,"snaptrim_duration":0.000944677,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":166,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.144325+0000","last_change":"2024-09-15T03:47:39.930675+0000","last_active":"2024-09-15T03:48:36.144325+0000","last_peered":"2024-09-15T03:48:36.144325+0000","last_clean":"2024-09-15T03:48:36.144325+0000","last_became_active":"2024-09-15T03:47:33.998802+0000","last_became_peered":"2024-09-15T03:47:33.998802+0000","last_unstale":"2024-09-15T03:48:36.144325+0000","last_undegraded":"2024-09-15T03:48:36.144325+0000","last_fullsized":"2024-09-15T03:48:36.144325+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:39.930628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:39.930628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:03.535396+0000","scrub_duration":0.001146333,"objects_trimmed":0,"snaptrim_duration":0.00034123999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":142,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.995379+0000","last_change":"2024-09-15T03:47:30.025153+0000","last_active":"2024-09-15T03:48:36.995379+0000","last_peered":"2024-09-15T03:48:36.995379+0000","last_clean":"2024-09-15T03:48:36.995379+0000","last_became_active":"2024-09-15T03:46:24.038945+0000","last_became_peered":"2024-09-15T03:46:24.038945+0000","last_unstale":"2024-09-15T03:48:36.995379+0000","last_undegraded":"2024-09-15T03:48:36.995379+0000","last_fullsized":"2024-09-15T03:48:36.995379+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:30.025107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:30.025107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013039429999999999,"objects_trimmed":0,"snaptrim_duration":0.00032787499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":142,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.503241+0000","last_change":"2024-09-15T03:47:27.297139+0000","last_active":"2024-09-15T03:48:36.503241+0000","last_peered":"2024-09-15T03:48:36.503241+0000","last_clean":"2024-09-15T03:48:36.503241+0000","last_became_active":"2024-09-15T03:46:24.063380+0000","last_became_peered":"2024-09-15T03:46:24.063380+0000","last_unstale":"2024-09-15T03:48:36.503241+0000","last_undegraded":"2024-09-15T03:48:36.503241+0000","last_fullsized":"2024-09-15T03:48:36.503241+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.297083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.297083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:48.875998+0000","scrub_duration":0.0018328789999999999,"objects_trimmed":0,"snaptrim_duration":0.00034490600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":140,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.795873+0000","last_change":"2024-09-15T03:47:42.675933+0000","last_active":"2024-09-15T03:48:36.795873+0000","last_peered":"2024-09-15T03:48:36.795873+0000","last_clean":"2024-09-15T03:48:36.795873+0000","last_became_active":"2024-09-15T03:46:24.063335+0000","last_became_peered":"2024-09-15T03:46:24.063335+0000","last_unstale":"2024-09-15T03:48:36.795873+0000","last_undegraded":"2024-09-15T03:48:36.795873+0000","last_fullsized":"2024-09-15T03:48:36.795873+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:42.675882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:42.675882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:02.697036+0000","scrub_duration":0.001445478,"objects_trimmed":0,"snaptrim_duration":0.00065928000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":11,"up_from":17,"seq":73014444063,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797940,"kb_used_data":92881720,"kb_used_omap":5019,"kb_used_meta":0,"kb_avail":92881676,"statfs":{"total":95927926784,"available":95110836224,"internally_reserved":45056,"allocated":95110881280,"data_stored":95110881280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5139614,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52700000000000002,"15min":0.53100000000000003},"min":{"1min":0.36699999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.79200000000000004,"5min":1.002,"15min":1.002},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.503,"15min":0.502},"min":{"1min":0.33600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.82199999999999995,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.73099999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":121.575,"5min":121.575,"15min":121.575},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2060.797,"5min":2060.797,"15min":2060.797},"last":0.74199999999999999},{"interface":"front","average":{"1min":121.56,"5min":121.56,"15min":121.56},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2060.7159999999999,"5min":2060.7159999999999,"15min":2060.7159999999999},"last":0.60499999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.53300000000000003,"15min":0.52300000000000002},"min":{"1min":0.35099999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.40999999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.91800000000000004,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.53700000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76100000000000001,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.40699999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.72099999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.61499999999999999,"15min":0.60699999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.68000000000000005,"15min":0.67500000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.50700000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65500000000000003,"15min":0.65400000000000003},"min":{"1min":0.48099999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.94399999999999995,"5min":1.103,"15min":1.103},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.64600000000000002,"15min":0.63900000000000001},"min":{"1min":0.48199999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.61499999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.65800000000000003,"15min":0.65200000000000002},"min":{"1min":0.50800000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67700000000000005,"15min":0.67500000000000004},"min":{"1min":0.497,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0149999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.48499999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65000000000000002,"15min":0.64900000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67000000000000004,"15min":0.66800000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.879,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.628}]},{"osd":8,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.84499999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.43099999999999999}]},{"osd":9,"last update":"Sun Sep 15 03:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61699999999999999,"15min":0.60799999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.65200000000000002,"15min":0.63900000000000001},"min":{"1min":0.45300000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.58599999999999997}]},{"osd":10,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.68500000000000005}]}]},{"osd":10,"up_from":105,"seq":450971566091,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798968,"kb_used_data":92880684,"kb_used_omap":3934,"kb_used_meta":0,"kb_avail":92880648,"statfs":{"total":95927926784,"available":95109783552,"internally_reserved":36864,"allocated":95109820416,"data_stored":95109820416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4028936,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]}]},{"osd":9,"up_from":17,"seq":73014444063,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797984,"kb_used_data":92881684,"kb_used_omap":5374,"kb_used_meta":0,"kb_avail":92881632,"statfs":{"total":95927926784,"available":95110791168,"internally_reserved":53248,"allocated":95110844416,"data_stored":95110844416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5503334,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.33400000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78200000000000003,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57799999999999996,"15min":0.57899999999999996},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.748,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.55600000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57199999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.82199999999999995,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.40300000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.75800000000000001,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.46400000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.54800000000000004,"15min":0.55400000000000005},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.71099999999999997,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.55500000000000005,"15min":0.56499999999999995},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69399999999999995,"5min":1.008,"15min":1.008},"last":0.40999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59499999999999997,"15min":0.58699999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.91300000000000003,"5min":1.03,"15min":1.03},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.64400000000000002,"15min":0.64800000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.83599999999999997,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.82899999999999996}]},{"osd":5,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.48799999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.86899999999999999,"5min":1.131,"15min":1.131},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.63800000000000001,"15min":0.64100000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82299999999999995,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.67300000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63700000000000001,"15min":0.63500000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.81299999999999994,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66800000000000004,"15min":0.67100000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.89300000000000002,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.84099999999999997}]},{"osd":7,"last update":"Sun Sep 15 03:48:37 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.65500000000000003,"15min":0.64600000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.751},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.64200000000000002,"15min":0.622},"min":{"1min":0.55000000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.79300000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51500000000000001,"15min":0.51700000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.68000000000000005,"5min":0.748,"15min":0.748},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.36799999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71099999999999997,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.78100000000000003}]},{"osd":10,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.90500000000000003}]},{"osd":11,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.65400000000000003,"15min":0.66600000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.874},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.66900000000000004,"15min":0.68300000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80900000000000005,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.624}]}]},{"osd":8,"up_from":17,"seq":73014444064,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799204,"kb_used_data":92880456,"kb_used_omap":5416,"kb_used_meta":0,"kb_avail":92880412,"statfs":{"total":95927926784,"available":95109541888,"internally_reserved":45056,"allocated":95109586944,"data_stored":95109586944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5546341,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.35199999999999998,"5min":0.313,"15min":0.313},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60399999999999998,"15min":0.60799999999999998},"min":{"1min":0.371,"5min":0.318,"15min":0.318},"max":{"1min":0.83199999999999996,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.50900000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":34.012,"15min":39.591000000000001},"min":{"1min":0.38200000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76400000000000001,"5min":963.98400000000004,"15min":963.98400000000004},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":34.064999999999998,"15min":39.652999999999999},"min":{"1min":0.40200000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91400000000000003,"5min":964.79600000000005,"15min":964.79600000000005},"last":0.41699999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.437}]},{"osd":3,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.54800000000000004,"15min":0.53900000000000003},"min":{"1min":0.373,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.58099999999999996,"15min":0.57099999999999995},"min":{"1min":0.44,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.55900000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.46200000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.82199999999999995,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.67700000000000005,"15min":0.68200000000000005},"min":{"1min":0.48399999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83299999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.621}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:48:30 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.68600000000000005,"15min":0.68200000000000005},"min":{"1min":0.5,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.93100000000000005,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.66300000000000003,"15min":0.65300000000000002},"min":{"1min":0.56200000000000006,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.48999999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69399999999999995,"15min":0.69799999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.67500000000000004,"15min":0.67900000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97899999999999998,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.60599999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.629,"15min":0.63300000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84799999999999998,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.68999999999999995}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.77800000000000002}]},{"osd":11,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63400000000000001,"15min":0.64200000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.627,"15min":0.63400000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.75800000000000001}]}]},{"osd":7,"up_from":17,"seq":73014444063,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798048,"kb_used_data":92881628,"kb_used_omap":5904,"kb_used_meta":0,"kb_avail":92881568,"statfs":{"total":95927926784,"available":95110725632,"internally_reserved":61440,"allocated":95110787072,"data_stored":95110787072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045714,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.50900000000000001,"15min":0.50800000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.40000000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.74399999999999999,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.56100000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.66100000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.38200000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.69599999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.54400000000000004,"15min":0.55200000000000005},"min":{"1min":0.34200000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71399999999999997,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.504},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.75,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.83799999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.49099999999999999,"15min":0.48799999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55100000000000005,"15min":0.55200000000000005},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.85899999999999999,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.39500000000000002}]},{"osd":5,"last update":"Sun Sep 15 03:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.32800000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.78200000000000003,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.60499999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.76500000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.623},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.872,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.66000000000000003}]},{"osd":8,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.1899999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":1.075},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.72699999999999998,"15min":0.73299999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.996,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.80400000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.76900000000000002,"15min":0.77500000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.747,"5min":0.79000000000000004,"15min":0.79700000000000004},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0860000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.98899999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.86099999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.80900000000000005,"15min":0.81200000000000006},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":1.038},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.145,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.74199999999999999}]}]},{"osd":6,"up_from":17,"seq":73014444063,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797924,"kb_used_data":92881752,"kb_used_omap":3996,"kb_used_meta":0,"kb_avail":92881692,"statfs":{"total":95927926784,"available":95110852608,"internally_reserved":61440,"allocated":95110914048,"data_stored":95110914048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4092729,"internal_metadata":0},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57299999999999995,"15min":0.57499999999999996},"min":{"1min":0.39000000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.84099999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.39000000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72499999999999998,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.45000000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:47:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.56299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67000000000000004,"15min":0.67400000000000004},"min":{"1min":0.45200000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78900000000000003,"5min":1.145,"15min":1.145},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.64100000000000001,"15min":0.64600000000000002},"min":{"1min":0.40699999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80900000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.51400000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.35799999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.75700000000000001,"5min":1.127,"15min":1.127},"last":0.62},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.755,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.63300000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:48:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.495,"15min":0.48499999999999999},"min":{"1min":0.32600000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.97099999999999997,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.51300000000000001,"15min":0.50600000000000001},"min":{"1min":0.32700000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.67900000000000005}]},{"osd":5,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.56100000000000005,"15min":0.55400000000000005},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.42499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81100000000000005,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.73699999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.59599999999999997,"15min":0.58799999999999997},"min":{"1min":0.40799999999999997,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85699999999999998,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.38,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82899999999999996,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.79600000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.80500000000000005,"15min":0.80700000000000005},"min":{"1min":0.57599999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.133,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.78200000000000003,"15min":0.78000000000000003},"min":{"1min":0.57099999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.1240000000000001,"5min":1.242,"15min":1.242},"last":0.96699999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.79500000000000004,"15min":0.79000000000000004},"min":{"1min":0.59199999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0940000000000001,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.79200000000000004,"15min":0.78800000000000003},"min":{"1min":0.60499999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.0620000000000001,"5min":1.21,"15min":1.21},"last":1.046}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.82899999999999996,"15min":0.82499999999999996},"min":{"1min":0.58799999999999997,"5min":0.505,"15min":0.505},"max":{"1min":1.482,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":1.026},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.79600000000000004,"15min":0.79400000000000004},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.0569999999999999,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.90500000000000003}]}]},{"osd":5,"up_from":17,"seq":73014444063,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797996,"kb_used_data":92881668,"kb_used_omap":1913,"kb_used_meta":0,"kb_avail":92881620,"statfs":{"total":95927926784,"available":95110778880,"internally_reserved":49152,"allocated":95110828032,"data_stored":95110828032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1959744,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.36899999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.59699999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":91.076999999999998,"15min":106.17},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.93000000000000005,"5min":2037.425,"15min":2037.425},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.60699999999999998,"5min":91.123000000000005,"15min":106.209},"min":{"1min":0.47999999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.90800000000000003,"5min":2037.3230000000001,"15min":2037.3230000000001},"last":0.66400000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60199999999999998,"15min":0.60699999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60599999999999998,"15min":0.60999999999999999},"min":{"1min":0.40699999999999997,"5min":0.379,"15min":0.379},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.63200000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:48:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.60999999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47899999999999998,"15min":0.47799999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.53100000000000003,"15min":0.53600000000000003},"min":{"1min":0.32100000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.751,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.53800000000000003}]},{"osd":6,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55400000000000005,"15min":0.55200000000000005},"min":{"1min":0.33400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.38300000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55900000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59999999999999998,"15min":0.60599999999999998},"min":{"1min":0.443,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.95299999999999996,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.60699999999999998,"15min":0.61599999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.80900000000000005,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.46800000000000003}]},{"osd":8,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.73499999999999999,"15min":0.74199999999999999},"min":{"1min":0.54000000000000004,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.127,"5min":1.218,"15min":1.218},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.75,"15min":0.754},"min":{"1min":0.57399999999999995,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.94699999999999995,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.67000000000000004}]},{"osd":9,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.77100000000000002,"15min":0.78100000000000003},"min":{"1min":0.55500000000000005,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.88600000000000001,"5min":1.177,"15min":1.177},"last":0.747},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.58699999999999997,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.75600000000000001}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.83399999999999996}]},{"osd":11,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.77800000000000002},"min":{"1min":0.56799999999999995,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.96499999999999997,"5min":1.155,"15min":1.155},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.78900000000000003,"15min":0.79800000000000004},"min":{"1min":0.55900000000000005,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.90800000000000003,"5min":1.194,"15min":1.194},"last":0.77600000000000002}]}]},{"osd":4,"up_from":17,"seq":73014444063,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797980,"kb_used_data":92881680,"kb_used_omap":5862,"kb_used_meta":0,"kb_avail":92881636,"statfs":{"total":95927926784,"available":95110795264,"internally_reserved":45056,"allocated":95110840320,"data_stored":95110840320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6002968,"internal_metadata":0},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.73099999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.61799999999999999,"15min":0.623},"min":{"1min":0.39300000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88100000000000001,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.51600000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":67.558000000000007,"15min":78.728999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.83099999999999996,"5min":1758.9780000000001,"15min":1758.9780000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.55600000000000005,"5min":67.602000000000004,"15min":78.775999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84799999999999998,"5min":1758.671,"15min":1758.671},"last":0.58499999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.65100000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.56599999999999995,"15min":0.57799999999999996},"min":{"1min":0.35899999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.73399999999999999,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75800000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.48499999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.51800000000000002,"15min":0.53200000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69999999999999996,"5min":0.996,"15min":0.996},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.498,"15min":0.50600000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72599999999999998,"5min":0.747,"15min":0.747},"last":0.63900000000000001}]},{"osd":6,"last update":"Sun Sep 15 03:48:29 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.90200000000000002}]},{"osd":7,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66000000000000003,"15min":0.67200000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.77800000000000002,"5min":1.113,"15min":1.113},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.64000000000000001,"15min":0.66000000000000003},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.81000000000000005,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.85699999999999998}]},{"osd":8,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.745,"15min":0.76500000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.99199999999999999,"5min":1.532,"15min":1.532},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.78300000000000003,"15min":0.79900000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.96899999999999997,"5min":1.363,"15min":1.363},"last":0.81000000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.75800000000000001,"15min":0.77600000000000002},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.86799999999999999,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.751,"15min":0.77400000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.85499999999999998,"5min":1.444,"15min":1.444},"last":0.71099999999999997}]},{"osd":10,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.79400000000000004}]},{"osd":11,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.84599999999999997,"15min":0.86899999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0069999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.746},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.79500000000000004,"15min":0.81499999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.91800000000000004,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.88300000000000001}]}]},{"osd":3,"up_from":17,"seq":73014444063,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797984,"kb_used_data":92881680,"kb_used_omap":4766,"kb_used_meta":0,"kb_avail":92881632,"statfs":{"total":95927926784,"available":95110791168,"internally_reserved":49152,"allocated":95110840320,"data_stored":95110840320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4880403,"internal_metadata":0},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.373,"15min":0.36699999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.98099999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:47:38 2024","interfaces":[{"interface":"back","average":{"1min":35.600000000000001,"5min":35.600000000000001,"15min":35.600000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":564.05100000000004,"5min":564.05100000000004,"15min":564.05100000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":35.677,"5min":35.677,"15min":35.677},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":564.57000000000005,"5min":564.57000000000005,"15min":564.57000000000005},"last":0.46400000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.45000000000000001,"15min":0.42199999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.1799999999999997,"5min":5.1799999999999997,"15min":5.1799999999999997},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.45700000000000002,"15min":0.44500000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.498}]},{"osd":4,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.57499999999999996,"15min":0.54900000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.57299999999999995,"15min":0.56499999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.3639999999999999,"5min":2.3639999999999999,"15min":2.3639999999999999},"last":0.79900000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":2.3940000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":5.2830000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.57499999999999996,"15min":0.56499999999999995},"min":{"1min":0.38200000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":1.133},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.65500000000000003,"15min":0.629},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.2489999999999997,"5min":5.2489999999999997,"15min":5.2489999999999997},"last":0.57899999999999996}]},{"osd":7,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.827,"5min":1.827,"15min":1.827},"last":1.1060000000000001},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.63,"15min":0.59999999999999998},"min":{"1min":0.42899999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":5.3090000000000002,"5min":5.3090000000000002,"15min":5.3090000000000002},"last":0.61499999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.68700000000000006,"15min":0.67100000000000004},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.66300000000000003,"15min":0.64900000000000002},"min":{"1min":0.54000000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.8,"5min":1.8,"15min":1.8},"last":1.026}]},{"osd":9,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.71899999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.495,"15min":0.495},"max":{"1min":5.3399999999999999,"5min":5.3399999999999999,"15min":5.3399999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.71999999999999997,"15min":0.70799999999999996},"min":{"1min":0.52200000000000002,"5min":0.498,"15min":0.498},"max":{"1min":2.4239999999999999,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":0.94699999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.70499999999999996,"15min":0.69599999999999995},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.859,"5min":1.859,"15min":1.859},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.73299999999999998,"15min":0.72399999999999998},"min":{"1min":0.51100000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.5430000000000001,"5min":2.5430000000000001,"15min":2.5430000000000001},"last":0.90500000000000003}]}]},{"osd":2,"up_from":17,"seq":73014444063,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797640,"kb_used_data":92881976,"kb_used_omap":5376,"kb_used_meta":0,"kb_avail":92881976,"statfs":{"total":95927926784,"available":95111143424,"internally_reserved":0,"allocated":95111143424,"data_stored":95111143424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5505295,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:37 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.45500000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":27.634,"15min":32.183},"min":{"1min":0.248,"5min":0.247,"15min":0.247},"max":{"1min":0.56499999999999995,"5min":647.92399999999998,"15min":647.92399999999998},"last":0.317},{"interface":"front","average":{"1min":0.40500000000000003,"5min":27.661000000000001,"15min":32.203000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.56599999999999995,"5min":648.02700000000004,"15min":648.02700000000004},"last":0.36899999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.65100000000000002,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38500000000000001,"15min":0.38900000000000001},"min":{"1min":0.27500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.51300000000000001,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.34300000000000003}]},{"osd":4,"last update":"Sun Sep 15 03:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.70999999999999996}]},{"osd":5,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.58799999999999997,"15min":0.58999999999999997},"min":{"1min":0.36499999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.95799999999999996,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.55000000000000004,"15min":0.53800000000000003},"min":{"1min":0.47799999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.84699999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.64700000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59499999999999997,"15min":0.58899999999999997},"min":{"1min":0.42199999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.84699999999999998,"5min":1.321,"15min":1.321},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.59099999999999997,"15min":0.58199999999999996},"min":{"1min":0.46700000000000003,"5min":0.374,"15min":0.374},"max":{"1min":0.90800000000000003,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.88500000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59799999999999998,"15min":0.59099999999999997},"min":{"1min":0.40500000000000003,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93300000000000005,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.751},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.48399999999999999,"5min":0.44,"15min":0.44},"max":{"1min":0.878,"5min":1.363,"15min":1.363},"last":0.76800000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":1.204,"15min":1.296},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90100000000000002,"5min":14.721,"15min":14.721},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.69199999999999995,"5min":1.2350000000000001,"15min":1.325},"min":{"1min":0.50700000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.873,"5min":14.970000000000001,"15min":14.970000000000001},"last":0.629}]},{"osd":9,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":1.2410000000000001,"15min":1.331},"min":{"1min":0.54800000000000004,"5min":0.502,"15min":0.502},"max":{"1min":0.875,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":1.244,"15min":1.3380000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.90900000000000003,"5min":14.888,"15min":14.888},"last":0.61599999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.2529999999999999,"15min":1.3440000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.006,"5min":14.654999999999999,"15min":14.654999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":1.266,"15min":1.361},"min":{"1min":0.48999999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.89400000000000002,"5min":15.096,"15min":15.096},"last":0.73799999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798000,"kb_used_data":92881668,"kb_used_omap":6208,"kb_used_meta":0,"kb_avail":92881616,"statfs":{"total":95927926784,"available":95110774784,"internally_reserved":53248,"allocated":95110828032,"data_stored":95110828032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6357472,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.40300000000000002,"15min":0.41299999999999998},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.53200000000000003,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":1.5660000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.38200000000000001,"15min":0.38600000000000001},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":1.681}]},{"osd":2,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.51500000000000001,"15min":0.53300000000000003},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.65300000000000002,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.5960000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.27200000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":1.6499999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.52000000000000002,"15min":0.53400000000000003},"min":{"1min":0.33400000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69099999999999995,"5min":1.907,"15min":1.907},"last":1.6200000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.51000000000000001,"15min":0.52400000000000002},"min":{"1min":0.30099999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.83199999999999996,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":1.7030000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.33900000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.93000000000000005,"5min":1.482,"15min":1.482},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.68000000000000005,"15min":0.69399999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.79500000000000004,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":0.374}]},{"osd":5,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.98499999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.71099999999999997,"15min":0.73299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.71099999999999997,"5min":2.7440000000000002,"15min":2.7440000000000002},"last":0.41499999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.64400000000000002,"15min":0.65800000000000003},"min":{"1min":0.38900000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.95599999999999996,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.71599999999999997,"15min":0.73299999999999998},"min":{"1min":0.45400000000000001,"5min":0.432,"15min":0.432},"max":{"1min":1.083,"5min":3.278,"15min":3.278},"last":0.68999999999999995}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.81399999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":96.073999999999998,"15min":111.98399999999999},"min":{"1min":0.438,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.0649999999999999,"5min":2861.7190000000001,"15min":2861.7190000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.77700000000000002,"15min":0.80200000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.84799999999999998,"5min":1.946,"15min":1.946},"last":0.63800000000000001}]},{"osd":9,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":96.081000000000003,"15min":111.985},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.44},"max":{"1min":1.024,"5min":2861.6010000000001,"15min":2861.6010000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.70399999999999996,"15min":0.71399999999999997},"min":{"1min":0.44900000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.006,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.56599999999999995}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":119.96599999999999,"5min":119.96599999999999,"15min":119.96599999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2861.5639999999999,"5min":2861.5639999999999,"15min":2861.5639999999999},"last":0.58999999999999997}]},{"osd":11,"last update":"Sun Sep 15 03:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.71099999999999997}]}]},{"osd":0,"up_from":17,"seq":73014444063,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798564,"kb_used_data":92881100,"kb_used_omap":4435,"kb_used_meta":0,"kb_avail":92881052,"statfs":{"total":95927926784,"available":95110197248,"internally_reserved":49152,"allocated":95110246400,"data_stored":95110246400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4541645,"internal_metadata":0},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.42299999999999999,"15min":0.433},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.53300000000000003,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.44700000000000001,"15min":0.45300000000000001},"min":{"1min":0.29899999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.60399999999999998,"5min":1.155,"15min":1.155},"last":0.313}]},{"osd":2,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.804,"5min":1.804,"15min":1.804},"last":0.379}]},{"osd":3,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.45900000000000002,"15min":0.47199999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.60399999999999998,"5min":1.52,"15min":1.52},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.33300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.65000000000000002,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.36499999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.57199999999999995}]},{"osd":5,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.60599999999999998,"15min":0.61599999999999999},"min":{"1min":0.41199999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.65600000000000003,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.68100000000000005,"15min":0.70099999999999996},"min":{"1min":0.41999999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.871,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.55500000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.63800000000000001,"15min":0.65900000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.72499999999999998,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.68400000000000005,"15min":0.70799999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81999999999999995,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.68400000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.67500000000000004,"15min":0.69699999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93400000000000005,"5min":1.359,"15min":1.359},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.74399999999999999,"5min":1.216,"15min":1.216},"last":0.61599999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.72899999999999998,"15min":0.74199999999999999},"min":{"1min":0.46000000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.93600000000000005,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.433},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.73799999999999999,"15min":0.75800000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.86499999999999999,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.54800000000000004}]},{"osd":9,"last update":"Sun Sep 15 03:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.52100000000000002}]},{"osd":10,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.48499999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.70899999999999996,"15min":0.71599999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.83899999999999997,"5min":1.603,"15min":1.603},"last":0.498},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.78700000000000003,"15min":0.80400000000000005},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89200000000000002,"5min":1.494,"15min":1.494},"last":0.63300000000000001}]}]}],"pool_statfs":[]}} 2024-09-15T03:48:41.122 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:48:41.399 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:48:41.400 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-15T03:48:41.411 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-09-15T03:48:40.170557+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":125,"num_write_kb":2152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":2,"num_per_pool_omap_osds":0,"kb":1124155392,"kb_used":9578232,"kb_used_data":1114577696,"kb_used_omap":58207,"kb_used_meta":0,"kb_avail":1114577160,"statfs":{"total":1151135121408,"available":1141327011840,"internally_reserved":548864,"allocated":1141327560704,"data_stored":1141327560704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59604195,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001275"},"pg_stats":[{"pgid":"1.0","version":"20'92","reported_seq":251,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.795857+0000","last_change":"2024-09-15T03:47:39.774326+0000","last_active":"2024-09-15T03:48:36.795857+0000","last_peered":"2024-09-15T03:48:36.795857+0000","last_clean":"2024-09-15T03:48:36.795857+0000","last_became_active":"2024-09-15T03:46:18.011130+0000","last_became_peered":"2024-09-15T03:46:18.011130+0000","last_unstale":"2024-09-15T03:48:36.795857+0000","last_undegraded":"2024-09-15T03:48:36.795857+0000","last_fullsized":"2024-09-15T03:48:36.795857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-15T03:47:39.774262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_clean_scrub_stamp":"2024-09-15T03:47:39.774262+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:43.691013+0000","scrub_duration":0.0027413759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"22'1","reported_seq":145,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.144868+0000","last_change":"2024-09-15T03:47:52.292975+0000","last_active":"2024-09-15T03:48:36.144868+0000","last_peered":"2024-09-15T03:48:36.144868+0000","last_clean":"2024-09-15T03:48:36.144868+0000","last_became_active":"2024-09-15T03:46:24.038633+0000","last_became_peered":"2024-09-15T03:46:24.038633+0000","last_unstale":"2024-09-15T03:48:36.144868+0000","last_undegraded":"2024-09-15T03:48:36.144868+0000","last_fullsized":"2024-09-15T03:48:36.144868+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'1","last_scrub_stamp":"2024-09-15T03:47:52.292926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:52.292926+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:56.362597+0000","scrub_duration":0.001409429,"objects_trimmed":0,"snaptrim_duration":0.00030178200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":140,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.795838+0000","last_change":"2024-09-15T03:47:27.779257+0000","last_active":"2024-09-15T03:48:36.795838+0000","last_peered":"2024-09-15T03:48:36.795838+0000","last_clean":"2024-09-15T03:48:36.795838+0000","last_became_active":"2024-09-15T03:46:24.063106+0000","last_became_peered":"2024-09-15T03:46:24.063106+0000","last_unstale":"2024-09-15T03:48:36.795838+0000","last_undegraded":"2024-09-15T03:48:36.795838+0000","last_fullsized":"2024-09-15T03:48:36.795838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.779210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.779210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:42.717969+0000","scrub_duration":0.0021030710000000002,"objects_trimmed":0,"snaptrim_duration":0.000393075,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":148,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.995562+0000","last_change":"2024-09-15T03:47:42.997012+0000","last_active":"2024-09-15T03:48:36.995562+0000","last_peered":"2024-09-15T03:48:36.995562+0000","last_clean":"2024-09-15T03:48:36.995562+0000","last_became_active":"2024-09-15T03:46:24.037471+0000","last_became_peered":"2024-09-15T03:46:24.037471+0000","last_unstale":"2024-09-15T03:48:36.995562+0000","last_undegraded":"2024-09-15T03:48:36.995562+0000","last_fullsized":"2024-09-15T03:48:36.995562+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:42.996957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:42.996957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:53.443125+0000","scrub_duration":0.0014164309999999999,"objects_trimmed":0,"snaptrim_duration":0.00026490299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"24'2","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.795873+0000","last_change":"2024-09-15T03:48:04.781084+0000","last_active":"2024-09-15T03:48:36.795873+0000","last_peered":"2024-09-15T03:48:36.795873+0000","last_clean":"2024-09-15T03:48:36.795873+0000","last_became_active":"2024-09-15T03:47:36.848076+0000","last_became_peered":"2024-09-15T03:47:36.848076+0000","last_unstale":"2024-09-15T03:48:36.795873+0000","last_undegraded":"2024-09-15T03:48:36.795873+0000","last_fullsized":"2024-09-15T03:48:36.795873+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'2","last_scrub_stamp":"2024-09-15T03:48:04.781025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:48:04.781025+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:13.832081+0000","scrub_duration":0.0016375750000000001,"objects_trimmed":0,"snaptrim_duration":0.000944677,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":166,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.144325+0000","last_change":"2024-09-15T03:47:39.930675+0000","last_active":"2024-09-15T03:48:36.144325+0000","last_peered":"2024-09-15T03:48:36.144325+0000","last_clean":"2024-09-15T03:48:36.144325+0000","last_became_active":"2024-09-15T03:47:33.998802+0000","last_became_peered":"2024-09-15T03:47:33.998802+0000","last_unstale":"2024-09-15T03:48:36.144325+0000","last_undegraded":"2024-09-15T03:48:36.144325+0000","last_fullsized":"2024-09-15T03:48:36.144325+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:39.930628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:39.930628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:03.535396+0000","scrub_duration":0.001146333,"objects_trimmed":0,"snaptrim_duration":0.00034123999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":142,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.995379+0000","last_change":"2024-09-15T03:47:30.025153+0000","last_active":"2024-09-15T03:48:36.995379+0000","last_peered":"2024-09-15T03:48:36.995379+0000","last_clean":"2024-09-15T03:48:36.995379+0000","last_became_active":"2024-09-15T03:46:24.038945+0000","last_became_peered":"2024-09-15T03:46:24.038945+0000","last_unstale":"2024-09-15T03:48:36.995379+0000","last_undegraded":"2024-09-15T03:48:36.995379+0000","last_fullsized":"2024-09-15T03:48:36.995379+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:30.025107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:30.025107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013039429999999999,"objects_trimmed":0,"snaptrim_duration":0.00032787499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":142,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.503241+0000","last_change":"2024-09-15T03:47:27.297139+0000","last_active":"2024-09-15T03:48:36.503241+0000","last_peered":"2024-09-15T03:48:36.503241+0000","last_clean":"2024-09-15T03:48:36.503241+0000","last_became_active":"2024-09-15T03:46:24.063380+0000","last_became_peered":"2024-09-15T03:46:24.063380+0000","last_unstale":"2024-09-15T03:48:36.503241+0000","last_undegraded":"2024-09-15T03:48:36.503241+0000","last_fullsized":"2024-09-15T03:48:36.503241+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.297083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.297083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:48.875998+0000","scrub_duration":0.0018328789999999999,"objects_trimmed":0,"snaptrim_duration":0.00034490600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":140,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.795873+0000","last_change":"2024-09-15T03:47:42.675933+0000","last_active":"2024-09-15T03:48:36.795873+0000","last_peered":"2024-09-15T03:48:36.795873+0000","last_clean":"2024-09-15T03:48:36.795873+0000","last_became_active":"2024-09-15T03:46:24.063335+0000","last_became_peered":"2024-09-15T03:46:24.063335+0000","last_unstale":"2024-09-15T03:48:36.795873+0000","last_undegraded":"2024-09-15T03:48:36.795873+0000","last_fullsized":"2024-09-15T03:48:36.795873+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:42.675882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:42.675882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:02.697036+0000","scrub_duration":0.001445478,"objects_trimmed":0,"snaptrim_duration":0.00065928000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":11,"up_from":17,"seq":73014444063,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797940,"kb_used_data":92881720,"kb_used_omap":5019,"kb_used_meta":0,"kb_avail":92881676,"statfs":{"total":95927926784,"available":95110836224,"internally_reserved":45056,"allocated":95110881280,"data_stored":95110881280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5139614,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52700000000000002,"15min":0.53100000000000003},"min":{"1min":0.36699999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.79200000000000004,"5min":1.002,"15min":1.002},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.503,"15min":0.502},"min":{"1min":0.33600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.82199999999999995,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.73099999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":121.575,"5min":121.575,"15min":121.575},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2060.797,"5min":2060.797,"15min":2060.797},"last":0.74199999999999999},{"interface":"front","average":{"1min":121.56,"5min":121.56,"15min":121.56},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2060.7159999999999,"5min":2060.7159999999999,"15min":2060.7159999999999},"last":0.60499999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.53300000000000003,"15min":0.52300000000000002},"min":{"1min":0.35099999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.40999999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.91800000000000004,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.53700000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76100000000000001,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.40699999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.72099999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.61499999999999999,"15min":0.60699999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.68000000000000005,"15min":0.67500000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.50700000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65500000000000003,"15min":0.65400000000000003},"min":{"1min":0.48099999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.94399999999999995,"5min":1.103,"15min":1.103},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.64600000000000002,"15min":0.63900000000000001},"min":{"1min":0.48199999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.61499999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.65800000000000003,"15min":0.65200000000000002},"min":{"1min":0.50800000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67700000000000005,"15min":0.67500000000000004},"min":{"1min":0.497,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0149999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.48499999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65000000000000002,"15min":0.64900000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67000000000000004,"15min":0.66800000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.879,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.628}]},{"osd":8,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.84499999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.43099999999999999}]},{"osd":9,"last update":"Sun Sep 15 03:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61699999999999999,"15min":0.60799999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.65200000000000002,"15min":0.63900000000000001},"min":{"1min":0.45300000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.58599999999999997}]},{"osd":10,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.68500000000000005}]}]},{"osd":10,"up_from":105,"seq":450971566091,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798968,"kb_used_data":92880684,"kb_used_omap":3934,"kb_used_meta":0,"kb_avail":92880648,"statfs":{"total":95927926784,"available":95109783552,"internally_reserved":36864,"allocated":95109820416,"data_stored":95109820416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4028936,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]}]},{"osd":9,"up_from":17,"seq":73014444063,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797984,"kb_used_data":92881684,"kb_used_omap":5374,"kb_used_meta":0,"kb_avail":92881632,"statfs":{"total":95927926784,"available":95110791168,"internally_reserved":53248,"allocated":95110844416,"data_stored":95110844416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5503334,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.33400000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78200000000000003,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57799999999999996,"15min":0.57899999999999996},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.748,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.55600000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57199999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.82199999999999995,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.40300000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.75800000000000001,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.46400000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.54800000000000004,"15min":0.55400000000000005},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.71099999999999997,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.55500000000000005,"15min":0.56499999999999995},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69399999999999995,"5min":1.008,"15min":1.008},"last":0.40999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59499999999999997,"15min":0.58699999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.91300000000000003,"5min":1.03,"15min":1.03},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.64400000000000002,"15min":0.64800000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.83599999999999997,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.82899999999999996}]},{"osd":5,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.48799999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.86899999999999999,"5min":1.131,"15min":1.131},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.63800000000000001,"15min":0.64100000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82299999999999995,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.67300000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63700000000000001,"15min":0.63500000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.81299999999999994,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66800000000000004,"15min":0.67100000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.89300000000000002,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.84099999999999997}]},{"osd":7,"last update":"Sun Sep 15 03:48:37 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.65500000000000003,"15min":0.64600000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.751},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.64200000000000002,"15min":0.622},"min":{"1min":0.55000000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.79300000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51500000000000001,"15min":0.51700000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.68000000000000005,"5min":0.748,"15min":0.748},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.36799999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71099999999999997,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.78100000000000003}]},{"osd":10,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.90500000000000003}]},{"osd":11,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.65400000000000003,"15min":0.66600000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.874},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.66900000000000004,"15min":0.68300000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80900000000000005,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.624}]}]},{"osd":8,"up_from":17,"seq":73014444064,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799204,"kb_used_data":92880456,"kb_used_omap":5416,"kb_used_meta":0,"kb_avail":92880412,"statfs":{"total":95927926784,"available":95109541888,"internally_reserved":45056,"allocated":95109586944,"data_stored":95109586944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5546341,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.35199999999999998,"5min":0.313,"15min":0.313},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60399999999999998,"15min":0.60799999999999998},"min":{"1min":0.371,"5min":0.318,"15min":0.318},"max":{"1min":0.83199999999999996,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.50900000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":34.012,"15min":39.591000000000001},"min":{"1min":0.38200000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76400000000000001,"5min":963.98400000000004,"15min":963.98400000000004},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":34.064999999999998,"15min":39.652999999999999},"min":{"1min":0.40200000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91400000000000003,"5min":964.79600000000005,"15min":964.79600000000005},"last":0.41699999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.437}]},{"osd":3,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.54800000000000004,"15min":0.53900000000000003},"min":{"1min":0.373,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.58099999999999996,"15min":0.57099999999999995},"min":{"1min":0.44,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.55900000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.46200000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.82199999999999995,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.67700000000000005,"15min":0.68200000000000005},"min":{"1min":0.48399999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83299999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.621}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:48:30 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.68600000000000005,"15min":0.68200000000000005},"min":{"1min":0.5,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.93100000000000005,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.66300000000000003,"15min":0.65300000000000002},"min":{"1min":0.56200000000000006,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.48999999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69399999999999995,"15min":0.69799999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.67500000000000004,"15min":0.67900000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97899999999999998,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.60599999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.629,"15min":0.63300000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84799999999999998,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.68999999999999995}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.77800000000000002}]},{"osd":11,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63400000000000001,"15min":0.64200000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.627,"15min":0.63400000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.75800000000000001}]}]},{"osd":7,"up_from":17,"seq":73014444063,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798048,"kb_used_data":92881628,"kb_used_omap":5904,"kb_used_meta":0,"kb_avail":92881568,"statfs":{"total":95927926784,"available":95110725632,"internally_reserved":61440,"allocated":95110787072,"data_stored":95110787072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045714,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.50900000000000001,"15min":0.50800000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.40000000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.74399999999999999,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.56100000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.66100000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.38200000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.69599999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.54400000000000004,"15min":0.55200000000000005},"min":{"1min":0.34200000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71399999999999997,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.504},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.75,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.83799999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.49099999999999999,"15min":0.48799999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55100000000000005,"15min":0.55200000000000005},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.85899999999999999,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.39500000000000002}]},{"osd":5,"last update":"Sun Sep 15 03:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.32800000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.78200000000000003,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.60499999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.76500000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.623},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.872,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.66000000000000003}]},{"osd":8,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.1899999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":1.075},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.72699999999999998,"15min":0.73299999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.996,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.80400000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.76900000000000002,"15min":0.77500000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.747,"5min":0.79000000000000004,"15min":0.79700000000000004},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0860000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.98899999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.86099999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.80900000000000005,"15min":0.81200000000000006},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":1.038},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.145,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.74199999999999999}]}]},{"osd":6,"up_from":17,"seq":73014444063,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797924,"kb_used_data":92881752,"kb_used_omap":3996,"kb_used_meta":0,"kb_avail":92881692,"statfs":{"total":95927926784,"available":95110852608,"internally_reserved":61440,"allocated":95110914048,"data_stored":95110914048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4092729,"internal_metadata":0},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57299999999999995,"15min":0.57499999999999996},"min":{"1min":0.39000000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.84099999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.39000000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72499999999999998,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.45000000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:47:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.56299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67000000000000004,"15min":0.67400000000000004},"min":{"1min":0.45200000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78900000000000003,"5min":1.145,"15min":1.145},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.64100000000000001,"15min":0.64600000000000002},"min":{"1min":0.40699999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80900000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.51400000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.35799999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.75700000000000001,"5min":1.127,"15min":1.127},"last":0.62},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.755,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.63300000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:48:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.495,"15min":0.48499999999999999},"min":{"1min":0.32600000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.97099999999999997,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.51300000000000001,"15min":0.50600000000000001},"min":{"1min":0.32700000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.67900000000000005}]},{"osd":5,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.56100000000000005,"15min":0.55400000000000005},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.42499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81100000000000005,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.73699999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.59599999999999997,"15min":0.58799999999999997},"min":{"1min":0.40799999999999997,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85699999999999998,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.38,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82899999999999996,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.79600000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.80500000000000005,"15min":0.80700000000000005},"min":{"1min":0.57599999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.133,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.78200000000000003,"15min":0.78000000000000003},"min":{"1min":0.57099999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.1240000000000001,"5min":1.242,"15min":1.242},"last":0.96699999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.79500000000000004,"15min":0.79000000000000004},"min":{"1min":0.59199999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0940000000000001,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.79200000000000004,"15min":0.78800000000000003},"min":{"1min":0.60499999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.0620000000000001,"5min":1.21,"15min":1.21},"last":1.046}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.82899999999999996,"15min":0.82499999999999996},"min":{"1min":0.58799999999999997,"5min":0.505,"15min":0.505},"max":{"1min":1.482,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":1.026},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.79600000000000004,"15min":0.79400000000000004},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.0569999999999999,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.90500000000000003}]}]},{"osd":5,"up_from":17,"seq":73014444063,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797996,"kb_used_data":92881668,"kb_used_omap":1913,"kb_used_meta":0,"kb_avail":92881620,"statfs":{"total":95927926784,"available":95110778880,"internally_reserved":49152,"allocated":95110828032,"data_stored":95110828032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1959744,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.36899999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.59699999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":91.076999999999998,"15min":106.17},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.93000000000000005,"5min":2037.425,"15min":2037.425},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.60699999999999998,"5min":91.123000000000005,"15min":106.209},"min":{"1min":0.47999999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.90800000000000003,"5min":2037.3230000000001,"15min":2037.3230000000001},"last":0.66400000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60199999999999998,"15min":0.60699999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60599999999999998,"15min":0.60999999999999999},"min":{"1min":0.40699999999999997,"5min":0.379,"15min":0.379},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.63200000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:48:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.60999999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47899999999999998,"15min":0.47799999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.53100000000000003,"15min":0.53600000000000003},"min":{"1min":0.32100000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.751,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.53800000000000003}]},{"osd":6,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55400000000000005,"15min":0.55200000000000005},"min":{"1min":0.33400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.38300000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55900000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59999999999999998,"15min":0.60599999999999998},"min":{"1min":0.443,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.95299999999999996,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.60699999999999998,"15min":0.61599999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.80900000000000005,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.46800000000000003}]},{"osd":8,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.73499999999999999,"15min":0.74199999999999999},"min":{"1min":0.54000000000000004,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.127,"5min":1.218,"15min":1.218},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.75,"15min":0.754},"min":{"1min":0.57399999999999995,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.94699999999999995,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.67000000000000004}]},{"osd":9,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.77100000000000002,"15min":0.78100000000000003},"min":{"1min":0.55500000000000005,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.88600000000000001,"5min":1.177,"15min":1.177},"last":0.747},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.58699999999999997,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.75600000000000001}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.83399999999999996}]},{"osd":11,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.77800000000000002},"min":{"1min":0.56799999999999995,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.96499999999999997,"5min":1.155,"15min":1.155},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.78900000000000003,"15min":0.79800000000000004},"min":{"1min":0.55900000000000005,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.90800000000000003,"5min":1.194,"15min":1.194},"last":0.77600000000000002}]}]},{"osd":4,"up_from":17,"seq":73014444063,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797980,"kb_used_data":92881680,"kb_used_omap":5862,"kb_used_meta":0,"kb_avail":92881636,"statfs":{"total":95927926784,"available":95110795264,"internally_reserved":45056,"allocated":95110840320,"data_stored":95110840320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6002968,"internal_metadata":0},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.73099999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.61799999999999999,"15min":0.623},"min":{"1min":0.39300000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88100000000000001,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.51600000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":67.558000000000007,"15min":78.728999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.83099999999999996,"5min":1758.9780000000001,"15min":1758.9780000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.55600000000000005,"5min":67.602000000000004,"15min":78.775999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84799999999999998,"5min":1758.671,"15min":1758.671},"last":0.58499999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.65100000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.56599999999999995,"15min":0.57799999999999996},"min":{"1min":0.35899999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.73399999999999999,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75800000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.48499999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.51800000000000002,"15min":0.53200000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69999999999999996,"5min":0.996,"15min":0.996},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.498,"15min":0.50600000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72599999999999998,"5min":0.747,"15min":0.747},"last":0.63900000000000001}]},{"osd":6,"last update":"Sun Sep 15 03:48:29 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.90200000000000002}]},{"osd":7,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66000000000000003,"15min":0.67200000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.77800000000000002,"5min":1.113,"15min":1.113},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.64000000000000001,"15min":0.66000000000000003},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.81000000000000005,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.85699999999999998}]},{"osd":8,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.745,"15min":0.76500000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.99199999999999999,"5min":1.532,"15min":1.532},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.78300000000000003,"15min":0.79900000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.96899999999999997,"5min":1.363,"15min":1.363},"last":0.81000000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.75800000000000001,"15min":0.77600000000000002},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.86799999999999999,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.751,"15min":0.77400000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.85499999999999998,"5min":1.444,"15min":1.444},"last":0.71099999999999997}]},{"osd":10,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.79400000000000004}]},{"osd":11,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.84599999999999997,"15min":0.86899999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0069999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.746},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.79500000000000004,"15min":0.81499999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.91800000000000004,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.88300000000000001}]}]},{"osd":3,"up_from":17,"seq":73014444063,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797984,"kb_used_data":92881680,"kb_used_omap":4766,"kb_used_meta":0,"kb_avail":92881632,"statfs":{"total":95927926784,"available":95110791168,"internally_reserved":49152,"allocated":95110840320,"data_stored":95110840320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4880403,"internal_metadata":0},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.373,"15min":0.36699999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.98099999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:47:38 2024","interfaces":[{"interface":"back","average":{"1min":35.600000000000001,"5min":35.600000000000001,"15min":35.600000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":564.05100000000004,"5min":564.05100000000004,"15min":564.05100000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":35.677,"5min":35.677,"15min":35.677},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":564.57000000000005,"5min":564.57000000000005,"15min":564.57000000000005},"last":0.46400000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.45000000000000001,"15min":0.42199999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.1799999999999997,"5min":5.1799999999999997,"15min":5.1799999999999997},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.45700000000000002,"15min":0.44500000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.498}]},{"osd":4,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.57499999999999996,"15min":0.54900000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.57299999999999995,"15min":0.56499999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.3639999999999999,"5min":2.3639999999999999,"15min":2.3639999999999999},"last":0.79900000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":2.3940000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":5.2830000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.57499999999999996,"15min":0.56499999999999995},"min":{"1min":0.38200000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":1.133},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.65500000000000003,"15min":0.629},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.2489999999999997,"5min":5.2489999999999997,"15min":5.2489999999999997},"last":0.57899999999999996}]},{"osd":7,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.827,"5min":1.827,"15min":1.827},"last":1.1060000000000001},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.63,"15min":0.59999999999999998},"min":{"1min":0.42899999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":5.3090000000000002,"5min":5.3090000000000002,"15min":5.3090000000000002},"last":0.61499999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.68700000000000006,"15min":0.67100000000000004},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.66300000000000003,"15min":0.64900000000000002},"min":{"1min":0.54000000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.8,"5min":1.8,"15min":1.8},"last":1.026}]},{"osd":9,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.71899999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.495,"15min":0.495},"max":{"1min":5.3399999999999999,"5min":5.3399999999999999,"15min":5.3399999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.71999999999999997,"15min":0.70799999999999996},"min":{"1min":0.52200000000000002,"5min":0.498,"15min":0.498},"max":{"1min":2.4239999999999999,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":0.94699999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.70499999999999996,"15min":0.69599999999999995},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.859,"5min":1.859,"15min":1.859},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.73299999999999998,"15min":0.72399999999999998},"min":{"1min":0.51100000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.5430000000000001,"5min":2.5430000000000001,"15min":2.5430000000000001},"last":0.90500000000000003}]}]},{"osd":2,"up_from":17,"seq":73014444063,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797640,"kb_used_data":92881976,"kb_used_omap":5376,"kb_used_meta":0,"kb_avail":92881976,"statfs":{"total":95927926784,"available":95111143424,"internally_reserved":0,"allocated":95111143424,"data_stored":95111143424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5505295,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:37 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.45500000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":27.634,"15min":32.183},"min":{"1min":0.248,"5min":0.247,"15min":0.247},"max":{"1min":0.56499999999999995,"5min":647.92399999999998,"15min":647.92399999999998},"last":0.317},{"interface":"front","average":{"1min":0.40500000000000003,"5min":27.661000000000001,"15min":32.203000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.56599999999999995,"5min":648.02700000000004,"15min":648.02700000000004},"last":0.36899999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.65100000000000002,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38500000000000001,"15min":0.38900000000000001},"min":{"1min":0.27500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.51300000000000001,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.34300000000000003}]},{"osd":4,"last update":"Sun Sep 15 03:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.70999999999999996}]},{"osd":5,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.58799999999999997,"15min":0.58999999999999997},"min":{"1min":0.36499999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.95799999999999996,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.55000000000000004,"15min":0.53800000000000003},"min":{"1min":0.47799999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.84699999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.64700000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59499999999999997,"15min":0.58899999999999997},"min":{"1min":0.42199999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.84699999999999998,"5min":1.321,"15min":1.321},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.59099999999999997,"15min":0.58199999999999996},"min":{"1min":0.46700000000000003,"5min":0.374,"15min":0.374},"max":{"1min":0.90800000000000003,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.88500000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59799999999999998,"15min":0.59099999999999997},"min":{"1min":0.40500000000000003,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93300000000000005,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.751},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.48399999999999999,"5min":0.44,"15min":0.44},"max":{"1min":0.878,"5min":1.363,"15min":1.363},"last":0.76800000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":1.204,"15min":1.296},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90100000000000002,"5min":14.721,"15min":14.721},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.69199999999999995,"5min":1.2350000000000001,"15min":1.325},"min":{"1min":0.50700000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.873,"5min":14.970000000000001,"15min":14.970000000000001},"last":0.629}]},{"osd":9,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":1.2410000000000001,"15min":1.331},"min":{"1min":0.54800000000000004,"5min":0.502,"15min":0.502},"max":{"1min":0.875,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":1.244,"15min":1.3380000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.90900000000000003,"5min":14.888,"15min":14.888},"last":0.61599999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.2529999999999999,"15min":1.3440000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.006,"5min":14.654999999999999,"15min":14.654999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":1.266,"15min":1.361},"min":{"1min":0.48999999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.89400000000000002,"5min":15.096,"15min":15.096},"last":0.73799999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798000,"kb_used_data":92881668,"kb_used_omap":6208,"kb_used_meta":0,"kb_avail":92881616,"statfs":{"total":95927926784,"available":95110774784,"internally_reserved":53248,"allocated":95110828032,"data_stored":95110828032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6357472,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.40300000000000002,"15min":0.41299999999999998},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.53200000000000003,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":1.5660000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.38200000000000001,"15min":0.38600000000000001},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":1.681}]},{"osd":2,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.51500000000000001,"15min":0.53300000000000003},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.65300000000000002,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.5960000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.27200000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":1.6499999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.52000000000000002,"15min":0.53400000000000003},"min":{"1min":0.33400000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69099999999999995,"5min":1.907,"15min":1.907},"last":1.6200000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.51000000000000001,"15min":0.52400000000000002},"min":{"1min":0.30099999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.83199999999999996,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":1.7030000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.33900000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.93000000000000005,"5min":1.482,"15min":1.482},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.68000000000000005,"15min":0.69399999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.79500000000000004,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":0.374}]},{"osd":5,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.98499999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.71099999999999997,"15min":0.73299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.71099999999999997,"5min":2.7440000000000002,"15min":2.7440000000000002},"last":0.41499999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.64400000000000002,"15min":0.65800000000000003},"min":{"1min":0.38900000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.95599999999999996,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.71599999999999997,"15min":0.73299999999999998},"min":{"1min":0.45400000000000001,"5min":0.432,"15min":0.432},"max":{"1min":1.083,"5min":3.278,"15min":3.278},"last":0.68999999999999995}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.81399999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":96.073999999999998,"15min":111.98399999999999},"min":{"1min":0.438,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.0649999999999999,"5min":2861.7190000000001,"15min":2861.7190000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.77700000000000002,"15min":0.80200000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.84799999999999998,"5min":1.946,"15min":1.946},"last":0.63800000000000001}]},{"osd":9,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":96.081000000000003,"15min":111.985},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.44},"max":{"1min":1.024,"5min":2861.6010000000001,"15min":2861.6010000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.70399999999999996,"15min":0.71399999999999997},"min":{"1min":0.44900000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.006,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.56599999999999995}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":119.96599999999999,"5min":119.96599999999999,"15min":119.96599999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2861.5639999999999,"5min":2861.5639999999999,"15min":2861.5639999999999},"last":0.58999999999999997}]},{"osd":11,"last update":"Sun Sep 15 03:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.71099999999999997}]}]},{"osd":0,"up_from":17,"seq":73014444063,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798564,"kb_used_data":92881100,"kb_used_omap":4435,"kb_used_meta":0,"kb_avail":92881052,"statfs":{"total":95927926784,"available":95110197248,"internally_reserved":49152,"allocated":95110246400,"data_stored":95110246400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4541645,"internal_metadata":0},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.42299999999999999,"15min":0.433},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.53300000000000003,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.44700000000000001,"15min":0.45300000000000001},"min":{"1min":0.29899999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.60399999999999998,"5min":1.155,"15min":1.155},"last":0.313}]},{"osd":2,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.804,"5min":1.804,"15min":1.804},"last":0.379}]},{"osd":3,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.45900000000000002,"15min":0.47199999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.60399999999999998,"5min":1.52,"15min":1.52},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.33300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.65000000000000002,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.36499999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.57199999999999995}]},{"osd":5,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.60599999999999998,"15min":0.61599999999999999},"min":{"1min":0.41199999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.65600000000000003,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.68100000000000005,"15min":0.70099999999999996},"min":{"1min":0.41999999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.871,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.55500000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.63800000000000001,"15min":0.65900000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.72499999999999998,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.68400000000000005,"15min":0.70799999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81999999999999995,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.68400000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.67500000000000004,"15min":0.69699999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93400000000000005,"5min":1.359,"15min":1.359},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.74399999999999999,"5min":1.216,"15min":1.216},"last":0.61599999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.72899999999999998,"15min":0.74199999999999999},"min":{"1min":0.46000000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.93600000000000005,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.433},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.73799999999999999,"15min":0.75800000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.86499999999999999,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.54800000000000004}]},{"osd":9,"last update":"Sun Sep 15 03:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.52100000000000002}]},{"osd":10,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.48499999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.70899999999999996,"15min":0.71599999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.83899999999999997,"5min":1.603,"15min":1.603},"last":0.498},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.78700000000000003,"15min":0.80400000000000005},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89200000000000002,"5min":1.494,"15min":1.494},"last":0.63300000000000001}]}]}],"pool_statfs":[]}} 2024-09-15T03:48:41.414 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-15T03:48:41.414 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-15T03:48:41.423 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-09-15T03:48:41.423 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:48:41.696 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:48:41.696 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":147,"fsid":"f27b540e-e5e3-4e95-afe8-c95ee66b0a62","created":"2024-09-15T03:46:09.843094+0000","modified":"2024-09-15T03:48:37.149253+0000","last_up_change":"2024-09-15T03:47:51.883425+0000","last_in_change":"2024-09-15T03:47:32.982084+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:46:16.185530+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:46:22.284406+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":24,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"cd433a6a-c875-4e6e-b89c-938caecbae3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":28908},{"type":"v1","addr":"172.21.15.1:6801","nonce":28908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":28908},{"type":"v1","addr":"172.21.15.1:6803","nonce":28908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":28908},{"type":"v1","addr":"172.21.15.1:6807","nonce":28908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":28908},{"type":"v1","addr":"172.21.15.1:6805","nonce":28908}]},"public_addr":"172.21.15.1:6801/28908","cluster_addr":"172.21.15.1:6803/28908","heartbeat_back_addr":"172.21.15.1:6807/28908","heartbeat_front_addr":"172.21.15.1:6805/28908","state":["exists","up"]},{"osd":1,"uuid":"789930f9-42bf-476b-9f09-a21bffe33a79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":28913},{"type":"v1","addr":"172.21.15.1:6809","nonce":28913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":28913},{"type":"v1","addr":"172.21.15.1:6811","nonce":28913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":28913},{"type":"v1","addr":"172.21.15.1:6815","nonce":28913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":28913},{"type":"v1","addr":"172.21.15.1:6813","nonce":28913}]},"public_addr":"172.21.15.1:6809/28913","cluster_addr":"172.21.15.1:6811/28913","heartbeat_back_addr":"172.21.15.1:6815/28913","heartbeat_front_addr":"172.21.15.1:6813/28913","state":["exists","up"]},{"osd":2,"uuid":"9a83fec5-0d63-461c-b516-289443b3882b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":28911},{"type":"v1","addr":"172.21.15.1:6823","nonce":28911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":28911},{"type":"v1","addr":"172.21.15.1:6827","nonce":28911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":28911},{"type":"v1","addr":"172.21.15.1:6831","nonce":28911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":28911},{"type":"v1","addr":"172.21.15.1:6829","nonce":28911}]},"public_addr":"172.21.15.1:6823/28911","cluster_addr":"172.21.15.1:6827/28911","heartbeat_back_addr":"172.21.15.1:6831/28911","heartbeat_front_addr":"172.21.15.1:6829/28911","state":["exists","up"]},{"osd":3,"uuid":"d944bf89-e183-496a-9ca3-7949741b4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":28914},{"type":"v1","addr":"172.21.15.1:6817","nonce":28914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":28914},{"type":"v1","addr":"172.21.15.1:6819","nonce":28914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":28914},{"type":"v1","addr":"172.21.15.1:6826","nonce":28914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":28914},{"type":"v1","addr":"172.21.15.1:6822","nonce":28914}]},"public_addr":"172.21.15.1:6817/28914","cluster_addr":"172.21.15.1:6819/28914","heartbeat_back_addr":"172.21.15.1:6826/28914","heartbeat_front_addr":"172.21.15.1:6822/28914","state":["exists","up"]},{"osd":4,"uuid":"15ec2dbd-d099-4aff-94b2-1bf9dbb00b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":28468},{"type":"v1","addr":"172.21.15.67:6805","nonce":28468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6807","nonce":28468},{"type":"v1","addr":"172.21.15.67:6809","nonce":28468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":28468},{"type":"v1","addr":"172.21.15.67:6815","nonce":28468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":28468},{"type":"v1","addr":"172.21.15.67:6813","nonce":28468}]},"public_addr":"172.21.15.67:6805/28468","cluster_addr":"172.21.15.67:6809/28468","heartbeat_back_addr":"172.21.15.67:6815/28468","heartbeat_front_addr":"172.21.15.67:6813/28468","state":["exists","up"]},{"osd":5,"uuid":"1e45de07-746b-4612-9a52-d05509cae9a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":28495},{"type":"v1","addr":"172.21.15.67:6825","nonce":28495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":28495},{"type":"v1","addr":"172.21.15.67:6827","nonce":28495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":28495},{"type":"v1","addr":"172.21.15.67:6831","nonce":28495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":28495},{"type":"v1","addr":"172.21.15.67:6829","nonce":28495}]},"public_addr":"172.21.15.67:6825/28495","cluster_addr":"172.21.15.67:6827/28495","heartbeat_back_addr":"172.21.15.67:6831/28495","heartbeat_front_addr":"172.21.15.67:6829/28495","state":["exists","up"]},{"osd":6,"uuid":"ccf785f8-751d-473e-b17c-3df9346be9f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":28469},{"type":"v1","addr":"172.21.15.67:6801","nonce":28469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":28469},{"type":"v1","addr":"172.21.15.67:6804","nonce":28469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":28469},{"type":"v1","addr":"172.21.15.67:6812","nonce":28469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":28469},{"type":"v1","addr":"172.21.15.67:6808","nonce":28469}]},"public_addr":"172.21.15.67:6801/28469","cluster_addr":"172.21.15.67:6804/28469","heartbeat_back_addr":"172.21.15.67:6812/28469","heartbeat_front_addr":"172.21.15.67:6808/28469","state":["exists","up"]},{"osd":7,"uuid":"f9f59987-869d-42eb-9eb6-08c39e3afe6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":28470},{"type":"v1","addr":"172.21.15.67:6817","nonce":28470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":28470},{"type":"v1","addr":"172.21.15.67:6819","nonce":28470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":28470},{"type":"v1","addr":"172.21.15.67:6823","nonce":28470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":28470},{"type":"v1","addr":"172.21.15.67:6821","nonce":28470}]},"public_addr":"172.21.15.67:6817/28470","cluster_addr":"172.21.15.67:6819/28470","heartbeat_back_addr":"172.21.15.67:6823/28470","heartbeat_front_addr":"172.21.15.67:6821/28470","state":["exists","up"]},{"osd":8,"uuid":"69ddfe52-11df-4608-9a10-e8907c4cd87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":29472},{"type":"v1","addr":"172.21.15.190:6809","nonce":29472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":29472},{"type":"v1","addr":"172.21.15.190:6811","nonce":29472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":29472},{"type":"v1","addr":"172.21.15.190:6815","nonce":29472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":29472},{"type":"v1","addr":"172.21.15.190:6813","nonce":29472}]},"public_addr":"172.21.15.190:6809/29472","cluster_addr":"172.21.15.190:6811/29472","heartbeat_back_addr":"172.21.15.190:6815/29472","heartbeat_front_addr":"172.21.15.190:6813/29472","state":["exists","up"]},{"osd":9,"uuid":"f92e1609-2a1a-4814-9ac0-44a73373d229","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":29473},{"type":"v1","addr":"172.21.15.190:6801","nonce":29473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":29473},{"type":"v1","addr":"172.21.15.190:6803","nonce":29473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":29473},{"type":"v1","addr":"172.21.15.190:6807","nonce":29473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":29473},{"type":"v1","addr":"172.21.15.190:6805","nonce":29473}]},"public_addr":"172.21.15.190:6801/29473","cluster_addr":"172.21.15.190:6803/29473","heartbeat_back_addr":"172.21.15.190:6807/29473","heartbeat_front_addr":"172.21.15.190:6805/29473","state":["exists","up"]},{"osd":10,"uuid":"5f2e8ca3-8c65-4595-8e82-27dc3768ab99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":78,"up_from":105,"up_thru":108,"down_at":80,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":33741},{"type":"v1","addr":"172.21.15.190:6817","nonce":33741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":33741},{"type":"v1","addr":"172.21.15.190:6819","nonce":33741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":33741},{"type":"v1","addr":"172.21.15.190:6823","nonce":33741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":33741},{"type":"v1","addr":"172.21.15.190:6821","nonce":33741}]},"public_addr":"172.21.15.190:6817/33741","cluster_addr":"172.21.15.190:6819/33741","heartbeat_back_addr":"172.21.15.190:6823/33741","heartbeat_front_addr":"172.21.15.190:6821/33741","state":["exists","up"]},{"osd":11,"uuid":"cedf7df4-f875-42f5-a8cf-13cb82f05cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":29475},{"type":"v1","addr":"172.21.15.190:6825","nonce":29475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":29475},{"type":"v1","addr":"172.21.15.190:6827","nonce":29475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":29475},{"type":"v1","addr":"172.21.15.190:6831","nonce":29475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":29475},{"type":"v1","addr":"172.21.15.190:6829","nonce":29475}]},"public_addr":"172.21.15.190:6825/29475","cluster_addr":"172.21.15.190:6827/29475","heartbeat_back_addr":"172.21.15.190:6831/29475","heartbeat_front_addr":"172.21.15.190:6829/29475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:37.565633+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:48:35.370424+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:25.983954+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:48:30.827938+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:48:30.042695+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:34.158155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:37.031163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:18.853134+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:29.597174+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:37.409413+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:51.125326+0000","dead_epoch":80},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:48:36.717248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lrcprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","l":"3","m":"2","name":"lrcprofile","plugin":"lrc"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:48:41.709 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T03:48:41.709 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:48:41.981 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:48:41.982 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-15T03:48:41.993 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-09-15T03:48:40.170557+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":125,"num_write_kb":2152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":2,"num_per_pool_omap_osds":0,"kb":1124155392,"kb_used":9578232,"kb_used_data":1114577696,"kb_used_omap":58207,"kb_used_meta":0,"kb_avail":1114577160,"statfs":{"total":1151135121408,"available":1141327011840,"internally_reserved":548864,"allocated":1141327560704,"data_stored":1141327560704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59604195,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001275"},"pg_stats":[{"pgid":"1.0","version":"20'92","reported_seq":251,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.795857+0000","last_change":"2024-09-15T03:47:39.774326+0000","last_active":"2024-09-15T03:48:36.795857+0000","last_peered":"2024-09-15T03:48:36.795857+0000","last_clean":"2024-09-15T03:48:36.795857+0000","last_became_active":"2024-09-15T03:46:18.011130+0000","last_became_peered":"2024-09-15T03:46:18.011130+0000","last_unstale":"2024-09-15T03:48:36.795857+0000","last_undegraded":"2024-09-15T03:48:36.795857+0000","last_fullsized":"2024-09-15T03:48:36.795857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-15T03:47:39.774262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_clean_scrub_stamp":"2024-09-15T03:47:39.774262+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:43.691013+0000","scrub_duration":0.0027413759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"22'1","reported_seq":145,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.144868+0000","last_change":"2024-09-15T03:47:52.292975+0000","last_active":"2024-09-15T03:48:36.144868+0000","last_peered":"2024-09-15T03:48:36.144868+0000","last_clean":"2024-09-15T03:48:36.144868+0000","last_became_active":"2024-09-15T03:46:24.038633+0000","last_became_peered":"2024-09-15T03:46:24.038633+0000","last_unstale":"2024-09-15T03:48:36.144868+0000","last_undegraded":"2024-09-15T03:48:36.144868+0000","last_fullsized":"2024-09-15T03:48:36.144868+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'1","last_scrub_stamp":"2024-09-15T03:47:52.292926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:52.292926+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:56.362597+0000","scrub_duration":0.001409429,"objects_trimmed":0,"snaptrim_duration":0.00030178200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":140,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.795838+0000","last_change":"2024-09-15T03:47:27.779257+0000","last_active":"2024-09-15T03:48:36.795838+0000","last_peered":"2024-09-15T03:48:36.795838+0000","last_clean":"2024-09-15T03:48:36.795838+0000","last_became_active":"2024-09-15T03:46:24.063106+0000","last_became_peered":"2024-09-15T03:46:24.063106+0000","last_unstale":"2024-09-15T03:48:36.795838+0000","last_undegraded":"2024-09-15T03:48:36.795838+0000","last_fullsized":"2024-09-15T03:48:36.795838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.779210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.779210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:42.717969+0000","scrub_duration":0.0021030710000000002,"objects_trimmed":0,"snaptrim_duration":0.000393075,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":148,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.995562+0000","last_change":"2024-09-15T03:47:42.997012+0000","last_active":"2024-09-15T03:48:36.995562+0000","last_peered":"2024-09-15T03:48:36.995562+0000","last_clean":"2024-09-15T03:48:36.995562+0000","last_became_active":"2024-09-15T03:46:24.037471+0000","last_became_peered":"2024-09-15T03:46:24.037471+0000","last_unstale":"2024-09-15T03:48:36.995562+0000","last_undegraded":"2024-09-15T03:48:36.995562+0000","last_fullsized":"2024-09-15T03:48:36.995562+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:42.996957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:42.996957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:53.443125+0000","scrub_duration":0.0014164309999999999,"objects_trimmed":0,"snaptrim_duration":0.00026490299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"24'2","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.795873+0000","last_change":"2024-09-15T03:48:04.781084+0000","last_active":"2024-09-15T03:48:36.795873+0000","last_peered":"2024-09-15T03:48:36.795873+0000","last_clean":"2024-09-15T03:48:36.795873+0000","last_became_active":"2024-09-15T03:47:36.848076+0000","last_became_peered":"2024-09-15T03:47:36.848076+0000","last_unstale":"2024-09-15T03:48:36.795873+0000","last_undegraded":"2024-09-15T03:48:36.795873+0000","last_fullsized":"2024-09-15T03:48:36.795873+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'2","last_scrub_stamp":"2024-09-15T03:48:04.781025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:48:04.781025+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:13.832081+0000","scrub_duration":0.0016375750000000001,"objects_trimmed":0,"snaptrim_duration":0.000944677,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":166,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.144325+0000","last_change":"2024-09-15T03:47:39.930675+0000","last_active":"2024-09-15T03:48:36.144325+0000","last_peered":"2024-09-15T03:48:36.144325+0000","last_clean":"2024-09-15T03:48:36.144325+0000","last_became_active":"2024-09-15T03:47:33.998802+0000","last_became_peered":"2024-09-15T03:47:33.998802+0000","last_unstale":"2024-09-15T03:48:36.144325+0000","last_undegraded":"2024-09-15T03:48:36.144325+0000","last_fullsized":"2024-09-15T03:48:36.144325+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:39.930628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:39.930628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:03.535396+0000","scrub_duration":0.001146333,"objects_trimmed":0,"snaptrim_duration":0.00034123999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":142,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.995379+0000","last_change":"2024-09-15T03:47:30.025153+0000","last_active":"2024-09-15T03:48:36.995379+0000","last_peered":"2024-09-15T03:48:36.995379+0000","last_clean":"2024-09-15T03:48:36.995379+0000","last_became_active":"2024-09-15T03:46:24.038945+0000","last_became_peered":"2024-09-15T03:46:24.038945+0000","last_unstale":"2024-09-15T03:48:36.995379+0000","last_undegraded":"2024-09-15T03:48:36.995379+0000","last_fullsized":"2024-09-15T03:48:36.995379+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:30.025107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:30.025107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013039429999999999,"objects_trimmed":0,"snaptrim_duration":0.00032787499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":142,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.503241+0000","last_change":"2024-09-15T03:47:27.297139+0000","last_active":"2024-09-15T03:48:36.503241+0000","last_peered":"2024-09-15T03:48:36.503241+0000","last_clean":"2024-09-15T03:48:36.503241+0000","last_became_active":"2024-09-15T03:46:24.063380+0000","last_became_peered":"2024-09-15T03:46:24.063380+0000","last_unstale":"2024-09-15T03:48:36.503241+0000","last_undegraded":"2024-09-15T03:48:36.503241+0000","last_fullsized":"2024-09-15T03:48:36.503241+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.297083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.297083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:48.875998+0000","scrub_duration":0.0018328789999999999,"objects_trimmed":0,"snaptrim_duration":0.00034490600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":140,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.795873+0000","last_change":"2024-09-15T03:47:42.675933+0000","last_active":"2024-09-15T03:48:36.795873+0000","last_peered":"2024-09-15T03:48:36.795873+0000","last_clean":"2024-09-15T03:48:36.795873+0000","last_became_active":"2024-09-15T03:46:24.063335+0000","last_became_peered":"2024-09-15T03:46:24.063335+0000","last_unstale":"2024-09-15T03:48:36.795873+0000","last_undegraded":"2024-09-15T03:48:36.795873+0000","last_fullsized":"2024-09-15T03:48:36.795873+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:42.675882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:42.675882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:02.697036+0000","scrub_duration":0.001445478,"objects_trimmed":0,"snaptrim_duration":0.00065928000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":11,"up_from":17,"seq":73014444063,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797940,"kb_used_data":92881720,"kb_used_omap":5019,"kb_used_meta":0,"kb_avail":92881676,"statfs":{"total":95927926784,"available":95110836224,"internally_reserved":45056,"allocated":95110881280,"data_stored":95110881280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5139614,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52700000000000002,"15min":0.53100000000000003},"min":{"1min":0.36699999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.79200000000000004,"5min":1.002,"15min":1.002},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.503,"15min":0.502},"min":{"1min":0.33600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.82199999999999995,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.73099999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":121.575,"5min":121.575,"15min":121.575},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2060.797,"5min":2060.797,"15min":2060.797},"last":0.74199999999999999},{"interface":"front","average":{"1min":121.56,"5min":121.56,"15min":121.56},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2060.7159999999999,"5min":2060.7159999999999,"15min":2060.7159999999999},"last":0.60499999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.53300000000000003,"15min":0.52300000000000002},"min":{"1min":0.35099999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.40999999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.91800000000000004,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.53700000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76100000000000001,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.40699999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.72099999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.61499999999999999,"15min":0.60699999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.68000000000000005,"15min":0.67500000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.50700000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65500000000000003,"15min":0.65400000000000003},"min":{"1min":0.48099999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.94399999999999995,"5min":1.103,"15min":1.103},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.64600000000000002,"15min":0.63900000000000001},"min":{"1min":0.48199999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.61499999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.65800000000000003,"15min":0.65200000000000002},"min":{"1min":0.50800000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67700000000000005,"15min":0.67500000000000004},"min":{"1min":0.497,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0149999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.48499999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65000000000000002,"15min":0.64900000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67000000000000004,"15min":0.66800000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.879,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.628}]},{"osd":8,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.84499999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.43099999999999999}]},{"osd":9,"last update":"Sun Sep 15 03:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61699999999999999,"15min":0.60799999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.65200000000000002,"15min":0.63900000000000001},"min":{"1min":0.45300000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.58599999999999997}]},{"osd":10,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.68500000000000005}]}]},{"osd":10,"up_from":105,"seq":450971566091,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798968,"kb_used_data":92880684,"kb_used_omap":3934,"kb_used_meta":0,"kb_avail":92880648,"statfs":{"total":95927926784,"available":95109783552,"internally_reserved":36864,"allocated":95109820416,"data_stored":95109820416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4028936,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]}]},{"osd":9,"up_from":17,"seq":73014444063,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797984,"kb_used_data":92881684,"kb_used_omap":5374,"kb_used_meta":0,"kb_avail":92881632,"statfs":{"total":95927926784,"available":95110791168,"internally_reserved":53248,"allocated":95110844416,"data_stored":95110844416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5503334,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.33400000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78200000000000003,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57799999999999996,"15min":0.57899999999999996},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.748,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.55600000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57199999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.82199999999999995,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.40300000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.75800000000000001,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.46400000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.54800000000000004,"15min":0.55400000000000005},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.71099999999999997,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.55500000000000005,"15min":0.56499999999999995},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69399999999999995,"5min":1.008,"15min":1.008},"last":0.40999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59499999999999997,"15min":0.58699999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.91300000000000003,"5min":1.03,"15min":1.03},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.64400000000000002,"15min":0.64800000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.83599999999999997,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.82899999999999996}]},{"osd":5,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.48799999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.86899999999999999,"5min":1.131,"15min":1.131},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.63800000000000001,"15min":0.64100000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82299999999999995,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.67300000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63700000000000001,"15min":0.63500000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.81299999999999994,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66800000000000004,"15min":0.67100000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.89300000000000002,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.84099999999999997}]},{"osd":7,"last update":"Sun Sep 15 03:48:37 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.65500000000000003,"15min":0.64600000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.751},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.64200000000000002,"15min":0.622},"min":{"1min":0.55000000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.79300000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51500000000000001,"15min":0.51700000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.68000000000000005,"5min":0.748,"15min":0.748},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.36799999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71099999999999997,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.78100000000000003}]},{"osd":10,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.90500000000000003}]},{"osd":11,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.65400000000000003,"15min":0.66600000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.874},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.66900000000000004,"15min":0.68300000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80900000000000005,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.624}]}]},{"osd":8,"up_from":17,"seq":73014444064,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799204,"kb_used_data":92880456,"kb_used_omap":5416,"kb_used_meta":0,"kb_avail":92880412,"statfs":{"total":95927926784,"available":95109541888,"internally_reserved":45056,"allocated":95109586944,"data_stored":95109586944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5546341,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.35199999999999998,"5min":0.313,"15min":0.313},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60399999999999998,"15min":0.60799999999999998},"min":{"1min":0.371,"5min":0.318,"15min":0.318},"max":{"1min":0.83199999999999996,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.50900000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":34.012,"15min":39.591000000000001},"min":{"1min":0.38200000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76400000000000001,"5min":963.98400000000004,"15min":963.98400000000004},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":34.064999999999998,"15min":39.652999999999999},"min":{"1min":0.40200000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91400000000000003,"5min":964.79600000000005,"15min":964.79600000000005},"last":0.41699999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.437}]},{"osd":3,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.54800000000000004,"15min":0.53900000000000003},"min":{"1min":0.373,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.58099999999999996,"15min":0.57099999999999995},"min":{"1min":0.44,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.55900000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.46200000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.82199999999999995,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.67700000000000005,"15min":0.68200000000000005},"min":{"1min":0.48399999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83299999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.621}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:48:30 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.68600000000000005,"15min":0.68200000000000005},"min":{"1min":0.5,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.93100000000000005,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.66300000000000003,"15min":0.65300000000000002},"min":{"1min":0.56200000000000006,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.48999999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69399999999999995,"15min":0.69799999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.67500000000000004,"15min":0.67900000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97899999999999998,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.60599999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.629,"15min":0.63300000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84799999999999998,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.68999999999999995}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.77800000000000002}]},{"osd":11,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63400000000000001,"15min":0.64200000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.627,"15min":0.63400000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.75800000000000001}]}]},{"osd":7,"up_from":17,"seq":73014444063,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798048,"kb_used_data":92881628,"kb_used_omap":5904,"kb_used_meta":0,"kb_avail":92881568,"statfs":{"total":95927926784,"available":95110725632,"internally_reserved":61440,"allocated":95110787072,"data_stored":95110787072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045714,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.50900000000000001,"15min":0.50800000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.40000000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.74399999999999999,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.56100000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.66100000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.38200000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.69599999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.54400000000000004,"15min":0.55200000000000005},"min":{"1min":0.34200000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71399999999999997,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.504},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.75,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.83799999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.49099999999999999,"15min":0.48799999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55100000000000005,"15min":0.55200000000000005},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.85899999999999999,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.39500000000000002}]},{"osd":5,"last update":"Sun Sep 15 03:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.32800000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.78200000000000003,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.60499999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.76500000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.623},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.872,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.66000000000000003}]},{"osd":8,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.1899999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":1.075},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.72699999999999998,"15min":0.73299999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.996,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.80400000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.76900000000000002,"15min":0.77500000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.747,"5min":0.79000000000000004,"15min":0.79700000000000004},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0860000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.98899999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.86099999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.80900000000000005,"15min":0.81200000000000006},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":1.038},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.145,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.74199999999999999}]}]},{"osd":6,"up_from":17,"seq":73014444063,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797924,"kb_used_data":92881752,"kb_used_omap":3996,"kb_used_meta":0,"kb_avail":92881692,"statfs":{"total":95927926784,"available":95110852608,"internally_reserved":61440,"allocated":95110914048,"data_stored":95110914048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4092729,"internal_metadata":0},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57299999999999995,"15min":0.57499999999999996},"min":{"1min":0.39000000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.84099999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.39000000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72499999999999998,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.45000000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:47:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.56299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67000000000000004,"15min":0.67400000000000004},"min":{"1min":0.45200000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78900000000000003,"5min":1.145,"15min":1.145},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.64100000000000001,"15min":0.64600000000000002},"min":{"1min":0.40699999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80900000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.51400000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.35799999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.75700000000000001,"5min":1.127,"15min":1.127},"last":0.62},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.755,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.63300000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:48:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.495,"15min":0.48499999999999999},"min":{"1min":0.32600000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.97099999999999997,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.51300000000000001,"15min":0.50600000000000001},"min":{"1min":0.32700000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.67900000000000005}]},{"osd":5,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.56100000000000005,"15min":0.55400000000000005},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.42499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81100000000000005,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.73699999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.59599999999999997,"15min":0.58799999999999997},"min":{"1min":0.40799999999999997,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85699999999999998,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.38,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82899999999999996,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.79600000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.80500000000000005,"15min":0.80700000000000005},"min":{"1min":0.57599999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.133,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.78200000000000003,"15min":0.78000000000000003},"min":{"1min":0.57099999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.1240000000000001,"5min":1.242,"15min":1.242},"last":0.96699999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.79500000000000004,"15min":0.79000000000000004},"min":{"1min":0.59199999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0940000000000001,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.79200000000000004,"15min":0.78800000000000003},"min":{"1min":0.60499999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.0620000000000001,"5min":1.21,"15min":1.21},"last":1.046}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.82899999999999996,"15min":0.82499999999999996},"min":{"1min":0.58799999999999997,"5min":0.505,"15min":0.505},"max":{"1min":1.482,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":1.026},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.79600000000000004,"15min":0.79400000000000004},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.0569999999999999,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.90500000000000003}]}]},{"osd":5,"up_from":17,"seq":73014444063,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797996,"kb_used_data":92881668,"kb_used_omap":1913,"kb_used_meta":0,"kb_avail":92881620,"statfs":{"total":95927926784,"available":95110778880,"internally_reserved":49152,"allocated":95110828032,"data_stored":95110828032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1959744,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.36899999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.59699999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":91.076999999999998,"15min":106.17},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.93000000000000005,"5min":2037.425,"15min":2037.425},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.60699999999999998,"5min":91.123000000000005,"15min":106.209},"min":{"1min":0.47999999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.90800000000000003,"5min":2037.3230000000001,"15min":2037.3230000000001},"last":0.66400000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60199999999999998,"15min":0.60699999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60599999999999998,"15min":0.60999999999999999},"min":{"1min":0.40699999999999997,"5min":0.379,"15min":0.379},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.63200000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:48:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.60999999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47899999999999998,"15min":0.47799999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.53100000000000003,"15min":0.53600000000000003},"min":{"1min":0.32100000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.751,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.53800000000000003}]},{"osd":6,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55400000000000005,"15min":0.55200000000000005},"min":{"1min":0.33400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.38300000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55900000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59999999999999998,"15min":0.60599999999999998},"min":{"1min":0.443,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.95299999999999996,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.60699999999999998,"15min":0.61599999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.80900000000000005,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.46800000000000003}]},{"osd":8,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.73499999999999999,"15min":0.74199999999999999},"min":{"1min":0.54000000000000004,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.127,"5min":1.218,"15min":1.218},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.75,"15min":0.754},"min":{"1min":0.57399999999999995,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.94699999999999995,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.67000000000000004}]},{"osd":9,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.77100000000000002,"15min":0.78100000000000003},"min":{"1min":0.55500000000000005,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.88600000000000001,"5min":1.177,"15min":1.177},"last":0.747},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.58699999999999997,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.75600000000000001}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.83399999999999996}]},{"osd":11,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.77800000000000002},"min":{"1min":0.56799999999999995,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.96499999999999997,"5min":1.155,"15min":1.155},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.78900000000000003,"15min":0.79800000000000004},"min":{"1min":0.55900000000000005,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.90800000000000003,"5min":1.194,"15min":1.194},"last":0.77600000000000002}]}]},{"osd":4,"up_from":17,"seq":73014444063,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797980,"kb_used_data":92881680,"kb_used_omap":5862,"kb_used_meta":0,"kb_avail":92881636,"statfs":{"total":95927926784,"available":95110795264,"internally_reserved":45056,"allocated":95110840320,"data_stored":95110840320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6002968,"internal_metadata":0},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.73099999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.61799999999999999,"15min":0.623},"min":{"1min":0.39300000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88100000000000001,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.51600000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":67.558000000000007,"15min":78.728999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.83099999999999996,"5min":1758.9780000000001,"15min":1758.9780000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.55600000000000005,"5min":67.602000000000004,"15min":78.775999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84799999999999998,"5min":1758.671,"15min":1758.671},"last":0.58499999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.65100000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.56599999999999995,"15min":0.57799999999999996},"min":{"1min":0.35899999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.73399999999999999,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75800000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.48499999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.51800000000000002,"15min":0.53200000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69999999999999996,"5min":0.996,"15min":0.996},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.498,"15min":0.50600000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72599999999999998,"5min":0.747,"15min":0.747},"last":0.63900000000000001}]},{"osd":6,"last update":"Sun Sep 15 03:48:29 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.90200000000000002}]},{"osd":7,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66000000000000003,"15min":0.67200000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.77800000000000002,"5min":1.113,"15min":1.113},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.64000000000000001,"15min":0.66000000000000003},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.81000000000000005,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.85699999999999998}]},{"osd":8,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.745,"15min":0.76500000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.99199999999999999,"5min":1.532,"15min":1.532},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.78300000000000003,"15min":0.79900000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.96899999999999997,"5min":1.363,"15min":1.363},"last":0.81000000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.75800000000000001,"15min":0.77600000000000002},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.86799999999999999,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.751,"15min":0.77400000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.85499999999999998,"5min":1.444,"15min":1.444},"last":0.71099999999999997}]},{"osd":10,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.79400000000000004}]},{"osd":11,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.84599999999999997,"15min":0.86899999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0069999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.746},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.79500000000000004,"15min":0.81499999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.91800000000000004,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.88300000000000001}]}]},{"osd":3,"up_from":17,"seq":73014444063,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797984,"kb_used_data":92881680,"kb_used_omap":4766,"kb_used_meta":0,"kb_avail":92881632,"statfs":{"total":95927926784,"available":95110791168,"internally_reserved":49152,"allocated":95110840320,"data_stored":95110840320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4880403,"internal_metadata":0},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.373,"15min":0.36699999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.98099999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:47:38 2024","interfaces":[{"interface":"back","average":{"1min":35.600000000000001,"5min":35.600000000000001,"15min":35.600000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":564.05100000000004,"5min":564.05100000000004,"15min":564.05100000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":35.677,"5min":35.677,"15min":35.677},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":564.57000000000005,"5min":564.57000000000005,"15min":564.57000000000005},"last":0.46400000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.45000000000000001,"15min":0.42199999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.1799999999999997,"5min":5.1799999999999997,"15min":5.1799999999999997},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.45700000000000002,"15min":0.44500000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.498}]},{"osd":4,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.57499999999999996,"15min":0.54900000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.57299999999999995,"15min":0.56499999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.3639999999999999,"5min":2.3639999999999999,"15min":2.3639999999999999},"last":0.79900000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":2.3940000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":5.2830000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.57499999999999996,"15min":0.56499999999999995},"min":{"1min":0.38200000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":1.133},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.65500000000000003,"15min":0.629},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.2489999999999997,"5min":5.2489999999999997,"15min":5.2489999999999997},"last":0.57899999999999996}]},{"osd":7,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.827,"5min":1.827,"15min":1.827},"last":1.1060000000000001},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.63,"15min":0.59999999999999998},"min":{"1min":0.42899999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":5.3090000000000002,"5min":5.3090000000000002,"15min":5.3090000000000002},"last":0.61499999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.68700000000000006,"15min":0.67100000000000004},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.66300000000000003,"15min":0.64900000000000002},"min":{"1min":0.54000000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.8,"5min":1.8,"15min":1.8},"last":1.026}]},{"osd":9,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.71899999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.495,"15min":0.495},"max":{"1min":5.3399999999999999,"5min":5.3399999999999999,"15min":5.3399999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.71999999999999997,"15min":0.70799999999999996},"min":{"1min":0.52200000000000002,"5min":0.498,"15min":0.498},"max":{"1min":2.4239999999999999,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":0.94699999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.70499999999999996,"15min":0.69599999999999995},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.859,"5min":1.859,"15min":1.859},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.73299999999999998,"15min":0.72399999999999998},"min":{"1min":0.51100000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.5430000000000001,"5min":2.5430000000000001,"15min":2.5430000000000001},"last":0.90500000000000003}]}]},{"osd":2,"up_from":17,"seq":73014444063,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797640,"kb_used_data":92881976,"kb_used_omap":5376,"kb_used_meta":0,"kb_avail":92881976,"statfs":{"total":95927926784,"available":95111143424,"internally_reserved":0,"allocated":95111143424,"data_stored":95111143424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5505295,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:47:37 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.45500000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":27.634,"15min":32.183},"min":{"1min":0.248,"5min":0.247,"15min":0.247},"max":{"1min":0.56499999999999995,"5min":647.92399999999998,"15min":647.92399999999998},"last":0.317},{"interface":"front","average":{"1min":0.40500000000000003,"5min":27.661000000000001,"15min":32.203000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.56599999999999995,"5min":648.02700000000004,"15min":648.02700000000004},"last":0.36899999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.65100000000000002,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38500000000000001,"15min":0.38900000000000001},"min":{"1min":0.27500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.51300000000000001,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.34300000000000003}]},{"osd":4,"last update":"Sun Sep 15 03:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.70999999999999996}]},{"osd":5,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.58799999999999997,"15min":0.58999999999999997},"min":{"1min":0.36499999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.95799999999999996,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.55000000000000004,"15min":0.53800000000000003},"min":{"1min":0.47799999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.84699999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.64700000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59499999999999997,"15min":0.58899999999999997},"min":{"1min":0.42199999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.84699999999999998,"5min":1.321,"15min":1.321},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.59099999999999997,"15min":0.58199999999999996},"min":{"1min":0.46700000000000003,"5min":0.374,"15min":0.374},"max":{"1min":0.90800000000000003,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.88500000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59799999999999998,"15min":0.59099999999999997},"min":{"1min":0.40500000000000003,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93300000000000005,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.751},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.48399999999999999,"5min":0.44,"15min":0.44},"max":{"1min":0.878,"5min":1.363,"15min":1.363},"last":0.76800000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":1.204,"15min":1.296},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90100000000000002,"5min":14.721,"15min":14.721},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.69199999999999995,"5min":1.2350000000000001,"15min":1.325},"min":{"1min":0.50700000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.873,"5min":14.970000000000001,"15min":14.970000000000001},"last":0.629}]},{"osd":9,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":1.2410000000000001,"15min":1.331},"min":{"1min":0.54800000000000004,"5min":0.502,"15min":0.502},"max":{"1min":0.875,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":1.244,"15min":1.3380000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.90900000000000003,"5min":14.888,"15min":14.888},"last":0.61599999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.2529999999999999,"15min":1.3440000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.006,"5min":14.654999999999999,"15min":14.654999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":1.266,"15min":1.361},"min":{"1min":0.48999999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.89400000000000002,"5min":15.096,"15min":15.096},"last":0.73799999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798000,"kb_used_data":92881668,"kb_used_omap":6208,"kb_used_meta":0,"kb_avail":92881616,"statfs":{"total":95927926784,"available":95110774784,"internally_reserved":53248,"allocated":95110828032,"data_stored":95110828032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6357472,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.40300000000000002,"15min":0.41299999999999998},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.53200000000000003,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":1.5660000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.38200000000000001,"15min":0.38600000000000001},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":1.681}]},{"osd":2,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.51500000000000001,"15min":0.53300000000000003},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.65300000000000002,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.5960000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.27200000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":1.6499999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.52000000000000002,"15min":0.53400000000000003},"min":{"1min":0.33400000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69099999999999995,"5min":1.907,"15min":1.907},"last":1.6200000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.51000000000000001,"15min":0.52400000000000002},"min":{"1min":0.30099999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.83199999999999996,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":1.7030000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.33900000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.93000000000000005,"5min":1.482,"15min":1.482},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.68000000000000005,"15min":0.69399999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.79500000000000004,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":0.374}]},{"osd":5,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.98499999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.71099999999999997,"15min":0.73299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.71099999999999997,"5min":2.7440000000000002,"15min":2.7440000000000002},"last":0.41499999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.64400000000000002,"15min":0.65800000000000003},"min":{"1min":0.38900000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.95599999999999996,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.71599999999999997,"15min":0.73299999999999998},"min":{"1min":0.45400000000000001,"5min":0.432,"15min":0.432},"max":{"1min":1.083,"5min":3.278,"15min":3.278},"last":0.68999999999999995}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.81399999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":96.073999999999998,"15min":111.98399999999999},"min":{"1min":0.438,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.0649999999999999,"5min":2861.7190000000001,"15min":2861.7190000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.77700000000000002,"15min":0.80200000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.84799999999999998,"5min":1.946,"15min":1.946},"last":0.63800000000000001}]},{"osd":9,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":96.081000000000003,"15min":111.985},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.44},"max":{"1min":1.024,"5min":2861.6010000000001,"15min":2861.6010000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.70399999999999996,"15min":0.71399999999999997},"min":{"1min":0.44900000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.006,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.56599999999999995}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":119.96599999999999,"5min":119.96599999999999,"15min":119.96599999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2861.5639999999999,"5min":2861.5639999999999,"15min":2861.5639999999999},"last":0.58999999999999997}]},{"osd":11,"last update":"Sun Sep 15 03:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.71099999999999997}]}]},{"osd":0,"up_from":17,"seq":73014444063,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798564,"kb_used_data":92881100,"kb_used_omap":4435,"kb_used_meta":0,"kb_avail":92881052,"statfs":{"total":95927926784,"available":95110197248,"internally_reserved":49152,"allocated":95110246400,"data_stored":95110246400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4541645,"internal_metadata":0},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.42299999999999999,"15min":0.433},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.53300000000000003,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.44700000000000001,"15min":0.45300000000000001},"min":{"1min":0.29899999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.60399999999999998,"5min":1.155,"15min":1.155},"last":0.313}]},{"osd":2,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.804,"5min":1.804,"15min":1.804},"last":0.379}]},{"osd":3,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.45900000000000002,"15min":0.47199999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.60399999999999998,"5min":1.52,"15min":1.52},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.33300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.65000000000000002,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.36499999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.57199999999999995}]},{"osd":5,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.60599999999999998,"15min":0.61599999999999999},"min":{"1min":0.41199999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.65600000000000003,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.68100000000000005,"15min":0.70099999999999996},"min":{"1min":0.41999999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.871,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.55500000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.63800000000000001,"15min":0.65900000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.72499999999999998,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.68400000000000005,"15min":0.70799999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81999999999999995,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.68400000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.67500000000000004,"15min":0.69699999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93400000000000005,"5min":1.359,"15min":1.359},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.74399999999999999,"5min":1.216,"15min":1.216},"last":0.61599999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.72899999999999998,"15min":0.74199999999999999},"min":{"1min":0.46000000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.93600000000000005,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.433},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.73799999999999999,"15min":0.75800000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.86499999999999999,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.54800000000000004}]},{"osd":9,"last update":"Sun Sep 15 03:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.52100000000000002}]},{"osd":10,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.48499999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.70899999999999996,"15min":0.71599999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.83899999999999997,"5min":1.603,"15min":1.603},"last":0.498},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.78700000000000003,"15min":0.80400000000000005},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89200000000000002,"5min":1.494,"15min":1.494},"last":0.63300000000000001}]}]}],"pool_statfs":[]}} 2024-09-15T03:48:42.046 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:48:42.325 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:48:42.325 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-15T03:48:42.337 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-09-15T03:48:42.170936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":125,"num_write_kb":2152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":2,"num_per_pool_omap_osds":0,"kb":1124155392,"kb_used":9576020,"kb_used_data":1114579424,"kb_used_omap":58211,"kb_used_meta":0,"kb_avail":1114579372,"statfs":{"total":1151135121408,"available":1141329276928,"internally_reserved":53248,"allocated":1141329330176,"data_stored":1141329330176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59608883,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.032084"},"pg_stats":[{"pgid":"1.0","version":"20'92","reported_seq":251,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.795857+0000","last_change":"2024-09-15T03:47:39.774326+0000","last_active":"2024-09-15T03:48:36.795857+0000","last_peered":"2024-09-15T03:48:36.795857+0000","last_clean":"2024-09-15T03:48:36.795857+0000","last_became_active":"2024-09-15T03:46:18.011130+0000","last_became_peered":"2024-09-15T03:46:18.011130+0000","last_unstale":"2024-09-15T03:48:36.795857+0000","last_undegraded":"2024-09-15T03:48:36.795857+0000","last_fullsized":"2024-09-15T03:48:36.795857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-15T03:47:39.774262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_clean_scrub_stamp":"2024-09-15T03:47:39.774262+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:43.691013+0000","scrub_duration":0.0027413759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"22'1","reported_seq":145,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.144868+0000","last_change":"2024-09-15T03:47:52.292975+0000","last_active":"2024-09-15T03:48:36.144868+0000","last_peered":"2024-09-15T03:48:36.144868+0000","last_clean":"2024-09-15T03:48:36.144868+0000","last_became_active":"2024-09-15T03:46:24.038633+0000","last_became_peered":"2024-09-15T03:46:24.038633+0000","last_unstale":"2024-09-15T03:48:36.144868+0000","last_undegraded":"2024-09-15T03:48:36.144868+0000","last_fullsized":"2024-09-15T03:48:36.144868+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'1","last_scrub_stamp":"2024-09-15T03:47:52.292926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:52.292926+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:56.362597+0000","scrub_duration":0.001409429,"objects_trimmed":0,"snaptrim_duration":0.00030178200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":140,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.795838+0000","last_change":"2024-09-15T03:47:27.779257+0000","last_active":"2024-09-15T03:48:36.795838+0000","last_peered":"2024-09-15T03:48:36.795838+0000","last_clean":"2024-09-15T03:48:36.795838+0000","last_became_active":"2024-09-15T03:46:24.063106+0000","last_became_peered":"2024-09-15T03:46:24.063106+0000","last_unstale":"2024-09-15T03:48:36.795838+0000","last_undegraded":"2024-09-15T03:48:36.795838+0000","last_fullsized":"2024-09-15T03:48:36.795838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.779210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.779210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:42.717969+0000","scrub_duration":0.0021030710000000002,"objects_trimmed":0,"snaptrim_duration":0.000393075,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":148,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.995562+0000","last_change":"2024-09-15T03:47:42.997012+0000","last_active":"2024-09-15T03:48:36.995562+0000","last_peered":"2024-09-15T03:48:36.995562+0000","last_clean":"2024-09-15T03:48:36.995562+0000","last_became_active":"2024-09-15T03:46:24.037471+0000","last_became_peered":"2024-09-15T03:46:24.037471+0000","last_unstale":"2024-09-15T03:48:36.995562+0000","last_undegraded":"2024-09-15T03:48:36.995562+0000","last_fullsized":"2024-09-15T03:48:36.995562+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:42.996957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:42.996957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:53.443125+0000","scrub_duration":0.0014164309999999999,"objects_trimmed":0,"snaptrim_duration":0.00026490299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"24'2","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.795873+0000","last_change":"2024-09-15T03:48:04.781084+0000","last_active":"2024-09-15T03:48:36.795873+0000","last_peered":"2024-09-15T03:48:36.795873+0000","last_clean":"2024-09-15T03:48:36.795873+0000","last_became_active":"2024-09-15T03:47:36.848076+0000","last_became_peered":"2024-09-15T03:47:36.848076+0000","last_unstale":"2024-09-15T03:48:36.795873+0000","last_undegraded":"2024-09-15T03:48:36.795873+0000","last_fullsized":"2024-09-15T03:48:36.795873+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'2","last_scrub_stamp":"2024-09-15T03:48:04.781025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:48:04.781025+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:13.832081+0000","scrub_duration":0.0016375750000000001,"objects_trimmed":0,"snaptrim_duration":0.000944677,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":166,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.144325+0000","last_change":"2024-09-15T03:47:39.930675+0000","last_active":"2024-09-15T03:48:36.144325+0000","last_peered":"2024-09-15T03:48:36.144325+0000","last_clean":"2024-09-15T03:48:36.144325+0000","last_became_active":"2024-09-15T03:47:33.998802+0000","last_became_peered":"2024-09-15T03:47:33.998802+0000","last_unstale":"2024-09-15T03:48:36.144325+0000","last_undegraded":"2024-09-15T03:48:36.144325+0000","last_fullsized":"2024-09-15T03:48:36.144325+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:39.930628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:39.930628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:03.535396+0000","scrub_duration":0.001146333,"objects_trimmed":0,"snaptrim_duration":0.00034123999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":142,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.995379+0000","last_change":"2024-09-15T03:47:30.025153+0000","last_active":"2024-09-15T03:48:36.995379+0000","last_peered":"2024-09-15T03:48:36.995379+0000","last_clean":"2024-09-15T03:48:36.995379+0000","last_became_active":"2024-09-15T03:46:24.038945+0000","last_became_peered":"2024-09-15T03:46:24.038945+0000","last_unstale":"2024-09-15T03:48:36.995379+0000","last_undegraded":"2024-09-15T03:48:36.995379+0000","last_fullsized":"2024-09-15T03:48:36.995379+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:30.025107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:30.025107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013039429999999999,"objects_trimmed":0,"snaptrim_duration":0.00032787499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":142,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.503241+0000","last_change":"2024-09-15T03:47:27.297139+0000","last_active":"2024-09-15T03:48:36.503241+0000","last_peered":"2024-09-15T03:48:36.503241+0000","last_clean":"2024-09-15T03:48:36.503241+0000","last_became_active":"2024-09-15T03:46:24.063380+0000","last_became_peered":"2024-09-15T03:46:24.063380+0000","last_unstale":"2024-09-15T03:48:36.503241+0000","last_undegraded":"2024-09-15T03:48:36.503241+0000","last_fullsized":"2024-09-15T03:48:36.503241+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.297083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.297083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:48.875998+0000","scrub_duration":0.0018328789999999999,"objects_trimmed":0,"snaptrim_duration":0.00034490600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":140,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.795873+0000","last_change":"2024-09-15T03:47:42.675933+0000","last_active":"2024-09-15T03:48:36.795873+0000","last_peered":"2024-09-15T03:48:36.795873+0000","last_clean":"2024-09-15T03:48:36.795873+0000","last_became_active":"2024-09-15T03:46:24.063335+0000","last_became_peered":"2024-09-15T03:46:24.063335+0000","last_unstale":"2024-09-15T03:48:36.795873+0000","last_undegraded":"2024-09-15T03:48:36.795873+0000","last_fullsized":"2024-09-15T03:48:36.795873+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:42.675882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:42.675882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:02.697036+0000","scrub_duration":0.001445478,"objects_trimmed":0,"snaptrim_duration":0.00065928000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":11,"up_from":17,"seq":73014444064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797768,"kb_used_data":92881848,"kb_used_omap":5019,"kb_used_meta":0,"kb_avail":92881848,"statfs":{"total":95927926784,"available":95111012352,"internally_reserved":0,"allocated":95111012352,"data_stored":95111012352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5140362,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52700000000000002,"15min":0.53100000000000003},"min":{"1min":0.36699999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.79200000000000004,"5min":1.002,"15min":1.002},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.503,"15min":0.502},"min":{"1min":0.33600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.82199999999999995,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.47099999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":121.575,"5min":121.575,"15min":121.575},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2060.797,"5min":2060.797,"15min":2060.797},"last":0.74199999999999999},{"interface":"front","average":{"1min":121.56,"5min":121.56,"15min":121.56},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2060.7159999999999,"5min":2060.7159999999999,"15min":2060.7159999999999},"last":0.60499999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.53300000000000003,"15min":0.52300000000000002},"min":{"1min":0.35099999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.499},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.40999999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.91800000000000004,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.53700000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76100000000000001,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.40699999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.64800000000000002}]},{"osd":4,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.61499999999999999,"15min":0.60699999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.68000000000000005,"15min":0.67500000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.92300000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65500000000000003,"15min":0.65400000000000003},"min":{"1min":0.48099999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.94399999999999995,"5min":1.103,"15min":1.103},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.64600000000000002,"15min":0.63900000000000001},"min":{"1min":0.48199999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.79100000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.65800000000000003,"15min":0.65200000000000002},"min":{"1min":0.50800000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67700000000000005,"15min":0.67500000000000004},"min":{"1min":0.497,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0149999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.93400000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65000000000000002,"15min":0.64900000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67000000000000004,"15min":0.66800000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.879,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.94799999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.84499999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.75}]},{"osd":9,"last update":"Sun Sep 15 03:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61699999999999999,"15min":0.60799999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.65200000000000002,"15min":0.63900000000000001},"min":{"1min":0.45300000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.82699999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.87}]}]},{"osd":10,"up_from":105,"seq":450971566092,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798792,"kb_used_data":92880824,"kb_used_omap":3934,"kb_used_meta":0,"kb_avail":92880824,"statfs":{"total":95927926784,"available":95109963776,"internally_reserved":0,"allocated":95109963776,"data_stored":95109963776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4028986,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]}]},{"osd":9,"up_from":17,"seq":73014444064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797764,"kb_used_data":92881852,"kb_used_omap":5375,"kb_used_meta":0,"kb_avail":92881852,"statfs":{"total":95927926784,"available":95111016448,"internally_reserved":0,"allocated":95111016448,"data_stored":95111016448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5504082,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.33400000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78200000000000003,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57799999999999996,"15min":0.57899999999999996},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.748,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.53500000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57199999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.82199999999999995,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.40300000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.75800000000000001,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.64100000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.54800000000000004,"15min":0.55400000000000005},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.71099999999999997,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.55500000000000005,"15min":0.56499999999999995},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69399999999999995,"5min":1.008,"15min":1.008},"last":0.54800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59499999999999997,"15min":0.58699999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.91300000000000003,"5min":1.03,"15min":1.03},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.64400000000000002,"15min":0.64800000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.83599999999999997,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.66300000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.48799999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.86899999999999999,"5min":1.131,"15min":1.131},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.63800000000000001,"15min":0.64100000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82299999999999995,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.625}]},{"osd":6,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63700000000000001,"15min":0.63500000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.81299999999999994,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66800000000000004,"15min":0.67100000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.89300000000000002,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.67600000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:48:37 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.65500000000000003,"15min":0.64600000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.64200000000000002,"15min":0.622},"min":{"1min":0.55000000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.55900000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51500000000000001,"15min":0.51700000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.68000000000000005,"5min":0.748,"15min":0.748},"last":0.436},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.36799999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71099999999999997,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.48299999999999998}]},{"osd":10,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.40100000000000002}]},{"osd":11,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.65400000000000003,"15min":0.66600000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.66900000000000004,"15min":0.68300000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80900000000000005,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.504}]}]},{"osd":8,"up_from":17,"seq":73014444065,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798992,"kb_used_data":92880624,"kb_used_omap":5417,"kb_used_meta":0,"kb_avail":92880624,"statfs":{"total":95927926784,"available":95109758976,"internally_reserved":0,"allocated":95109758976,"data_stored":95109758976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5547089,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.35199999999999998,"5min":0.313,"15min":0.313},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60399999999999998,"15min":0.60799999999999998},"min":{"1min":0.371,"5min":0.318,"15min":0.318},"max":{"1min":0.83199999999999996,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.50900000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":34.012,"15min":39.591000000000001},"min":{"1min":0.38200000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76400000000000001,"5min":963.98400000000004,"15min":963.98400000000004},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":34.064999999999998,"15min":39.652999999999999},"min":{"1min":0.40200000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91400000000000003,"5min":964.79600000000005,"15min":964.79600000000005},"last":0.41699999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.437}]},{"osd":3,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.54800000000000004,"15min":0.53900000000000003},"min":{"1min":0.373,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.58099999999999996,"15min":0.57099999999999995},"min":{"1min":0.44,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.55900000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.46200000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.82199999999999995,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.67700000000000005,"15min":0.68200000000000005},"min":{"1min":0.48399999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83299999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.621}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:48:30 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.68600000000000005,"15min":0.68200000000000005},"min":{"1min":0.5,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.93100000000000005,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.66300000000000003,"15min":0.65300000000000002},"min":{"1min":0.56200000000000006,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.48999999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69399999999999995,"15min":0.69799999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.67500000000000004,"15min":0.67900000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97899999999999998,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.60599999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.629,"15min":0.63300000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84799999999999998,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.68999999999999995}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.77800000000000002}]},{"osd":11,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63400000000000001,"15min":0.64200000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.627,"15min":0.63400000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.75800000000000001}]}]},{"osd":7,"up_from":17,"seq":73014444064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797768,"kb_used_data":92881848,"kb_used_omap":5904,"kb_used_meta":0,"kb_avail":92881848,"statfs":{"total":95927926784,"available":95111012352,"internally_reserved":0,"allocated":95111012352,"data_stored":95111012352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045764,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.50900000000000001,"15min":0.50800000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.40000000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.74399999999999999,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.55800000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.66100000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.38200000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.629}]},{"osd":3,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.54400000000000004,"15min":0.55200000000000005},"min":{"1min":0.34200000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71399999999999997,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.75,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.72499999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.49099999999999999,"15min":0.48799999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55100000000000005,"15min":0.55200000000000005},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.85899999999999999,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.68500000000000005}]},{"osd":5,"last update":"Sun Sep 15 03:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.32800000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.78200000000000003,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.76800000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.76500000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.872,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.70799999999999996}]},{"osd":8,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.1899999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.72699999999999998,"15min":0.73299999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.996,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.69899999999999995}]},{"osd":9,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.76900000000000002,"15min":0.77500000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.747,"5min":0.79000000000000004,"15min":0.79700000000000004},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0860000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.86399999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.79800000000000004}]},{"osd":11,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.80900000000000005,"15min":0.81200000000000006},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.145,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.879}]}]},{"osd":6,"up_from":17,"seq":73014444064,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797644,"kb_used_data":92881972,"kb_used_omap":3996,"kb_used_meta":0,"kb_avail":92881972,"statfs":{"total":95927926784,"available":95111139328,"internally_reserved":0,"allocated":95111139328,"data_stored":95111139328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4092779,"internal_metadata":0},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57299999999999995,"15min":0.57499999999999996},"min":{"1min":0.39000000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.84099999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.39000000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72499999999999998,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.45000000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:47:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.56299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67000000000000004,"15min":0.67400000000000004},"min":{"1min":0.45200000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78900000000000003,"5min":1.145,"15min":1.145},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.64100000000000001,"15min":0.64600000000000002},"min":{"1min":0.40699999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80900000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.51400000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.35799999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.75700000000000001,"5min":1.127,"15min":1.127},"last":0.62},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.755,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.63300000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:48:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.495,"15min":0.48499999999999999},"min":{"1min":0.32600000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.97099999999999997,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.51300000000000001,"15min":0.50600000000000001},"min":{"1min":0.32700000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.67900000000000005}]},{"osd":5,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.56100000000000005,"15min":0.55400000000000005},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.42499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81100000000000005,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.73699999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.59599999999999997,"15min":0.58799999999999997},"min":{"1min":0.40799999999999997,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85699999999999998,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.38,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82899999999999996,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.79600000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.80500000000000005,"15min":0.80700000000000005},"min":{"1min":0.57599999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.133,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.78200000000000003,"15min":0.78000000000000003},"min":{"1min":0.57099999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.1240000000000001,"5min":1.242,"15min":1.242},"last":0.96699999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.79500000000000004,"15min":0.79000000000000004},"min":{"1min":0.59199999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0940000000000001,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.79200000000000004,"15min":0.78800000000000003},"min":{"1min":0.60499999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.0620000000000001,"5min":1.21,"15min":1.21},"last":1.046}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.82899999999999996,"15min":0.82499999999999996},"min":{"1min":0.58799999999999997,"5min":0.505,"15min":0.505},"max":{"1min":1.482,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":1.026},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.79600000000000004,"15min":0.79400000000000004},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.0569999999999999,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.90500000000000003}]}]},{"osd":5,"up_from":17,"seq":73014444064,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797768,"kb_used_data":92881848,"kb_used_omap":1913,"kb_used_meta":0,"kb_avail":92881848,"statfs":{"total":95927926784,"available":95111012352,"internally_reserved":0,"allocated":95111012352,"data_stored":95111012352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1959794,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.36899999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.64900000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":91.076999999999998,"15min":106.17},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.93000000000000005,"5min":2037.425,"15min":2037.425},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":91.123000000000005,"15min":106.209},"min":{"1min":0.47999999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.90800000000000003,"5min":2037.3230000000001,"15min":2037.3230000000001},"last":0.68799999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60199999999999998,"15min":0.60699999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60599999999999998,"15min":0.60999999999999999},"min":{"1min":0.40699999999999997,"5min":0.379,"15min":0.379},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.85399999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:48:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.61199999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47899999999999998,"15min":0.47799999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.53100000000000003,"15min":0.53600000000000003},"min":{"1min":0.32100000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.751,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.46500000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55400000000000005,"15min":0.55200000000000005},"min":{"1min":0.33400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.495},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.38300000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58499999999999996}]},{"osd":7,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59999999999999998,"15min":0.60599999999999998},"min":{"1min":0.443,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.95299999999999996,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.60699999999999998,"15min":0.61599999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.80900000000000005,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.56399999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.73499999999999999,"15min":0.74199999999999999},"min":{"1min":0.54000000000000004,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.127,"5min":1.218,"15min":1.218},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.75,"15min":0.754},"min":{"1min":0.57399999999999995,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.94699999999999995,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.79200000000000004}]},{"osd":9,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.77100000000000002,"15min":0.78100000000000003},"min":{"1min":0.55500000000000005,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.88600000000000001,"5min":1.177,"15min":1.177},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.58699999999999997,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.73099999999999998}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.83399999999999996}]},{"osd":11,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.77800000000000002},"min":{"1min":0.56799999999999995,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.96499999999999997,"5min":1.155,"15min":1.155},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.78900000000000003,"15min":0.79800000000000004},"min":{"1min":0.55900000000000005,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.90800000000000003,"5min":1.194,"15min":1.194},"last":0.77100000000000002}]}]},{"osd":4,"up_from":17,"seq":73014444064,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797768,"kb_used_data":92881848,"kb_used_omap":5863,"kb_used_meta":0,"kb_avail":92881848,"statfs":{"total":95927926784,"available":95111012352,"internally_reserved":0,"allocated":95111012352,"data_stored":95111012352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6003716,"internal_metadata":0},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.73099999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.61799999999999999,"15min":0.623},"min":{"1min":0.39300000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88100000000000001,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.46100000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":67.558000000000007,"15min":78.728999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.83099999999999996,"5min":1758.9780000000001,"15min":1758.9780000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":67.602000000000004,"15min":78.775999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84799999999999998,"5min":1758.671,"15min":1758.671},"last":0.48899999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.65100000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.56599999999999995,"15min":0.57799999999999996},"min":{"1min":0.35899999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.73399999999999999,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.373},{"interface":"front","average":{"1min":0.497,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75800000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.35299999999999998}]},{"osd":5,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.51800000000000002,"15min":0.53200000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69999999999999996,"5min":0.996,"15min":0.996},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.498,"15min":0.50600000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72599999999999998,"5min":0.747,"15min":0.747},"last":0.47399999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:48:29 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.51700000000000002}]},{"osd":7,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66000000000000003,"15min":0.67200000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.77800000000000002,"5min":1.113,"15min":1.113},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.64000000000000001,"15min":0.66000000000000003},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.81000000000000005,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.54500000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.745,"15min":0.76500000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.99199999999999999,"5min":1.532,"15min":1.532},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.78300000000000003,"15min":0.79900000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.96899999999999997,"5min":1.363,"15min":1.363},"last":0.64700000000000002}]},{"osd":9,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.75800000000000001,"15min":0.77600000000000002},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.86799999999999999,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.751,"15min":0.77400000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.85499999999999998,"5min":1.444,"15min":1.444},"last":0.52900000000000003}]},{"osd":10,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.68999999999999995}]},{"osd":11,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.84599999999999997,"15min":0.86899999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0069999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.63},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.79500000000000004,"15min":0.81499999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.91800000000000004,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.67900000000000005}]}]},{"osd":3,"up_from":17,"seq":73014444064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797768,"kb_used_data":92881848,"kb_used_omap":4766,"kb_used_meta":0,"kb_avail":92881848,"statfs":{"total":95927926784,"available":95111012352,"internally_reserved":0,"allocated":95111012352,"data_stored":95111012352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4881151,"internal_metadata":0},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":1.222},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.373,"15min":0.36699999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.67300000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":28.617000000000001,"15min":33.271999999999998},"min":{"1min":0.27600000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":7.0350000000000001,"5min":564.05100000000004,"15min":564.05100000000004},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.55700000000000005,"5min":28.652999999999999,"15min":33.335000000000001},"min":{"1min":0.33500000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.448,"5min":564.57000000000005,"15min":564.57000000000005},"last":1.0820000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.45000000000000001,"15min":0.42199999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.1799999999999997,"5min":5.1799999999999997,"15min":5.1799999999999997},"last":1.004},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.45700000000000002,"15min":0.44500000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":1.196}]},{"osd":4,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.57499999999999996,"15min":0.54900000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"last":0.44},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.57299999999999995,"15min":0.56499999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.3639999999999999,"5min":2.3639999999999999,"15min":2.3639999999999999},"last":1.048}]},{"osd":5,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":2.3940000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":5.2830000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.57499999999999996,"15min":0.56499999999999995},"min":{"1min":0.38200000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.65500000000000003,"15min":0.629},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.2489999999999997,"5min":5.2489999999999997,"15min":5.2489999999999997},"last":0.51600000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.827,"5min":1.827,"15min":1.827},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.63,"15min":0.59999999999999998},"min":{"1min":0.42899999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":5.3090000000000002,"5min":5.3090000000000002,"15min":5.3090000000000002},"last":0.47999999999999998}]},{"osd":8,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.68700000000000006,"15min":0.67100000000000004},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":1.1699999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.66300000000000003,"15min":0.64900000000000002},"min":{"1min":0.54000000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.8,"5min":1.8,"15min":1.8},"last":0.72499999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.71899999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.495,"15min":0.495},"max":{"1min":5.3399999999999999,"5min":5.3399999999999999,"15min":5.3399999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.71999999999999997,"15min":0.70799999999999996},"min":{"1min":0.52200000000000002,"5min":0.498,"15min":0.498},"max":{"1min":2.4239999999999999,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":1.143}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.70499999999999996,"15min":0.69599999999999995},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.859,"5min":1.859,"15min":1.859},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.73299999999999998,"15min":0.72399999999999998},"min":{"1min":0.51100000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.5430000000000001,"5min":2.5430000000000001,"15min":2.5430000000000001},"last":1.115}]}]},{"osd":2,"up_from":17,"seq":73014444064,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797640,"kb_used_data":92881976,"kb_used_omap":5376,"kb_used_meta":0,"kb_avail":92881976,"statfs":{"total":95927926784,"available":95111143424,"internally_reserved":0,"allocated":95111143424,"data_stored":95111143424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5505295,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.36299999999999999,"15min":0.36599999999999999},"min":{"1min":0.22900000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.56399999999999995,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.27400000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":27.634,"15min":32.183},"min":{"1min":0.248,"5min":0.247,"15min":0.247},"max":{"1min":0.56499999999999995,"5min":647.92399999999998,"15min":647.92399999999998},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":27.661000000000001,"15min":32.203000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.56599999999999995,"5min":648.02700000000004,"15min":648.02700000000004},"last":0.375}]},{"osd":3,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.65100000000000002,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38500000000000001,"15min":0.38900000000000001},"min":{"1min":0.27500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.51300000000000001,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.31900000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.69899999999999995}]},{"osd":5,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.58799999999999997,"15min":0.58999999999999997},"min":{"1min":0.36499999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.95799999999999996,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.55000000000000004,"15min":0.53800000000000003},"min":{"1min":0.47799999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.84699999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.59199999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59499999999999997,"15min":0.58899999999999997},"min":{"1min":0.42199999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.84699999999999998,"5min":1.321,"15min":1.321},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.59099999999999997,"15min":0.58199999999999996},"min":{"1min":0.46700000000000003,"5min":0.374,"15min":0.374},"max":{"1min":0.90800000000000003,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.56200000000000006}]},{"osd":7,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59799999999999998,"15min":0.59099999999999997},"min":{"1min":0.40500000000000003,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93300000000000005,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.48399999999999999,"5min":0.44,"15min":0.44},"max":{"1min":0.878,"5min":1.363,"15min":1.363},"last":0.66800000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":1.204,"15min":1.296},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90100000000000002,"5min":14.721,"15min":14.721},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.69199999999999995,"5min":1.2350000000000001,"15min":1.325},"min":{"1min":0.50700000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.873,"5min":14.970000000000001,"15min":14.970000000000001},"last":0.64800000000000002}]},{"osd":9,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":1.2410000000000001,"15min":1.331},"min":{"1min":0.54800000000000004,"5min":0.502,"15min":0.502},"max":{"1min":0.875,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":1.244,"15min":1.3380000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.90900000000000003,"5min":14.888,"15min":14.888},"last":0.63900000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.2529999999999999,"15min":1.3440000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.006,"5min":14.654999999999999,"15min":14.654999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":1.266,"15min":1.361},"min":{"1min":0.48999999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.89400000000000002,"5min":15.096,"15min":15.096},"last":0.72699999999999998}]}]},{"osd":1,"up_from":17,"seq":73014444064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798000,"kb_used_data":92881668,"kb_used_omap":6208,"kb_used_meta":0,"kb_avail":92881616,"statfs":{"total":95927926784,"available":95110774784,"internally_reserved":53248,"allocated":95110828032,"data_stored":95110828032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6357472,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.40300000000000002,"15min":0.41299999999999998},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.53200000000000003,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":1.5660000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.38200000000000001,"15min":0.38600000000000001},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":1.681}]},{"osd":2,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.51500000000000001,"15min":0.53300000000000003},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.65300000000000002,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.5960000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.27200000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":1.6499999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.52000000000000002,"15min":0.53400000000000003},"min":{"1min":0.33400000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69099999999999995,"5min":1.907,"15min":1.907},"last":1.6200000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.51000000000000001,"15min":0.52400000000000002},"min":{"1min":0.30099999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.83199999999999996,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":1.7030000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.33900000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.93000000000000005,"5min":1.482,"15min":1.482},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.68000000000000005,"15min":0.69399999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.79500000000000004,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":0.374}]},{"osd":5,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.98499999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.71099999999999997,"15min":0.73299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.71099999999999997,"5min":2.7440000000000002,"15min":2.7440000000000002},"last":0.41499999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.64400000000000002,"15min":0.65800000000000003},"min":{"1min":0.38900000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.95599999999999996,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.71599999999999997,"15min":0.73299999999999998},"min":{"1min":0.45400000000000001,"5min":0.432,"15min":0.432},"max":{"1min":1.083,"5min":3.278,"15min":3.278},"last":0.68999999999999995}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.81399999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":96.073999999999998,"15min":111.98399999999999},"min":{"1min":0.438,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.0649999999999999,"5min":2861.7190000000001,"15min":2861.7190000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.77700000000000002,"15min":0.80200000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.84799999999999998,"5min":1.946,"15min":1.946},"last":0.63800000000000001}]},{"osd":9,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":96.081000000000003,"15min":111.985},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.44},"max":{"1min":1.024,"5min":2861.6010000000001,"15min":2861.6010000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.70399999999999996,"15min":0.71399999999999997},"min":{"1min":0.44900000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.006,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.56599999999999995}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":119.96599999999999,"5min":119.96599999999999,"15min":119.96599999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2861.5639999999999,"5min":2861.5639999999999,"15min":2861.5639999999999},"last":0.58999999999999997}]},{"osd":11,"last update":"Sun Sep 15 03:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.71099999999999997}]}]},{"osd":0,"up_from":17,"seq":73014444064,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798348,"kb_used_data":92881268,"kb_used_omap":4435,"kb_used_meta":0,"kb_avail":92881268,"statfs":{"total":95927926784,"available":95110418432,"internally_reserved":0,"allocated":95110418432,"data_stored":95110418432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4542393,"internal_metadata":0},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.42299999999999999,"15min":0.433},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.53300000000000003,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.44700000000000001,"15min":0.45300000000000001},"min":{"1min":0.29899999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.60399999999999998,"5min":1.155,"15min":1.155},"last":0.89300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.804,"5min":1.804,"15min":1.804},"last":0.379}]},{"osd":3,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.45900000000000002,"15min":0.47199999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.60399999999999998,"5min":1.52,"15min":1.52},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.33300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.65000000000000002,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":1.24}]},{"osd":4,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.996}]},{"osd":5,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.60599999999999998,"15min":0.61599999999999999},"min":{"1min":0.41199999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.65600000000000003,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":1.1950000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.68100000000000005,"15min":0.70099999999999996},"min":{"1min":0.41999999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.871,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.93899999999999995}]},{"osd":6,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.63800000000000001,"15min":0.65900000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.72499999999999998,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.68400000000000005,"15min":0.70799999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81999999999999995,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.59099999999999997}]},{"osd":7,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.67500000000000004,"15min":0.69699999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93400000000000005,"5min":1.359,"15min":1.359},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.74399999999999999,"5min":1.216,"15min":1.216},"last":1.159}]},{"osd":8,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.72899999999999998,"15min":0.74199999999999999},"min":{"1min":0.46000000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.93600000000000005,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.73799999999999999,"15min":0.75800000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.86499999999999999,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.65100000000000002}]},{"osd":9,"last update":"Sun Sep 15 03:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":1.073},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.83099999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.623}]},{"osd":11,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.70899999999999996,"15min":0.71599999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.83899999999999997,"5min":1.603,"15min":1.603},"last":1.2190000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.78700000000000003,"15min":0.80400000000000005},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89200000000000002,"5min":1.494,"15min":1.494},"last":0.76500000000000001}]}]}],"pool_statfs":[]}} 2024-09-15T03:48:42.340 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T03:48:42.340 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:48:42.612 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:48:42.613 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-15T03:48:42.624 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-09-15T03:48:42.170936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":125,"num_write_kb":2152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":2,"num_per_pool_omap_osds":0,"kb":1124155392,"kb_used":9576020,"kb_used_data":1114579424,"kb_used_omap":58211,"kb_used_meta":0,"kb_avail":1114579372,"statfs":{"total":1151135121408,"available":1141329276928,"internally_reserved":53248,"allocated":1141329330176,"data_stored":1141329330176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59608883,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.032084"},"pg_stats":[{"pgid":"1.0","version":"20'92","reported_seq":251,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.795857+0000","last_change":"2024-09-15T03:47:39.774326+0000","last_active":"2024-09-15T03:48:36.795857+0000","last_peered":"2024-09-15T03:48:36.795857+0000","last_clean":"2024-09-15T03:48:36.795857+0000","last_became_active":"2024-09-15T03:46:18.011130+0000","last_became_peered":"2024-09-15T03:46:18.011130+0000","last_unstale":"2024-09-15T03:48:36.795857+0000","last_undegraded":"2024-09-15T03:48:36.795857+0000","last_fullsized":"2024-09-15T03:48:36.795857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-15T03:47:39.774262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_clean_scrub_stamp":"2024-09-15T03:47:39.774262+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:43.691013+0000","scrub_duration":0.0027413759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"22'1","reported_seq":145,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.144868+0000","last_change":"2024-09-15T03:47:52.292975+0000","last_active":"2024-09-15T03:48:36.144868+0000","last_peered":"2024-09-15T03:48:36.144868+0000","last_clean":"2024-09-15T03:48:36.144868+0000","last_became_active":"2024-09-15T03:46:24.038633+0000","last_became_peered":"2024-09-15T03:46:24.038633+0000","last_unstale":"2024-09-15T03:48:36.144868+0000","last_undegraded":"2024-09-15T03:48:36.144868+0000","last_fullsized":"2024-09-15T03:48:36.144868+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'1","last_scrub_stamp":"2024-09-15T03:47:52.292926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:52.292926+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:56.362597+0000","scrub_duration":0.001409429,"objects_trimmed":0,"snaptrim_duration":0.00030178200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":140,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.795838+0000","last_change":"2024-09-15T03:47:27.779257+0000","last_active":"2024-09-15T03:48:36.795838+0000","last_peered":"2024-09-15T03:48:36.795838+0000","last_clean":"2024-09-15T03:48:36.795838+0000","last_became_active":"2024-09-15T03:46:24.063106+0000","last_became_peered":"2024-09-15T03:46:24.063106+0000","last_unstale":"2024-09-15T03:48:36.795838+0000","last_undegraded":"2024-09-15T03:48:36.795838+0000","last_fullsized":"2024-09-15T03:48:36.795838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.779210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.779210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:42.717969+0000","scrub_duration":0.0021030710000000002,"objects_trimmed":0,"snaptrim_duration":0.000393075,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":148,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.995562+0000","last_change":"2024-09-15T03:47:42.997012+0000","last_active":"2024-09-15T03:48:36.995562+0000","last_peered":"2024-09-15T03:48:36.995562+0000","last_clean":"2024-09-15T03:48:36.995562+0000","last_became_active":"2024-09-15T03:46:24.037471+0000","last_became_peered":"2024-09-15T03:46:24.037471+0000","last_unstale":"2024-09-15T03:48:36.995562+0000","last_undegraded":"2024-09-15T03:48:36.995562+0000","last_fullsized":"2024-09-15T03:48:36.995562+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:42.996957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:42.996957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:53.443125+0000","scrub_duration":0.0014164309999999999,"objects_trimmed":0,"snaptrim_duration":0.00026490299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"24'2","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.795873+0000","last_change":"2024-09-15T03:48:04.781084+0000","last_active":"2024-09-15T03:48:36.795873+0000","last_peered":"2024-09-15T03:48:36.795873+0000","last_clean":"2024-09-15T03:48:36.795873+0000","last_became_active":"2024-09-15T03:47:36.848076+0000","last_became_peered":"2024-09-15T03:47:36.848076+0000","last_unstale":"2024-09-15T03:48:36.795873+0000","last_undegraded":"2024-09-15T03:48:36.795873+0000","last_fullsized":"2024-09-15T03:48:36.795873+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'2","last_scrub_stamp":"2024-09-15T03:48:04.781025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:48:04.781025+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:13.832081+0000","scrub_duration":0.0016375750000000001,"objects_trimmed":0,"snaptrim_duration":0.000944677,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":166,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.144325+0000","last_change":"2024-09-15T03:47:39.930675+0000","last_active":"2024-09-15T03:48:36.144325+0000","last_peered":"2024-09-15T03:48:36.144325+0000","last_clean":"2024-09-15T03:48:36.144325+0000","last_became_active":"2024-09-15T03:47:33.998802+0000","last_became_peered":"2024-09-15T03:47:33.998802+0000","last_unstale":"2024-09-15T03:48:36.144325+0000","last_undegraded":"2024-09-15T03:48:36.144325+0000","last_fullsized":"2024-09-15T03:48:36.144325+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:39.930628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:39.930628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:03.535396+0000","scrub_duration":0.001146333,"objects_trimmed":0,"snaptrim_duration":0.00034123999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":142,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.995379+0000","last_change":"2024-09-15T03:47:30.025153+0000","last_active":"2024-09-15T03:48:36.995379+0000","last_peered":"2024-09-15T03:48:36.995379+0000","last_clean":"2024-09-15T03:48:36.995379+0000","last_became_active":"2024-09-15T03:46:24.038945+0000","last_became_peered":"2024-09-15T03:46:24.038945+0000","last_unstale":"2024-09-15T03:48:36.995379+0000","last_undegraded":"2024-09-15T03:48:36.995379+0000","last_fullsized":"2024-09-15T03:48:36.995379+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:30.025107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:30.025107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013039429999999999,"objects_trimmed":0,"snaptrim_duration":0.00032787499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":142,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.503241+0000","last_change":"2024-09-15T03:47:27.297139+0000","last_active":"2024-09-15T03:48:36.503241+0000","last_peered":"2024-09-15T03:48:36.503241+0000","last_clean":"2024-09-15T03:48:36.503241+0000","last_became_active":"2024-09-15T03:46:24.063380+0000","last_became_peered":"2024-09-15T03:46:24.063380+0000","last_unstale":"2024-09-15T03:48:36.503241+0000","last_undegraded":"2024-09-15T03:48:36.503241+0000","last_fullsized":"2024-09-15T03:48:36.503241+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.297083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.297083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:48.875998+0000","scrub_duration":0.0018328789999999999,"objects_trimmed":0,"snaptrim_duration":0.00034490600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":140,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:48:36.795873+0000","last_change":"2024-09-15T03:47:42.675933+0000","last_active":"2024-09-15T03:48:36.795873+0000","last_peered":"2024-09-15T03:48:36.795873+0000","last_clean":"2024-09-15T03:48:36.795873+0000","last_became_active":"2024-09-15T03:46:24.063335+0000","last_became_peered":"2024-09-15T03:46:24.063335+0000","last_unstale":"2024-09-15T03:48:36.795873+0000","last_undegraded":"2024-09-15T03:48:36.795873+0000","last_fullsized":"2024-09-15T03:48:36.795873+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:42.675882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:42.675882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:02.697036+0000","scrub_duration":0.001445478,"objects_trimmed":0,"snaptrim_duration":0.00065928000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":11,"up_from":17,"seq":73014444064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797768,"kb_used_data":92881848,"kb_used_omap":5019,"kb_used_meta":0,"kb_avail":92881848,"statfs":{"total":95927926784,"available":95111012352,"internally_reserved":0,"allocated":95111012352,"data_stored":95111012352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5140362,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52700000000000002,"15min":0.53100000000000003},"min":{"1min":0.36699999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.79200000000000004,"5min":1.002,"15min":1.002},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.503,"15min":0.502},"min":{"1min":0.33600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.82199999999999995,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.47099999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":121.575,"5min":121.575,"15min":121.575},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2060.797,"5min":2060.797,"15min":2060.797},"last":0.74199999999999999},{"interface":"front","average":{"1min":121.56,"5min":121.56,"15min":121.56},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2060.7159999999999,"5min":2060.7159999999999,"15min":2060.7159999999999},"last":0.60499999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.53300000000000003,"15min":0.52300000000000002},"min":{"1min":0.35099999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.499},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.40999999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.91800000000000004,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.53700000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76100000000000001,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.40699999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.64800000000000002}]},{"osd":4,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.61499999999999999,"15min":0.60699999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.68000000000000005,"15min":0.67500000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.92300000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65500000000000003,"15min":0.65400000000000003},"min":{"1min":0.48099999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.94399999999999995,"5min":1.103,"15min":1.103},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.64600000000000002,"15min":0.63900000000000001},"min":{"1min":0.48199999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.79100000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.65800000000000003,"15min":0.65200000000000002},"min":{"1min":0.50800000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67700000000000005,"15min":0.67500000000000004},"min":{"1min":0.497,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0149999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.93400000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65000000000000002,"15min":0.64900000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67000000000000004,"15min":0.66800000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.879,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.94799999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.84499999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.75}]},{"osd":9,"last update":"Sun Sep 15 03:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61699999999999999,"15min":0.60799999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.65200000000000002,"15min":0.63900000000000001},"min":{"1min":0.45300000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.82699999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.87}]}]},{"osd":10,"up_from":105,"seq":450971566092,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798792,"kb_used_data":92880824,"kb_used_omap":3934,"kb_used_meta":0,"kb_avail":92880824,"statfs":{"total":95927926784,"available":95109963776,"internally_reserved":0,"allocated":95109963776,"data_stored":95109963776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4028986,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]}]},{"osd":9,"up_from":17,"seq":73014444064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797764,"kb_used_data":92881852,"kb_used_omap":5375,"kb_used_meta":0,"kb_avail":92881852,"statfs":{"total":95927926784,"available":95111016448,"internally_reserved":0,"allocated":95111016448,"data_stored":95111016448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5504082,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.33400000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78200000000000003,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57799999999999996,"15min":0.57899999999999996},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.748,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.53500000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57199999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.82199999999999995,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.40300000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.75800000000000001,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.64100000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.54800000000000004,"15min":0.55400000000000005},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.71099999999999997,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.55500000000000005,"15min":0.56499999999999995},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69399999999999995,"5min":1.008,"15min":1.008},"last":0.54800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59499999999999997,"15min":0.58699999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.91300000000000003,"5min":1.03,"15min":1.03},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.64400000000000002,"15min":0.64800000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.83599999999999997,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.66300000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.48799999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.86899999999999999,"5min":1.131,"15min":1.131},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.63800000000000001,"15min":0.64100000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82299999999999995,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.625}]},{"osd":6,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63700000000000001,"15min":0.63500000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.81299999999999994,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66800000000000004,"15min":0.67100000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.89300000000000002,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.67600000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:48:37 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.65500000000000003,"15min":0.64600000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.64200000000000002,"15min":0.622},"min":{"1min":0.55000000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.55900000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51500000000000001,"15min":0.51700000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.68000000000000005,"5min":0.748,"15min":0.748},"last":0.436},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.36799999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71099999999999997,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.48299999999999998}]},{"osd":10,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.40100000000000002}]},{"osd":11,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.65400000000000003,"15min":0.66600000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.66900000000000004,"15min":0.68300000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80900000000000005,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.504}]}]},{"osd":8,"up_from":17,"seq":73014444065,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798992,"kb_used_data":92880624,"kb_used_omap":5417,"kb_used_meta":0,"kb_avail":92880624,"statfs":{"total":95927926784,"available":95109758976,"internally_reserved":0,"allocated":95109758976,"data_stored":95109758976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5547089,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.35199999999999998,"5min":0.313,"15min":0.313},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60399999999999998,"15min":0.60799999999999998},"min":{"1min":0.371,"5min":0.318,"15min":0.318},"max":{"1min":0.83199999999999996,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.50900000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":34.012,"15min":39.591000000000001},"min":{"1min":0.38200000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76400000000000001,"5min":963.98400000000004,"15min":963.98400000000004},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":34.064999999999998,"15min":39.652999999999999},"min":{"1min":0.40200000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91400000000000003,"5min":964.79600000000005,"15min":964.79600000000005},"last":0.41699999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.437}]},{"osd":3,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.54800000000000004,"15min":0.53900000000000003},"min":{"1min":0.373,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.58099999999999996,"15min":0.57099999999999995},"min":{"1min":0.44,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.55900000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.46200000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.82199999999999995,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.67700000000000005,"15min":0.68200000000000005},"min":{"1min":0.48399999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83299999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.621}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:48:30 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.68600000000000005,"15min":0.68200000000000005},"min":{"1min":0.5,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.93100000000000005,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.66300000000000003,"15min":0.65300000000000002},"min":{"1min":0.56200000000000006,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.48999999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69399999999999995,"15min":0.69799999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.67500000000000004,"15min":0.67900000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97899999999999998,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.60599999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.629,"15min":0.63300000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84799999999999998,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.68999999999999995}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.77800000000000002}]},{"osd":11,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63400000000000001,"15min":0.64200000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.627,"15min":0.63400000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.75800000000000001}]}]},{"osd":7,"up_from":17,"seq":73014444064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797768,"kb_used_data":92881848,"kb_used_omap":5904,"kb_used_meta":0,"kb_avail":92881848,"statfs":{"total":95927926784,"available":95111012352,"internally_reserved":0,"allocated":95111012352,"data_stored":95111012352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045764,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.50900000000000001,"15min":0.50800000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.40000000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.74399999999999999,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.55800000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.66100000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.38200000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.629}]},{"osd":3,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.54400000000000004,"15min":0.55200000000000005},"min":{"1min":0.34200000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71399999999999997,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.75,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.72499999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.49099999999999999,"15min":0.48799999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55100000000000005,"15min":0.55200000000000005},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.85899999999999999,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.68500000000000005}]},{"osd":5,"last update":"Sun Sep 15 03:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.32800000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.78200000000000003,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.76800000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.76500000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.872,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.70799999999999996}]},{"osd":8,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.1899999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.72699999999999998,"15min":0.73299999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.996,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.69899999999999995}]},{"osd":9,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.76900000000000002,"15min":0.77500000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.747,"5min":0.79000000000000004,"15min":0.79700000000000004},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0860000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.86399999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.79800000000000004}]},{"osd":11,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.80900000000000005,"15min":0.81200000000000006},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.145,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.879}]}]},{"osd":6,"up_from":17,"seq":73014444064,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797644,"kb_used_data":92881972,"kb_used_omap":3996,"kb_used_meta":0,"kb_avail":92881972,"statfs":{"total":95927926784,"available":95111139328,"internally_reserved":0,"allocated":95111139328,"data_stored":95111139328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4092779,"internal_metadata":0},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57299999999999995,"15min":0.57499999999999996},"min":{"1min":0.39000000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.84099999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.39000000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72499999999999998,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.45000000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:47:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.56299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67000000000000004,"15min":0.67400000000000004},"min":{"1min":0.45200000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78900000000000003,"5min":1.145,"15min":1.145},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.64100000000000001,"15min":0.64600000000000002},"min":{"1min":0.40699999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80900000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.51400000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.35799999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.75700000000000001,"5min":1.127,"15min":1.127},"last":0.62},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.755,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.63300000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:48:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.495,"15min":0.48499999999999999},"min":{"1min":0.32600000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.97099999999999997,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.51300000000000001,"15min":0.50600000000000001},"min":{"1min":0.32700000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.67900000000000005}]},{"osd":5,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.56100000000000005,"15min":0.55400000000000005},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.42499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81100000000000005,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.73699999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.59599999999999997,"15min":0.58799999999999997},"min":{"1min":0.40799999999999997,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85699999999999998,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.38,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82899999999999996,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.79600000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.80500000000000005,"15min":0.80700000000000005},"min":{"1min":0.57599999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.133,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.78200000000000003,"15min":0.78000000000000003},"min":{"1min":0.57099999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.1240000000000001,"5min":1.242,"15min":1.242},"last":0.96699999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.79500000000000004,"15min":0.79000000000000004},"min":{"1min":0.59199999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0940000000000001,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.79200000000000004,"15min":0.78800000000000003},"min":{"1min":0.60499999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.0620000000000001,"5min":1.21,"15min":1.21},"last":1.046}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.82899999999999996,"15min":0.82499999999999996},"min":{"1min":0.58799999999999997,"5min":0.505,"15min":0.505},"max":{"1min":1.482,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":1.026},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.79600000000000004,"15min":0.79400000000000004},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.0569999999999999,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.90500000000000003}]}]},{"osd":5,"up_from":17,"seq":73014444064,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797768,"kb_used_data":92881848,"kb_used_omap":1913,"kb_used_meta":0,"kb_avail":92881848,"statfs":{"total":95927926784,"available":95111012352,"internally_reserved":0,"allocated":95111012352,"data_stored":95111012352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1959794,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.36899999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.64900000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":91.076999999999998,"15min":106.17},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.93000000000000005,"5min":2037.425,"15min":2037.425},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":91.123000000000005,"15min":106.209},"min":{"1min":0.47999999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.90800000000000003,"5min":2037.3230000000001,"15min":2037.3230000000001},"last":0.68799999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60199999999999998,"15min":0.60699999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60599999999999998,"15min":0.60999999999999999},"min":{"1min":0.40699999999999997,"5min":0.379,"15min":0.379},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.85399999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:48:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.61199999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47899999999999998,"15min":0.47799999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.53100000000000003,"15min":0.53600000000000003},"min":{"1min":0.32100000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.751,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.46500000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55400000000000005,"15min":0.55200000000000005},"min":{"1min":0.33400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.495},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.38300000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58499999999999996}]},{"osd":7,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59999999999999998,"15min":0.60599999999999998},"min":{"1min":0.443,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.95299999999999996,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.60699999999999998,"15min":0.61599999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.80900000000000005,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.56399999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.73499999999999999,"15min":0.74199999999999999},"min":{"1min":0.54000000000000004,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.127,"5min":1.218,"15min":1.218},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.75,"15min":0.754},"min":{"1min":0.57399999999999995,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.94699999999999995,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.79200000000000004}]},{"osd":9,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.77100000000000002,"15min":0.78100000000000003},"min":{"1min":0.55500000000000005,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.88600000000000001,"5min":1.177,"15min":1.177},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.58699999999999997,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.73099999999999998}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.83399999999999996}]},{"osd":11,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.77800000000000002},"min":{"1min":0.56799999999999995,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.96499999999999997,"5min":1.155,"15min":1.155},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.78900000000000003,"15min":0.79800000000000004},"min":{"1min":0.55900000000000005,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.90800000000000003,"5min":1.194,"15min":1.194},"last":0.77100000000000002}]}]},{"osd":4,"up_from":17,"seq":73014444064,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797768,"kb_used_data":92881848,"kb_used_omap":5863,"kb_used_meta":0,"kb_avail":92881848,"statfs":{"total":95927926784,"available":95111012352,"internally_reserved":0,"allocated":95111012352,"data_stored":95111012352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6003716,"internal_metadata":0},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.73099999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.61799999999999999,"15min":0.623},"min":{"1min":0.39300000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88100000000000001,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.46100000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":67.558000000000007,"15min":78.728999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.83099999999999996,"5min":1758.9780000000001,"15min":1758.9780000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":67.602000000000004,"15min":78.775999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84799999999999998,"5min":1758.671,"15min":1758.671},"last":0.48899999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.65100000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.56599999999999995,"15min":0.57799999999999996},"min":{"1min":0.35899999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.73399999999999999,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.373},{"interface":"front","average":{"1min":0.497,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75800000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.35299999999999998}]},{"osd":5,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.51800000000000002,"15min":0.53200000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69999999999999996,"5min":0.996,"15min":0.996},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.498,"15min":0.50600000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72599999999999998,"5min":0.747,"15min":0.747},"last":0.47399999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:48:29 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.51700000000000002}]},{"osd":7,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66000000000000003,"15min":0.67200000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.77800000000000002,"5min":1.113,"15min":1.113},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.64000000000000001,"15min":0.66000000000000003},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.81000000000000005,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.54500000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.745,"15min":0.76500000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.99199999999999999,"5min":1.532,"15min":1.532},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.78300000000000003,"15min":0.79900000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.96899999999999997,"5min":1.363,"15min":1.363},"last":0.64700000000000002}]},{"osd":9,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.75800000000000001,"15min":0.77600000000000002},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.86799999999999999,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.751,"15min":0.77400000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.85499999999999998,"5min":1.444,"15min":1.444},"last":0.52900000000000003}]},{"osd":10,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.68999999999999995}]},{"osd":11,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.84599999999999997,"15min":0.86899999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0069999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.63},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.79500000000000004,"15min":0.81499999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.91800000000000004,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.67900000000000005}]}]},{"osd":3,"up_from":17,"seq":73014444064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797768,"kb_used_data":92881848,"kb_used_omap":4766,"kb_used_meta":0,"kb_avail":92881848,"statfs":{"total":95927926784,"available":95111012352,"internally_reserved":0,"allocated":95111012352,"data_stored":95111012352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4881151,"internal_metadata":0},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":1.222},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.373,"15min":0.36699999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.67300000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":28.617000000000001,"15min":33.271999999999998},"min":{"1min":0.27600000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":7.0350000000000001,"5min":564.05100000000004,"15min":564.05100000000004},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.55700000000000005,"5min":28.652999999999999,"15min":33.335000000000001},"min":{"1min":0.33500000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.448,"5min":564.57000000000005,"15min":564.57000000000005},"last":1.0820000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.45000000000000001,"15min":0.42199999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.1799999999999997,"5min":5.1799999999999997,"15min":5.1799999999999997},"last":1.004},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.45700000000000002,"15min":0.44500000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":1.196}]},{"osd":4,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.57499999999999996,"15min":0.54900000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"last":0.44},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.57299999999999995,"15min":0.56499999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.3639999999999999,"5min":2.3639999999999999,"15min":2.3639999999999999},"last":1.048}]},{"osd":5,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":2.3940000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":5.2830000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.57499999999999996,"15min":0.56499999999999995},"min":{"1min":0.38200000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.65500000000000003,"15min":0.629},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.2489999999999997,"5min":5.2489999999999997,"15min":5.2489999999999997},"last":0.51600000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.827,"5min":1.827,"15min":1.827},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.63,"15min":0.59999999999999998},"min":{"1min":0.42899999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":5.3090000000000002,"5min":5.3090000000000002,"15min":5.3090000000000002},"last":0.47999999999999998}]},{"osd":8,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.68700000000000006,"15min":0.67100000000000004},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":1.1699999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.66300000000000003,"15min":0.64900000000000002},"min":{"1min":0.54000000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.8,"5min":1.8,"15min":1.8},"last":0.72499999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.71899999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.495,"15min":0.495},"max":{"1min":5.3399999999999999,"5min":5.3399999999999999,"15min":5.3399999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.71999999999999997,"15min":0.70799999999999996},"min":{"1min":0.52200000000000002,"5min":0.498,"15min":0.498},"max":{"1min":2.4239999999999999,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":1.143}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.70499999999999996,"15min":0.69599999999999995},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.859,"5min":1.859,"15min":1.859},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.73299999999999998,"15min":0.72399999999999998},"min":{"1min":0.51100000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.5430000000000001,"5min":2.5430000000000001,"15min":2.5430000000000001},"last":1.115}]}]},{"osd":2,"up_from":17,"seq":73014444064,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797640,"kb_used_data":92881976,"kb_used_omap":5376,"kb_used_meta":0,"kb_avail":92881976,"statfs":{"total":95927926784,"available":95111143424,"internally_reserved":0,"allocated":95111143424,"data_stored":95111143424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5505295,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.36299999999999999,"15min":0.36599999999999999},"min":{"1min":0.22900000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.56399999999999995,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.27400000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":27.634,"15min":32.183},"min":{"1min":0.248,"5min":0.247,"15min":0.247},"max":{"1min":0.56499999999999995,"5min":647.92399999999998,"15min":647.92399999999998},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":27.661000000000001,"15min":32.203000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.56599999999999995,"5min":648.02700000000004,"15min":648.02700000000004},"last":0.375}]},{"osd":3,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.65100000000000002,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38500000000000001,"15min":0.38900000000000001},"min":{"1min":0.27500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.51300000000000001,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.31900000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.69899999999999995}]},{"osd":5,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.58799999999999997,"15min":0.58999999999999997},"min":{"1min":0.36499999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.95799999999999996,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.55000000000000004,"15min":0.53800000000000003},"min":{"1min":0.47799999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.84699999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.59199999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59499999999999997,"15min":0.58899999999999997},"min":{"1min":0.42199999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.84699999999999998,"5min":1.321,"15min":1.321},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.59099999999999997,"15min":0.58199999999999996},"min":{"1min":0.46700000000000003,"5min":0.374,"15min":0.374},"max":{"1min":0.90800000000000003,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.56200000000000006}]},{"osd":7,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59799999999999998,"15min":0.59099999999999997},"min":{"1min":0.40500000000000003,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93300000000000005,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.48399999999999999,"5min":0.44,"15min":0.44},"max":{"1min":0.878,"5min":1.363,"15min":1.363},"last":0.66800000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":1.204,"15min":1.296},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90100000000000002,"5min":14.721,"15min":14.721},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.69199999999999995,"5min":1.2350000000000001,"15min":1.325},"min":{"1min":0.50700000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.873,"5min":14.970000000000001,"15min":14.970000000000001},"last":0.64800000000000002}]},{"osd":9,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":1.2410000000000001,"15min":1.331},"min":{"1min":0.54800000000000004,"5min":0.502,"15min":0.502},"max":{"1min":0.875,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":1.244,"15min":1.3380000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.90900000000000003,"5min":14.888,"15min":14.888},"last":0.63900000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.2529999999999999,"15min":1.3440000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.006,"5min":14.654999999999999,"15min":14.654999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":1.266,"15min":1.361},"min":{"1min":0.48999999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.89400000000000002,"5min":15.096,"15min":15.096},"last":0.72699999999999998}]}]},{"osd":1,"up_from":17,"seq":73014444064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798000,"kb_used_data":92881668,"kb_used_omap":6208,"kb_used_meta":0,"kb_avail":92881616,"statfs":{"total":95927926784,"available":95110774784,"internally_reserved":53248,"allocated":95110828032,"data_stored":95110828032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6357472,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.40300000000000002,"15min":0.41299999999999998},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.53200000000000003,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":1.5660000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.38200000000000001,"15min":0.38600000000000001},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":1.681}]},{"osd":2,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.51500000000000001,"15min":0.53300000000000003},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.65300000000000002,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.5960000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.27200000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":1.6499999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.52000000000000002,"15min":0.53400000000000003},"min":{"1min":0.33400000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69099999999999995,"5min":1.907,"15min":1.907},"last":1.6200000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.51000000000000001,"15min":0.52400000000000002},"min":{"1min":0.30099999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.83199999999999996,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":1.7030000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.33900000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.93000000000000005,"5min":1.482,"15min":1.482},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.68000000000000005,"15min":0.69399999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.79500000000000004,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":0.374}]},{"osd":5,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.98499999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.71099999999999997,"15min":0.73299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.71099999999999997,"5min":2.7440000000000002,"15min":2.7440000000000002},"last":0.41499999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.64400000000000002,"15min":0.65800000000000003},"min":{"1min":0.38900000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.95599999999999996,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.71599999999999997,"15min":0.73299999999999998},"min":{"1min":0.45400000000000001,"5min":0.432,"15min":0.432},"max":{"1min":1.083,"5min":3.278,"15min":3.278},"last":0.68999999999999995}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.81399999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":96.073999999999998,"15min":111.98399999999999},"min":{"1min":0.438,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.0649999999999999,"5min":2861.7190000000001,"15min":2861.7190000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.77700000000000002,"15min":0.80200000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.84799999999999998,"5min":1.946,"15min":1.946},"last":0.63800000000000001}]},{"osd":9,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":96.081000000000003,"15min":111.985},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.44},"max":{"1min":1.024,"5min":2861.6010000000001,"15min":2861.6010000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.70399999999999996,"15min":0.71399999999999997},"min":{"1min":0.44900000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.006,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.56599999999999995}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":119.96599999999999,"5min":119.96599999999999,"15min":119.96599999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2861.5639999999999,"5min":2861.5639999999999,"15min":2861.5639999999999},"last":0.58999999999999997}]},{"osd":11,"last update":"Sun Sep 15 03:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.71099999999999997}]}]},{"osd":0,"up_from":17,"seq":73014444064,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798348,"kb_used_data":92881268,"kb_used_omap":4435,"kb_used_meta":0,"kb_avail":92881268,"statfs":{"total":95927926784,"available":95110418432,"internally_reserved":0,"allocated":95110418432,"data_stored":95110418432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4542393,"internal_metadata":0},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.42299999999999999,"15min":0.433},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.53300000000000003,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.44700000000000001,"15min":0.45300000000000001},"min":{"1min":0.29899999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.60399999999999998,"5min":1.155,"15min":1.155},"last":0.89300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.804,"5min":1.804,"15min":1.804},"last":0.379}]},{"osd":3,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.45900000000000002,"15min":0.47199999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.60399999999999998,"5min":1.52,"15min":1.52},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.33300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.65000000000000002,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":1.24}]},{"osd":4,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.996}]},{"osd":5,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.60599999999999998,"15min":0.61599999999999999},"min":{"1min":0.41199999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.65600000000000003,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":1.1950000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.68100000000000005,"15min":0.70099999999999996},"min":{"1min":0.41999999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.871,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.93899999999999995}]},{"osd":6,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.63800000000000001,"15min":0.65900000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.72499999999999998,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.68400000000000005,"15min":0.70799999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81999999999999995,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.59099999999999997}]},{"osd":7,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.67500000000000004,"15min":0.69699999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93400000000000005,"5min":1.359,"15min":1.359},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.74399999999999999,"5min":1.216,"15min":1.216},"last":1.159}]},{"osd":8,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.72899999999999998,"15min":0.74199999999999999},"min":{"1min":0.46000000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.93600000000000005,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.73799999999999999,"15min":0.75800000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.86499999999999999,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.65100000000000002}]},{"osd":9,"last update":"Sun Sep 15 03:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":1.073},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.83099999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.623}]},{"osd":11,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.70899999999999996,"15min":0.71599999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.83899999999999997,"5min":1.603,"15min":1.603},"last":1.2190000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.78700000000000003,"15min":0.80400000000000005},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89200000000000002,"5min":1.494,"15min":1.494},"last":0.76500000000000001}]}]}],"pool_statfs":[]}} 2024-09-15T03:48:42.627 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:48:42.898 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:48:42.898 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":147,"fsid":"f27b540e-e5e3-4e95-afe8-c95ee66b0a62","created":"2024-09-15T03:46:09.843094+0000","modified":"2024-09-15T03:48:37.149253+0000","last_up_change":"2024-09-15T03:47:51.883425+0000","last_in_change":"2024-09-15T03:47:32.982084+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:46:16.185530+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:46:22.284406+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":24,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"cd433a6a-c875-4e6e-b89c-938caecbae3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":28908},{"type":"v1","addr":"172.21.15.1:6801","nonce":28908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":28908},{"type":"v1","addr":"172.21.15.1:6803","nonce":28908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":28908},{"type":"v1","addr":"172.21.15.1:6807","nonce":28908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":28908},{"type":"v1","addr":"172.21.15.1:6805","nonce":28908}]},"public_addr":"172.21.15.1:6801/28908","cluster_addr":"172.21.15.1:6803/28908","heartbeat_back_addr":"172.21.15.1:6807/28908","heartbeat_front_addr":"172.21.15.1:6805/28908","state":["exists","up"]},{"osd":1,"uuid":"789930f9-42bf-476b-9f09-a21bffe33a79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":28913},{"type":"v1","addr":"172.21.15.1:6809","nonce":28913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":28913},{"type":"v1","addr":"172.21.15.1:6811","nonce":28913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":28913},{"type":"v1","addr":"172.21.15.1:6815","nonce":28913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":28913},{"type":"v1","addr":"172.21.15.1:6813","nonce":28913}]},"public_addr":"172.21.15.1:6809/28913","cluster_addr":"172.21.15.1:6811/28913","heartbeat_back_addr":"172.21.15.1:6815/28913","heartbeat_front_addr":"172.21.15.1:6813/28913","state":["exists","up"]},{"osd":2,"uuid":"9a83fec5-0d63-461c-b516-289443b3882b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":28911},{"type":"v1","addr":"172.21.15.1:6823","nonce":28911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":28911},{"type":"v1","addr":"172.21.15.1:6827","nonce":28911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":28911},{"type":"v1","addr":"172.21.15.1:6831","nonce":28911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":28911},{"type":"v1","addr":"172.21.15.1:6829","nonce":28911}]},"public_addr":"172.21.15.1:6823/28911","cluster_addr":"172.21.15.1:6827/28911","heartbeat_back_addr":"172.21.15.1:6831/28911","heartbeat_front_addr":"172.21.15.1:6829/28911","state":["exists","up"]},{"osd":3,"uuid":"d944bf89-e183-496a-9ca3-7949741b4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":28914},{"type":"v1","addr":"172.21.15.1:6817","nonce":28914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":28914},{"type":"v1","addr":"172.21.15.1:6819","nonce":28914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":28914},{"type":"v1","addr":"172.21.15.1:6826","nonce":28914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":28914},{"type":"v1","addr":"172.21.15.1:6822","nonce":28914}]},"public_addr":"172.21.15.1:6817/28914","cluster_addr":"172.21.15.1:6819/28914","heartbeat_back_addr":"172.21.15.1:6826/28914","heartbeat_front_addr":"172.21.15.1:6822/28914","state":["exists","up"]},{"osd":4,"uuid":"15ec2dbd-d099-4aff-94b2-1bf9dbb00b56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":28468},{"type":"v1","addr":"172.21.15.67:6805","nonce":28468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6807","nonce":28468},{"type":"v1","addr":"172.21.15.67:6809","nonce":28468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":28468},{"type":"v1","addr":"172.21.15.67:6815","nonce":28468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":28468},{"type":"v1","addr":"172.21.15.67:6813","nonce":28468}]},"public_addr":"172.21.15.67:6805/28468","cluster_addr":"172.21.15.67:6809/28468","heartbeat_back_addr":"172.21.15.67:6815/28468","heartbeat_front_addr":"172.21.15.67:6813/28468","state":["exists","up"]},{"osd":5,"uuid":"1e45de07-746b-4612-9a52-d05509cae9a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":28495},{"type":"v1","addr":"172.21.15.67:6825","nonce":28495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":28495},{"type":"v1","addr":"172.21.15.67:6827","nonce":28495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":28495},{"type":"v1","addr":"172.21.15.67:6831","nonce":28495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":28495},{"type":"v1","addr":"172.21.15.67:6829","nonce":28495}]},"public_addr":"172.21.15.67:6825/28495","cluster_addr":"172.21.15.67:6827/28495","heartbeat_back_addr":"172.21.15.67:6831/28495","heartbeat_front_addr":"172.21.15.67:6829/28495","state":["exists","up"]},{"osd":6,"uuid":"ccf785f8-751d-473e-b17c-3df9346be9f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":28469},{"type":"v1","addr":"172.21.15.67:6801","nonce":28469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":28469},{"type":"v1","addr":"172.21.15.67:6804","nonce":28469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":28469},{"type":"v1","addr":"172.21.15.67:6812","nonce":28469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":28469},{"type":"v1","addr":"172.21.15.67:6808","nonce":28469}]},"public_addr":"172.21.15.67:6801/28469","cluster_addr":"172.21.15.67:6804/28469","heartbeat_back_addr":"172.21.15.67:6812/28469","heartbeat_front_addr":"172.21.15.67:6808/28469","state":["exists","up"]},{"osd":7,"uuid":"f9f59987-869d-42eb-9eb6-08c39e3afe6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":28470},{"type":"v1","addr":"172.21.15.67:6817","nonce":28470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":28470},{"type":"v1","addr":"172.21.15.67:6819","nonce":28470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":28470},{"type":"v1","addr":"172.21.15.67:6823","nonce":28470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":28470},{"type":"v1","addr":"172.21.15.67:6821","nonce":28470}]},"public_addr":"172.21.15.67:6817/28470","cluster_addr":"172.21.15.67:6819/28470","heartbeat_back_addr":"172.21.15.67:6823/28470","heartbeat_front_addr":"172.21.15.67:6821/28470","state":["exists","up"]},{"osd":8,"uuid":"69ddfe52-11df-4608-9a10-e8907c4cd87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":29472},{"type":"v1","addr":"172.21.15.190:6809","nonce":29472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":29472},{"type":"v1","addr":"172.21.15.190:6811","nonce":29472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":29472},{"type":"v1","addr":"172.21.15.190:6815","nonce":29472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":29472},{"type":"v1","addr":"172.21.15.190:6813","nonce":29472}]},"public_addr":"172.21.15.190:6809/29472","cluster_addr":"172.21.15.190:6811/29472","heartbeat_back_addr":"172.21.15.190:6815/29472","heartbeat_front_addr":"172.21.15.190:6813/29472","state":["exists","up"]},{"osd":9,"uuid":"f92e1609-2a1a-4814-9ac0-44a73373d229","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":29473},{"type":"v1","addr":"172.21.15.190:6801","nonce":29473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":29473},{"type":"v1","addr":"172.21.15.190:6803","nonce":29473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":29473},{"type":"v1","addr":"172.21.15.190:6807","nonce":29473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":29473},{"type":"v1","addr":"172.21.15.190:6805","nonce":29473}]},"public_addr":"172.21.15.190:6801/29473","cluster_addr":"172.21.15.190:6803/29473","heartbeat_back_addr":"172.21.15.190:6807/29473","heartbeat_front_addr":"172.21.15.190:6805/29473","state":["exists","up"]},{"osd":10,"uuid":"5f2e8ca3-8c65-4595-8e82-27dc3768ab99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":78,"up_from":105,"up_thru":108,"down_at":80,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":33741},{"type":"v1","addr":"172.21.15.190:6817","nonce":33741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":33741},{"type":"v1","addr":"172.21.15.190:6819","nonce":33741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":33741},{"type":"v1","addr":"172.21.15.190:6823","nonce":33741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":33741},{"type":"v1","addr":"172.21.15.190:6821","nonce":33741}]},"public_addr":"172.21.15.190:6817/33741","cluster_addr":"172.21.15.190:6819/33741","heartbeat_back_addr":"172.21.15.190:6823/33741","heartbeat_front_addr":"172.21.15.190:6821/33741","state":["exists","up"]},{"osd":11,"uuid":"cedf7df4-f875-42f5-a8cf-13cb82f05cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":29475},{"type":"v1","addr":"172.21.15.190:6825","nonce":29475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":29475},{"type":"v1","addr":"172.21.15.190:6827","nonce":29475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":29475},{"type":"v1","addr":"172.21.15.190:6831","nonce":29475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":29475},{"type":"v1","addr":"172.21.15.190:6829","nonce":29475}]},"public_addr":"172.21.15.190:6825/29475","cluster_addr":"172.21.15.190:6827/29475","heartbeat_back_addr":"172.21.15.190:6831/29475","heartbeat_front_addr":"172.21.15.190:6829/29475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:37.565633+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:48:35.370424+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:25.983954+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:48:30.827938+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:48:30.042695+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:34.158155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:37.031163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:18.853134+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:29.597174+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:37.409413+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:47:51.125326+0000","dead_epoch":80},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:48:36.717248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lrcprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","l":"3","m":"2","name":"lrcprofile","plugin":"lrc"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:48:43.911 INFO:tasks.ceph:Scrubbing osd.0 2024-09-15T03:48:43.911 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:48:44.043 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:44.043 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:48:44.043 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:44.054 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-15T03:48:44.325 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 0 to deep-scrub 2024-09-15T03:48:44.338 INFO:tasks.ceph:Scrubbing osd.1 2024-09-15T03:48:44.339 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:48:44.457 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:44.457 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:48:44.457 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:44.467 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-15T03:48:44.739 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 1 to deep-scrub 2024-09-15T03:48:44.751 INFO:tasks.ceph:Scrubbing osd.2 2024-09-15T03:48:44.752 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:48:44.871 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:44.871 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:48:44.871 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:44.881 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-15T03:48:45.153 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 2 to deep-scrub 2024-09-15T03:48:45.166 INFO:tasks.ceph:Scrubbing osd.3 2024-09-15T03:48:45.166 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:48:45.286 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:45.289 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:48:45.289 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:45.297 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-15T03:48:45.571 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 3 to deep-scrub 2024-09-15T03:48:45.584 INFO:tasks.ceph:Scrubbing osd.4 2024-09-15T03:48:45.584 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:48:45.703 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:45.703 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:48:45.703 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:45.713 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-15T03:48:45.987 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 4 to deep-scrub 2024-09-15T03:48:45.999 INFO:tasks.ceph:Scrubbing osd.5 2024-09-15T03:48:45.999 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:48:46.118 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:46.118 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:48:46.118 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:46.130 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-15T03:48:46.408 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 5 to deep-scrub 2024-09-15T03:48:46.420 INFO:tasks.ceph:Scrubbing osd.6 2024-09-15T03:48:46.420 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:48:46.540 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:46.540 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:48:46.540 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:46.550 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-15T03:48:46.822 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 6 to deep-scrub 2024-09-15T03:48:46.835 INFO:tasks.ceph:Scrubbing osd.7 2024-09-15T03:48:46.835 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:48:46.954 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:46.954 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:48:46.954 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:46.964 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-15T03:48:47.237 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 7 to deep-scrub 2024-09-15T03:48:47.250 INFO:tasks.ceph:Scrubbing osd.8 2024-09-15T03:48:47.250 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:48:47.370 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:47.370 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:48:47.370 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:47.381 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-09-15T03:48:47.654 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 8 to deep-scrub 2024-09-15T03:48:47.666 INFO:tasks.ceph:Scrubbing osd.9 2024-09-15T03:48:47.666 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:48:47.786 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:47.786 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:48:47.786 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:47.796 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-09-15T03:48:48.067 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 9 to deep-scrub 2024-09-15T03:48:48.079 INFO:tasks.ceph:Scrubbing osd.10 2024-09-15T03:48:48.079 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:48:48.199 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:48.199 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:48:48.199 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:48.209 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 10 2024-09-15T03:48:48.481 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 10 to deep-scrub 2024-09-15T03:48:48.494 INFO:tasks.ceph:Scrubbing osd.11 2024-09-15T03:48:48.494 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:48:48.613 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-15T03:48:48.614 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:48:48.614 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-15T03:48:48.624 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 11 2024-09-15T03:48:48.895 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 11 to deep-scrub 2024-09-15T03:48:48.908 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:48:49.182 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:48:49.182 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-15T03:48:49.193 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-09-15T03:48:48.172089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":125,"num_write_kb":2152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":2,"num_per_pool_omap_osds":0,"kb":1124155392,"kb_used":9576012,"kb_used_data":1114579444,"kb_used_omap":58219,"kb_used_meta":0,"kb_avail":1114579380,"statfs":{"total":1151135121408,"available":1141329285120,"internally_reserved":65536,"allocated":1141329350656,"data_stored":1141329350656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59616719,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001531"},"pg_stats":[{"pgid":"1.0","version":"20'92","reported_seq":262,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-15T03:48:45.380270+0000","last_change":"2024-09-15T03:48:45.380224+0000","last_active":"2024-09-15T03:48:45.380270+0000","last_peered":"2024-09-15T03:48:45.380270+0000","last_clean":"2024-09-15T03:48:45.380270+0000","last_became_active":"2024-09-15T03:46:18.011130+0000","last_became_peered":"2024-09-15T03:46:18.011130+0000","last_unstale":"2024-09-15T03:48:45.380270+0000","last_undegraded":"2024-09-15T03:48:45.380270+0000","last_fullsized":"2024-09-15T03:48:45.380270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-15T03:48:45.380164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:16.886982+0000","last_clean_scrub_stamp":"2024-09-15T03:48:45.380164+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:05.445301+0000","scrub_duration":0.002127477,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"22'1","reported_seq":146,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-15T03:48:37.826817+0000","last_change":"2024-09-15T03:47:52.292975+0000","last_active":"2024-09-15T03:48:37.826817+0000","last_peered":"2024-09-15T03:48:37.826817+0000","last_clean":"2024-09-15T03:48:37.826817+0000","last_became_active":"2024-09-15T03:46:24.038633+0000","last_became_peered":"2024-09-15T03:46:24.038633+0000","last_unstale":"2024-09-15T03:48:37.826817+0000","last_undegraded":"2024-09-15T03:48:37.826817+0000","last_fullsized":"2024-09-15T03:48:37.826817+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'1","last_scrub_stamp":"2024-09-15T03:47:52.292926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:52.292926+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:56.362597+0000","scrub_duration":0.001409429,"objects_trimmed":0,"snaptrim_duration":0.00030178200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":151,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-15T03:48:44.343815+0000","last_change":"2024-09-15T03:48:44.343760+0000","last_active":"2024-09-15T03:48:44.343815+0000","last_peered":"2024-09-15T03:48:44.343815+0000","last_clean":"2024-09-15T03:48:44.343815+0000","last_became_active":"2024-09-15T03:46:24.063106+0000","last_became_peered":"2024-09-15T03:46:24.063106+0000","last_unstale":"2024-09-15T03:48:44.343815+0000","last_undegraded":"2024-09-15T03:48:44.343815+0000","last_fullsized":"2024-09-15T03:48:44.343815+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:48:44.343703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:48:44.343703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:52.596382+0000","scrub_duration":0.0016175230000000001,"objects_trimmed":0,"snaptrim_duration":0.000393075,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":149,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-15T03:48:37.152618+0000","last_change":"2024-09-15T03:47:42.997012+0000","last_active":"2024-09-15T03:48:37.152618+0000","last_peered":"2024-09-15T03:48:37.152618+0000","last_clean":"2024-09-15T03:48:37.152618+0000","last_became_active":"2024-09-15T03:46:24.037471+0000","last_became_peered":"2024-09-15T03:46:24.037471+0000","last_unstale":"2024-09-15T03:48:37.152618+0000","last_undegraded":"2024-09-15T03:48:37.152618+0000","last_fullsized":"2024-09-15T03:48:37.152618+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:42.996957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:42.996957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:53.443125+0000","scrub_duration":0.0014164309999999999,"objects_trimmed":0,"snaptrim_duration":0.00026490299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"24'2","reported_seq":187,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-15T03:48:37.826964+0000","last_change":"2024-09-15T03:48:04.781084+0000","last_active":"2024-09-15T03:48:37.826964+0000","last_peered":"2024-09-15T03:48:37.826964+0000","last_clean":"2024-09-15T03:48:37.826964+0000","last_became_active":"2024-09-15T03:47:36.848076+0000","last_became_peered":"2024-09-15T03:47:36.848076+0000","last_unstale":"2024-09-15T03:48:37.826964+0000","last_undegraded":"2024-09-15T03:48:37.826964+0000","last_fullsized":"2024-09-15T03:48:37.826964+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'2","last_scrub_stamp":"2024-09-15T03:48:04.781025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:48:04.781025+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:13.832081+0000","scrub_duration":0.0016375750000000001,"objects_trimmed":0,"snaptrim_duration":0.000944677,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":167,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-15T03:48:37.151594+0000","last_change":"2024-09-15T03:47:39.930675+0000","last_active":"2024-09-15T03:48:37.151594+0000","last_peered":"2024-09-15T03:48:37.151594+0000","last_clean":"2024-09-15T03:48:37.151594+0000","last_became_active":"2024-09-15T03:47:33.998802+0000","last_became_peered":"2024-09-15T03:47:33.998802+0000","last_unstale":"2024-09-15T03:48:37.151594+0000","last_undegraded":"2024-09-15T03:48:37.151594+0000","last_fullsized":"2024-09-15T03:48:37.151594+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:39.930628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:39.930628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:03.535396+0000","scrub_duration":0.001146333,"objects_trimmed":0,"snaptrim_duration":0.00034123999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":154,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-15T03:48:46.097239+0000","last_change":"2024-09-15T03:48:41.258291+0000","last_active":"2024-09-15T03:48:46.097239+0000","last_peered":"2024-09-15T03:48:46.097239+0000","last_clean":"2024-09-15T03:48:46.097239+0000","last_became_active":"2024-09-15T03:46:24.038945+0000","last_became_peered":"2024-09-15T03:46:24.038945+0000","last_unstale":"2024-09-15T03:48:46.097239+0000","last_undegraded":"2024-09-15T03:48:46.097239+0000","last_fullsized":"2024-09-15T03:48:46.097239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:48:41.258243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:48:41.258243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.001533478,"objects_trimmed":0,"snaptrim_duration":0.00032787499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":144,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-15T03:48:46.097317+0000","last_change":"2024-09-15T03:47:27.297139+0000","last_active":"2024-09-15T03:48:46.097317+0000","last_peered":"2024-09-15T03:48:46.097317+0000","last_clean":"2024-09-15T03:48:46.097317+0000","last_became_active":"2024-09-15T03:46:24.063380+0000","last_became_peered":"2024-09-15T03:46:24.063380+0000","last_unstale":"2024-09-15T03:48:46.097317+0000","last_undegraded":"2024-09-15T03:48:46.097317+0000","last_fullsized":"2024-09-15T03:48:46.097317+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:27.297083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:27.297083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.0018328789999999999,"objects_trimmed":0,"snaptrim_duration":0.00034490600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":141,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-15T03:48:37.826941+0000","last_change":"2024-09-15T03:47:42.675933+0000","last_active":"2024-09-15T03:48:37.826941+0000","last_peered":"2024-09-15T03:48:37.826941+0000","last_clean":"2024-09-15T03:48:37.826941+0000","last_became_active":"2024-09-15T03:46:24.063335+0000","last_became_peered":"2024-09-15T03:46:24.063335+0000","last_unstale":"2024-09-15T03:48:37.826941+0000","last_undegraded":"2024-09-15T03:48:37.826941+0000","last_fullsized":"2024-09-15T03:48:37.826941+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:47:42.675882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:46:23.024039+0000","last_clean_scrub_stamp":"2024-09-15T03:47:42.675882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:02.697036+0000","scrub_duration":0.001445478,"objects_trimmed":0,"snaptrim_duration":0.00065928000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":11,"up_from":17,"seq":73014444065,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797852,"kb_used_data":92881776,"kb_used_omap":5020,"kb_used_meta":0,"kb_avail":92881764,"statfs":{"total":95927926784,"available":95110926336,"internally_reserved":12288,"allocated":95110938624,"data_stored":95110938624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5141060,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52700000000000002,"15min":0.53100000000000003},"min":{"1min":0.36699999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.79200000000000004,"5min":1.002,"15min":1.002},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.503,"15min":0.502},"min":{"1min":0.33600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.82199999999999995,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.40100000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":121.575,"5min":121.575,"15min":121.575},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2060.797,"5min":2060.797,"15min":2060.797},"last":0.74199999999999999},{"interface":"front","average":{"1min":121.56,"5min":121.56,"15min":121.56},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2060.7159999999999,"5min":2060.7159999999999,"15min":2060.7159999999999},"last":0.60499999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.53300000000000003,"15min":0.52300000000000002},"min":{"1min":0.35099999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.40999999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.91800000000000004,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.34699999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76100000000000001,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.40699999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.499}]},{"osd":4,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.61499999999999999,"15min":0.60699999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.63},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.68000000000000005,"15min":0.67500000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.76600000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65500000000000003,"15min":0.65400000000000003},"min":{"1min":0.48099999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.94399999999999995,"5min":1.103,"15min":1.103},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.64600000000000002,"15min":0.63900000000000001},"min":{"1min":0.48199999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.60199999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.65800000000000003,"15min":0.65200000000000002},"min":{"1min":0.50800000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67700000000000005,"15min":0.67500000000000004},"min":{"1min":0.497,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0149999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.57099999999999995}]},{"osd":7,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65000000000000002,"15min":0.64900000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67000000000000004,"15min":0.66800000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.879,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.748}]},{"osd":8,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.84499999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.48299999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61699999999999999,"15min":0.60799999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.622},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.65200000000000002,"15min":0.63900000000000001},"min":{"1min":0.45300000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.69199999999999995}]},{"osd":10,"last update":"Sun Sep 15 03:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.65400000000000003}]}]},{"osd":10,"up_from":105,"seq":450971566093,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798792,"kb_used_data":92880824,"kb_used_omap":3934,"kb_used_meta":0,"kb_avail":92880824,"statfs":{"total":95927926784,"available":95109963776,"internally_reserved":0,"allocated":95109963776,"data_stored":95109963776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4028986,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]}]},{"osd":9,"up_from":17,"seq":73014444065,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797848,"kb_used_data":92881780,"kb_used_omap":5375,"kb_used_meta":0,"kb_avail":92881768,"statfs":{"total":95927926784,"available":95110930432,"internally_reserved":12288,"allocated":95110942720,"data_stored":95110942720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5504780,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.33400000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78200000000000003,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57799999999999996,"15min":0.57899999999999996},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.748,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.54300000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57199999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.82199999999999995,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.40300000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.75800000000000001,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.73499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.54800000000000004,"15min":0.55400000000000005},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.71099999999999997,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.55500000000000005,"15min":0.56499999999999995},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69399999999999995,"5min":1.008,"15min":1.008},"last":0.52000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59499999999999997,"15min":0.58699999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.91300000000000003,"5min":1.03,"15min":1.03},"last":0.628},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.64400000000000002,"15min":0.64800000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.83599999999999997,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.70999999999999996}]},{"osd":5,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.48799999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.86899999999999999,"5min":1.131,"15min":1.131},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.63800000000000001,"15min":0.64100000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82299999999999995,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.65200000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63700000000000001,"15min":0.63500000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.81299999999999994,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66800000000000004,"15min":0.67100000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.89300000000000002,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.53800000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:48:37 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.65500000000000003,"15min":0.64600000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.64200000000000002,"15min":0.622},"min":{"1min":0.55000000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.69399999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51500000000000001,"15min":0.51700000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.68000000000000005,"5min":0.748,"15min":0.748},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.36799999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71099999999999997,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.46000000000000002}]},{"osd":10,"last update":"Sun Sep 15 03:47:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.40600000000000003}]},{"osd":11,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.65400000000000003,"15min":0.66600000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.66900000000000004,"15min":0.68300000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80900000000000005,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.48499999999999999}]}]},{"osd":8,"up_from":17,"seq":73014444066,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798872,"kb_used_data":92880752,"kb_used_omap":5418,"kb_used_meta":0,"kb_avail":92880744,"statfs":{"total":95927926784,"available":95109881856,"internally_reserved":8192,"allocated":95109890048,"data_stored":95109890048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5548256,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.35199999999999998,"5min":0.313,"15min":0.313},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60399999999999998,"15min":0.60799999999999998},"min":{"1min":0.371,"5min":0.318,"15min":0.318},"max":{"1min":0.83199999999999996,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.65100000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":34.012,"15min":39.591000000000001},"min":{"1min":0.38200000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76400000000000001,"5min":963.98400000000004,"15min":963.98400000000004},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.53700000000000003,"5min":34.064999999999998,"15min":39.652999999999999},"min":{"1min":0.40200000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91400000000000003,"5min":964.79600000000005,"15min":964.79600000000005},"last":0.55000000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.70299999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.54800000000000004,"15min":0.53900000000000003},"min":{"1min":0.373,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.624},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.58099999999999996,"15min":0.57099999999999995},"min":{"1min":0.44,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.71499999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.46200000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.82199999999999995,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.67700000000000005,"15min":0.68200000000000005},"min":{"1min":0.48399999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83299999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.65900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:48:30 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.68600000000000005,"15min":0.68200000000000005},"min":{"1min":0.5,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.93100000000000005,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.66300000000000003,"15min":0.65300000000000002},"min":{"1min":0.56200000000000006,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.56299999999999994}]},{"osd":7,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69399999999999995,"15min":0.69799999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.67500000000000004,"15min":0.67900000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97899999999999998,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.53500000000000003}]},{"osd":9,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.629,"15min":0.63300000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84799999999999998,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.501}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.59299999999999997}]},{"osd":11,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63400000000000001,"15min":0.64200000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.627,"15min":0.63400000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.63900000000000001}]}]},{"osd":7,"up_from":17,"seq":73014444065,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797856,"kb_used_data":92881776,"kb_used_omap":5905,"kb_used_meta":0,"kb_avail":92881760,"statfs":{"total":95927926784,"available":95110922240,"internally_reserved":16384,"allocated":95110938624,"data_stored":95110938624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047122,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.50900000000000001,"15min":0.50800000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.40000000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.74399999999999999,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.55800000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.66100000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.38200000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.629}]},{"osd":3,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.54400000000000004,"15min":0.55200000000000005},"min":{"1min":0.34200000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71399999999999997,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.75,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.72499999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.49099999999999999,"15min":0.48799999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55100000000000005,"15min":0.55200000000000005},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.85899999999999999,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.68500000000000005}]},{"osd":5,"last update":"Sun Sep 15 03:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.32800000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.78200000000000003,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.76800000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.76500000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.872,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.70799999999999996}]},{"osd":8,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.1899999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.72699999999999998,"15min":0.73299999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.996,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.69899999999999995}]},{"osd":9,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.76900000000000002,"15min":0.77500000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.747,"5min":0.79000000000000004,"15min":0.79700000000000004},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0860000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.86399999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:47:15 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.79800000000000004}]},{"osd":11,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.80900000000000005,"15min":0.81200000000000006},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.145,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.879}]}]},{"osd":6,"up_from":17,"seq":73014444065,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797644,"kb_used_data":92881972,"kb_used_omap":3996,"kb_used_meta":0,"kb_avail":92881972,"statfs":{"total":95927926784,"available":95111139328,"internally_reserved":0,"allocated":95111139328,"data_stored":95111139328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4092779,"internal_metadata":0},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57299999999999995,"15min":0.57499999999999996},"min":{"1min":0.39000000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.84099999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.39000000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72499999999999998,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.58599999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:47:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.56299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67000000000000004,"15min":0.67400000000000004},"min":{"1min":0.45200000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78900000000000003,"5min":1.145,"15min":1.145},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.64100000000000001,"15min":0.64600000000000002},"min":{"1min":0.40699999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80900000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.60399999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.35799999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.75700000000000001,"5min":1.127,"15min":1.127},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.755,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.56699999999999995}]},{"osd":4,"last update":"Sun Sep 15 03:48:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.495,"15min":0.48499999999999999},"min":{"1min":0.32600000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.97099999999999997,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.436},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.51300000000000001,"15min":0.50600000000000001},"min":{"1min":0.32700000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.40600000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.56100000000000005,"15min":0.55400000000000005},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.42499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81100000000000005,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.67900000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.59599999999999997,"15min":0.58799999999999997},"min":{"1min":0.40799999999999997,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85699999999999998,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.38,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82899999999999996,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.46899999999999997}]},{"osd":8,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.80500000000000005,"15min":0.80700000000000005},"min":{"1min":0.57599999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.133,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.78200000000000003,"15min":0.78000000000000003},"min":{"1min":0.57099999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.1240000000000001,"5min":1.242,"15min":1.242},"last":0.69799999999999995}]},{"osd":9,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.79500000000000004,"15min":0.79000000000000004},"min":{"1min":0.59199999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0940000000000001,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.79200000000000004,"15min":0.78800000000000003},"min":{"1min":0.60499999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.0620000000000001,"5min":1.21,"15min":1.21},"last":0.75}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":11,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.82899999999999996,"15min":0.82499999999999996},"min":{"1min":0.58799999999999997,"5min":0.505,"15min":0.505},"max":{"1min":1.482,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.79600000000000004,"15min":0.79400000000000004},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.0569999999999999,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.66200000000000003}]}]},{"osd":5,"up_from":17,"seq":73014444065,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797856,"kb_used_data":92881776,"kb_used_omap":1915,"kb_used_meta":0,"kb_avail":92881760,"statfs":{"total":95927926784,"available":95110922240,"internally_reserved":16384,"allocated":95110938624,"data_stored":95110938624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1961152,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.36899999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.41099999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":91.076999999999998,"15min":106.17},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.93000000000000005,"5min":2037.425,"15min":2037.425},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":91.123000000000005,"15min":106.209},"min":{"1min":0.47999999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.90800000000000003,"5min":2037.3230000000001,"15min":2037.3230000000001},"last":0.432}]},{"osd":2,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60199999999999998,"15min":0.60699999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60599999999999998,"15min":0.60999999999999999},"min":{"1min":0.40699999999999997,"5min":0.379,"15min":0.379},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.38500000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:48:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.53200000000000003}]},{"osd":4,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47899999999999998,"15min":0.47799999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.53100000000000003,"15min":0.53600000000000003},"min":{"1min":0.32100000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.751,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.31}]},{"osd":6,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55400000000000005,"15min":0.55200000000000005},"min":{"1min":0.33400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.38300000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.36899999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59999999999999998,"15min":0.60599999999999998},"min":{"1min":0.443,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.95299999999999996,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.60699999999999998,"15min":0.61599999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.80900000000000005,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.64300000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.73499999999999999,"15min":0.74199999999999999},"min":{"1min":0.54000000000000004,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.127,"5min":1.218,"15min":1.218},"last":0.623},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.75,"15min":0.754},"min":{"1min":0.57399999999999995,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.94699999999999995,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.55300000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.77100000000000002,"15min":0.78100000000000003},"min":{"1min":0.55500000000000005,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.88600000000000001,"5min":1.177,"15min":1.177},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.58699999999999997,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.57399999999999995}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.83399999999999996}]},{"osd":11,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.77800000000000002},"min":{"1min":0.56799999999999995,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.96499999999999997,"5min":1.155,"15min":1.155},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.78900000000000003,"15min":0.79800000000000004},"min":{"1min":0.55900000000000005,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.90800000000000003,"5min":1.194,"15min":1.194},"last":0.69199999999999995}]}]},{"osd":4,"up_from":17,"seq":73014444065,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797768,"kb_used_data":92881848,"kb_used_omap":5864,"kb_used_meta":0,"kb_avail":92881848,"statfs":{"total":95927926784,"available":95111012352,"internally_reserved":0,"allocated":95111012352,"data_stored":95111012352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6005006,"internal_metadata":0},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.73099999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.61799999999999999,"15min":0.623},"min":{"1min":0.39300000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88100000000000001,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.59399999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":67.558000000000007,"15min":78.728999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.83099999999999996,"5min":1758.9780000000001,"15min":1758.9780000000001},"last":0.503},{"interface":"front","average":{"1min":0.55600000000000005,"5min":67.602000000000004,"15min":78.775999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84799999999999998,"5min":1758.671,"15min":1758.671},"last":0.76300000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.65100000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.56599999999999995,"15min":0.57799999999999996},"min":{"1min":0.35899999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.73399999999999999,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.497,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75800000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.66800000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.51800000000000002,"15min":0.53200000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69999999999999996,"5min":0.996,"15min":0.996},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.498,"15min":0.50600000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72599999999999998,"5min":0.747,"15min":0.747},"last":0.65300000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:48:29 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.84799999999999998}]},{"osd":7,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66000000000000003,"15min":0.67200000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.77800000000000002,"5min":1.113,"15min":1.113},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.64000000000000001,"15min":0.66000000000000003},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.81000000000000005,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.626}]},{"osd":8,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.745,"15min":0.76500000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.99199999999999999,"5min":1.532,"15min":1.532},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.78300000000000003,"15min":0.79900000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.96899999999999997,"5min":1.363,"15min":1.363},"last":0.81399999999999995}]},{"osd":9,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.75800000000000001,"15min":0.77600000000000002},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.86799999999999999,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.751,"15min":0.77400000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.85499999999999998,"5min":1.444,"15min":1.444},"last":0.86399999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:47:19 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.88800000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.84599999999999997,"15min":0.86899999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0069999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.79500000000000004,"15min":0.81499999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.91800000000000004,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.78400000000000003}]}]},{"osd":3,"up_from":17,"seq":73014444065,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797768,"kb_used_data":92881848,"kb_used_omap":4767,"kb_used_meta":0,"kb_avail":92881848,"statfs":{"total":95927926784,"available":95111012352,"internally_reserved":0,"allocated":95111012352,"data_stored":95111012352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4882368,"internal_metadata":0},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.373,"15min":0.36699999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.441}]},{"osd":1,"last update":"Sun Sep 15 03:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":28.617000000000001,"15min":33.271999999999998},"min":{"1min":0.27600000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":7.0350000000000001,"5min":564.05100000000004,"15min":564.05100000000004},"last":0.313},{"interface":"front","average":{"1min":0.55700000000000005,"5min":28.652999999999999,"15min":33.335000000000001},"min":{"1min":0.33500000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.448,"5min":564.57000000000005,"15min":564.57000000000005},"last":0.46999999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.45000000000000001,"15min":0.42199999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.1799999999999997,"5min":5.1799999999999997,"15min":5.1799999999999997},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.45700000000000002,"15min":0.44500000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.42699999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.57499999999999996,"15min":0.54900000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.57299999999999995,"15min":0.56499999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.3639999999999999,"5min":2.3639999999999999,"15min":2.3639999999999999},"last":0.50700000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":2.3940000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":5.2830000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.57499999999999996,"15min":0.56499999999999995},"min":{"1min":0.38200000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.65500000000000003,"15min":0.629},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.2489999999999997,"5min":5.2489999999999997,"15min":5.2489999999999997},"last":0.48299999999999998}]},{"osd":7,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.827,"5min":1.827,"15min":1.827},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.63,"15min":0.59999999999999998},"min":{"1min":0.42899999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":5.3090000000000002,"5min":5.3090000000000002,"15min":5.3090000000000002},"last":0.499}]},{"osd":8,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.68700000000000006,"15min":0.67100000000000004},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.66300000000000003,"15min":0.64900000000000002},"min":{"1min":0.54000000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.8,"5min":1.8,"15min":1.8},"last":0.59199999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.71899999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.495,"15min":0.495},"max":{"1min":5.3399999999999999,"5min":5.3399999999999999,"15min":5.3399999999999999},"last":0.621},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.71999999999999997,"15min":0.70799999999999996},"min":{"1min":0.52200000000000002,"5min":0.498,"15min":0.498},"max":{"1min":2.4239999999999999,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":0.72699999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.70499999999999996,"15min":0.69599999999999995},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.859,"5min":1.859,"15min":1.859},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.73299999999999998,"15min":0.72399999999999998},"min":{"1min":0.51100000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.5430000000000001,"5min":2.5430000000000001,"15min":2.5430000000000001},"last":0.65300000000000002}]}]},{"osd":2,"up_from":17,"seq":73014444065,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797640,"kb_used_data":92881976,"kb_used_omap":5376,"kb_used_meta":0,"kb_avail":92881976,"statfs":{"total":95927926784,"available":95111143424,"internally_reserved":0,"allocated":95111143424,"data_stored":95111143424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5505295,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.36299999999999999,"15min":0.36599999999999999},"min":{"1min":0.22900000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.56399999999999995,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.53500000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":27.634,"15min":32.183},"min":{"1min":0.248,"5min":0.247,"15min":0.247},"max":{"1min":0.56499999999999995,"5min":647.92399999999998,"15min":647.92399999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":27.661000000000001,"15min":32.203000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.56599999999999995,"5min":648.02700000000004,"15min":648.02700000000004},"last":0.52400000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.65100000000000002,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.505},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38500000000000001,"15min":0.38900000000000001},"min":{"1min":0.27500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.51300000000000001,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.44800000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.60699999999999998}]},{"osd":5,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.58799999999999997,"15min":0.58999999999999997},"min":{"1min":0.36499999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.95799999999999996,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.748},{"interface":"front","average":{"1min":0.624,"5min":0.55000000000000004,"15min":0.53800000000000003},"min":{"1min":0.47799999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.84699999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.71999999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59499999999999997,"15min":0.58899999999999997},"min":{"1min":0.42199999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.84699999999999998,"5min":1.321,"15min":1.321},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.59099999999999997,"15min":0.58199999999999996},"min":{"1min":0.46700000000000003,"5min":0.374,"15min":0.374},"max":{"1min":0.90800000000000003,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.78100000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59799999999999998,"15min":0.59099999999999997},"min":{"1min":0.40500000000000003,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93300000000000005,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.48399999999999999,"5min":0.44,"15min":0.44},"max":{"1min":0.878,"5min":1.363,"15min":1.363},"last":0.81399999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":1.204,"15min":1.296},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90100000000000002,"5min":14.721,"15min":14.721},"last":0.875},{"interface":"front","average":{"1min":0.69199999999999995,"5min":1.2350000000000001,"15min":1.325},"min":{"1min":0.50700000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.873,"5min":14.970000000000001,"15min":14.970000000000001},"last":0.92600000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":1.2410000000000001,"15min":1.331},"min":{"1min":0.54800000000000004,"5min":0.502,"15min":0.502},"max":{"1min":0.875,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":1.244,"15min":1.3380000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.90900000000000003,"5min":14.888,"15min":14.888},"last":0.88700000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.2529999999999999,"15min":1.3440000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.006,"5min":14.654999999999999,"15min":14.654999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.69799999999999995,"5min":1.266,"15min":1.361},"min":{"1min":0.48999999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.89400000000000002,"5min":15.096,"15min":15.096},"last":0.97899999999999998}]}]},{"osd":1,"up_from":17,"seq":73014444065,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797768,"kb_used_data":92881848,"kb_used_omap":6208,"kb_used_meta":0,"kb_avail":92881848,"statfs":{"total":95927926784,"available":95111012352,"internally_reserved":0,"allocated":95111012352,"data_stored":95111012352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6357522,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.40300000000000002,"15min":0.41299999999999998},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.53200000000000003,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.38200000000000001,"15min":0.38600000000000001},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.47799999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.51500000000000001,"15min":0.53300000000000003},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.65300000000000002,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.27200000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.34499999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.52000000000000002,"15min":0.53400000000000003},"min":{"1min":0.33400000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69099999999999995,"5min":1.907,"15min":1.907},"last":0.438},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.51000000000000001,"15min":0.52400000000000002},"min":{"1min":0.30099999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.83199999999999996,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.35799999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.33900000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.93000000000000005,"5min":1.482,"15min":1.482},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.68000000000000005,"15min":0.69399999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.79500000000000004,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":0.46700000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.98499999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.71099999999999997,"15min":0.73299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.71099999999999997,"5min":2.7440000000000002,"15min":2.7440000000000002},"last":0.51700000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.64400000000000002,"15min":0.65800000000000003},"min":{"1min":0.38900000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.95599999999999996,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.503},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.71599999999999997,"15min":0.73299999999999998},"min":{"1min":0.45400000000000001,"5min":0.432,"15min":0.432},"max":{"1min":1.083,"5min":3.278,"15min":3.278},"last":0.55600000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.81399999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":96.073999999999998,"15min":111.98399999999999},"min":{"1min":0.438,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.0649999999999999,"5min":2861.7190000000001,"15min":2861.7190000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.77700000000000002,"15min":0.80200000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.84799999999999998,"5min":1.946,"15min":1.946},"last":0.70999999999999996}]},{"osd":9,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":96.081000000000003,"15min":111.985},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.44},"max":{"1min":1.024,"5min":2861.6010000000001,"15min":2861.6010000000001},"last":0.626},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.70399999999999996,"15min":0.71399999999999997},"min":{"1min":0.44900000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.006,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.79100000000000004}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":119.96599999999999,"5min":119.96599999999999,"15min":119.96599999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2861.5639999999999,"5min":2861.5639999999999,"15min":2861.5639999999999},"last":0.76200000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.81899999999999995}]}]},{"osd":0,"up_from":17,"seq":73014444065,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798348,"kb_used_data":92881268,"kb_used_omap":4435,"kb_used_meta":0,"kb_avail":92881268,"statfs":{"total":95927926784,"available":95110418432,"internally_reserved":0,"allocated":95110418432,"data_stored":95110418432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4542393,"internal_metadata":0},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.42299999999999999,"15min":0.433},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.53300000000000003,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.44700000000000001,"15min":0.45300000000000001},"min":{"1min":0.29899999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.60399999999999998,"5min":1.155,"15min":1.155},"last":0.33800000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.804,"5min":1.804,"15min":1.804},"last":0.379}]},{"osd":3,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.45900000000000002,"15min":0.47199999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.60399999999999998,"5min":1.52,"15min":1.52},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.33300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.65000000000000002,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.37}]},{"osd":4,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.72899999999999998}]},{"osd":5,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.60599999999999998,"15min":0.61599999999999999},"min":{"1min":0.41199999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.65600000000000003,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.68100000000000005,"15min":0.70099999999999996},"min":{"1min":0.41999999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.871,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.64300000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.63800000000000001,"15min":0.65900000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.72499999999999998,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.68400000000000005,"15min":0.70799999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81999999999999995,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.57199999999999995}]},{"osd":7,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.67500000000000004,"15min":0.69699999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93400000000000005,"5min":1.359,"15min":1.359},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.74399999999999999,"5min":1.216,"15min":1.216},"last":0.53600000000000003}]},{"osd":8,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.72899999999999998,"15min":0.74199999999999999},"min":{"1min":0.46000000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.93600000000000005,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.73799999999999999,"15min":0.75800000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.86499999999999999,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.77200000000000002}]},{"osd":9,"last update":"Sun Sep 15 03:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.745},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.68400000000000005}]},{"osd":10,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.627}]},{"osd":11,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.70899999999999996,"15min":0.71599999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.83899999999999997,"5min":1.603,"15min":1.603},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.78700000000000003,"15min":0.80400000000000005},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89200000000000002,"5min":1.494,"15min":1.494},"last":0.80600000000000005}]}]}],"pool_statfs":[]}} 2024-09-15T03:48:49.199 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-15T03:47:52.292926+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=47, tm_sec=52, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:48:49.199 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-15T03:47:42.996957+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=47, tm_sec=42, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:48:49.199 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-15T03:48:04.781025+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=48, tm_sec=4, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:48:49.199 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-15T03:47:39.930628+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=47, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:48:49.199 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-15T03:48:41.258243+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:48:49.200 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-15T03:47:27.297083+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=47, tm_sec=27, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:48:49.200 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-15T03:47:42.675882+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=47, tm_sec=42, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:48:49.200 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-15T03:49:09.201 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:49:09.489 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-15T03:49:09.490 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-15T03:49:09.501 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2024-09-15T03:49:08.176836+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":125,"num_write_kb":2152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":2,"num_per_pool_omap_osds":0,"kb":1124155392,"kb_used":9577236,"kb_used_data":1114578280,"kb_used_omap":58304,"kb_used_meta":0,"kb_avail":1114578156,"statfs":{"total":1151135121408,"available":1141328031744,"internally_reserved":126976,"allocated":1141328158720,"data_stored":1141328158720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59703890,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.858861"},"pg_stats":[{"pgid":"1.0","version":"150'94","reported_seq":280,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-15T03:49:01.500042+0000","last_change":"2024-09-15T03:48:54.587234+0000","last_active":"2024-09-15T03:49:01.500042+0000","last_peered":"2024-09-15T03:49:01.500042+0000","last_clean":"2024-09-15T03:49:01.500042+0000","last_became_active":"2024-09-15T03:46:18.011130+0000","last_became_peered":"2024-09-15T03:46:18.011130+0000","last_unstale":"2024-09-15T03:49:01.500042+0000","last_undegraded":"2024-09-15T03:49:01.500042+0000","last_fullsized":"2024-09-15T03:49:01.500042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'94","last_scrub_stamp":"2024-09-15T03:48:54.587175+0000","last_deep_scrub":"150'94","last_deep_scrub_stamp":"2024-09-15T03:48:54.587175+0000","last_clean_scrub_stamp":"2024-09-15T03:48:54.587175+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:18.594545+0000","scrub_duration":0.0038480720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"151'2","reported_seq":163,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-15T03:49:01.500211+0000","last_change":"2024-09-15T03:48:55.803287+0000","last_active":"2024-09-15T03:49:01.500211+0000","last_peered":"2024-09-15T03:49:01.500211+0000","last_clean":"2024-09-15T03:49:01.500211+0000","last_became_active":"2024-09-15T03:46:24.038633+0000","last_became_peered":"2024-09-15T03:46:24.038633+0000","last_unstale":"2024-09-15T03:49:01.500211+0000","last_undegraded":"2024-09-15T03:49:01.500211+0000","last_fullsized":"2024-09-15T03:49:01.500211+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'2","last_scrub_stamp":"2024-09-15T03:48:55.803230+0000","last_deep_scrub":"151'2","last_deep_scrub_stamp":"2024-09-15T03:48:55.803230+0000","last_clean_scrub_stamp":"2024-09-15T03:48:55.803230+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:08.014227+0000","scrub_duration":0.0021932330000000002,"objects_trimmed":0,"snaptrim_duration":0.00030178200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":167,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-15T03:49:01.500050+0000","last_change":"2024-09-15T03:48:56.573058+0000","last_active":"2024-09-15T03:49:01.500050+0000","last_peered":"2024-09-15T03:49:01.500050+0000","last_clean":"2024-09-15T03:49:01.500050+0000","last_became_active":"2024-09-15T03:46:24.063106+0000","last_became_peered":"2024-09-15T03:46:24.063106+0000","last_unstale":"2024-09-15T03:49:01.500050+0000","last_undegraded":"2024-09-15T03:49:01.500050+0000","last_fullsized":"2024-09-15T03:49:01.500050+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:48:56.572999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:48:56.572999+0000","last_clean_scrub_stamp":"2024-09-15T03:48:56.572999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:00.873295+0000","scrub_duration":0.0015825430000000001,"objects_trimmed":0,"snaptrim_duration":0.000393075,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":165,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-15T03:49:00.317552+0000","last_change":"2024-09-15T03:48:48.857062+0000","last_active":"2024-09-15T03:49:00.317552+0000","last_peered":"2024-09-15T03:49:00.317552+0000","last_clean":"2024-09-15T03:49:00.317552+0000","last_became_active":"2024-09-15T03:46:24.037471+0000","last_became_peered":"2024-09-15T03:46:24.037471+0000","last_unstale":"2024-09-15T03:49:00.317552+0000","last_undegraded":"2024-09-15T03:49:00.317552+0000","last_fullsized":"2024-09-15T03:49:00.317552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:48:48.857000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:48:48.857000+0000","last_clean_scrub_stamp":"2024-09-15T03:48:48.857000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:05.983068+0000","scrub_duration":0.001492169,"objects_trimmed":0,"snaptrim_duration":0.00026490299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"151'3","reported_seq":204,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-15T03:49:01.500047+0000","last_change":"2024-09-15T03:48:55.581746+0000","last_active":"2024-09-15T03:49:01.500047+0000","last_peered":"2024-09-15T03:49:01.500047+0000","last_clean":"2024-09-15T03:49:01.500047+0000","last_became_active":"2024-09-15T03:47:36.848076+0000","last_became_peered":"2024-09-15T03:47:36.848076+0000","last_unstale":"2024-09-15T03:49:01.500047+0000","last_undegraded":"2024-09-15T03:49:01.500047+0000","last_fullsized":"2024-09-15T03:49:01.500047+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'3","last_scrub_stamp":"2024-09-15T03:48:55.581654+0000","last_deep_scrub":"151'3","last_deep_scrub_stamp":"2024-09-15T03:48:55.581654+0000","last_clean_scrub_stamp":"2024-09-15T03:48:55.581654+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:15.758107+0000","scrub_duration":0.0031295659999999999,"objects_trimmed":0,"snaptrim_duration":0.000944677,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":183,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-15T03:49:00.317582+0000","last_change":"2024-09-15T03:48:48.608364+0000","last_active":"2024-09-15T03:49:00.317582+0000","last_peered":"2024-09-15T03:49:00.317582+0000","last_clean":"2024-09-15T03:49:00.317582+0000","last_became_active":"2024-09-15T03:47:33.998802+0000","last_became_peered":"2024-09-15T03:47:33.998802+0000","last_unstale":"2024-09-15T03:49:00.317582+0000","last_undegraded":"2024-09-15T03:49:00.317582+0000","last_fullsized":"2024-09-15T03:49:00.317582+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:48:48.608299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:48:48.608299+0000","last_clean_scrub_stamp":"2024-09-15T03:48:48.608299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:14.819550+0000","scrub_duration":0.0015397169999999999,"objects_trimmed":0,"snaptrim_duration":0.00034123999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":169,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-15T03:49:01.828555+0000","last_change":"2024-09-15T03:48:46.236553+0000","last_active":"2024-09-15T03:49:01.828555+0000","last_peered":"2024-09-15T03:49:01.828555+0000","last_clean":"2024-09-15T03:49:01.828555+0000","last_became_active":"2024-09-15T03:46:24.038945+0000","last_became_peered":"2024-09-15T03:46:24.038945+0000","last_unstale":"2024-09-15T03:49:01.828555+0000","last_undegraded":"2024-09-15T03:49:01.828555+0000","last_fullsized":"2024-09-15T03:49:01.828555+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:48:46.236497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:48:46.236497+0000","last_clean_scrub_stamp":"2024-09-15T03:48:46.236497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:02.778091+0000","scrub_duration":0.001444545,"objects_trimmed":0,"snaptrim_duration":0.00032787499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":159,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-15T03:49:01.090136+0000","last_change":"2024-09-15T03:48:48.180170+0000","last_active":"2024-09-15T03:49:01.090136+0000","last_peered":"2024-09-15T03:49:01.090136+0000","last_clean":"2024-09-15T03:49:01.090136+0000","last_became_active":"2024-09-15T03:46:24.063380+0000","last_became_peered":"2024-09-15T03:46:24.063380+0000","last_unstale":"2024-09-15T03:49:01.090136+0000","last_undegraded":"2024-09-15T03:49:01.090136+0000","last_fullsized":"2024-09-15T03:49:01.090136+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:48:48.180112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:48:48.180112+0000","last_clean_scrub_stamp":"2024-09-15T03:48:48.180112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:59.997912+0000","scrub_duration":0.001674434,"objects_trimmed":0,"snaptrim_duration":0.00034490600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":157,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-15T03:49:01.500055+0000","last_change":"2024-09-15T03:48:57.623242+0000","last_active":"2024-09-15T03:49:01.500055+0000","last_peered":"2024-09-15T03:49:01.500055+0000","last_clean":"2024-09-15T03:49:01.500055+0000","last_became_active":"2024-09-15T03:46:24.063335+0000","last_became_peered":"2024-09-15T03:46:24.063335+0000","last_unstale":"2024-09-15T03:49:01.500055+0000","last_undegraded":"2024-09-15T03:49:01.500055+0000","last_fullsized":"2024-09-15T03:49:01.500055+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:48:57.623183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:48:57.623183+0000","last_clean_scrub_stamp":"2024-09-15T03:48:57.623183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:08.852697+0000","scrub_duration":0.0019310079999999999,"objects_trimmed":0,"snaptrim_duration":0.00065928000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":11,"up_from":17,"seq":73014444069,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797840,"kb_used_data":92881776,"kb_used_omap":5026,"kb_used_meta":0,"kb_avail":92881776,"statfs":{"total":95927926784,"available":95110938624,"internally_reserved":0,"allocated":95110938624,"data_stored":95110938624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5147193,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52700000000000002,"15min":0.53100000000000003},"min":{"1min":0.36699999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.79200000000000004,"5min":1.002,"15min":1.002},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.503,"15min":0.502},"min":{"1min":0.33600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.82199999999999995,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.50700000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":121.575,"5min":121.575,"15min":121.575},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2060.797,"5min":2060.797,"15min":2060.797},"last":0.74199999999999999},{"interface":"front","average":{"1min":121.56,"5min":121.56,"15min":121.56},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2060.7159999999999,"5min":2060.7159999999999,"15min":2060.7159999999999},"last":0.60499999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.53300000000000003,"15min":0.52300000000000002},"min":{"1min":0.35099999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.40999999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.91800000000000004,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.53700000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76100000000000001,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.40699999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.52800000000000002}]},{"osd":4,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.61499999999999999,"15min":0.60699999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.68000000000000005,"15min":0.67500000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.72099999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65500000000000003,"15min":0.65400000000000003},"min":{"1min":0.48099999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.94399999999999995,"5min":1.103,"15min":1.103},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.64600000000000002,"15min":0.63900000000000001},"min":{"1min":0.48199999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.64600000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.65800000000000003,"15min":0.65200000000000002},"min":{"1min":0.50800000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67700000000000005,"15min":0.67500000000000004},"min":{"1min":0.497,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0149999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.58999999999999997}]},{"osd":7,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65000000000000002,"15min":0.64900000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67000000000000004,"15min":0.66800000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.879,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.66000000000000003}]},{"osd":8,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.84499999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.60799999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61699999999999999,"15min":0.60799999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.65200000000000002,"15min":0.63900000000000001},"min":{"1min":0.45300000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.623}]},{"osd":10,"last update":"Sun Sep 15 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.57399999999999995}]}]},{"osd":10,"up_from":105,"seq":450971566097,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798864,"kb_used_data":92880752,"kb_used_omap":3939,"kb_used_meta":0,"kb_avail":92880752,"statfs":{"total":95927926784,"available":95109890048,"internally_reserved":0,"allocated":95109890048,"data_stored":95109890048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4034034,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.57199999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.59499999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.69799999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.76100000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.79900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Sun Sep 15 03:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.73599999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.71199999999999997}]},{"osd":8,"last update":"Sun Sep 15 03:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.434},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.65400000000000003}]},{"osd":9,"last update":"Sun Sep 15 03:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.505},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.61199999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:48:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.623}]}]},{"osd":9,"up_from":17,"seq":73014444069,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797912,"kb_used_data":92881720,"kb_used_omap":5382,"kb_used_meta":0,"kb_avail":92881704,"statfs":{"total":95927926784,"available":95110864896,"internally_reserved":16384,"allocated":95110881280,"data_stored":95110881280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5511646,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.33400000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78200000000000003,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57799999999999996,"15min":0.57899999999999996},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.748,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.57399999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57199999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.82199999999999995,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.754},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.40300000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.75800000000000001,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.74199999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.54800000000000004,"15min":0.55400000000000005},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.71099999999999997,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.625},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.55500000000000005,"15min":0.56499999999999995},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69399999999999995,"5min":1.008,"15min":1.008},"last":0.48099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59499999999999997,"15min":0.58699999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.91300000000000003,"5min":1.03,"15min":1.03},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.64400000000000002,"15min":0.64800000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.83599999999999997,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.60899999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.48799999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.86899999999999999,"5min":1.131,"15min":1.131},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.63800000000000001,"15min":0.64100000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82299999999999995,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.52900000000000003}]},{"osd":6,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63700000000000001,"15min":0.63500000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.81299999999999994,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66800000000000004,"15min":0.67100000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.89300000000000002,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.50700000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:48:37 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.65500000000000003,"15min":0.64600000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.64200000000000002,"15min":0.622},"min":{"1min":0.55000000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.52100000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51500000000000001,"15min":0.51700000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.68000000000000005,"5min":0.748,"15min":0.748},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.36799999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71099999999999997,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.42999999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.45400000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.65400000000000003,"15min":0.66600000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.66900000000000004,"15min":0.68300000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80900000000000005,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.40400000000000003}]}]},{"osd":8,"up_from":17,"seq":73014444070,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799008,"kb_used_data":92880620,"kb_used_omap":5431,"kb_used_meta":0,"kb_avail":92880608,"statfs":{"total":95927926784,"available":95109742592,"internally_reserved":12288,"allocated":95109754880,"data_stored":95109754880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5562288,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.35199999999999998,"5min":0.313,"15min":0.313},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60399999999999998,"15min":0.60799999999999998},"min":{"1min":0.371,"5min":0.318,"15min":0.318},"max":{"1min":0.83199999999999996,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.59999999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":34.012,"15min":39.591000000000001},"min":{"1min":0.38200000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76400000000000001,"5min":963.98400000000004,"15min":963.98400000000004},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":34.064999999999998,"15min":39.652999999999999},"min":{"1min":0.40200000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91400000000000003,"5min":964.79600000000005,"15min":964.79600000000005},"last":0.502}]},{"osd":2,"last update":"Sun Sep 15 03:48:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.44600000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.54800000000000004,"15min":0.53900000000000003},"min":{"1min":0.373,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.58099999999999996,"15min":0.57099999999999995},"min":{"1min":0.44,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.66800000000000004}]},{"osd":4,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.46200000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.82199999999999995,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.67700000000000005,"15min":0.68200000000000005},"min":{"1min":0.48399999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83299999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.68600000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:48:30 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.68600000000000005,"15min":0.68200000000000005},"min":{"1min":0.5,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.93100000000000005,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.66300000000000003,"15min":0.65300000000000002},"min":{"1min":0.56200000000000006,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.65000000000000002}]},{"osd":7,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69399999999999995,"15min":0.69799999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.67500000000000004,"15min":0.67900000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97899999999999998,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.66000000000000003}]},{"osd":9,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.629,"15min":0.63300000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84799999999999998,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.56799999999999995}]},{"osd":10,"last update":"Sun Sep 15 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.51900000000000002}]},{"osd":11,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63400000000000001,"15min":0.64200000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.627,"15min":0.63400000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.629}]}]},{"osd":7,"up_from":17,"seq":73014444069,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797912,"kb_used_data":92881716,"kb_used_omap":5910,"kb_used_meta":0,"kb_avail":92881704,"statfs":{"total":95927926784,"available":95110864896,"internally_reserved":12288,"allocated":95110877184,"data_stored":95110877184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6051979,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.50900000000000001,"15min":0.50800000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.75},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.40000000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.74399999999999999,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.78600000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.66100000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.38200000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.55300000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.54400000000000004,"15min":0.55200000000000005},"min":{"1min":0.34200000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71399999999999997,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.75,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.91700000000000004}]},{"osd":4,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.49099999999999999,"15min":0.48799999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55100000000000005,"15min":0.55200000000000005},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.85899999999999999,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.36199999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.32800000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.78200000000000003,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.66200000000000003}]},{"osd":6,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.76500000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.621},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.872,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.64100000000000001}]},{"osd":8,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.1899999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.72699999999999998,"15min":0.73299999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.996,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.98499999999999999}]},{"osd":9,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.76900000000000002,"15min":0.77500000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.747,"5min":0.79000000000000004,"15min":0.79700000000000004},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0860000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":1.0960000000000001}]},{"osd":10,"last update":"Sun Sep 15 03:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.69599999999999995}]},{"osd":11,"last update":"Sun Sep 15 03:48:17 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.80900000000000005,"15min":0.81200000000000006},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":1.014},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.145,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.879}]}]},{"osd":6,"up_from":17,"seq":73014444069,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797788,"kb_used_data":92881840,"kb_used_omap":4005,"kb_used_meta":0,"kb_avail":92881828,"statfs":{"total":95927926784,"available":95110991872,"internally_reserved":12288,"allocated":95111004160,"data_stored":95111004160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4101632,"internal_metadata":0},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57299999999999995,"15min":0.57499999999999996},"min":{"1min":0.39000000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.84099999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.39000000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72499999999999998,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.79400000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:47:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.56299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67000000000000004,"15min":0.67400000000000004},"min":{"1min":0.45200000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78900000000000003,"5min":1.145,"15min":1.145},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.64100000000000001,"15min":0.64600000000000002},"min":{"1min":0.40699999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80900000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.70799999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.35799999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.75700000000000001,"5min":1.127,"15min":1.127},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.755,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.58399999999999996}]},{"osd":4,"last update":"Sun Sep 15 03:48:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.495,"15min":0.48499999999999999},"min":{"1min":0.32600000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.97099999999999997,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.51300000000000001,"15min":0.50600000000000001},"min":{"1min":0.32700000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.59799999999999998}]},{"osd":5,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.56100000000000005,"15min":0.55400000000000005},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.42499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81100000000000005,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.93500000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.59599999999999997,"15min":0.58799999999999997},"min":{"1min":0.40799999999999997,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85699999999999998,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.38,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82899999999999996,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.78100000000000003}]},{"osd":8,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.80500000000000005,"15min":0.80700000000000005},"min":{"1min":0.57599999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.133,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":1.228},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.78200000000000003,"15min":0.78000000000000003},"min":{"1min":0.57099999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.1240000000000001,"5min":1.242,"15min":1.242},"last":1.0900000000000001}]},{"osd":9,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.79500000000000004,"15min":0.79000000000000004},"min":{"1min":0.59199999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0940000000000001,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":1.1080000000000001},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.79200000000000004,"15min":0.78800000000000003},"min":{"1min":0.60499999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.0620000000000001,"5min":1.21,"15min":1.21},"last":1.0620000000000001}]},{"osd":10,"last update":"Sun Sep 15 03:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":1.0720000000000001},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":1.123}]},{"osd":11,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.82899999999999996,"15min":0.82499999999999996},"min":{"1min":0.58799999999999997,"5min":0.505,"15min":0.505},"max":{"1min":1.482,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":1.012},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.79600000000000004,"15min":0.79400000000000004},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.0569999999999999,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":1.048}]}]},{"osd":5,"up_from":17,"seq":73014444069,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797912,"kb_used_data":92881716,"kb_used_omap":1921,"kb_used_meta":0,"kb_avail":92881704,"statfs":{"total":95927926784,"available":95110864896,"internally_reserved":12288,"allocated":95110877184,"data_stored":95110877184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1967256,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.36899999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.66600000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":91.076999999999998,"15min":106.17},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.93000000000000005,"5min":2037.425,"15min":2037.425},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.60699999999999998,"5min":91.123000000000005,"15min":106.209},"min":{"1min":0.47999999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.90800000000000003,"5min":2037.3230000000001,"15min":2037.3230000000001},"last":0.70399999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60199999999999998,"15min":0.60699999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60599999999999998,"15min":0.60999999999999999},"min":{"1min":0.40699999999999997,"5min":0.379,"15min":0.379},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.78900000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:48:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.67800000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47899999999999998,"15min":0.47799999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.53100000000000003,"15min":0.53600000000000003},"min":{"1min":0.32100000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.751,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.52900000000000003}]},{"osd":6,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55400000000000005,"15min":0.55200000000000005},"min":{"1min":0.33400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.38300000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.441}]},{"osd":7,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.59999999999999998,"15min":0.60599999999999998},"min":{"1min":0.443,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.95299999999999996,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.60699999999999998,"15min":0.61599999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.80900000000000005,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.47199999999999998}]},{"osd":8,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.73499999999999999,"15min":0.74199999999999999},"min":{"1min":0.54000000000000004,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.127,"5min":1.218,"15min":1.218},"last":0.75},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.75,"15min":0.754},"min":{"1min":0.57399999999999995,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.94699999999999995,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.61399999999999999}]},{"osd":9,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.77100000000000002,"15min":0.78100000000000003},"min":{"1min":0.55500000000000005,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.88600000000000001,"5min":1.177,"15min":1.177},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.58699999999999997,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.82599999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.83399999999999996}]},{"osd":11,"last update":"Sun Sep 15 03:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.77800000000000002},"min":{"1min":0.56799999999999995,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.96499999999999997,"5min":1.155,"15min":1.155},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.78900000000000003,"15min":0.79800000000000004},"min":{"1min":0.55900000000000005,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.90800000000000003,"5min":1.194,"15min":1.194},"last":0.63900000000000001}]}]},{"osd":4,"up_from":17,"seq":73014444069,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797912,"kb_used_data":92881716,"kb_used_omap":5870,"kb_used_meta":0,"kb_avail":92881704,"statfs":{"total":95927926784,"available":95110864896,"internally_reserved":12288,"allocated":95110877184,"data_stored":95110877184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6011801,"internal_metadata":0},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.73099999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.61799999999999999,"15min":0.623},"min":{"1min":0.39300000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88100000000000001,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.755}]},{"osd":1,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":67.558000000000007,"15min":78.728999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.83099999999999996,"5min":1758.9780000000001,"15min":1758.9780000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":67.602000000000004,"15min":78.775999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84799999999999998,"5min":1758.671,"15min":1758.671},"last":0.71399999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.65100000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.56599999999999995,"15min":0.57799999999999996},"min":{"1min":0.35899999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.73399999999999999,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75800000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.59899999999999998}]},{"osd":5,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.51800000000000002,"15min":0.53200000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69999999999999996,"5min":0.996,"15min":0.996},"last":0.437},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.498,"15min":0.50600000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72599999999999998,"5min":0.747,"15min":0.747},"last":0.47699999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:48:29 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.69499999999999995}]},{"osd":7,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66000000000000003,"15min":0.67200000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.77800000000000002,"5min":1.113,"15min":1.113},"last":0.627},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.64000000000000001,"15min":0.66000000000000003},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.81000000000000005,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.65400000000000003}]},{"osd":8,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.745,"15min":0.76500000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.99199999999999999,"5min":1.532,"15min":1.532},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.78300000000000003,"15min":0.79900000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.96899999999999997,"5min":1.363,"15min":1.363},"last":0.80300000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.75800000000000001,"15min":0.77600000000000002},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.86799999999999999,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.751,"15min":0.77400000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.85499999999999998,"5min":1.444,"15min":1.444},"last":0.76900000000000002}]},{"osd":10,"last update":"Sun Sep 15 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.85799999999999998}]},{"osd":11,"last update":"Sun Sep 15 03:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.84599999999999997,"15min":0.86899999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0069999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.79500000000000004,"15min":0.81499999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.91800000000000004,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.88}]}]},{"osd":3,"up_from":17,"seq":73014444069,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797912,"kb_used_data":92881716,"kb_used_omap":4773,"kb_used_meta":0,"kb_avail":92881704,"statfs":{"total":95927926784,"available":95110864896,"internally_reserved":12288,"allocated":95110877184,"data_stored":95110877184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4887880,"internal_metadata":0},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.373,"15min":0.36699999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.41899999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":28.617000000000001,"15min":33.271999999999998},"min":{"1min":0.27600000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":7.0350000000000001,"5min":564.05100000000004,"15min":564.05100000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":28.652999999999999,"15min":33.335000000000001},"min":{"1min":0.33500000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.448,"5min":564.57000000000005,"15min":564.57000000000005},"last":0.47599999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.45000000000000001,"15min":0.42199999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.1799999999999997,"5min":5.1799999999999997,"15min":5.1799999999999997},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.45700000000000002,"15min":0.44500000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.44800000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.57499999999999996,"15min":0.54900000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.57299999999999995,"15min":0.56499999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.3639999999999999,"5min":2.3639999999999999,"15min":2.3639999999999999},"last":0.502}]},{"osd":5,"last update":"Sun Sep 15 03:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":2.3940000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":5.2830000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.57499999999999996,"15min":0.56499999999999995},"min":{"1min":0.38200000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.65500000000000003,"15min":0.629},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.2489999999999997,"5min":5.2489999999999997,"15min":5.2489999999999997},"last":0.70999999999999996}]},{"osd":7,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.827,"5min":1.827,"15min":1.827},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.63,"15min":0.59999999999999998},"min":{"1min":0.42899999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":5.3090000000000002,"5min":5.3090000000000002,"15min":5.3090000000000002},"last":0.63800000000000001}]},{"osd":8,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.68700000000000006,"15min":0.67100000000000004},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.66300000000000003,"15min":0.64900000000000002},"min":{"1min":0.54000000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.8,"5min":1.8,"15min":1.8},"last":1.044}]},{"osd":9,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.71899999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.495,"15min":0.495},"max":{"1min":5.3399999999999999,"5min":5.3399999999999999,"15min":5.3399999999999999},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.71999999999999997,"15min":0.70799999999999996},"min":{"1min":0.52200000000000002,"5min":0.498,"15min":0.498},"max":{"1min":2.4239999999999999,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":0.99399999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":2.113,"5min":2.113,"15min":2.113},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":5.6150000000000002,"5min":5.6150000000000002,"15min":5.6150000000000002},"last":1.0269999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.70499999999999996,"15min":0.69599999999999995},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.859,"5min":1.859,"15min":1.859},"last":0.875},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.73299999999999998,"15min":0.72399999999999998},"min":{"1min":0.51100000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.5430000000000001,"5min":2.5430000000000001,"15min":2.5430000000000001},"last":0.96299999999999997}]}]},{"osd":2,"up_from":17,"seq":73014444069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797784,"kb_used_data":92881844,"kb_used_omap":5381,"kb_used_meta":0,"kb_avail":92881832,"statfs":{"total":95927926784,"available":95110995968,"internally_reserved":12288,"allocated":95111008256,"data_stored":95111008256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5510300,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.36299999999999999,"15min":0.36599999999999999},"min":{"1min":0.22900000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.56399999999999995,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.65300000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":27.634,"15min":32.183},"min":{"1min":0.248,"5min":0.247,"15min":0.247},"max":{"1min":0.56499999999999995,"5min":647.92399999999998,"15min":647.92399999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.40500000000000003,"5min":27.661000000000001,"15min":32.203000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.56599999999999995,"5min":648.02700000000004,"15min":648.02700000000004},"last":0.60799999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.65100000000000002,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38500000000000001,"15min":0.38900000000000001},"min":{"1min":0.27500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.51300000000000001,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.53800000000000003}]},{"osd":4,"last update":"Sun Sep 15 03:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.755},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.89300000000000002}]},{"osd":5,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.58799999999999997,"15min":0.58999999999999997},"min":{"1min":0.36499999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.95799999999999996,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.624,"5min":0.55000000000000004,"15min":0.53800000000000003},"min":{"1min":0.47799999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.84699999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.86499999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59499999999999997,"15min":0.58899999999999997},"min":{"1min":0.42199999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.84699999999999998,"5min":1.321,"15min":1.321},"last":0.746},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.59099999999999997,"15min":0.58199999999999996},"min":{"1min":0.46700000000000003,"5min":0.374,"15min":0.374},"max":{"1min":0.90800000000000003,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.90400000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59799999999999998,"15min":0.59099999999999997},"min":{"1min":0.40500000000000003,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93300000000000005,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.48399999999999999,"5min":0.44,"15min":0.44},"max":{"1min":0.878,"5min":1.363,"15min":1.363},"last":0.91800000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":1.204,"15min":1.296},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90100000000000002,"5min":14.721,"15min":14.721},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.69199999999999995,"5min":1.2350000000000001,"15min":1.325},"min":{"1min":0.50700000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.873,"5min":14.970000000000001,"15min":14.970000000000001},"last":0.79700000000000004}]},{"osd":9,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":1.2410000000000001,"15min":1.331},"min":{"1min":0.54800000000000004,"5min":0.502,"15min":0.502},"max":{"1min":0.875,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":1.244,"15min":1.3380000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.90900000000000003,"5min":14.888,"15min":14.888},"last":0.82099999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:48:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.2529999999999999,"15min":1.3440000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.006,"5min":14.654999999999999,"15min":14.654999999999999},"last":0.88},{"interface":"front","average":{"1min":0.69799999999999995,"5min":1.266,"15min":1.361},"min":{"1min":0.48999999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.89400000000000002,"5min":15.096,"15min":15.096},"last":0.94099999999999995}]}]},{"osd":1,"up_from":17,"seq":73014444069,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797900,"kb_used_data":92881728,"kb_used_omap":6214,"kb_used_meta":0,"kb_avail":92881716,"statfs":{"total":95927926784,"available":95110877184,"internally_reserved":12288,"allocated":95110889472,"data_stored":95110889472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6363599,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.40300000000000002,"15min":0.41299999999999998},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.53200000000000003,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.38200000000000001,"15min":0.38600000000000001},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.35999999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.51500000000000001,"15min":0.53300000000000003},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.65300000000000002,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.27200000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.47699999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.52000000000000002,"15min":0.53400000000000003},"min":{"1min":0.33400000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69099999999999995,"5min":1.907,"15min":1.907},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.51000000000000001,"15min":0.52400000000000002},"min":{"1min":0.30099999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.83199999999999996,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.437}]},{"osd":4,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.33900000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.93000000000000005,"5min":1.482,"15min":1.482},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.68000000000000005,"15min":0.69399999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.79500000000000004,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":0.53100000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.98499999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.71099999999999997,"15min":0.73299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.71099999999999997,"5min":2.7440000000000002,"15min":2.7440000000000002},"last":0.56799999999999995}]},{"osd":6,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.64400000000000002,"15min":0.65800000000000003},"min":{"1min":0.38900000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.95599999999999996,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.754},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.71599999999999997,"15min":0.73299999999999998},"min":{"1min":0.45400000000000001,"5min":0.432,"15min":0.432},"max":{"1min":1.083,"5min":3.278,"15min":3.278},"last":0.624}]},{"osd":7,"last update":"Sun Sep 15 03:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.81399999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":96.073999999999998,"15min":111.98399999999999},"min":{"1min":0.438,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.0649999999999999,"5min":2861.7190000000001,"15min":2861.7190000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.77700000000000002,"15min":0.80200000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.84799999999999998,"5min":1.946,"15min":1.946},"last":0.60699999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:48:19 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":96.081000000000003,"15min":111.985},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.44},"max":{"1min":1.024,"5min":2861.6010000000001,"15min":2861.6010000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.70399999999999996,"15min":0.71399999999999997},"min":{"1min":0.44900000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.006,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.64100000000000001}]},{"osd":10,"last update":"Sun Sep 15 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.53700000000000003}]},{"osd":11,"last update":"Sun Sep 15 03:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.59399999999999997}]}]},{"osd":0,"up_from":17,"seq":73014444069,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798492,"kb_used_data":92881136,"kb_used_omap":4447,"kb_used_meta":0,"kb_avail":92881124,"statfs":{"total":95927926784,"available":95110270976,"internally_reserved":12288,"allocated":95110283264,"data_stored":95110283264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4554282,"internal_metadata":0},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.42299999999999999,"15min":0.433},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.53300000000000003,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.318},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.44700000000000001,"15min":0.45300000000000001},"min":{"1min":0.29899999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.60399999999999998,"5min":1.155,"15min":1.155},"last":0.42299999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:47:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.804,"5min":1.804,"15min":1.804},"last":0.379}]},{"osd":3,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.45900000000000002,"15min":0.47199999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.60399999999999998,"5min":1.52,"15min":1.52},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.33300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.65000000000000002,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.55200000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.44500000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.60599999999999998,"15min":0.61599999999999999},"min":{"1min":0.41199999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.65600000000000003,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.68100000000000005,"15min":0.70099999999999996},"min":{"1min":0.41999999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.871,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.72099999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.63800000000000001,"15min":0.65900000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.72499999999999998,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.625},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.68400000000000005,"15min":0.70799999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81999999999999995,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.50900000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.67500000000000004,"15min":0.69699999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93400000000000005,"5min":1.359,"15min":1.359},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.74399999999999999,"5min":1.216,"15min":1.216},"last":0.60599999999999998}]},{"osd":8,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.72899999999999998,"15min":0.74199999999999999},"min":{"1min":0.46000000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.93600000000000005,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.73799999999999999,"15min":0.75800000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.86499999999999999,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.64300000000000002}]},{"osd":9,"last update":"Sun Sep 15 03:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.70399999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.76600000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.70899999999999996,"15min":0.71599999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.83899999999999997,"5min":1.603,"15min":1.603},"last":0.746},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.78700000000000003,"15min":0.80400000000000005},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89200000000000002,"5min":1.494,"15min":1.494},"last":0.79900000000000004}]}]}],"pool_statfs":[]}} 2024-09-15T03:49:09.509 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-15T03:49:09.799 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-15T03:49:09.799 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-15T03:49:09.800 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-15T03:49:09.800 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:49:09.813 INFO:tasks.ceph.osd.0:Stopped 2024-09-15T03:49:09.813 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-15T03:49:09.813 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:49:09.868 INFO:tasks.ceph.osd.1:Stopped 2024-09-15T03:49:09.869 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-15T03:49:09.869 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:49:09.924 INFO:tasks.ceph.osd.2:Stopped 2024-09-15T03:49:09.925 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-15T03:49:09.926 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:49:09.979 INFO:tasks.ceph.osd.3:Stopped 2024-09-15T03:49:09.980 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-15T03:49:09.980 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:49:09.998 INFO:tasks.ceph.osd.4:Stopped 2024-09-15T03:49:09.998 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-15T03:49:09.998 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:49:10.051 INFO:tasks.ceph.osd.5:Stopped 2024-09-15T03:49:10.052 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-15T03:49:10.052 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:49:10.107 INFO:tasks.ceph.osd.6:Stopped 2024-09-15T03:49:10.108 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-15T03:49:10.108 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:49:10.163 INFO:tasks.ceph.osd.7:Stopped 2024-09-15T03:49:10.164 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-09-15T03:49:10.164 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:49:10.181 INFO:tasks.ceph.osd.8:Stopped 2024-09-15T03:49:10.182 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-09-15T03:49:10.182 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:49:10.238 INFO:tasks.ceph.osd.9:Stopped 2024-09-15T03:49:10.239 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-09-15T03:49:10.239 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:49:10.294 INFO:tasks.ceph.osd.10:Stopped 2024-09-15T03:49:10.294 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-09-15T03:49:10.294 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:49:10.351 INFO:tasks.ceph.osd.11:Stopped 2024-09-15T03:49:10.351 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-15T03:49:10.352 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-15T03:49:10.352 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:49:10.380 INFO:tasks.ceph.mgr.x:Stopped 2024-09-15T03:49:10.380 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-15T03:49:10.380 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T03:49:10.380 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:49:10.392 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T03:49:10.392 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T03:49:10.392 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:49:10.432 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T03:49:10.433 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T03:49:10.433 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:49:10.446 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T03:49:10.446 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-15T03:49:10.447 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-09-15T03:49:10.479 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-15T03:49:10.479 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-15T03:49:10.632 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-15T03:49:10.632 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-15T03:49:10.731 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-15T03:49:10.731 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-15T03:49:10.797 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-15T03:49:10.797 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-15T03:49:10.972 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-15T03:49:10.972 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-15T03:49:11.108 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-15T03:49:11.108 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-15T03:49:11.181 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-15T03:49:11.181 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-15T03:49:11.238 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-15T03:49:11.238 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-15T03:49:11.288 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-15T03:49:11.289 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2024-09-15T03:49:11.395 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-15T03:49:11.395 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2024-09-15T03:49:11.469 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-15T03:49:11.469 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2024-09-15T03:49:11.515 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-15T03:49:11.516 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2024-09-15T03:49:11.574 INFO:tasks.ceph:Archiving mon data... 2024-09-15T03:49:11.575 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905308/data/mon.a.tgz 2024-09-15T03:49:11.575 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-09-15T03:49:11.595 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.Qi7rB5NtGG 2024-09-15T03:49:11.595 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Qi7rB5NtGG 2024-09-15T03:49:11.753 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.Qi7rB5NtGG 2024-09-15T03:49:11.841 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.Qi7rB5NtGG is 344KB 2024-09-15T03:49:11.912 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.Qi7rB5NtGG 2024-09-15T03:49:11.929 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905308/data/mon.b.tgz 2024-09-15T03:49:11.929 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-09-15T03:49:11.949 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.g0jBacodMp 2024-09-15T03:49:11.949 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.g0jBacodMp 2024-09-15T03:49:12.145 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.g0jBacodMp 2024-09-15T03:49:12.231 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.g0jBacodMp is 464KB 2024-09-15T03:49:12.331 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.g0jBacodMp 2024-09-15T03:49:12.347 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905308/data/mon.c.tgz 2024-09-15T03:49:12.347 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-09-15T03:49:12.366 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.THpTeaI7gV 2024-09-15T03:49:12.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.THpTeaI7gV 2024-09-15T03:49:12.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.THpTeaI7gV 2024-09-15T03:49:12.628 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.THpTeaI7gV is 464KB 2024-09-15T03:49:12.728 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.THpTeaI7gV 2024-09-15T03:49:12.744 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-15T03:49:12.744 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T03:49:12.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T03:49:12.750 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T03:49:12.808 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-15T03:49:12.808 DEBUG:teuthology.orchestra.run.smithi001:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-15T03:49:12.818 DEBUG:teuthology.orchestra.run.smithi067:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-15T03:49:12.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-15T03:49:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-15T03:49:12.849 INFO:teuthology.orchestra.run.smithi067.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-15T03:49:12.874 INFO:teuthology.orchestra.run.smithi190.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-15T03:49:12.879 INFO:tasks.ceph:Archiving crash dumps... 2024-09-15T03:49:12.880 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905308/remote/smithi001/crash 2024-09-15T03:49:12.881 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T03:49:12.922 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905308/remote/smithi067/crash 2024-09-15T03:49:12.923 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T03:49:12.951 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905308/remote/smithi190/crash 2024-09-15T03:49:12.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T03:49:12.977 INFO:tasks.ceph:Compressing logs... 2024-09-15T03:49:12.978 DEBUG:teuthology.orchestra.run.smithi001:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T03:49:12.980 DEBUG:teuthology.orchestra.run.smithi067:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T03:49:12.998 DEBUG:teuthology.orchestra.run.smithi190:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T03:49:13.006 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-15T03:49:13.006 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36678.log 2024-09-15T03:49:13.007 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32098.log 2024-09-15T03:49:13.007 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30400.log 2024-09-15T03:49:13.007 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36678.log.gz 2024-09-15T03:49:13.007 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32098.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.32098.log.gz -- 2024-09-15T03:49:13.007 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.40558.log 2024-09-15T03:49:13.034 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-15T03:49:13.034 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-15T03:49:13.034 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T03:49:13.034 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-15T03:49:13.035 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-15T03:49:13.035 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T03:49:13.035 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-15T03:49:13.035 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-15T03:49:13.035 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz/var/log/ceph/ceph-mgr.x.log: 2024-09-15T03:49:13.035 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mon.b.log: 83.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T03:49:13.065 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2024-09-15T03:49:13.065 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29253.log 2024-09-15T03:49:13.065 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.32517.log 2024-09-15T03:49:13.065 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.10.log: /var/log/ceph/ceph-client.admin.29253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29133.log 2024-09-15T03:49:13.066 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29253.log.gz 2024-09-15T03:49:13.066 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/objectstore_tool.32517.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-09-15T03:49:13.066 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29133.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.32577.log 2024-09-15T03:49:13.066 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29133.log.gz 2024-09-15T03:49:13.066 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.9.log: 84.9% -- replaced with /var/log/ceph/objectstore_tool.32517.log.gz 2024-09-15T03:49:13.066 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28973.log 2024-09-15T03:49:13.066 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/objectstore_tool.32577.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T03:49:13.066 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28973.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28973.log.gz 2024-09-15T03:49:13.066 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29053.log 2024-09-15T03:49:13.066 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29093.log 2024-09-15T03:49:13.067 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph.audit.log: 83.8% -- replaced with /var/log/ceph/objectstore_tool.32577.log.gz 2024-09-15T03:49:13.067 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29053.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-15T03:49:13.067 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29053.log.gz 2024-09-15T03:49:13.067 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29093.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.29093.log.gz 2024-09-15T03:49:13.067 INFO:teuthology.orchestra.run.smithi190.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.29293.log 2024-09-15T03:49:13.067 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29173.log 2024-09-15T03:49:13.067 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29293.log.gz 2024-09-15T03:49:13.067 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.32437.log 2024-09-15T03:49:13.067 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.11.log 2024-09-15T03:49:13.067 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29173.log.gz 2024-09-15T03:49:13.067 INFO:teuthology.orchestra.run.smithi190.stderr: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T03:49:13.068 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/objectstore_tool.32437.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.32374.log 2024-09-15T03:49:13.068 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29373.log 2024-09-15T03:49:13.068 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/objectstore_tool.32374.log: gzip -5 --verbose -- 82.9% -- replaced with /var/log/ceph/objectstore_tool.32437.log.gz 2024-09-15T03:49:13.068 INFO:teuthology.orchestra.run.smithi190.stderr: /var/log/ceph/ceph-client.admin.29013.log 2024-09-15T03:49:13.068 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29373.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29373.log.gz 2024-09-15T03:49:13.068 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T03:49:13.068 INFO:teuthology.orchestra.run.smithi190.stderr: 82.9% -- replaced with /var/log/ceph/objectstore_tool.32374.log.gz 2024-09-15T03:49:13.068 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29333.log 2024-09-15T03:49:13.068 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29013.log.gz 2024-09-15T03:49:13.068 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-09-15T03:49:13.068 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29333.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29333.log.gz 2024-09-15T03:49:13.068 INFO:teuthology.orchestra.run.smithi190.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29213.log 2024-09-15T03:49:13.068 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool..log 2024-09-15T03:49:13.069 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29213.log: 0.0% 83.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T03:49:13.069 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28933.log 2024-09-15T03:49:13.069 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29213.log.gz 2024-09-15T03:49:13.069 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/objectstore_tool..log: /var/log/ceph/ceph-client.admin.28933.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28933.log.gz 2024-09-15T03:49:13.069 INFO:teuthology.orchestra.run.smithi190.stderr: 90.4% -- replaced with /var/log/ceph/objectstore_tool..log.gz 2024-09-15T03:49:13.070 INFO:teuthology.orchestra.run.smithi067.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-15T03:49:13.108 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44899.log 2024-09-15T03:49:13.108 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30400.log.gz 2024-09-15T03:49:13.109 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39854.log 2024-09-15T03:49:13.109 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40558.log.gz 2024-09-15T03:49:13.109 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45109.log 2024-09-15T03:49:13.109 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44899.log.gz 2024-09-15T03:49:13.109 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32626.log 2024-09-15T03:49:13.109 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39854.log.gz 2024-09-15T03:49:13.109 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46117.log 2024-09-15T03:49:13.109 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45109.log.gz 2024-09-15T03:49:13.109 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40617.log 2024-09-15T03:49:13.109 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32626.log.gz 2024-09-15T03:49:13.109 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43481.log 2024-09-15T03:49:13.109 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46117.log.gz 2024-09-15T03:49:13.110 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41264.log 2024-09-15T03:49:13.110 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40617.log.gz 2024-09-15T03:49:13.110 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29542.log 2024-09-15T03:49:13.110 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43481.log.gz 2024-09-15T03:49:13.110 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45529.log 2024-09-15T03:49:13.110 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41264.log.gz 2024-09-15T03:49:13.110 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36424.log 2024-09-15T03:49:13.110 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29542.log.gz 2024-09-15T03:49:13.110 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31575.log 2024-09-15T03:49:13.110 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45529.log.gz 2024-09-15T03:49:13.110 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38172.log 2024-09-15T03:49:13.110 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36424.log.gz 2024-09-15T03:49:13.111 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29880.log 2024-09-15T03:49:13.111 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31575.log.gz 2024-09-15T03:49:13.111 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32997.log 2024-09-15T03:49:13.111 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38172.log.gz 2024-09-15T03:49:13.111 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29885.log 2024-09-15T03:49:13.111 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29880.log.gz 2024-09-15T03:49:13.111 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45865.log 2024-09-15T03:49:13.111 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32997.log.gz 2024-09-15T03:49:13.111 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30292.log 2024-09-15T03:49:13.111 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29885.log.gz 2024-09-15T03:49:13.111 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45025.log 2024-09-15T03:49:13.111 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45865.log.gz 2024-09-15T03:49:13.111 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44387.log 2024-09-15T03:49:13.112 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30292.log.gz 2024-09-15T03:49:13.112 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41476.log 2024-09-15T03:49:13.112 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45025.log.gz 2024-09-15T03:49:13.112 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44387.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44387.log.gz 2024-09-15T03:49:13.112 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44389.log 2024-09-15T03:49:13.112 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45487.log 2024-09-15T03:49:13.112 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41476.log.gz 2024-09-15T03:49:13.112 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42883.log 2024-09-15T03:49:13.112 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44389.log.gz 2024-09-15T03:49:13.112 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45361.log 2024-09-15T03:49:13.112 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45487.log.gz 2024-09-15T03:49:13.112 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40852.log 2024-09-15T03:49:13.113 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42883.log.gz 2024-09-15T03:49:13.113 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45361.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-15T03:49:13.113 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45361.log.gz 2024-09-15T03:49:13.113 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42337.log 2024-09-15T03:49:13.113 INFO:teuthology.orchestra.run.smithi001.stderr: 28.1% -- replaced with /var/log/ceph/ceph-client.admin.40852.log.gz 2024-09-15T03:49:13.113 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41847.log 2024-09-15T03:49:13.113 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42505.log 2024-09-15T03:49:13.113 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42337.log.gz 2024-09-15T03:49:13.113 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38953.log 2024-09-15T03:49:13.113 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41847.log.gz 2024-09-15T03:49:13.113 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42505.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42505.log.gz 2024-09-15T03:49:13.113 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41416.log 2024-09-15T03:49:13.113 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45151.log 2024-09-15T03:49:13.114 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38953.log.gz 2024-09-15T03:49:13.114 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41416.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41416.log.gz 2024-09-15T03:49:13.114 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44056.log 2024-09-15T03:49:13.114 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43051.log 2024-09-15T03:49:13.114 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45151.log.gz 2024-09-15T03:49:13.114 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44056.log.gz 2024-09-15T03:49:13.114 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29884.log 2024-09-15T03:49:13.114 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43051.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.43051.log.gz /var/log/ceph/ceph-client.admin.42799.log 2024-09-15T03:49:13.114 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-09-15T03:49:13.114 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43480.log 2024-09-15T03:49:13.114 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29884.log.gz 2024-09-15T03:49:13.114 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30328.log 2024-09-15T03:49:13.114 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42799.log.gz 2024-09-15T03:49:13.115 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43480.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43480.log.gz 2024-09-15T03:49:13.115 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31947.log 2024-09-15T03:49:13.115 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34523.log 2024-09-15T03:49:13.115 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30328.log.gz 2024-09-15T03:49:13.115 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31947.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31947.log.gz 2024-09-15T03:49:13.115 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37927.log 2024-09-15T03:49:13.115 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33152.log 2024-09-15T03:49:13.115 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34523.log.gz 2024-09-15T03:49:13.115 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37927.log.gz 2024-09-15T03:49:13.115 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42038.log 2024-09-15T03:49:13.115 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34092.log 2024-09-15T03:49:13.115 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33152.log.gz 2024-09-15T03:49:13.115 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36382.log 2024-09-15T03:49:13.116 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42038.log.gz 2024-09-15T03:49:13.116 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36505.log 2024-09-15T03:49:13.116 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34092.log.gz 2024-09-15T03:49:13.116 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36382.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36382.log.gz 2024-09-15T03:49:13.116 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38357.log 2024-09-15T03:49:13.116 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29864.log 2024-09-15T03:49:13.116 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36505.log.gz 2024-09-15T03:49:13.116 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38357.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38357.log.gz 2024-09-15T03:49:13.116 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43461.log 2024-09-15T03:49:13.116 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41952.log 2024-09-15T03:49:13.116 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29864.log.gz 2024-09-15T03:49:13.116 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43461.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43461.log.gz 2024-09-15T03:49:13.117 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43802.log 2024-09-15T03:49:13.117 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33428.log 2024-09-15T03:49:13.117 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41952.log: /var/log/ceph/ceph-client.admin.43802.log: 26.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.41952.log.gz -5 2024-09-15T03:49:13.117 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41912.log 2024-09-15T03:49:13.117 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43802.log.gz 2024-09-15T03:49:13.117 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33428.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33428.log.gz 2024-09-15T03:49:13.117 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30796.log 2024-09-15T03:49:13.117 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33619.log 2024-09-15T03:49:13.117 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41912.log.gz 2024-09-15T03:49:13.117 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28912.log 2024-09-15T03:49:13.117 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30796.log.gz 2024-09-15T03:49:13.117 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33619.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33619.log.gz 2024-09-15T03:49:13.117 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43978.log 2024-09-15T03:49:13.118 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.28912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34978.log 2024-09-15T03:49:13.118 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28912.log.gz 2024-09-15T03:49:13.118 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43978.log.gz 2024-09-15T03:49:13.118 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45235.log 2024-09-15T03:49:13.118 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38848.log 2024-09-15T03:49:13.118 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34978.log.gz 2024-09-15T03:49:13.118 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45235.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45235.log.gz 2024-09-15T03:49:13.118 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45907.log 2024-09-15T03:49:13.118 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43927.log 2024-09-15T03:49:13.118 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38848.log.gz 2024-09-15T03:49:13.118 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45907.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45907.log.gz 2024-09-15T03:49:13.118 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33386.log 2024-09-15T03:49:13.118 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41645.log 2024-09-15T03:49:13.119 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43927.log.gz 2024-09-15T03:49:13.119 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33386.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33386.log.gz 2024-09-15T03:49:13.119 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40409.log 2024-09-15T03:49:13.119 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32034.log 2024-09-15T03:49:13.119 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41645.log.gz 2024-09-15T03:49:13.119 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40409.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40409.log.gz 2024-09-15T03:49:13.119 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36464.log 2024-09-15T03:49:13.119 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45949.log 2024-09-15T03:49:13.119 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32034.log.gz 2024-09-15T03:49:13.119 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36464.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36464.log.gz 2024-09-15T03:49:13.119 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33470.log 2024-09-15T03:49:13.119 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31017.log 2024-09-15T03:49:13.119 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45949.log.gz 2024-09-15T03:49:13.120 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33470.log.gz 2024-09-15T03:49:13.120 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35884.log 2024-09-15T03:49:13.120 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40517.log 2024-09-15T03:49:13.120 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31017.log.gz 2024-09-15T03:49:13.120 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35884.log.gz 2024-09-15T03:49:13.120 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34589.log 2024-09-15T03:49:13.120 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29329.log 2024-09-15T03:49:13.120 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40517.log.gz 2024-09-15T03:49:13.120 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34589.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34589.log.gz 2024-09-15T03:49:13.120 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42231.log 2024-09-15T03:49:13.120 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31804.log 2024-09-15T03:49:13.120 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29329.log.gz 2024-09-15T03:49:13.120 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42231.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42231.log.gz 2024-09-15T03:49:13.121 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40166.log 2024-09-15T03:49:13.121 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41183.log 2024-09-15T03:49:13.121 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31804.log.gz 2024-09-15T03:49:13.121 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40166.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40166.log.gz 2024-09-15T03:49:13.121 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39403.log 2024-09-15T03:49:13.121 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43881.log 2024-09-15T03:49:13.121 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41183.log.gz 2024-09-15T03:49:13.121 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39403.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.39403.log.gz 2024-09-15T03:49:13.121 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.43456.log 2024-09-15T03:49:13.121 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32520.log 2024-09-15T03:49:13.121 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43881.log.gz 2024-09-15T03:49:13.121 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43456.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43456.log.gz 2024-09-15T03:49:13.121 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44857.log 2024-09-15T03:49:13.122 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46075.log 2024-09-15T03:49:13.122 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32520.log.gz 2024-09-15T03:49:13.122 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44857.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44857.log.gz 2024-09-15T03:49:13.122 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29865.log 2024-09-15T03:49:13.122 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46075.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T03:49:13.122 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46075.log.gz 2024-09-15T03:49:13.122 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29865.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29865.log.gz 2024-09-15T03:49:13.122 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30295.log 2024-09-15T03:49:13.122 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43219.log 2024-09-15T03:49:13.122 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30295.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30295.log.gz 2024-09-15T03:49:13.122 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46157.log 2024-09-15T03:49:13.122 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44983.log 2024-09-15T03:49:13.122 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43219.log.gz 2024-09-15T03:49:13.123 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36073.log 2024-09-15T03:49:13.123 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46157.log.gz 2024-09-15T03:49:13.123 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44983.log: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T03:49:13.123 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44983.log.gz 2024-09-15T03:49:13.123 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42230.log 2024-09-15T03:49:13.123 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34051.log 2024-09-15T03:49:13.123 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36073.log.gz 2024-09-15T03:49:13.123 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42230.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42230.log.gz 2024-09-15T03:49:13.123 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37969.log 2024-09-15T03:49:13.123 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.34051.log: -- /var/log/ceph/ceph-client.admin.42421.log 2024-09-15T03:49:13.123 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34051.log.gz 2024-09-15T03:49:13.123 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37969.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37969.log.gz 2024-09-15T03:49:13.123 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35679.log 2024-09-15T03:49:13.124 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44329.log 2024-09-15T03:49:13.124 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42421.log.gz 2024-09-15T03:49:13.124 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35679.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35679.log.gz 2024-09-15T03:49:13.124 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43966.log 2024-09-15T03:49:13.124 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40729.log 2024-09-15T03:49:13.124 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44329.log.gz 2024-09-15T03:49:13.124 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43966.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43966.log.gz 2024-09-15T03:49:13.124 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38544.log 2024-09-15T03:49:13.124 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45781.log 2024-09-15T03:49:13.124 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40729.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40729.log.gz 2024-09-15T03:49:13.124 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38544.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38544.log.gz 2024-09-15T03:49:13.124 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27229.log 2024-09-15T03:49:13.125 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37528.log 2024-09-15T03:49:13.125 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45781.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45781.log.gz 2024-09-15T03:49:13.125 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.tmp-client.admin.27229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44441.log 2024-09-15T03:49:13.125 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27229.log.gz 2024-09-15T03:49:13.125 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40730.log 2024-09-15T03:49:13.125 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37528.log.gz 2024-09-15T03:49:13.125 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44441.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44441.log.gz 2024-09-15T03:49:13.125 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42841.log 2024-09-15T03:49:13.125 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30515.log 2024-09-15T03:49:13.125 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40730.log.gz 2024-09-15T03:49:13.125 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42841.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.42841.log.gz --verbose 2024-09-15T03:49:13.125 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.42631.log 2024-09-15T03:49:13.125 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35497.log 2024-09-15T03:49:13.126 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30515.log.gz 2024-09-15T03:49:13.126 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42631.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.42631.log.gz 2024-09-15T03:49:13.126 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40004.log 2024-09-15T03:49:13.126 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37477.log 2024-09-15T03:49:13.126 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35497.log.gz 2024-09-15T03:49:13.126 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40004.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.40004.log.gz -- 2024-09-15T03:49:13.126 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.46033.log 2024-09-15T03:49:13.126 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29888.log 2024-09-15T03:49:13.126 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37477.log.gz 2024-09-15T03:49:13.126 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36985.log 2024-09-15T03:49:13.126 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46033.log.gz 2024-09-15T03:49:13.126 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42967.log 2024-09-15T03:49:13.126 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29888.log.gz 2024-09-15T03:49:13.127 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36985.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36985.log.gz 2024-09-15T03:49:13.127 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45991.log 2024-09-15T03:49:13.127 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29874.log 2024-09-15T03:49:13.127 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42967.log.gz 2024-09-15T03:49:13.127 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45991.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45991.log.gz 2024-09-15T03:49:13.127 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28682.log 2024-09-15T03:49:13.127 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43093.log 2024-09-15T03:49:13.127 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29874.log.gz 2024-09-15T03:49:13.127 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.28682.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28682.log.gz -5 2024-09-15T03:49:13.127 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43923.log 2024-09-15T03:49:13.127 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29886.log 2024-09-15T03:49:13.127 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43093.log.gz 2024-09-15T03:49:13.127 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43923.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.43923.log.gz 2024-09-15T03:49:13.128 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.39813.log 2024-09-15T03:49:13.128 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40959.log 2024-09-15T03:49:13.128 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29886.log.gz 2024-09-15T03:49:13.128 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39813.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39813.log.gz 2024-09-15T03:49:13.128 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43934.log 2024-09-15T03:49:13.128 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30745.log 2024-09-15T03:49:13.128 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40959.log.gz 2024-09-15T03:49:13.128 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29410.log 2024-09-15T03:49:13.128 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43934.log.gz 2024-09-15T03:49:13.128 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.32172.log 2024-09-15T03:49:13.128 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30745.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30745.log.gz 2024-09-15T03:49:13.128 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29410.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29410.log.gz 2024-09-15T03:49:13.128 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45613.log 2024-09-15T03:49:13.129 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.32172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41213.log 2024-09-15T03:49:13.129 INFO:teuthology.orchestra.run.smithi001.stderr: 79.5% -- replaced with /var/log/ceph/ceph-client.0.32172.log.gz 2024-09-15T03:49:13.129 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45613.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45613.log.gz 2024-09-15T03:49:13.129 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43452.log 2024-09-15T03:49:13.129 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38274.log 2024-09-15T03:49:13.129 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41213.log.gz 2024-09-15T03:49:13.129 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43452.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.43452.log.gz 2024-09-15T03:49:13.129 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35619.log 2024-09-15T03:49:13.129 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38417.log 2024-09-15T03:49:13.129 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38274.log.gz 2024-09-15T03:49:13.129 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35619.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.35619.log.gz --verbose 2024-09-15T03:49:13.129 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.43009.log 2024-09-15T03:49:13.129 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30858.log 2024-09-15T03:49:13.130 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38417.log.gz 2024-09-15T03:49:13.130 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43009.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43009.log.gz -5 2024-09-15T03:49:13.130 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29626.log 2024-09-15T03:49:13.130 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44617.log 2024-09-15T03:49:13.130 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30858.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30858.log.gz 2024-09-15T03:49:13.130 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31023.log 2024-09-15T03:49:13.130 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29626.log.gz 2024-09-15T03:49:13.130 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43472.log 2024-09-15T03:49:13.130 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44617.log.gz 2024-09-15T03:49:13.130 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44328.log 2024-09-15T03:49:13.130 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31023.log.gz 2024-09-15T03:49:13.130 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42085.log 2024-09-15T03:49:13.130 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43472.log.gz 2024-09-15T03:49:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44328.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44328.log.gz 2024-09-15T03:49:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31222.log 2024-09-15T03:49:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.42085.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39626.log 2024-09-15T03:49:13.131 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42085.log.gz 2024-09-15T03:49:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31222.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.31222.log.gz 2024-09-15T03:49:13.131 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.30387.log 2024-09-15T03:49:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37026.log 2024-09-15T03:49:13.131 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39626.log.gz 2024-09-15T03:49:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30387.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.30387.log.gz -- 2024-09-15T03:49:13.131 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.45277.log 2024-09-15T03:49:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45445.log 2024-09-15T03:49:13.131 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37026.log.gz 2024-09-15T03:49:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45277.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.45277.log.gz 2024-09-15T03:49:13.131 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39910.log 2024-09-15T03:49:13.132 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41141.log 2024-09-15T03:49:13.132 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45445.log.gz 2024-09-15T03:49:13.132 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42589.log 2024-09-15T03:49:13.132 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39910.log.gz 2024-09-15T03:49:13.132 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29875.log 2024-09-15T03:49:13.132 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41141.log.gz 2024-09-15T03:49:13.132 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42589.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42589.log.gz 2024-09-15T03:49:13.132 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45571.log 2024-09-15T03:49:13.132 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37069.log 2024-09-15T03:49:13.132 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29875.log.gz 2024-09-15T03:49:13.132 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45571.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45571.log.gz 2024-09-15T03:49:13.132 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-15T03:49:13.133 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44941.log 2024-09-15T03:49:13.133 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37069.log.gz 2024-09-15T03:49:13.133 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30951.log 2024-09-15T03:49:13.133 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45193.log 2024-09-15T03:49:13.133 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44941.log.gz 2024-09-15T03:49:13.133 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31452.log 2024-09-15T03:49:13.133 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30951.log.gz 2024-09-15T03:49:13.133 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45193.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.45193.log.gz -5 2024-09-15T03:49:13.133 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44444.log 2024-09-15T03:49:13.133 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32585.log 2024-09-15T03:49:13.133 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31452.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31452.log.gz 2024-09-15T03:49:13.133 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44815.log 2024-09-15T03:49:13.133 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44444.log.gz 2024-09-15T03:49:13.134 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30345.log 2024-09-15T03:49:13.134 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32585.log.gz 2024-09-15T03:49:13.134 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44815.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.44815.log.gz /var/log/ceph/ceph-client.admin.31573.log 2024-09-15T03:49:13.134 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-09-15T03:49:13.134 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41374.log 2024-09-15T03:49:13.134 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30345.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30345.log.gz 2024-09-15T03:49:13.134 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31573.log.gz 2024-09-15T03:49:13.134 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45319.log 2024-09-15T03:49:13.134 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43819.log 2024-09-15T03:49:13.134 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41374.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41374.log.gz 2024-09-15T03:49:13.134 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31348.log 2024-09-15T03:49:13.134 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45319.log.gz 2024-09-15T03:49:13.134 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43819.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-15T03:49:13.135 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43819.log.gz 2024-09-15T03:49:13.135 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31348.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31348.log.gz 2024-09-15T03:49:13.135 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35266.log 2024-09-15T03:49:13.135 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43135.log 2024-09-15T03:49:13.135 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.35266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36320.log 2024-09-15T03:49:13.135 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35266.log.gz 2024-09-15T03:49:13.135 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43135.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43135.log.gz -5 2024-09-15T03:49:13.135 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44539.log 2024-09-15T03:49:13.135 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5/var/log/ceph/ceph-client.admin.36320.log: --verbose -- /var/log/ceph/ceph-client.admin.37332.log 2024-09-15T03:49:13.135 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36320.log.gz 2024-09-15T03:49:13.135 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44773.log 2024-09-15T03:49:13.135 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44539.log.gz 2024-09-15T03:49:13.135 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37332.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37332.log.gz 2024-09-15T03:49:13.136 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30276.log 2024-09-15T03:49:13.136 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30319.log 2024-09-15T03:49:13.136 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44773.log.gz 2024-09-15T03:49:13.136 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35928.log 2024-09-15T03:49:13.136 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30276.log.gz 2024-09-15T03:49:13.136 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30319.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30319.log.gz 2024-09-15T03:49:13.136 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33965.log 2024-09-15T03:49:13.136 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45655.log 2024-09-15T03:49:13.136 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35928.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35928.log.gz 2024-09-15T03:49:13.136 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38112.log 2024-09-15T03:49:13.136 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33965.log.gz 2024-09-15T03:49:13.136 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45655.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45655.log.gz -5 2024-09-15T03:49:13.137 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39934.log 2024-09-15T03:49:13.137 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43953.log 2024-09-15T03:49:13.137 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38112.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38112.log.gz 2024-09-15T03:49:13.137 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29491.log 2024-09-15T03:49:13.137 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39934.log.gz 2024-09-15T03:49:13.137 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40206.log 2024-09-15T03:49:13.137 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43953.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43953.log.gz 2024-09-15T03:49:13.137 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34378.log 2024-09-15T03:49:13.137 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29491.log.gz 2024-09-15T03:49:13.137 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32256.log 2024-09-15T03:49:13.137 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40206.log.gz 2024-09-15T03:49:13.137 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39361.log 2024-09-15T03:49:13.137 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34378.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34378.log.gz 2024-09-15T03:49:13.138 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29881.log 2024-09-15T03:49:13.138 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32256.log.gz 2024-09-15T03:49:13.138 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35184.log 2024-09-15T03:49:13.138 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39361.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39361.log.gz 2024-09-15T03:49:13.138 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43475.log 2024-09-15T03:49:13.138 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29881.log.gz 2024-09-15T03:49:13.138 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35184.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T03:49:13.138 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35184.log.gz 2024-09-15T03:49:13.138 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42757.log 2024-09-15T03:49:13.138 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43475.log.gz 2024-09-15T03:49:13.138 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45739.log 2024-09-15T03:49:13.138 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43459.log 2024-09-15T03:49:13.139 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42757.log.gz 2024-09-15T03:49:13.139 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29584.log 2024-09-15T03:49:13.139 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45739.log.gz 2024-09-15T03:49:13.139 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43459.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.43459.log.gz 2024-09-15T03:49:13.139 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43945.log 2024-09-15T03:49:13.139 INFO:teuthology.orchestra.run.smithi001.stderr: 83.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T03:49:13.139 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30746.log 2024-09-15T03:49:13.139 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29584.log.gz 2024-09-15T03:49:13.139 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42379.log 2024-09-15T03:49:13.139 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43945.log.gz 2024-09-15T03:49:13.139 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32013.log 2024-09-15T03:49:13.139 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30746.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30746.log.gz 2024-09-15T03:49:13.140 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35432.log 2024-09-15T03:49:13.140 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42379.log.gz 2024-09-15T03:49:13.140 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32013.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.42463.log -- replaced with /var/log/ceph/ceph-client.admin.32013.log.gz 2024-09-15T03:49:13.140 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-09-15T03:49:13.140 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44269.log 2024-09-15T03:49:13.140 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35432.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35432.log.gz 2024-09-15T03:49:13.140 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33904.log 2024-09-15T03:49:13.140 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42463.log.gz 2024-09-15T03:49:13.140 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30806.log 2024-09-15T03:49:13.140 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44269.log.gz 2024-09-15T03:49:13.140 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36573.log 2024-09-15T03:49:13.141 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33904.log.gz 2024-09-15T03:49:13.141 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30806.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.31390.log -- replaced with /var/log/ceph/ceph-client.admin.30806.log.gz 2024-09-15T03:49:13.141 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-09-15T03:49:13.141 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40668.log 2024-09-15T03:49:13.141 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36573.log.gz 2024-09-15T03:49:13.141 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33511.log 2024-09-15T03:49:13.141 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31390.log.gz 2024-09-15T03:49:13.141 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40668.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40668.log.gz -5 2024-09-15T03:49:13.141 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43931.log 2024-09-15T03:49:13.141 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41984.log 2024-09-15T03:49:13.141 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33511.log.gz 2024-09-15T03:49:13.142 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45067.log 2024-09-15T03:49:13.142 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43931.log.gz 2024-09-15T03:49:13.142 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41984.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.41984.log.gz --verbose 2024-09-15T03:49:13.142 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.34440.log 2024-09-15T03:49:13.142 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32438.log 2024-09-15T03:49:13.142 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45067.log.gz 2024-09-15T03:49:13.142 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44608.log 2024-09-15T03:49:13.142 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34440.log.gz 2024-09-15T03:49:13.142 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45403.log 2024-09-15T03:49:13.142 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32438.log.gz 2024-09-15T03:49:13.142 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39727.log 2024-09-15T03:49:13.142 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44608.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44608.log.gz 2024-09-15T03:49:13.143 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43458.log 2024-09-15T03:49:13.143 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45403.log.gz 2024-09-15T03:49:13.143 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39727.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39727.log.gz -5 2024-09-15T03:49:13.143 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40413.log 2024-09-15T03:49:13.143 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42295.log 2024-09-15T03:49:13.143 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43458.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43458.log.gz 2024-09-15T03:49:13.143 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32932.log 2024-09-15T03:49:13.143 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40413.log.gz 2024-09-15T03:49:13.143 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41555.log 2024-09-15T03:49:13.143 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42295.log.gz 2024-09-15T03:49:13.143 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30859.log 2024-09-15T03:49:13.143 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.32932.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32932.log.gz 2024-09-15T03:49:13.144 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40477.log 2024-09-15T03:49:13.144 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41555.log.gz 2024-09-15T03:49:13.144 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30859.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30859.log.gz -5 2024-09-15T03:49:13.144 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43462.log 2024-09-15T03:49:13.144 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42925.log 2024-09-15T03:49:13.144 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40477.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40477.log.gz 2024-09-15T03:49:13.144 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33577.log 2024-09-15T03:49:13.144 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43462.log.gz 2024-09-15T03:49:13.144 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42925.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42925.log.gz 2024-09-15T03:49:13.144 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30676.log 2024-09-15T03:49:13.144 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31264.log 2024-09-15T03:49:13.144 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33577.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33577.log.gz 2024-09-15T03:49:13.145 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31574.log 2024-09-15T03:49:13.145 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30676.log.gz 2024-09-15T03:49:13.145 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44529.log 2024-09-15T03:49:13.145 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31264.log.gz 2024-09-15T03:49:13.145 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39016.log 2024-09-15T03:49:13.145 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31574.log.gz 2024-09-15T03:49:13.145 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44529.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.44529.log.gz 2024-09-15T03:49:13.145 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39258.log 2024-09-15T03:49:13.145 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42147.log 2024-09-15T03:49:13.145 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39016.log.gz 2024-09-15T03:49:13.145 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42127.log 2024-09-15T03:49:13.145 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39258.log.gz 2024-09-15T03:49:13.145 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36920.log 2024-09-15T03:49:13.145 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42147.log.gz 2024-09-15T03:49:13.146 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42127.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42127.log.gz -5 2024-09-15T03:49:13.146 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34936.log 2024-09-15T03:49:13.146 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35993.log 2024-09-15T03:49:13.146 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36920.log.gz 2024-09-15T03:49:13.146 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34936.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.34936.log.gz -- 2024-09-15T03:49:13.146 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.38909.log 2024-09-15T03:49:13.146 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34482.log 2024-09-15T03:49:13.146 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35993.log.gz 2024-09-15T03:49:13.146 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38909.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38909.log.gz 2024-09-15T03:49:13.146 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30696.log 2024-09-15T03:49:13.146 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37580.log 2024-09-15T03:49:13.146 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34482.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34482.log.gz 2024-09-15T03:49:13.146 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30696.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.30696.log.gz -5 2024-09-15T03:49:13.147 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29872.log 2024-09-15T03:49:13.147 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31180.log 2024-09-15T03:49:13.147 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37580.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37580.log.gz 2024-09-15T03:49:13.147 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29872.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.29872.log.gz -5 2024-09-15T03:49:13.147 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.30234.log 2024-09-15T03:49:13.147 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42673.log 2024-09-15T03:49:13.147 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31180.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31180.log.gz 2024-09-15T03:49:13.147 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30327.log 2024-09-15T03:49:13.147 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30234.log.gz 2024-09-15T03:49:13.147 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45823.log 2024-09-15T03:49:13.147 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42673.log.gz 2024-09-15T03:49:13.147 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33039.log 2024-09-15T03:49:13.147 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30327.log.gz 2024-09-15T03:49:13.148 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34830.log 2024-09-15T03:49:13.148 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45823.log.gz 2024-09-15T03:49:13.148 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.33039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37413.log 2024-09-15T03:49:13.148 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33039.log.gz 2024-09-15T03:49:13.148 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34830.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34830.log.gz 2024-09-15T03:49:13.148 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42547.log 2024-09-15T03:49:13.148 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42715.log 2024-09-15T03:49:13.148 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37413.log.gz 2024-09-15T03:49:13.148 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43473.log 2024-09-15T03:49:13.148 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42547.log.gz 2024-09-15T03:49:13.148 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38503.log 2024-09-15T03:49:13.148 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42715.log.gz 2024-09-15T03:49:13.148 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-15T03:49:13.149 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43473.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43473.log.gz 2024-09-15T03:49:13.149 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43471.log 2024-09-15T03:49:13.149 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38503.log.gz 2024-09-15T03:49:13.149 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30308.log 2024-09-15T03:49:13.149 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41332.log 2024-09-15T03:49:13.149 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43471.log.gz 2024-09-15T03:49:13.149 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35043.log 2024-09-15T03:49:13.149 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30308.log.gz 2024-09-15T03:49:13.149 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30955.log 2024-09-15T03:49:13.149 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41332.log: /var/log/ceph/ceph-client.admin.35043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29466.log 2024-09-15T03:49:13.149 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35043.log.gz 2024-09-15T03:49:13.149 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41332.log.gz 2024-09-15T03:49:13.149 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.30955.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30955.log.gz 2024-09-15T03:49:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44385.log 2024-09-15T03:49:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31306.log 2024-09-15T03:49:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29466.log: /var/log/ceph/ceph-client.admin.44385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37864.log 2024-09-15T03:49:13.150 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44385.log.gz 2024-09-15T03:49:13.150 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29466.log.gz 2024-09-15T03:49:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.31306.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31306.log.gz 2024-09-15T03:49:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45697.log 2024-09-15T03:49:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29887.log 2024-09-15T03:49:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37864.log: /var/log/ceph/ceph-client.admin.45697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43177.log 2024-09-15T03:49:13.150 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45697.log.gz 2024-09-15T03:49:13.150 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37864.log.gz 2024-09-15T03:49:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.29887.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29887.log.gz 2024-09-15T03:49:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43177.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43177.log.gz 2024-09-15T03:49:13.151 INFO:teuthology.orchestra.run.smithi190.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-15T03:49:13.180 INFO:teuthology.orchestra.run.smithi067.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-15T03:49:13.201 INFO:teuthology.orchestra.run.smithi067.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-15T03:49:13.288 INFO:teuthology.orchestra.run.smithi001.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-15T03:49:13.318 INFO:teuthology.orchestra.run.smithi190.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2024-09-15T03:49:13.380 INFO:teuthology.orchestra.run.smithi190.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-09-15T03:49:13.397 INFO:teuthology.orchestra.run.smithi001.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-15T03:49:13.464 INFO:teuthology.orchestra.run.smithi067.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-15T03:49:13.467 INFO:teuthology.orchestra.run.smithi001.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-15T03:49:13.510 INFO:teuthology.orchestra.run.smithi190.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2024-09-15T03:49:13.536 INFO:teuthology.orchestra.run.smithi001.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-15T03:49:13.550 INFO:teuthology.orchestra.run.smithi001.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-15T03:49:13.552 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-09-15T03:49:13.553 INFO:teuthology.orchestra.run.smithi001.stderr:real 0m0.558s 2024-09-15T03:49:13.553 INFO:teuthology.orchestra.run.smithi001.stderr:user 0m2.036s 2024-09-15T03:49:13.553 INFO:teuthology.orchestra.run.smithi001.stderr:sys 0m0.208s 2024-09-15T03:49:13.585 INFO:teuthology.orchestra.run.smithi190.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-09-15T03:49:13.587 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-15T03:49:13.587 INFO:teuthology.orchestra.run.smithi190.stderr:real 0m0.551s 2024-09-15T03:49:13.587 INFO:teuthology.orchestra.run.smithi190.stderr:user 0m1.631s 2024-09-15T03:49:13.587 INFO:teuthology.orchestra.run.smithi190.stderr:sys 0m0.100s 2024-09-15T03:49:13.595 INFO:teuthology.orchestra.run.smithi067.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-15T03:49:13.819 INFO:teuthology.orchestra.run.smithi067.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-15T03:49:13.821 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T03:49:13.821 INFO:teuthology.orchestra.run.smithi067.stderr:real 0m0.809s 2024-09-15T03:49:13.821 INFO:teuthology.orchestra.run.smithi067.stderr:user 0m2.121s 2024-09-15T03:49:13.821 INFO:teuthology.orchestra.run.smithi067.stderr:sys 0m0.087s 2024-09-15T03:49:13.821 INFO:tasks.ceph:Archiving logs... 2024-09-15T03:49:13.822 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905308/remote/smithi001/log 2024-09-15T03:49:13.822 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T03:49:14.338 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905308/remote/smithi067/log 2024-09-15T03:49:14.339 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T03:49:14.655 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905308/remote/smithi190/log 2024-09-15T03:49:14.656 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T03:49:14.934 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-15T03:49:14.947 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-15T03:49:14.947 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T03:49:14.949 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T03:49:14.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T03:49:15.050 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-15T03:49:15.061 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-15T03:49:15.062 DEBUG:teuthology.orchestra.run.smithi001:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:49:15.064 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:49:15.067 DEBUG:teuthology.orchestra.run.smithi190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:49:15.080 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:49:15.082 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:49:15.082 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:49:15.085 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:49:15.085 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-09-15T03:49:15.085 INFO:teuthology.orchestra.run.smithi001.stdout:^* hv01.front.sepia.ceph.com 3 6 377 19 +201us[ +204us] +/- 57ms 2024-09-15T03:49:15.086 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 20 -85us[ -81us] +/- 71ms 2024-09-15T03:49:15.086 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 -158us[ -154us] +/- 82ms 2024-09-15T03:49:15.086 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:49:15.087 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:49:15.087 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-09-15T03:49:15.087 INFO:teuthology.orchestra.run.smithi190.stdout:^* hv01.front.sepia.ceph.com 3 6 377 20 +200us[ +209us] +/- 57ms 2024-09-15T03:49:15.087 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 21 -88us[ -79us] +/- 71ms 2024-09-15T03:49:15.087 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 -171us[ -162us] +/- 82ms 2024-09-15T03:49:15.087 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:49:15.089 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:49:15.089 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-09-15T03:49:15.089 INFO:teuthology.orchestra.run.smithi067.stdout:^* hv01.front.sepia.ceph.com 3 6 377 20 +204us[ +208us] +/- 57ms 2024-09-15T03:49:15.089 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 20 -79us[ -79us] +/- 71ms 2024-09-15T03:49:15.089 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 -158us[ -158us] +/- 82ms 2024-09-15T03:49:15.089 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:49:15.089 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-15T03:49:15.101 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-15T03:49:15.101 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-15T03:49:15.115 DEBUG:teuthology.orchestra.run.smithi001:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T03:49:15.209 DEBUG:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T03:49:15.285 DEBUG:teuthology.orchestra.run.smithi190:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T03:49:15.352 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:49:15.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:49:15.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:49:15.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:49:15.417 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:49:15.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:49:15.446 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-15T03:49:15.455 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-15T03:49:15.466 INFO:teuthology.task.internal:Duration was 580.783292 seconds 2024-09-15T03:49:15.467 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-15T03:49:15.476 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-15T03:49:15.476 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T03:49:15.478 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T03:49:15.481 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T03:49:15.522 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:49:15.522 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:49:15.523 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:49:15.941 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-15T03:49:15.941 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi001.front.sepia.ceph.com 2024-09-15T03:49:15.941 DEBUG:teuthology.orchestra.run.smithi001:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T03:49:15.969 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-09-15T03:49:15.969 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T03:49:16.010 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi190.front.sepia.ceph.com 2024-09-15T03:49:16.010 DEBUG:teuthology.orchestra.run.smithi190:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T03:49:16.039 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-15T03:49:16.039 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T03:49:16.042 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T03:49:16.053 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T03:49:16.106 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-15T03:49:16.106 DEBUG:teuthology.orchestra.run.smithi001:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T03:49:16.115 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T03:49:16.124 DEBUG:teuthology.orchestra.run.smithi190:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T03:49:16.254 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-15T03:49:16.459 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-15T03:49:16.460 DEBUG:teuthology.orchestra.run.smithi001:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T03:49:16.488 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T03:49:16.516 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T03:49:16.541 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-15T03:49:16.623 DEBUG:teuthology.orchestra.run.smithi001:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:49:16.626 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:49:16.629 DEBUG:teuthology.orchestra.run.smithi190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:49:16.650 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = core 2024-09-15T03:49:16.652 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core 2024-09-15T03:49:16.652 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-09-15T03:49:16.679 DEBUG:teuthology.orchestra.run.smithi001:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:49:16.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:49:16.734 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:49:16.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:49:16.750 DEBUG:teuthology.orchestra.run.smithi190:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:49:16.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:49:16.764 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-15T03:49:16.836 INFO:teuthology.task.internal:Transferring archived files... 2024-09-15T03:49:16.836 DEBUG:teuthology.misc:Transferring archived files from smithi001:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905308/remote/smithi001 2024-09-15T03:49:16.836 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T03:49:17.146 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905308/remote/smithi067 2024-09-15T03:49:17.146 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T03:49:17.218 DEBUG:teuthology.misc:Transferring archived files from smithi190:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905308/remote/smithi190 2024-09-15T03:49:17.219 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T03:49:17.252 INFO:teuthology.task.internal:Removing archive directory... 2024-09-15T03:49:17.252 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T03:49:17.255 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T03:49:17.258 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T03:49:17.302 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-15T03:49:17.314 INFO:teuthology.task.internal:Not uploading archives. 2024-09-15T03:49:17.315 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-15T03:49:17.328 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-15T03:49:17.328 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T03:49:17.331 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T03:49:17.333 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T03:49:17.347 INFO:teuthology.orchestra.run.smithi001.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 03:49 /home/ubuntu/cephtest 2024-09-15T03:49:17.348 INFO:teuthology.orchestra.run.smithi067.stdout: 265336 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 03:49 /home/ubuntu/cephtest 2024-09-15T03:49:17.357 INFO:teuthology.orchestra.run.smithi190.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 03:49 /home/ubuntu/cephtest 2024-09-15T03:49:17.358 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-15T03:49:17.370 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-15T03:49:17.402 INFO:teuthology.run:Summary data: description: rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} duration: 580.78329205513 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-15T03:49:17.402 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T03:49:17.478 INFO:teuthology.run:pass